Replay history and rebuild projections

Replay processes retained events again from an earlier Flowcore position. It lets teams rebuild replaceable state, recover from consumer defects, create new read models, and test improved processing logic against real history.

Replay is powerful because it repeats delivery. That also makes it dangerous for consumers with non-idempotent side effects.

Know when replay is appropriate

Common replay use cases include:

• Rebuilding a database, search index, graph, cache, or warehouse table
• Creating a new projection from existing history
• Recovering downstream state after data loss
• Correcting a handler bug
• Migrating to a new projection schema
• Comparing an old and new processing implementation
• Reproducing a production sequence in an isolated environment

Replay is not a general retry button for an unknown production failure. First determine what the consumer does, which history it needs, and whether repeated side effects are safe.

Understand the replay position

A Flowcore processing position uses a time bucket and, within that bucket, an event ID.

To replay from the beginning, find the first available bucket for the selected event types. To replay a bounded incident, identify the last known good position before the incorrect state began.

Record:

• Tenant and data core
• Included flow types and event types
• Starting time bucket and event ID
• Optional stopping position
• Projection target
• Consumer code version
• Initiator and approval

Do not choose a start timestamp from memory during an incident. Resolve it to an actual bucket and event position.

Separate replay from live state

The safest rebuild writes to a new target:

• A new database schema
• A new set of tables
• A new search index
• A new object-storage prefix
• A new warehouse dataset

This blue/green projection pattern preserves the current read model while the rebuild runs.

A typical sequence is:

1. Create the new target and apply its schema.
2. Configure the replay consumer to write only to that target.
3. Start from the approved Flowcore position.
4. Process retained history in ascending order.
5. Catch up with events produced during the rebuild.
6. Validate counts and business invariants.
7. Switch readers to the new target.
8. Keep the old target for a defined rollback window.
9. Remove it through the normal data lifecycle.

Clearing a live projection and rebuilding it in place creates an outage and removes your easiest rollback path.

Make handlers replay-safe

At-least-once delivery already requires idempotency. Replay makes the requirement visible at scale.

For database projections, store each Flowcore event ID and the resulting state change in one transaction. A unique constraint on event ID prevents the same event from being applied twice.

An example table shape is:

CREATE TABLE projection_events (
  event_id TEXT PRIMARY KEY,
  time_bucket TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

The handler begins a transaction, inserts the event ID, applies the state transition, and commits. If the insert conflicts, the event was already applied and can be acknowledged safely.

Do not use only an in-memory deduplication set. It disappears during deployment and does not coordinate multiple replicas.

Protect external side effects

A projection write can often be repeated safely. A payment, email, SMS, shipment request, or third-party API call may not be.

Before replaying a consumer with side effects, choose one of these strategies:

• Disable side-effect handlers during replay.
• Route replay into an isolated test destination.
• Use the Flowcore event ID as a provider idempotency key.
• Replay only projection handlers, not action handlers.
• Use an outbox that records whether the external action already occurred.
• Build a dedicated historical transformation consumer.

Never reset a broad production pathway until you know every registered handler's replay behavior.

Choose replay scope carefully

Replay the smallest history that can produce a correct result.

A full data-core replay may be appropriate when rebuilding a model that depends on many related event types. A single-flow or event-type replay is safer when the projection is isolated.

Scope also affects ordering. If a projection combines order.placed.0, payment.authorized.0, and order.cancelled.0, replaying only one stream may create impossible intermediate state.

Document which event types form the projection's complete source set.

Use ascending deterministic processing

Historical reconstruction normally processes events in ascending storage order within each selected stream and bucket.

If the projection depends on valid time rather than arrival order, encode that rule in the projection. Late-arriving events and corrections can otherwise produce different results during rebuild.

Concurrency can improve throughput, but it can also change processing order. Keep concurrency at one for order-sensitive flows or partition work by a key that preserves the required sequence.

Do not increase concurrency until correctness tests show that the projection is order-independent or safely partitioned.

Persist replay checkpoints

A long replay must survive process restarts. Persist the current time bucket and event ID for each flow type.

The Pathways Data Pump accepts a state-manager factory and provides a PostgreSQL-backed implementation:

import {
  createPostgresPumpStateManagerFactory,
} from "@flowcore/pathways"
 
const stateManagerFactory =
  await createPostgresPumpStateManagerFactory({
    connectionString: process.env.DATABASE_URL!,
    tableName: "order_projection_rebuild_state",
  })

Use a dedicated replay checkpoint table or namespace so the rebuild cannot overwrite the live pathway's position.

Advancing a replay checkpoint means all required work through that event is durable. If the process crashes after a projection commit but before a checkpoint update, duplicate detection makes the repeated event safe.

Bound the replay when appropriate

For incident recovery or migration verification, set a known stopping point. A bounded replay produces a stable dataset that can be compared against expected results.

For a live rebuild, historical replay eventually needs to catch up with ongoing writes. One strategy is:

1. Record a high-water position before the bulk replay.
2. Replay from the start through that position.
3. Validate the historical result.
4. Continue from the high-water position toward live traffic.
5. Switch readers once lag is acceptably low and invariants pass.

Do not pause producers for the entire rebuild unless the domain requires a strict maintenance window.

Validate more than event counts

Matching the number of processed events is necessary but insufficient. Different event sequences can produce the same count and incorrect state.

Validate domain invariants such as:

• Every active order has the expected line totals.
• Cancelled orders are not marked fulfillable.
• Aggregate balances equal the sum of contributing events.
• Search documents correspond to current source entities.
• No projection references a missing parent.
• The number of unique processed event IDs matches the expected source set.

Compare representative records between old and new projections. Run application-level read tests against the rebuilt target before switching traffic.

Handle bad historical events

A replay may reveal events that current schemas reject or current handlers no longer understand.

Do not silently skip them and declare the rebuild successful. Choose an explicit policy:

• Run the handler version compatible with that event version.
• Add a migration adapter that converts old payloads into the current internal model.
• Quarantine the event and block completion until reviewed.
• Correct the projection through a documented compensating process.

Retained events keep their original meaning. Do not mutate history to satisfy a newer schema.

Keep old contract schemas and migration code available as long as the history can be replayed.

Control load and backpressure

A replay can process years of history faster than events originally arrived. The projection database or external dependency may not support that rate.

Control:

• Page size
• Pump buffer size
• Per-flow-type concurrency
• Database connection pools
• Handler transaction duration
• Retry backoff
• Maximum acceptable live-traffic impact

Monitor CPU, memory, database locks, connection saturation, write latency, retry rates, buffer depth, and replay lag.

A faster pump is not useful if the projection target spends the entire rebuild in overload.

Use separate credentials and targets for rehearsals

Rehearse production replay in an isolated environment with read access to approved history and write access only to the isolated projection target.

The replay identity should not have permission to publish business events, administer IAM, truncate event history, or modify the live projection unless the procedure requires it.

If sensitive data is included, the rehearsal environment must satisfy the same security requirements as production. Do not export protected history to a developer laptop for convenience.

Operate replay as a tracked change

A production replay should have an owner, reason, approved scope, code version, start position, target, expected duration, validation plan, and abort condition.

During execution, record:

• Current bucket and event position
• Processed and duplicate counts
• Failure and retry counts
• Lag to the high-water mark
• Projection-target health
• Any quarantined events

After completion, preserve validation results and the final position. If the new projection replaces an old one, record the traffic-switch time and rollback window.

Diagnose replay problems

Replay starts from the wrong point

Stop before processing more history. Verify that the replay uses its own state store and that no live checkpoint was copied unintentionally.

Events repeat after restart

Check whether the checkpoint update was committed. Repetition is safe only if handlers use durable idempotency.

Projection differs from the live model

Determine whether the live model contains out-of-band writes, the replay omitted an event type, handler logic changed, or historical events fail current validation.

Replay never catches up

Compare incoming event rate with processing throughput. Increase capacity only after checking ordering and downstream limits. Consider a staged high-water approach.

One malformed event blocks progress

Capture the event ID and contract version without exposing sensitive payload data. Apply the documented compatibility or quarantine policy rather than advancing silently.

Finish with a deliberate cutover

Before switching readers, ensure the rebuilt projection has processed through the agreed position, passed domain validation, and reached acceptable lag.

Change traffic through a reversible deployment or configuration switch. Observe read errors, latency, and business metrics after cutover. Keep the previous projection intact for the agreed rollback window.

Replay is complete when the new state is validated and operationally adopted, not when the pump reaches the newest bucket.

Return to Query and follow event streams for SDK pagination and live observation patterns.