Recover delivery, replay, and projection gaps

Flowcore recovery begins with retained events and a known processing position. The event history is normally the evidence you preserve; checkpoints, projections, and external side effects are the state you repair.

Choose the smallest recovery that restores correctness. Restarting delivery, replaying a bounded window, and rebuilding an entire projection solve different problems and carry different risks.

Capture evidence before changing state

Record the affected tenant, event paths, last known successful event ID, current checkpoint, first missing record, and application version. Preserve relevant logs and error responses without copying secrets or unnecessary sensitive payloads.

This snapshot is essential if a restart or state reset changes what the system reports. Without it, operators cannot explain which events may have been duplicated, skipped, or replayed.

Resume a managed pathway

When matching events exist in Flowcore but managed delivery has stopped, verify the transform endpoint and its secret first. Restarting a pathway cannot fix an endpoint that still returns 401 or 500.

After correcting the endpoint, inspect the current pathway position and restart from the last known good event or an earlier bounded position. Assume events after that point can be delivered again. Confirm that handlers and external calls are idempotent before the restart.

Watch delivery lag and handler outcomes after recovery. A successful restart request only means the control action was accepted; it does not prove that the failed event was processed or that the projection caught up.

Recover a virtual pump cautiously

A virtual pump may stop advancing while the application and cluster heartbeats remain healthy. Compare the latest processed timestamp and per-flow cursor with recent Flowcore history.

SELECT max(created_at) AS last_processed_at
FROM pathway_state;
 
SELECT flow_type, time_bucket, event_id
FROM pathway_pump_state
ORDER BY flow_type;

Fix the handler, database, credential, or network problem before resetting state. If you must clear or rewrite a cursor, save the original value and calculate the exact replay window first.

Do not delete every state row as a routine restart technique. With no checkpoint, some pump versions begin near the current time, which can skip older events. Starting too far back can instead repeat years of side effects.

Define a bounded replay

A replay plan should name the consumer, event types, start and end positions, expected volume, and verification method. It should also state whether outbound email, billing, or partner calls are enabled during replay.

Use the narrowest window that contains the affected history. If one handler was broken for two days, replaying the entire data core from its beginning adds load and increases duplicate risk without improving correctness.

Before starting, estimate how quickly the consumer can process the window while live events continue. A replay that saturates the projection database can turn a contained data repair into a production outage.

Make replay safe in code

Projection handlers should use event identity for durable deduplication. When the projection update and processed marker share one transaction, replaying an already applied event becomes a no-op.

External side effects need a separate decision. Prefer consumers that pass a stable idempotency key to the downstream system. For a projection-only rebuild, an explicit replay mode can suppress integrations while preserving database updates.

async function handleOrderPlaced(event: OrderPlacedEvent, mode: "live" | "rebuild") {
  await projectOrderIdempotently(event)
 
  if (mode === "live") {
    await notifyFulfillment({
      idempotencyKey: event.eventId,
      orderId: event.payload.orderId,
    })
  }
}

Make the mode explicit in the replay job. Do not derive it from an ambiguous global environment setting that could disable live integrations accidentally.

Rebuild a projection into a shadow model

When projection logic or schema is broadly wrong, build a new projection version instead of repeatedly patching rows in place. Replay the required history into shadow tables, compare counts and representative business records, and then switch the query path.

Keep the previous projection available briefly for rollback. Once the new model has processed live events and remained current through the confidence window, retire the old tables and their consumer state through a reviewed cleanup.

A targeted partition rebuild is sufficient when the problem affects one tenant, entity range, or time period. Full rebuilds should be reserved for changes whose impact genuinely spans all retained history.

Close gaps introduced during migration

A migration gap can result from a partial historical import, cold-start cursor, or a period when neither the legacy strand nor the new pathway delivered events.

First determine whether the missing facts exist in Flowcore. If they do, replay the affected consumer from before the gap. If they do not, re-import from the source using stable source identities and a documented cutoff.

If old and new consumers ran simultaneously, reconciliation must also look for duplicates. Compare business identifiers and source event IDs rather than relying only on total row counts.

Handle poison events explicitly

A poison event fails every attempt until code, data, or configuration changes. Capture the event ID and error, reproduce the failure, and decide whether the correct response is a handler fix, a correcting event, or an approved skip.

Skipping an event can leave downstream state incomplete. Record that decision and either repair the projection manually with audit evidence or publish a durable compensating fact. Raising pathway capacity does not solve a deterministic failure.

Verify the recovered state

Recovery is complete when new events flow normally, replay lag reaches the expected level, and the affected projection matches trusted business evidence. Check representative records and important aggregates, not only event counts.

Confirm that no duplicate user-visible side effects occurred and that temporary replay permissions, feature flags, or elevated capacity have been removed. Record the final checkpoint and replay window in the incident or maintenance note.

Then improve detection. Add projection-freshness monitoring, alert when checkpoints stop advancing, and rehearse the corrected procedure in staging. The goal is to make the next recovery smaller and more predictable.

Return to Operate Data Pathways in production for ongoing delivery monitoring and restart practices.