Build reliable handlers and read models

A Pathways handler turns a durable event into useful state or behavior. Reliability depends less on the happy-path function body than on what happens during retries, process crashes, duplicate delivery, malformed history, and slow dependencies.

This guide focuses on handlers that build read models, but the same delivery principles apply to integrations and workflows.

Assume at-least-once delivery

A handler may receive the same Flowcore event more than once.

For example:

1. The handler updates the database.
2. The process crashes before acknowledging success.
3. The pathway retries the event.
4. The handler sees the same event ID again.

The retry is correct: the delivery system cannot know whether the database committed. The handler must make repetition safe.

Do not describe a consumer as exactly once unless the complete boundary, including external side effects and transaction coordination, truly provides that guarantee.

Use the Flowcore event ID for deduplication

Persist every applied event ID in the same durable system as the projection.

CREATE TABLE applied_flowcore_events (
  event_id TEXT PRIMARY KEY,
  time_bucket TEXT NOT NULL,
  flow_type TEXT NOT NULL,
  event_type TEXT NOT NULL,
  applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

A unique primary key lets concurrent or repeated delivery identify an event that was already applied.

Do not deduplicate only by payload identity. Two legitimate events can describe similar values. The Flowcore event ID identifies the delivery fact.

Apply projection state and deduplication atomically

For a relational read model, use one transaction:

async function handleOrderPlaced(event: FlowcoreEvent<OrderPlaced>) {
  await db.transaction(async (tx) => {
    const inserted = await tx.execute(sql`
      INSERT INTO applied_flowcore_events (
        event_id,
        time_bucket,
        flow_type,
        event_type
      ) VALUES (
        ${event.eventId},
        ${event.timeBucket},
        ${event.flowType},
        ${event.eventType}
      )
      ON CONFLICT (event_id) DO NOTHING
      RETURNING event_id
    `)
 
    if (inserted.rowCount === 0) {
      return
    }
 
    await tx.execute(sql`
      INSERT INTO orders (
        id,
        customer_id,
        total_minor,
        status
      ) VALUES (
        ${event.payload.orderId},
        ${event.payload.customerId},
        ${event.payload.totalMinor},
        'placed'
      )
      ON CONFLICT (id) DO UPDATE SET
        customer_id = EXCLUDED.customer_id,
        total_minor = EXCLUDED.total_minor,
        status = EXCLUDED.status
    `)
  })
}

If the transaction rolls back, neither the projection nor processed marker remains. If it commits and delivery repeats, the unique event ID turns the retry into a no-op.

Writing the marker before the projection in separate transactions can lose work. Writing it after the projection in a separate transaction can repeat work.

Distinguish the pump checkpoint from handler deduplication

A checkpoint stores how far a pump has progressed, typically as a time bucket and event ID for each flow type.

The processed-events table stores whether one projection applied one event.

They solve different failure windows:

• Checkpoints let fetching resume after restart.
• Deduplication makes repeated delivery safe.

Do not replace one with the other.

The runtime should advance the checkpoint only after required handlers complete successfully. If work fails, throwing keeps the event eligible for retry rather than falsely marking progress.

Design read models for their queries

A read model is not required to mirror the event hierarchy or producer database.

Choose a shape that serves the reader:

• Relational tables for transactional query patterns
• Documents for aggregate reads
• Search indexes for full-text and filtering
• Graphs for relationship traversal
• Time-series stores for measurements
• Warehouses for analytical scans
• Vector stores for similarity and retrieval workflows

Keep source event identity and useful provenance even when the projection shape differs substantially.

Flowcore history is the durable input. The read model is replaceable state.

Handle event ordering explicitly

Do not assume every event across a data core has one global business order.

Within a projection, identify the ordering requirement:

• Strict flow-type storage order
• Per-entity order
• Valid-time order
• Order-independent aggregation
• Last-write behavior based on a documented timestamp

Concurrency above one can process events from the same flow type simultaneously. Keep it at one for sequence-sensitive handlers or partition safely by entity.

Late-arriving events can have an older business time than events already projected. A valid-time model must account for that instead of blindly overwriting newer state.

Use upserts carefully

Upserts make retries convenient, but they can hide invalid state transitions.

An order.cancelled.0 handler should not necessarily create an order row that never received order.placed.0. Decide whether to:

• Reject and retry because the prerequisite event is expected shortly
• Store a pending transition
• Create a partial record
• Quarantine the event for investigation
• Rebuild using a broader source set

The correct choice depends on domain semantics and late-arrival behavior.

Separate projections from irreversible actions

A read-model handler writes replaceable state. A workflow handler may send money, email, SMS, or commands to another system.

For external actions:

• Use the Flowcore event ID as an idempotency key when the provider supports it.
• Store an outbox record before sending.
• Record provider responses and status transitions.
• Distinguish a retryable timeout from a permanent rejection.
• Avoid performing the action during historical replay unless explicitly intended.

A replay-safe projection does not automatically make every handler attached to the pathway replay-safe.

Classify errors before retrying

Transient errors

Database connection loss, temporary service unavailability, and network timeouts can succeed later. Throw and let bounded retry handle them.

Contract errors

An event that fails schema validation needs compatibility handling, not rapid retry. Keep old contract versions or migration adapters for retained history.

Permission and configuration errors

Expired keys, missing IAM permissions, unknown resources, or invalid endpoints require operator action. Repeated retry may increase lag without making progress.

Domain conflicts

A handler may receive a fact that conflicts with current projection assumptions. Decide whether late arrival, missing source events, or a code defect caused it.

Expose the classification in logs and metrics so operators know whether to wait, deploy, or change configuration.

Set bounded retry behavior

Pathways registrations support handler retry settings such as maximum retries, delay, retry status codes, and timeout.

app.register({
  flowType: "order.0",
  eventType: "order.placed.0",
  schema: orderPlacedSchema,
  maxRetries: 5,
  retryDelayMs: 1000,
  retryStatusCodes: [408, 429, 500, 502, 503, 504],
  timeoutMs: 15_000,
})

Choose values from dependency behavior and delivery timeout. A handler timeout should finish before the outer pathway considers delivery lost.

Do not retry every 4xx response. Authentication, authorization, validation, and missing-resource failures generally require intervention.

Prevent one bad event from hiding indefinitely

When bounded retries are exhausted, preserve enough information to investigate:

• Event ID
• Time bucket
• Flow and event type
• Pathway name
• Error classification
• Handler version
• Attempt count
• Correlation ID when safe

Keep sensitive payload fields out of the failed-event record unless the storage and access policy supports them.

A failed-event or quarantine workflow must not silently advance the main projection as if the event succeeded. Define whether processing blocks, isolates the entity, or continues with an operator-visible gap.

Control backpressure

When handlers process more slowly than events arrive, lag and buffer depth grow.

Respond in this order:

1. Measure handler latency and dependency saturation.
2. Identify the slow flow type or event type.
3. Remove unnecessary work from the synchronous handler.
4. Tune database indexes and transaction scope.
5. Increase concurrency only when ordering and capacity allow it.
6. Increase worker size or pathway capacity when the endpoint can use it.

Large buffers postpone failure but do not fix sustained under-capacity. Unbounded concurrency moves the overload to the database or external API.

Group processing intentionally

A virtual Pathways pump creates processing groups by flow type. Each group has its own state position and can restart independently.

This isolation means a failing order.0 pump does not have to stop customer.0, but it also means cross-flow projections cannot assume one shared checkpoint.

If a read model combines several flow types, record source positions separately and define what a consistent cut means for rebuild and cutover.

Make handlers observable

Record metrics for:

• Events received, applied, and deduplicated
• Handler duration
• Retry attempts and exhausted retries
• Validation failures
• Checkpoint age
• Delivery lag
• Buffer depth
• Database transaction failures
• External side-effect status

Use event ID, pathway, flow type, and event type as diagnostic context. Avoid high-cardinality labels such as customer ID in metrics.

A healthy HTTP response rate is not enough. An endpoint can return success while writing incorrect state, so add business invariants and projection consistency checks.

Test failure windows

Reliable-handler tests should deliberately simulate:

• Duplicate delivery
• Crash after projection commit but before acknowledgement
• Crash before commit
• Database timeout
• Malformed historical payload
• Late and out-of-order events
• Concurrent events for the same entity
• Replay into an empty target
• Replay when external actions are disabled

Test against realistic retained fixtures for every supported event version. A unit test with one current payload cannot prove replay compatibility.

Rebuild instead of repairing opaque state

When projection logic changes significantly, prefer building a new projection from retained events over applying an undocumented database repair.

A rebuild makes logic reproducible and validates that history is sufficient. If the projection cannot be rebuilt, document the missing source data or out-of-band mutations and correct the architecture.

Use an isolated target, catch up to live history, validate domain invariants, and switch readers reversibly.

Keep handlers small but complete

A handler should have one clear responsibility and commit one coherent result. Avoid combining unrelated projections and side effects in a single transaction-like function that cannot actually commit atomically across systems.

When several actions follow one event, use separate handlers or an outbox-backed workflow. Each can then own its retry and idempotency model.

Return to Run managed and virtual Data Pathways for delivery runtime and capacity configuration.