Operate Data Pathways in production

Operating a Data Pathway means proving that events continue to move from Flowcore into useful application state. Process uptime is only one part of that proof. You also need to observe delivery lag, handler outcomes, and checkpoint progress.

This guide applies to managed Data Pathways and virtual pumps. Flowcore operates the worker runtime for managed pathways; your team operates it for virtual pathways. In both modes, your handlers and projections must tolerate at-least-once delivery.

Measure progress, not only availability

A transform endpoint can return 200 from /health while its projection is hours behind. A virtual-pump pod can continue heartbeating after event processing has stopped. Operational monitoring therefore needs to answer when the last event was successfully processed and how far the consumer is behind current ingestion.

Track delivery rate, handler latency, retries, failures, and lag. For critical projections, measure freshness directly from the latest projected event rather than inferring it from CPU use or pod status.

Logs should include eventId, flow type, event type, handler name, duration, and outcome. Include correlation metadata when it helps trace a workflow, but do not log complete sensitive payloads or credentials.

Add a progress-aware health signal

For virtual pathways backed by PostgreSQL, the Pathways state tables provide useful diagnostic evidence. The exact schema depends on the library version, but common installations can inspect recent processing and per-flow cursors with queries like these:

SELECT max(created_at) AS last_processed_at
FROM pathway_state;
 
SELECT flow_type, time_bucket, event_id
FROM pathway_pump_state
ORDER BY flow_type;

Compare last_processed_at with recent Flowcore events. If Flowcore is receiving events while this timestamp remains old, the pump is stalled even if leader heartbeats are current.

Expose the age of the last processed event as a metric. Do not make liveness restart a quiet service merely because no events were expected; evaluate progress against actual ingestion or an agreed freshness threshold.

Size from measured handler cost

Pathway throughput depends on event volume, endpoint latency, downstream database cost, batch size, and retry behavior. Start with a conservative managed size class, then resize after observing sustained lag and handler duration under representative traffic.

For virtual pumps, verify what each concurrency option means in the installed library version. In some Flowcore data-pump versions, processor.concurrency controls the number of events passed to one handler invocation rather than the number of handlers running simultaneously. Increasing it can create larger failed batches without adding useful parallelism.

If the same event fails repeatedly, do not raise maxInFlight to force more throughput. That creates a wider retry storm. Identify the poison event or unavailable dependency first.

Make handlers safe to retry

Flowcore delivery is at-least-once. A timeout can happen after a database write or external request succeeded but before delivery was acknowledged. The event may then arrive again.

Use the event ID as a durable idempotency key. Projection updates should use upserts, unique constraints, or a processed-event record stored in the same transaction as the business change.

await db.transaction(async (tx) => {
  const inserted = await tx.insert(processedEvents)
    .values({ eventId: event.eventId })
    .onConflictDoNothing()
    .returning({ eventId: processedEvents.eventId })
 
  if (inserted.length === 0) return
 
  await tx.insert(orders)
    .values(toOrderProjection(event))
    .onConflictDoUpdate({
      target: orders.id,
      set: toOrderProjection(event),
    })
})

The processed marker and projection update belong in one transaction. An in-memory set is not sufficient because it disappears on restart and is not shared across replicas.

Classify failures before retrying

Network timeouts, temporary database unavailability, and downstream 503 responses are transient. Retry them with bounded exponential backoff.

Schema failures, unknown event versions, oversized payloads, and invalid business data are deterministic. Repeating them without a code or data change only blocks progress. Authorization errors also require action: a 401 usually points to invalid authentication, while a 403 points to missing permission or the wrong resource scope.

Let required handler failures propagate to the delivery runtime. Swallowing an exception may advance the checkpoint even though the projection was not updated. Use an error hook for logging and alerting, not as a replacement for failure signaling.

Set timeouts around real work

Fast projection handlers and file-processing workflows should not share one arbitrary timeout. Measure normal and high-percentile handler duration, then leave enough room for temporary variance without allowing a blocked dependency to hold delivery indefinitely.

Long-running work is often better split into two durable stages. The Flowcore handler records or publishes the accepted job, and a separate worker performs the expensive task idempotently. This keeps pathway delivery responsive while preserving traceability.

Restart from a known position

A restart is a data operation, not merely a process operation. Before repositioning a pathway, record the current checkpoint and decide whether events after the new position can be processed more than once safely.

Use a bounded restart to recover from a handler bug, rebuild a projection, or backfill a known period. For virtual pumps, take special care when no checkpoint exists: some versions initialize near the current hour and can miss earlier events created before startup.

After restart, verify that the checkpoint advances, lag decreases, and representative projection records are correct. A successful restart command does not prove successful recovery.

Diagnose a silent stall

A silent stall usually presents as fresh events in Flowcore but stale application data. Start at the transform endpoint or pump logs, then inspect checkpoint progress. Check for secret mismatches, endpoint 401 or 500 responses, schema errors, database timeouts, and a deployment that changed subscriptions or delivery mode.

For cluster-mode virtual pumps, leader election and instance heartbeats show coordination, not delivery. Confirm both leadership and recent event processing.

If recovery requires clearing pump state, first identify the resulting replay or skip behavior. Blindly deleting state can duplicate side effects or lose a historical gap. Preserve the old position in the incident record before changing it.

Control backpressure

Slow handlers eventually create lag. Bounded buffers and delivery concurrency prevent one consumer from exhausting memory, but they cannot compensate for permanently slow downstream work.

When lag grows, compare arrival rate with completed-handler rate. Optimize the handler, database indexes, or external dependency before increasing capacity. If throughput is healthy but traffic has genuinely grown, move to a larger managed size class or add correctly coordinated virtual capacity.

Continue with Run operational readiness and retention workflows to define SLOs, retention decisions, and operating routines.