Run managed and virtual Data Pathways

A Data Pathway moves selected Flowcore events to application code that processes them. It tracks progress, retries failures, exposes operational state, and supports controlled restart or replay.

Flowcore v2 provides two pathway modes:

• Managed: Flowcore-operated workers deliver events to your endpoint.
• Virtual: your application runs the pump and reports its state to the control plane.

The event contracts can remain the same in both modes. The runtime and operational ownership change.

Do not confuse pathway mode with platform hosting

Managed versus virtual describes event delivery. Managed versus dedicated Flowcore describes who operates the platform.

An application can use managed Data Pathways with either a managed or dedicated Flowcore platform, subject to the installation's capabilities. It can also run a virtual pathway while storing events on managed Flowcore.

Choose managed delivery

Managed mode is usually the simplest production model when the application can expose a stable HTTPS endpoint.

Flowcore workers:

1. Fetch matching events from retained history.
2. Track pathway position.
3. Buffer and schedule delivery.
4. POST events to configured endpoints.
5. Retry eligible failures.
6. Report lag, errors, and capacity signals.

Your application owns endpoint authentication, validation, idempotency, projection transactions, and business behavior.

Managed mode is a strong fit for serverless handlers and services that should not maintain a persistent pump.

Configure a managed pathway from Pathways

const app = new PathwaysBuilder({
  baseUrl: "https://webhook.api.flowcore.io",
  tenant: process.env.FLOWCORE_TENANT!,
  dataCore: "commerce",
  apiKey: process.env.FLOWCORE_API_KEY!,
  runtimeEnv: "production",
  pathwayMode: "managed",
  pathwayName: "order-projection",
  pathwayLabels: {
    name: "Order projection",
    description: "Maintains the production order read model",
  },
  managedConfig: {
    endpointUrl: "https://orders.example.com/api/flowcore/events",
    authHeaders: {
      Authorization: `Bearer ${process.env.FLOWCORE_DELIVERY_SECRET!}`,
    },
    sizeClass: "small",
  },
  autoProvision: {
    dataCore: false,
    flowType: false,
    eventType: false,
    pathway: true,
  },
})

A named pathway is upserted rather than recreated blindly. Labels provide operator-facing context in the control plane.

Managed production mode does not start an in-process pump. Calling the runtime startup path provisions or updates the pathway while delivery remains the responsibility of managed workers.

Authenticate managed deliveries

Treat the delivery endpoint as an externally reachable privileged interface.

Use HTTPS and require a dedicated secret or signature on every request. Keep the delivery credential separate from the Flowcore API key: the worker needs the delivery credential to call your endpoint, but it should not expose the application's platform key.

At the endpoint:

1. Validate authentication before parsing or processing protected payloads.
2. Enforce a body-size limit.
3. Validate the Flowcore envelope and registered event contract.
4. Process idempotently using the event ID.
5. Return success only after required durable work commits.
6. Return a retryable failure only when another attempt can plausibly succeed.

Do not place a managed endpoint behind interactive browser login or CSRF middleware. Give it a machine-to-machine authentication path.

Rotate delivery secrets with an overlap strategy so current workers and the endpoint can accept the transition without dropping delivery.

Size managed pathways from evidence

Managed pathways use size classes such as small, medium, and high. Start with measured volume and handler latency rather than picking the largest class.

A pathway's sustainable throughput depends on:

• Event arrival rate
• Payload size
• Endpoint response time
• Delivery batch size
• Retry volume
• Downstream database capacity
• Allowed concurrency

Monitor lag and handler latency. A larger worker cannot compensate for an endpoint that serializes every request behind a saturated connection pool.

Choose virtual delivery

Virtual mode runs the Data Pump with your application. It is appropriate when processing must stay in a private network, the team needs direct runtime control, or the application is already a persistent service.

A virtual pump actively fetches events, processes them through registered handlers, and persists a position per flow type.

It supports WebSocket, NATS, or poller notifications. Notifications wake the pump; Flowcore storage and the checkpoint remain the durable source.

Virtual mode is not suitable for request-only serverless runtimes that can be suspended between requests.

Start a virtual pump

import {
  createPostgresPumpStateManagerFactory,
} from "@flowcore/pathways"
 
const stateManagerFactory =
  await createPostgresPumpStateManagerFactory({
    connectionString: process.env.DATABASE_URL!,
  })
 
await app.startPump({
  stateManagerFactory,
  notifier: { type: "websocket" },
  bufferSize: 500,
  maxRedeliveryCount: 5,
  concurrency: {
    default: 2,
    byFlowType: {
      "order.0": 1,
      "analytics.0": 8,
    },
  },
  autoProvision: {
    dataCore: false,
    flowType: false,
    eventType: false,
    pathway: true,
  },
})

The PostgreSQL state manager stores each flow type's bucket and event ID. Use a dedicated database identity with only the required table permissions.

Select the notifier

WebSocket

The default low-latency option. The application keeps an outbound connection and fetches stored events when notified.

Use it when persistent outbound connections are supported and reconnect behavior is observable.

NATS

Use NATS notification mode when your environment already operates NATS and wants notifications through that infrastructure.

Poller

Polling periodically checks for new work. It is straightforward in restricted networks but trades latency for repeated API calls.

No notifier removes the need for checkpoints. After lost notifications or reconnects, the pump resumes from durable state.

Run one replica safely

A single persistent application instance can run one virtual pump directly. Ensure:

• The state manager is durable.
• Shutdown calls stopPump().
• The termination grace period allows active handlers to finish.
• Health checks distinguish a running process from a progressing pump.
• Restarts do not reset the checkpoint.

A process restart may still redeliver the last event. Handlers must remain idempotent.

Scale virtual processing with cluster mode

Multiple replicas cannot each run an independent pump against the same logical position without coordination.

Pathways cluster mode provides:

• PostgreSQL-backed lease leadership
• One active pump on the leader
• Worker registration and heartbeats
• WebSocket distribution from leader to workers
• Automatic leader takeover after lease expiry
• Local processing fallback when no workers are available

import {
  createPostgresPathwayCoordinator,
} from "@flowcore/pathways"
 
const coordinator = await createPostgresPathwayCoordinator({
  connectionString: process.env.DATABASE_URL!,
})
 
await app.startCluster({
  coordinator,
  advertisedAddress: `ws://${process.env.POD_ADDRESS}:9090`,
  port: 9090,
})
 
await app.startPump({
  stateManagerFactory,
  notifier: { type: "websocket" },
})

The advertised address must be reachable by other application instances. Network policy, service discovery, and graceful lease release are production concerns.

Cluster mode coordinates pump ownership. It does not make non-idempotent handlers safe.

Tune concurrency with ordering in mind

startPump() accepts one shared concurrency value or per-flow-type settings.

Concurrency above one allows multiple events from the flow type to be processed simultaneously. That can increase throughput, but strict processing order is no longer guaranteed.

Keep concurrency at one for flows whose state transitions depend on sequence. Increase it for order-independent or safely partitioned work after measuring downstream capacity.

Do not increase concurrency beyond the database pool, external API limit, or handler's lock strategy.

Understand buffering and backpressure

The pump buffers fetched events before handlers process them. When consumers slow down, buffer depth and delivery lag increase.

Backpressure protects memory and dependencies by limiting fetch and in-flight work. Tune:

• Buffer size
• Delivery or handler concurrency
• Page or batch size
• Endpoint timeout
• Retry delay
• Database pool size

A large buffer can hide a slow handler temporarily while increasing memory use and restart rework. A tiny buffer can reduce throughput under bursty traffic.

Use buffer depth together with lag and handler latency to diagnose capacity.

Classify retries

Retry timeouts, connection failures, and temporary dependency outages with bounded backoff.

Do not repeatedly retry invalid schemas, missing permissions, unsupported event versions, or permanent business rejection. Those require intervention or an explicit failed-event policy.

Managed workers and virtual pumps both operate under at-least-once semantics. A retry can deliver an event that the endpoint already committed before its response was lost.

Observe control-plane state

The Data Pathways control plane tracks pathway definitions, capacity slots, assignments, heartbeats, restarts, and metrics.

Useful signals include:

• Events fetched and delivered
• Delivery errors
• Buffer depth
• Lag
• Handler or endpoint latency
• Slot health
• Assignment lease state
• Pathway enabled state
• Restart progress

For managed pathways, slots represent worker capacity and assignments bind pathways to slots. These are normally platform operational details, but they help explain why an enabled pathway may wait for capacity or move after a worker restart.

Disable rather than delete during investigation

When delivery is unsafe, disable the pathway or stop the virtual pump before changing history or resources.

Disabling preserves configuration and position for investigation. Deleting or recreating resources can remove evidence and complicate recovery.

After correcting the endpoint or handler, resume from an approved position and monitor duplicates, errors, and lag.

Continue with Build reliable handlers and read models for transaction and idempotency patterns.