Prepare a Flowcore service for production

A local event pipeline proves that your contracts and handler work. Production requires additional decisions about ownership, credentials, pathway mode, checkpoints, idempotency, scaling, observability, and replay.

Use this guide as a release gate. Do not treat changing NODE_ENV to production as the entire migration.

Start with explicit environment boundaries

Use separate resources and credentials for development, staging, and production.

At minimum, separate:

• Tenant or agreed environment boundaries
• Data cores and event contracts where required by your platform model
• Application API keys
• CLI profiles
• Databases and checkpoint storage
• Delivery endpoints and authentication secrets
• Dashboards, alerts, and runbooks

A development key should not be able to publish to or administer production resources. Separate keys make rotation and incident response possible without disrupting unrelated applications.

Decide who owns resource provisioning

Pathways can provision data cores, flow types, and event types from code. The description fields act as ownership signals:

• dataCoreDescription declares ownership of the data core.
• flowTypeDescription declares ownership of a flow type.
• description declares ownership of an event type.

Provisioning is additive. It creates missing resources and updates owned descriptions, but it does not delete resources that disappear from code.

Choose one clear production workflow:

Application-owned provisioning

The service provisions its resources during a controlled bootstrap or startup step.

This is convenient, but the runtime key needs resource-management permissions and startup can depend on control-plane availability.

CI or bootstrap-owned provisioning

A deployment job calls provision() before the service is released. The runtime key can then have narrower event and processing permissions.

This usually provides a better approval trail and separates infrastructure failure from application startup.

Manifest-owned provisioning

Platform resources are managed with versioned v2 manifests and CLI diff/apply workflows. Application registrations expect the resources to exist.

This is useful when platform changes require centralized review.

Do not let both manifests and application startup independently own the same resource definitions without an explicit reconciliation policy.

Choose managed or virtual Data Pathways

Pathway mode determines where event delivery runs. It is separate from whether the Flowcore platform itself is managed or dedicated.

Managed mode

In managed mode, Flowcore operates the data pump and sends selected events to your authenticated endpoint.

Choose managed mode when:

• Your application can expose a stable HTTPS handler endpoint.
• You want Flowcore to operate delivery workers.
• Your runtime is serverless or should not host a persistent pump.
• You prefer scaling the handler separately from the event pump.

A managed production configuration includes a named pathway and endpoint settings:

const pathways = new PathwaysBuilder({
  baseUrl: "https://webhook.api.flowcore.io",
  tenant: process.env.FLOWCORE_TENANT!,
  dataCore: "orders",
  apiKey: process.env.FLOWCORE_API_KEY!,
  runtimeEnv: "production",
  pathwayMode: "managed",
  pathwayName: "order-projection",
  pathwayLabels: {
    name: "Order projection",
    description: "Builds the production order read model",
  },
  managedConfig: {
    endpointUrl: "https://orders.example.com/api/flowcore/events",
    authHeaders: {
      Authorization: `Bearer ${process.env.FLOWCORE_DELIVERY_SECRET!}`,
    },
    sizeClass: "small",
  },
  autoProvision: {
    dataCore: false,
    flowType: false,
    eventType: false,
    pathway: true,
  },
})

The endpoint must authenticate every delivery before processing the body. Store the delivery secret in a secret manager and rotate it independently from the Flowcore API key.

Managed mode does not start a local in-process pump. The managed workers own event fetching and delivery.

Virtual mode

In virtual mode, your application runs the data pump and reports its state to the Flowcore control plane.

Choose virtual mode when:

• Processing must run inside a private network.
• The application team needs direct control of the pump runtime.
• The workload runs as a persistent service rather than a request-only function.
• You can operate durable checkpoint storage and cluster coordination.

For multiple production replicas, start Pathways cluster mode. Cluster mode uses shared coordination so one leader owns the pump while work can be distributed to application instances.

A production virtual pathway needs:

• A persistent process model
• PostgreSQL or another supported durable state implementation
• Persistent pump checkpoints
• Cluster coordination for multiple replicas
• Reachable instance addresses for worker communication
• Graceful startup and shutdown
• Health, pulse, and leadership visibility

Virtual cluster mode is not suitable for runtimes that cannot maintain persistent processes or accept the required inter-instance communication. Use managed mode for typical serverless deployments.

Do not rely silently on environment defaults

Recent Pathways releases choose environment-aware defaults: production favors managed mode while development and test favor virtual mode.

Defaults are useful, but production architecture should be explicit. Set runtimeEnv and pathwayMode deliberately so a deployment cannot change behavior because an environment variable was missing or renamed.

Review package release notes before upgrading because provisioning and runtime defaults are operational behavior, not merely type-level changes.

Replace tutorial state with durable state

The getting-started tutorial uses memory for checkpoints and duplicate detection. Replace both before production.

Persist pump checkpoints

A virtual pathway must store its position durably. Losing a checkpoint can cause unintended replay; advancing it too early can skip work.

The checkpoint should only move after required processing has completed successfully.

Persist processed event IDs

At-least-once delivery means duplicates are possible. Store the Flowcore event ID with the projection change and enforce uniqueness.

For a database projection, the safest pattern is:

1. Begin a transaction.
2. Insert the event ID into a processed-events table.
3. Apply the projection change.
4. Commit both changes together.
5. Treat a duplicate event ID as already completed.

If the side effect is external, use an idempotency key, outbox, or explicit state machine. A local in-memory set is not sufficient.

Design retries intentionally

Classify failures before deciding to retry:

• Network timeouts and short dependency outages are usually transient.
• Invalid event payloads are not fixed by immediate retries.
• Missing permissions require configuration changes.
• Programming defects require a deployment.
• Capacity failures require throttling or scaling.

Use bounded backoff and expose repeated failures to operators. Rapid infinite retries can increase lag and overload the same dependency.

Make timeouts shorter than the surrounding delivery timeout so the consumer can fail clearly rather than being abandoned by the caller.

Secure every boundary

Before release:

• Use a private production data core unless public access is explicitly required.
• Enable delete protection for production data cores.
• Grant application keys only the required resources and actions.
• Separate provisioning credentials from runtime credentials where practical.
• Authenticate managed delivery endpoints.
• Encrypt transport with HTTPS.
• Keep API keys, delivery secrets, and encryption keys in a secret manager.
• Prevent secrets and sensitive event fields from entering logs.
• Define rotation and revocation procedures.

If payloads contain sensitive data, define retention, masking, encryption, and deletion workflows before ingestion. Durable history makes late governance fixes expensive.

Plan capacity and concurrency

Start with conservative concurrency and increase it using measured handler latency and dependency capacity.

Pathways supports shared or per-flow-type concurrency for virtual pumps. Different flow types may need different limits: order projections may scale broadly while audit processing remains serialized.

More concurrency does not help when the database connection pool, external API, or lock contention is already saturated. Monitor the whole processing path.

For managed pathways, select a size class and capacity based on measured event volume and latency requirements rather than the largest available option.

Add production observability

A production pipeline should expose enough information to answer:

• Is the service receiving events?
• How far behind is it?
• Which flow type is delayed?
• How long does processing take?
• How many events are retried or rejected?
• Which checkpoint was last committed?
• Is a virtual cluster leader healthy?
• Are managed endpoint responses succeeding?

Capture structured metrics and logs for:

• Delivery count and failure count
• Handler duration
• Retry attempts
• Current time bucket and event ID
• Delivery lag
• Buffer depth and backpressure
• Pathway pulse failures
• Provisioning failures
• Cluster leadership changes

Logs should include event IDs and pathway names for correlation, but should avoid entire sensitive payloads.

Alert on sustained lag and repeated failure, not only process crashes. A healthy process can still be making no progress.

Make shutdown graceful

On deployment or termination:

1. Stop accepting new application traffic if needed.
2. Stop or drain event processing.
3. Allow active handlers to commit.
4. Persist the final safe checkpoint.
5. Release cluster leadership and close worker connections.
6. Close database pools and exit.

The termination grace period must be longer than the maximum expected handler duration. Otherwise routine deployments can create avoidable duplicate delivery.

Test replay before you need it

A retained history is only useful if the team can safely replay it.

Before production launch:

1. Create an isolated projection target.
2. Replay representative history into it.
3. Verify counts and business invariants.
4. Confirm duplicate delivery does not corrupt state.
5. Confirm external side effects are disabled or protected.
6. Document how to select a starting checkpoint.
7. Record who can authorize a production replay.

Never test a replay by resetting a live payment, notification, or external API consumer without side-effect controls.

Production release checklist

Contracts and ownership

• Event names describe facts and include a versioning strategy.
• Payloads are runtime validated.
• Resource ownership is explicit: application, bootstrap job, or manifests.
• Production resources have reviewed descriptions and owners.

Security

• Development and production credentials are separate.
• Runtime permissions follow least privilege.
• The data core is private and delete-protected.
• Delivery endpoints authenticate requests.
• Secrets are stored and rotated outside source control.
• Sensitive-data handling is documented.

Processing

• Managed or virtual mode is set explicitly.
• Checkpoints are durable.
• Handlers are idempotent.
• Database changes and deduplication are atomic where possible.
• Retries and permanent failures have defined behavior.
• Concurrency respects downstream capacity.

Operations

• Health, lag, retry, and checkpoint signals are observable.
• Alerts identify sustained failure or stalled progress.
• Graceful shutdown is tested.
• Replay is tested against an isolated target.
• A rollback and recovery runbook exists.
• Package versions are pinned and upgrades are reviewed.

Where to go next

Return to Build your first event pipeline if you have not completed the local tutorial.

Read How events move through Flowcore for a conceptual explanation of delivery, checkpoints, idempotency, and replay.