How events move through Flowcore

A Flowcore system has two complementary sides:

• The write side records durable facts.
• The processing and read side turns those facts into useful behavior and queryable state.

Understanding that separation makes it easier to design services that can recover from failures, add new consumers, and rebuild state without changing producers.

The end-to-end lifecycle

A typical event moves through Flowcore in six stages:

1. A command or external input is validated by the producing application.
2. The producer publishes an event to a specific event type.
3. Flowcore stores the immutable event and its envelope.
4. A Data Pathway selects the event and delivers it to a consumer.
5. The consumer performs work and records successful progress.
6. If necessary, retained events are replayed to the same or a new consumer.

Each stage has a distinct responsibility. Keeping those responsibilities separate is what makes the architecture resilient.

1. Validate intent before publishing

Flowcore stores events; your application decides whether a requested action is valid.

For example, an order service may receive a PlaceOrder command. Before publishing, it should authenticate the caller, check business rules, validate the payload, and decide whether the order can be placed.

If validation succeeds, the service publishes an order.placed.0 event. If validation fails, it returns an error and does not create a false business fact.

Commands express intent. Events record outcomes. Keeping that distinction clear prevents consumers from treating unvalidated requests as completed facts.

2. Publish a typed event

Producers can publish through @flowcore/pathways, @flowcore/sdk, or the ingestion API, depending on the application and integration.

A producer selects the tenant, data core, flow type, and event type, then supplies the payload and any useful metadata. Production publishers should use service credentials with only the permissions they need.

A good producer also considers retry behavior. If a network timeout occurs after Flowcore accepted an event but before the producer received the response, retrying blindly can create a duplicate business fact. Use stable source identifiers or application-level idempotency where duplicate publication would be harmful.

Publishing successfully means the event is durably stored. It does not mean every downstream consumer has already processed it.

3. Store the event as durable history

Flowcore records the event with identity, routing, time, and storage information. The stored event becomes part of the history for that event type.

This durable boundary is important. Downstream consumers can be unavailable without forcing the producer to coordinate their recovery. A failed search index or analytics pipeline does not erase the original order, payment, or sensor event.

The event history can later support:

• Audit and investigation
• New integrations
• Projection rebuilds
• Recovery from consumer defects
• Historical testing
• Reprocessing with improved logic

Retention, sensitivity, and deletion requirements still need explicit governance. Immutable does not mean unmanaged.

4. Select and deliver events with Data Pathways

A Data Pathway connects retained events to an application that processes them. It selects events from configured sources, moves through history, and delivers work to a consumer.

Flowcore v2 supports two operating models.

Managed Data Pathway

A managed pathway runs its data pump in Flowcore-managed infrastructure. Flowcore fetches matching events and sends them to an authenticated application endpoint.

This model is useful when you want Flowcore to operate the delivery workers while your application owns the handler endpoint and business logic.

Virtual Data Pathway

A virtual pathway runs the data pump with your application while reporting its state to the Flowcore control plane.

This model can suit environments that need the processing runtime close to private services or under application-team control.

Both models are current v2 concepts. They replace legacy scenario and strand delivery patterns.

5. Process events into behavior and read models

A consumer receives an event and performs useful work. It might:

• Insert or update rows in PostgreSQL
• Build a document or search index
• Update a graph, vector store, cache, or aggregate
• Write to a warehouse or object store
• Call an external API
• Send a notification
• Trigger another business workflow

When a consumer transforms history into query-friendly state, that state is a projection or read model.

The read model should be designed for its queries. It does not need to copy the event hierarchy or preserve every event field. A customer-facing order table and an operations dashboard can project the same events into completely different schemas.

The retained event log remains the durable history. The projection is replaceable state.

Delivery is normally at least once

Reliable delivery systems may send an event more than once. A worker can process an event successfully and fail before acknowledging it, causing a retry.

Consumers must therefore be idempotent: processing the same event again should not corrupt the result or repeat an irreversible side effect.

A common database pattern is:

1. Read the Flowcore event ID.
2. Begin a database transaction.
3. Check whether that event ID has already been applied.
4. Apply the projection change.
5. Store the processed event ID.
6. Commit both changes atomically.

A unique constraint on the processed event ID makes duplicate detection reliable across multiple workers.

External side effects require similar care. Use provider idempotency keys, an outbox, or a local state machine instead of assuming a handler is called exactly once.

Checkpoints and progress

A pathway needs to know how far it has progressed. That durable position is commonly described as a checkpoint.

Checkpoints let processing resume after restarts and make delivery lag visible. They also define the point from which a replay or rebuild can begin.

Do not advance progress before required work is safely committed. Otherwise a crash can leave the checkpoint ahead of the projection, skipping work during recovery.

For multi-step handlers, decide what counts as success and which state must be committed together.

Retries and failure handling

Transient failures should be retried with bounded backoff. Permanent failures need visibility and an explicit operator decision.

A healthy consumer distinguishes among:

• Temporary dependency failures, such as a short database outage
• Invalid or unsupported event data
• Missing permissions or expired credentials
• Programming defects
• Capacity problems and sustained backpressure

Infinite rapid retries can overload the same failing dependency and increase lag. Capture enough context to diagnose the event, but do not leak payload secrets or sensitive fields into logs.

Replay and rebuilding state

Replay means processing retained events again from an earlier position. It is one of the main benefits of keeping durable history.

Common reasons to replay include:

• Rebuilding a damaged projection
• Creating a new read model from existing history
• Correcting a consumer bug
• Testing a new processing version
• Recovering data lost from a downstream system

A safe rebuild usually writes to a new schema, index, or database rather than clearing the live one in place. After replay, validate counts and business invariants, catch up with new events, and switch readers only when the new model is ready.

Replay also repeats event delivery, so idempotency and side-effect controls remain essential. Do not replay a notification or payment consumer without understanding what repeated processing will do.

CQRS without unnecessary complexity

This separation of writes and reads is commonly called CQRS:

• Commands and business rules decide which events may be written.
• Events record accepted facts.
• Projections create models optimized for reads.

CQRS does not require every service to replay its entire history for every request. User-facing queries should normally read a prepared projection. Replay is an operational and lifecycle capability, not the normal request path.

You can adopt this pattern incrementally. A service can publish important domain events and maintain one practical read model before introducing more consumers.

Flowcore v2 and legacy terminology

New systems should use the current v2 stack:

• REST-backed resource services
• @flowcore/sdk commands
• @flowcore/pathways for event contracts, publishing, and handlers
• Managed or virtual Data Pathways for delivery and replay
• Tenant IAM roles, policies, bindings, and API keys

Older documentation and repositories may use these legacy concepts:

• GraphQL-backed administration
• Scenarios
• Strands
• Adapters tied to the scenario model
• Hosted transformer shells

Those terms describe the previous delivery architecture. They are relevant when migrating an existing system, but they should not shape a new v2 design.

A practical example

Consider an order.placed.0 event:

1. The order API validates the cart and customer.
2. It publishes the event to the commerce data core.
3. Flowcore stores the event.
4. An order projection writes queryable order state to PostgreSQL.
5. A fulfillment consumer creates warehouse work.
6. An analytics pathway sends the event to a warehouse.
7. Months later, a fraud team replays the same history into a new detection model.

The producer did not synchronously coordinate all four consumers. Each consumer owns its processing state and can be rebuilt independently.

Questions to answer before production

For every event flow, document:

1. Who is allowed to publish the event?
2. What makes publication idempotent?
3. Which consumers receive it?
4. What state or side effects does each consumer own?
5. How does each consumer deduplicate delivery?
6. When is a checkpoint safe to advance?
7. How are retries, invalid events, and sustained failures surfaced?
8. What is the replay procedure?
9. Which retention and sensitivity rules apply?

These decisions turn an event demo into an operable production system.

What to read next

If any resource terms are unfamiliar, read Core concepts and the event data model.

Return to What is Flowcore? for the high-level platform overview.