Integrate external systems and downstream consumers

Most Flowcore systems exchange data with SaaS products, partner APIs, files, devices, and existing databases. A reliable integration makes that boundary explicit: it validates foreign data, converts it into a stable Flowcore fact, and preserves enough identity to reconcile failures later.

The same principle applies in the other direction. A Flowcore event is a durable trigger, while an email, partner API request, or database update is a side effect that may need to be retried.

Translate external data at the boundary

Do not publish an external payload unchanged merely because it already contains JSON. Source formats evolve according to another system's needs and often contain fields that should not become permanent Flowcore history.

An adapter should validate the source, normalize timestamps and identifiers, remove unnecessary sensitive data, and produce a documented event contract. Preserve the external record ID when it is required for reconciliation or idempotency.

const partnerOrderSchema = z.object({
  id: z.string(),
  customer_ref: z.string(),
  amount_minor: z.number().int().nonnegative(),
  created_at: z.string().datetime(),
})
 
function toOrderImported(source: unknown) {
  const order = partnerOrderSchema.parse(source)
 
  return {
    orderId: order.id,
    customerId: order.customer_ref,
    totalMinor: order.amount_minor,
    importedAt: order.created_at,
  }
}

This keeps partner naming out of the internal contract. If the source changes customer_ref, only the adapter needs to change.

Preserve source identity during ingestion

Polling jobs, webhook receivers, and import scripts should use stable source identities. A timeout can occur after Flowcore accepted an event but before the producer received the response. Retrying with a new identity makes duplicate detection difficult.

For large imports, divide work into bounded chunks and record which source records were accepted or rejected. Retry uncertain chunks rather than starting the entire import again. The Flowcore SDK batch limit still applies, and each downstream event remains independently processed even when ingestion was batched.

Invalid rows should remain visible in an import report or failure workflow. Returning success while silently dropping malformed data makes source and Flowcore counts impossible to reconcile.

Use reference events for files

Large files belong in controlled object storage, not inside ordinary event payloads. Publish a compact event containing a stable file ID, checksum, media type, ownership information, and storage reference.

The consumer should obtain authorized access to the file when processing begins. Avoid embedding a reusable signed URL or storage credential in durable event history.

If a file must be split, include part number, total part count, and checksum information. Consumers need to tolerate duplicate parts, detect missing parts, and avoid loading an unbounded file into memory.

Deliver through a focused transform endpoint

A managed Data Pathway normally sends matching events to an HTTP transform endpoint. The endpoint should authenticate the pathway, validate the envelope, invoke PathwayRouter, and return a status that accurately describes whether delivery succeeded.

app.post("/transform", async (c) => {
  const secret = c.req.header("x-secret") ?? ""
  const event = await c.req.json()
 
  try {
    const result = await router.processEvent(event, secret)
    return c.json(result, 200)
  } catch (error) {
    logger.error("Flowcore delivery failed", {
      eventId: event?.eventId,
      error,
    })
    return c.json({ error: "Delivery failed" }, 500)
  }
})

Use TLS and network controls where available. Validate secrets without logging them, limit request size, and avoid logging full event payloads. A 2xx response tells the delivery runtime that required processing succeeded, so do not return 200 after swallowing a failed projection update.

Make outbound calls idempotent

When a handler calls an external API, Flowcore may deliver the event more than once. Use the event ID or a stable business operation ID as the downstream idempotency key when the partner supports one.

await billingClient.createInvoice({
  idempotencyKey: event.eventId,
  customerId: event.payload.customerId,
  amountMinor: event.payload.totalMinor,
})

If the external API has no idempotency support, persist an outbound-operation record before or within the same transaction that claims the event. Record the request identity and final response so a retry can determine whether work already completed.

Retry network failures, timeouts, and temporary 5xx responses with bounded backoff. Validation failures and partner rejections require correction or an explicit failure workflow, not endless retries.

Separate unrelated consumers

One event can update the product read model, feed analytics, send a notification, and start a partner integration. Those consumers should not necessarily run in one handler.

Keep work together when it shares one transaction and one owner. Separate it when latency, failure policy, credentials, or operational ownership differs. A slow analytics warehouse should not block the projection used by the product UI.

When consumers coordinate a longer process, use a stable process ID and explicit follow-up events. Shared mutable flags in an unrelated database make replay and incident reconstruction harder.

Design integrations for replay

Replay can repeat external side effects. Before replaying a consumer that sends email, charges a card, or calls a partner, decide how duplicate actions will be prevented.

A projection rebuild can often disable outbound integrations and process only database state. A business workflow replay may require downstream idempotency keys or compensating actions. Treat the replay mode as part of the integration design rather than an emergency detail.

Historical backfills should use a bounded event range and stable source IDs. If only one mapping changed, replay the affected event types and window instead of all retained history.

Monitor the boundary

Integration health should show source-read failures, rejected input, accepted events, pathway lag, endpoint failures, and outbound partner errors. Reconcile counts for critical imports and scheduled jobs.

A webhook endpoint returning 200 while publishing no event is a silent failure. Measure the business outcome—accepted and stored facts—not only HTTP availability.

Continue with Build CQRS projections and read models for reliable query-side state and rebuilds.