Compose environments and migrate legacy manifests

Production, staging, and development often need the same resource structure with different tenants, endpoints, capacity, or access bindings. The Flowcore v2 CLI can merge resources from multiple inputs, allowing a shared base plus small environment-specific overlays.

The same workflow also provides a controlled destination when migrating legacy scenario manifests to v2 resources and Data Pathways.

Use a base-and-overlay structure

A repository can separate shared intent from environment values:

flowcore/
├── base/
│   ├── data-core.yaml
│   ├── events.yaml
│   └── iam.yaml
└── environments/
    ├── development.yaml
    ├── staging.yaml
    └── production.yaml

The base contains common resource definitions and policy. An overlay changes only values that genuinely differ.

Avoid copying complete manifests into every environment. Large duplicated files drift silently and make security differences difficult to review.

Understand merge identity

With v2 apply, resources from multiple -f inputs are merged when they share the same kind and metadata.name identity.

flowcore apply --v2 \
  -f flowcore/base \
  -f flowcore/environments/production.yaml \
  --profile acme-production

Later inputs can supply environment-specific fields for the same resource.

File order matters when the same field is defined more than once. Keep the order stable in local commands and CI.

Do not use several unrelated overlays that all redefine the same security field. Reviewers should be able to identify one authoritative value.

Keep stable intent in the base

Good base candidates include:

• Resource names
• Descriptions
• Data-core boundaries
• Flow and event type names
• Production-safe defaults such as private access
• Common policy structure
• Labels and ownership information

Environment overlays can supply:

• Tenant identifiers
• Environment-specific IAM subjects or bindings
• Managed endpoint URLs
• Capacity or size classes
• Dedicated installation details
• Approved differences in access or protection

Delete protection should normally remain enabled in the production base or production overlay. A development convenience should not weaken the production default through an unnoticed merge.

Never merge raw secrets

Do not put API keys, bearer tokens, delivery secrets, or private credentials into base or overlay files.

Use the secret-management mechanism approved for the deployment. If a manifest needs a secret resource reference, keep the reference in Git and the secret value in the protected system.

Generated merged files can also leak values into CI artifacts. Treat rendered configuration according to its contents.

Make the final configuration visible

Merge behavior becomes dangerous when reviewers only see individual inputs and cannot inspect the effective resource.

CI should validate the same ordered inputs used for deployment and expose the resulting diff against the target environment.

A pull request workflow should:

1. Install a pinned Flowcore CLI version.
2. Validate all YAML syntax and registered resource schemas.
3. Select the intended environment inputs in a deterministic order.
4. Authenticate with read-only or diff-appropriate credentials.
5. Run flowcore diff against the environment profile.
6. Attach or summarize the effective platform changes for review.

The production apply should consume the same commit and input order that produced the approved diff.

Keep profile and manifest environments aligned

A production overlay used with a development profile can target the wrong platform or fail unpredictably.

Name profiles and paths clearly, then verify both before every mutation:

flowcore config show --profile acme-production
flowcore whoami --profile acme-production
flowcore diff \
  -f flowcore/base \
  -f flowcore/environments/production.yaml \
  --profile acme-production

The tenant in the effective manifest should agree with the profile's intended environment.

Protect production apply in CI

Use an approved deployment environment with narrowly scoped credentials. Pull-request jobs should not hold production mutation credentials.

Separate stages:

• Validation requires no platform write access.
• Diff needs read access to the intended resources.
• Apply needs create or update permissions.
• Delete needs separately controlled destructive permissions.

A failure in one stage should not cause the workflow to retry with a broader credential.

Do not infer deletion from an overlay

If an event type is missing from one overlay, that does not necessarily mean it should be deleted from the platform.

Apply is additive and update-oriented. Manage deletion through a dedicated operation, explicit manifest, approval, and verification.

This distinction lets teams stop managing a resource from one file without erasing retained history accidentally.

Recognize legacy manifest structure

Legacy Flowcore projects may use documents with older version fields and structures that combine:

• Data cores and event type declarations
• Scenarios
• Strands
• Transformer endpoints
• Parameters and secrets
• Blueprints
• Hosted transformer-shell deployment settings

These documents belong to the legacy GraphQL and scenario command path. They are not v2 resources merely because they are YAML.

Do not add apiVersion and kind mechanically while keeping the old nested structure. Migration requires separating concerns.

Map legacy responsibilities to v2

A legacy scenario often combines event selection, transformation code, runtime deployment, endpoint configuration, and secrets.

In v2, those concerns become:

• DataCore, FlowType, and EventType resources for event organization
• Application code using @flowcore/pathways for contracts and handlers
• Managed or virtual Data Pathways for event delivery
• A normal service deployment for transformation logic
• IAM policies and bindings for access
• Secret management outside source-controlled manifests
• Observable checkpoints, retries, and pathway state

The migration target is not one replacement YAML object. It is a clearer division between platform resources and application runtime.

Inventory the legacy scenario

Before changing anything, identify:

• Every strand and selected event type
• The transformer code and its repository
• Runtime parameters and secrets
• Endpoint and network requirements
• Current deployment and scaling behavior
• Projection databases and external side effects
• Retry and checkpoint assumptions
• Event versions still in production
• Consumers that depend on the scenario output

Legacy behavior that is not documented can be lost during a purely syntactic migration.

Extract transformation logic into an application

Move transformer logic into a normal application service with explicit contracts and handlers.

Register every consumed event through Pathways, validate historical payload versions, and make handlers idempotent.

If the old transformer shell injected parameters or secrets, replace them with the application's normal configuration and secret-management path.

Do not copy broad platform credentials from the transformer deployment into the new service.

Create v2 event resources

Represent the durable event hierarchy with current resource manifests:

apiVersion: data-core.flowcore.io/v1
kind: DataCore
metadata:
  name: commerce
  tenant: acme-production
spec:
  accessControl: private
  deleteProtection: true
  description: Durable commerce events
 
---
 
apiVersion: data-core.flowcore.io/v1
kind: FlowType
metadata:
  name: order.0
  tenant: acme-production
spec:
  dataCore: commerce
  description: Order lifecycle events
  eventTypes:
    - name: order.placed.0
      description: An order was placed

Preserve existing event names and semantics while consumers still depend on them. A platform migration is not permission to reinterpret retained event history.

Replace strands with Data Pathways

For each legacy strand, define the corresponding v2 source set and destination behavior.

Choose managed mode when Flowcore should operate delivery to an authenticated service endpoint. Choose virtual mode when the application should run the pump with durable state and, for multiple replicas, cluster coordination.

Map:

• Strand event filters to pathway flow/event mappings
• Transformer endpoint to the managed delivery endpoint or local handler
• Runtime parameters to service configuration
• Strand progress to a deliberate starting checkpoint
• Old retries to bounded current retry behavior
• Legacy scaling to pathway capacity or virtual concurrency

Do not start from the beginning of history by accident. Record the old processing position and the approved v2 starting point.

Run old and new paths in parallel safely

A shadow period can compare behavior before cutover.

Write the new processor to an isolated projection or disable irreversible side effects. Compare event counts, state, errors, and lag with the legacy path.

If both paths call external systems, duplicate actions can occur. Use idempotency keys or keep only one path authoritative for side effects.

The shadow period should have an end condition and owner. Permanent dual processing doubles operational ambiguity.

Cut over deliberately

A controlled cutover usually follows this order:

1. Provision and validate v2 resources.
2. Deploy the new application and pathway in non-authoritative mode.
3. Catch up from the approved historical position.
4. Compare projections and business invariants.
5. Stop or disable the legacy strand.
6. Let the v2 pathway become authoritative.
7. Monitor delivery, errors, and lag.
8. Keep a rollback window.
9. Decommission transformer-shell resources only after confidence is established.

Preserve old contract schemas and migration records while retained legacy events remain replayable.

Handle compatibility bridges explicitly

Some migrations need a temporary translator between old and new event versions.

A compatibility processor should:

• Preserve source event ID and correlation metadata
• Be idempotent
• Document whether the translated event or source event is authoritative
• Avoid translating the same source more than once
• Have a retirement date or release condition

Do not let a temporary bridge become an undocumented permanent source of events.

Diagnose migration drift

If new and old projections differ, determine whether:

• The v2 pathway started from a different position
• An event type was omitted from the source mapping
• Legacy code performed out-of-band database writes
• Retry or ordering behavior differs
• Current schemas reject historical payloads
• The shadow target lacks required reference data
• Both processors applied the same side effect

Resolve the cause before disabling the legacy path. Matching total event counts alone does not prove equivalent state.

Return to Manage Flowcore v2 resource manifests for apply and deletion behavior.