Skip to content

Local CRE: require don_family on all nodesets and scope gateway pairing by family - #22733

Merged
cawthorne merged 44 commits into
developfrom
feature/cre-sharded-gateway-pairing
Jun 29, 2026
Merged

cawthorne merged 44 commits into
developfrom
feature/cre-sharded-gateway-pairing

Conversation

@cawthorne

@cawthorne cawthorne commented Jun 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds per-DON-family gateway↔workflow pairing for local CRE topologies. Every nodeset declares don_family; workflow DONs connect only to gateway DONs in the same family, and each gateway worker serves only workflow DONs in that family.

The same don_family value flows through capabilities registry (DonFamilies), workflow registry (SetDONLimit per family), and env workflow deploy.

Previously, env start injected all gateway connectors into every workflow DON and registered every gateway worker against all workflow DONs — separate gateway nodesets in TOML did not isolate families.

Solution

Every nodeset requires don_family (workflow, gateway, capabilities, bootstrap, etc.). Gateway topologies validate the workflow↔gateway graph at env start and fail if any family is missing or unmatched.

[[nodesets]]
  name = "feeds-zone-a"
  don_family = "feeds-zone-a"
  don_types = ["workflow"]

[[nodesets]]
  name = "gateway-zone-a"
  don_family = "feeds-zone-a"
  don_types = ["gateway"]
Component Before After
Workflow DON config All gateway connectors GatewayConnectorsForDonFamily
Gateway worker jobs All workflow DONs GatewayServiceConfigsForGateway
Capabilities registry Hardcoded default family Each nodeset's don_family (required)
Workflow registry setup Single family limit SetDONLimit per workflow family
Workflow deploy (DON selection) First workflow DON in state --don-family (+ --shard-index), or --workflow-don-name
Workflow deploy (docker cp) Manual --container-name-pattern Defaults to {nodesets.name}-node from resolved DON
Env start logging — DonFamilyGatewayPairingSummary()

Matching is by don_family string equality at env start — not index, zone name, or nodeset order. Existing single-family CI topologies use don_family = "test-don-family" on every nodeset.

Example env-start log:

Gateway don_family pairs: feeds-zone-a → gateway-zone-a (don_family=feeds-zone-a), feeds-zone-b → gateway-zone-b (don_family=feeds-zone-b)

don_family must match across the stack

Stage What uses don_family
Topology TOML (env start) Pairing, node config, resolver defaults
Capabilities registry DonFamilies: [don_family]
Workflow registry SetDONLimit per family
Workflow deploy Written into UpsertWorkflow
Workflow nodes (runtime) Registry workflow sync filter

Workflow deploy target resolution

When local CRE state exists, deploy picks the workflow DON in this order:

  1. --workflow-don-name — exact match on nodesets.name (escape hatch for shards and ambiguous families)
  2. --don-family — unique workflow DON with that family; use --shard-index when several shard DONs share the family (e.g. shard0 / shard1 both test-don-family)
  3. Single workflow DON — auto-selected when the topology has only one

--container-name-pattern is copy-only (docker cp substring). When omitted, it defaults to {resolvedDon.Name}-node (matches feeds-zone-a-node0, etc.).

Deploy guard: when both --workflow-don-name and --don-family are set, deploy errors if they disagree with local CRE state.

Multi-zone deploy (one family per zone — --don-family is enough):

go run . env workflow deploy \
  --don-family feeds-zone-a \
  --don-id <id> \
  ...

Sharded deploy (same family, multiple shard DONs):

go run . env workflow deploy \
  --don-family test-don-family \
  --shard-index 0 \
  ...

Explicit DON name:

go run . env workflow deploy \
  --workflow-don-name feeds-zone-a \
  ...

Deploy without local CRE state requires --don-family explicitly (no default family).

Test plan

  • go test ./system-tests/lib/cre/...
  • go test ./core/scripts/cre/environment/environment/...
  • go build ./core/scripts/cre/environment/...
  • CI CRE smoke/regression (grpc_source, E2E)
  • Multi-zone Docker E2E via chainlink-data-feeds monitoring/local-cre

@github-actions

github-actions Bot commented Jun 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@trunk-io

trunk-io Bot commented Jun 3, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

@cawthorne
cawthorne force-pushed the feature/cre-sharded-gateway-pairing branch from fff3cc1 to ea10e36 Compare June 4, 2026 12:57
@cawthorne
cawthorne marked this pull request as ready for review June 4, 2026 16:56
@cawthorne
cawthorne requested review from a team as code owners June 4, 2026 16:56
@product-security-plaid-production
product-security-plaid-production Bot requested review from chudilka1 and george-dorin and removed request for chudilka1 June 4, 2026 16:56
@cawthorne cawthorne changed the title Local CRE per DON gateway pairing Local CRE per Zone Gateway Node pairing Jun 8, 2026
Comment thread system-tests/lib/cre/topology.go Resolved
Comment thread system-tests/lib/cre/topology.go Resolved
Comment thread system-tests/lib/cre/topology.go Resolved
Comment thread system-tests/lib/cre/topology.go Resolved
Comment thread system-tests/lib/cre/topology.go Resolved
Comment thread system-tests/lib/cre/types.go Resolved
@cawthorne cawthorne changed the title Local CRE per Zone Gateway Node pairing Local CRE per DON Family Gateway Node pairing Jun 9, 2026
@cawthorne
cawthorne marked this pull request as draft June 9, 2026 10:43
RegisterWithContract now requires a DON family; expose DonFamily on Don
and default to test-don-family in test helpers when topology omits it.
@cawthorne cawthorne changed the title Local CRE per DON Family Gateway Node pairing Local CRE per DON family gateway pairing Jun 9, 2026
cawthorne added 4 commits June 9, 2026 12:06
Legacy single-gateway CI configs (bootstrap-gateway without don_family)
keep all-to-all wiring; pairing and validation activate only when a
workflow or gateway nodeset sets don_family.
Resolve config.go conflict by combining per-family connector scoping
with upstream ToConnectorGateway(). Log pairing via structured logger,
default legacy workflow deploy to test-don-family, and add tests for
partial don_family validation and pairing summary output.
Gateway worker jobs scope via GatewayServiceConfigsForGateway; deploy,
cap-reg, and workflow registry setup require explicit don_family. Add
don_family to capabilities nodesets in CI TOMLs and inline unit-test
fixtures after removing shared topology test helpers.
@cawthorne cawthorne changed the title Local CRE: don_family gateway pairing and deploy target resolution Local CRE: require don_family on all nodesets and scope gateway pairing by family Jun 24, 2026
cawthorne and others added 9 commits June 24, 2026 13:57
Drop redundant empty-family guards now that every nodeset requires
don_family, log pairing only on gateway topologies, and fix gateway URL
wording in state resolver comments.
Remove unused test topology helpers and apply goimports alignment in topology.go.
Gateway connector configs omit Incoming.Host by design; tests and deploy
must fall back to infra ExternalGatewayHost like the state resolver already does.
Required after making don_family mandatory on every nodeset.
Filter each gateway service's DON list by don_family instead of replacing
it with workflow DON names only. Vault registers on the capabilities DON;
replacing with workflow broke gateway→vault OCR routing and caused E2E timeouts.
@cawthorne
cawthorne marked this pull request as ready for review June 24, 2026 18:20
Tofel
Tofel previously approved these changes Jun 25, 2026
Resolve don_family from local CRE state for run-por-example and env start --with-example so registry registration satisfies the new required field.
@cl-sonarqube-production

Copy link
Copy Markdown

[[nodesets]]
nodes = 5
name = "workflow"
don_family = "test-don-family"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically one DON can belong to multiple families. We will need that for sharding but can be extended later. Also it might be better to use "zone-a" here to reflect what's in prod.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants