phased devenv: finish components and remove fallback by winder · Pull Request #1133 · smartcontractkit/chainlink-ccv

winder · 2026-05-21T22:15:31Z

PR Chain

There are multiple open pull requests that feed into each other:

Description

This PR continues work on the phased devenv to move all logic into components. The legacy fallback component has been removed.

Token Verifier Component: added the final new component.
Drop CL node support: this had odd dependencies with the committeeCCV and is not part of the long term vision. If needed, it can be added back in with a future PR.
CommitteeCCV Component: absorbed missing business logic (HMAC credential generation, verifier service management, TLS certificate logic, aggregator service management).
Executor Component: absorbed launch logic, it now has logic in multiple phases (to be addressed in future PR)

Testing

No new functionality, existing regression tests pass.

CI was temporarily updated to run all smoke tests with the phased environment to ensure more complete regression test coverage.

Checklist

Breaking changes documented in changelog (see changelog directory)
Cross link related PRs (in this or other repositories)
just lint fix - no new lint errors
just generate - mocks and protobufs are up to date

The phased devenv's legacy fallback no longer launches CL nodes, registers them with JD, or builds the per-chain impls/ClientLookup needed to drive them. With CL nodes gone from this path, HMAC credential generation no longer needs to precede CL node startup, unblocking a future move into CommitteeCCV. Monolith mode (environment_monolith.go) is untouched and continues to support CL nodes — shared helpers (launchCLNodes, ExpandForHA, NewNodeSetClientLookup, RegisterNodesWithJD, ConnectNodesToJD) and Cfg.NodeSets / Cfg.ClientLookup remain available for it. Changes: - legacy_component.go RunPhase2: remove launchCLNodes, the impls slice it required, CL-mode NOP alias extraction, NodeSetClientLookup construction, RegisterNodesWithJD/ConnectNodesToJD, the nodesets priorOutputs read, and the "_cl_client_lookup" output key. - legacy_component.go RunPhase4: drop the ClientLookup restore. - effect_executor.go: drop AcceptPendingJobs (had no ClientLookup to operate on); SyncAndVerifyJobProposals stays. - Drop now-unused imports: cciptestinterfaces, chainreg, ccvshared, simple_node_set.

Stage 2 of the Legacy elimination plan. Previously, CLDF lifecycle was split across three phases: Legacy Phase 2 created and published the CLDF pointer, protocol_contracts Phase 3 called Init/AddAddresses, and Legacy Phase 4 called AddEnvMetadata/PrintCLDFAddresses via runPhasedEnvironmentFinish. protocol_contracts now owns the full CLDF lifecycle: it creates the CLDF struct locally, initializes it, accumulates deployed addresses, finalizes env metadata, and prints the address table — all before returning from RunPhase3. It publishes _cldf in its output map. Legacy Phase 2 no longer publishes _cldf. Legacy Phase 4 reads _cldf from priorOutputs and copies Addresses/EnvMetadata back into Cfg.CLDF so that Store() still persists the correct data to disk. A PrintAddresses() method is added to cldf.CLDF so protocol_contracts can print the address table without importing the root ccv package.

protocol_contracts now decodes [environment_topology] and [protocol_contracts].use_legacy_configure_lane directly from rawConfig using the same toml marshal/unmarshal round-trip pattern used by other components (fake, executor, etc.). It no longer reads these values from the priorOutputs phase snapshot. Legacy RunPhase2 stops publishing "_environment_topology" and "_use_legacy_configure_lane". These were the only two keys in that map that existed solely as a pass-through for protocol_contracts — now that protocol_contracts reads rawConfig directly, the pass-through is unnecessary. A decodeTopology helper is added to protocol_contracts/component.go following the same pattern: marshal the raw any into a wrapper struct, then unmarshal into *ccvdeployment.EnvironmentTopology.

…v component Stage 4 of the Legacy component elimination. Moves HMAC credential generation, standalone verifier launch, JD registration, topology enrichment, aggregator config generation, and aggregator container launch from the legacy fallback and protocol_contracts Phase 3 into the CommitteeCCV Phase 4 component. - `committeeverifier/launch.go` (new): exports `LaunchStandaloneVerifiers` and `RegisterStandaloneVerifiersWithJD`, extracted from `environment.go`. The `modifiers` parameter avoids the `chainreg`→`committeeverifier` import cycle. - `deploy/deploy.go`: adds `EnrichTopologyWithVerifiers` exported wrapper around `enrichEnvironmentTopology` for use by CommitteeCCV. - `components/committeeccv/component.go`: expanded `RunPhase4` to perform all CommitteeCCV setup in one pass — HMAC gen, TLS assignment, verifier launch + JD registration, topology enrichment, aggregator config generation, and aggregator container launch. - `components/protocol_contracts/component.go`: removes GenerateAggregatorConfig loop and no longer reads verifiers/TLS from priorOutputs; passes `nil` verifiers to `BuildEnvironmentTopology`. - `legacy_component.go`: removes HMAC gen loop, verifier launch, and JD registration from RunPhase2; TLS cert gen and the `verifiers`/`aggregators` output keys remain for downstream Phase 4 components.

…tion Three bugs fixed to get all e2e tests passing after the committeeccv extraction (stage 4): - loadRaw used maps.Copy (shallow) to merge overlay config files, which replaced the entire environment_topology table rather than merging sub-keys. Replace with deepMergeMaps so overlays like env-one-exec-per-chain-standalone.toml can specify only executor_pools without wiping out nop_topology and committees. - The indexer component decodes its own TOML inputs (new pointers separate from cfg.Indexer). NewPhasedEnvironment now replaces cfg.Indexer with the component's started inputs before Store(cfg), so env-out.toml has populated Out fields and chaos tests can read the container name without a nil panic. - Verifier launch, HMAC credential generation, and JD registration were moved too early into CommitteeCCV Phase 4; ConnectAllChainsCanonical in protocol_contracts Phase 3 needs verifier JD node IDs. Restore those 40 lines to Legacy Phase 2 and pass verifiers through to BuildEnvironmentTopology.

…age 6) Move executor and verifier job spec generation out of the legacy runPhasedEnvironmentFinish and into their respective Phase 4 components. - Executor component gains RunPhase4: reads _env, _topology, _ds, and blockchainOutputs; inlines buildExecutorJobSpecs (from generateExecutorJobSpecs in environment.go) with a local executorJobSpec type; emits JobProposalEffect per standalone executor. - CommitteeCCV component RunPhase4 is extended: reads _ds and blockchainOutputs; after launching aggregators calls buildVerifierJobSpecEffects, which inlines generateVerifierJobSpecs and the verifier loop from runPhasedEnvironmentFinish. All verifier input side effects (GeneratedConfig, GeneratedJobSpecs, VerifierID, TLSCACertFile) are preserved. Returns verifier JobProposalEffects. - runPhasedEnvironmentFinish is simplified to (in, e, ds, timeTrack) -> error: retains aggregator endpoint collection and datastore seal only. - Legacy RunPhase4 drops reads for _topology, shared_tls_certs, _selectors (only needed by the removed job spec sections), updates the call site, and returns nil effects.

…→2 and committeeccv→3 Completes the phased devenv refactor by deleting the legacy fallback component and distributing its remaining responsibilities. - Delete legacy_component.go; SetFallback is no longer called in production code. - committeeccv: rename RunPhase4 → RunPhase3; absorb HMAC credential generation, standalone verifier launch + JD registration, and shared TLS cert generation that previously lived in legacy RunPhase2. - protocol_contracts: rename RunPhase3 → RunPhase2; pass nil verifiers to BuildEnvironmentTopology (enrichment now done by committeeccv Phase 3 via pointer mutation on the shared topology). - environment_phased.go: remove runPhasedEnvironmentFinish and all legacy key reads; reconstruct Cfg directly via Load[Cfg] + post-processing from runtime output map. Phase layout after this change: Phase 1: blockchains, jd, fake Phase 2: protocol_contracts Phase 3: committeeccv, executor, pricer Phase 4: indexer, tokenverifier, executor (job specs)

NewPhasedEnvironment was loading cfg.Blockchains from TOML (no Out populated), then storing that incomplete data to env-out.toml. Tests calling NewLibFromCCVEnv panicked at b.Out.Family because the blockchain output (RPC URLs, chain info) was absent. Fix: sync cfg.Blockchains from the Phase 1 runtime output before Store(), so the populated Out fields are written to env-out.toml.

After phased startup, `NewPhasedEnvironment` reconstructs `*Cfg` by loading from TOML then syncing runtime-populated `Out` fields back into cfg before `Store(cfg)`. The executor and fake service inputs were missing from this sync, causing nil pointer panics in tests that access `executor.Out.ContainerName` (chaos tests) and `fake.Out.ExternalHTTPURL` (token verification test). Add the two missing sync blocks to `environment_phased.go` so `env-out.toml` is written with fully-populated Out fields for all services.

Copilot

Pull request overview

This PR completes another step in the phased devenv migration by removing the legacy fallback component, shifting responsibilities into dedicated components, and explicitly dropping Chainlink node support from the phased startup path.

Changes:

Removed legacy_component.go fallback and updated phased environment reconstruction to derive *Cfg from TOML + component outputs.
Moved key business logic into components: committeeccv (Phase 3), executor (Phase 4 job proposals), and added a new tokenverifier (Phase 4) component.
Improved config overlay behavior by deep-merging nested TOML tables in loadRaw.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
changelog/2026-05-23_devenv_phased_legacy_elimination.md	Documents the phased devenv breaking changes (legacy removal, CL node removal, phase shifts).
build/devenv/services/committeeverifier/launch.go	Exposes verifier launch + JD registration helpers for component reuse.
build/devenv/legacy_component.go	Deletes the legacy fallback component and its legacy output keys.
build/devenv/environment.go	Registers the new tokenverifier component via side-effect import.
build/devenv/environment_phased.go	Rebuilds `*Cfg` from TOML + runtime outputs instead of reading legacy fallback output.
build/devenv/effect_executor.go	Removes CL-node-specific `AcceptPendingJobs` behavior from phased effects.
build/devenv/deploy/deploy.go	Exports `EnrichTopologyWithVerifiers` wrapper for component access.
build/devenv/config.go	Changes raw config merging from shallow copy to recursive deep merge.
build/devenv/components/tokenverifier/component.go	Adds tokenverifier Phase 4 component (changeset-driven config + container launch).
build/devenv/components/protocol_contracts/component.go	Moves protocol_contracts to Phase 2 and decodes topology from global config.
build/devenv/components/executor/component.go	Adds Phase 4 executor job spec generation and job proposal effects.
build/devenv/components/committeeccv/component.go	Moves committee business logic into Phase 3 (creds, TLS, verifier/agg launch, lane config, verifier job proposals).
build/devenv/cldf/cldf.go	Adds `CLDF.PrintAddresses()` helper used during phased startup.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

winder · 2026-05-25T13:45:47Z

-const legacyCfgKey = "_legacy_cfg"
-
-func init() {
-	devenvruntime.SetFallback(legacyFactory)


Removing this fallback component is the milestone I've been working towards. The business logic is now fully encapsulated in the different components.

winder · 2026-05-25T13:49:51Z

-	if cfg, ok := accumulated[legacyCfgKey].(*Cfg); ok && cfg != nil && cfg.ClientLookup != nil {
-		if err := jobs.AcceptPendingJobs(ctx, cfg.ClientLookup); err != nil {
-			return fmt.Errorf("accepting pending jobs: %w", err)
-		}
-	}


This is not needed because the standalone deployment will auto accept the job.

1. committeeccv: panic on empty aggNames — GetAggregatorNamesForCommittee can return an empty slice, causing a division-by-zero at ver.NodeIndex % len(aggNames). Add an explicit empty-check error before validation. Also remove the len > 1 guard so validateVerifierNodeIndices runs for single-aggregator committees too. 2. cldf: defer w.Flush() inside a loop — each iteration deferred a Flush on a fresh tabwriter.Writer; all defers fired LIFO at function return, producing reordered output. Replace with an explicit w.Flush() at the end of each loop iteration.

github-actions · 2026-05-26T14:50:21Z

Code coverage report:

Package	`main`	`will/committeeccv-component`	Diff
github.com/smartcontractkit/chainlink-ccv/aggregator	49.35%	49.31%	-0.04%
github.com/smartcontractkit/chainlink-ccv/bootstrap	54.14%	54.14%	+0.00%
github.com/smartcontractkit/chainlink-ccv/cli	65.13%	65.13%	+0.00%
github.com/smartcontractkit/chainlink-ccv/cmd	15.54%	15.54%	+0.00%
github.com/smartcontractkit/chainlink-ccv/common	56.54%	56.54%	+0.00%
github.com/smartcontractkit/chainlink-ccv/executor	45.97%	45.97%	+0.00%
github.com/smartcontractkit/chainlink-ccv/indexer	37.68%	37.72%	+0.04%
github.com/smartcontractkit/chainlink-ccv/integration	45.84%	45.84%	+0.00%
github.com/smartcontractkit/chainlink-ccv/pkg	84.62%	84.62%	+0.00%
github.com/smartcontractkit/chainlink-ccv/pricer	0.00%	0.00%	+0.00%
github.com/smartcontractkit/chainlink-ccv/protocol	65.88%	65.88%	+0.00%
github.com/smartcontractkit/chainlink-ccv/verifier	34.48%	34.48%	+0.00%
Total	46.50%	46.50%	+0.00%

ogtownsend · 2026-05-28T04:45:11Z

+// EnrichTopologyWithVerifiers enriches an existing topology in-place with signer addresses
+// derived from verifier bootstrap keys. Call this after verifiers are launched and their
+// Out.BootstrapKeys are populated. The topology pointer is mutated directly so that other
+// Phase 4 components reading the same pointer see the updated signer addresses.
+func EnrichTopologyWithVerifiers(topology *ccvdeployment.EnvironmentTopology, verifiers []*committeeverifier.Input) {
+	enrichEnvironmentTopology(topology, verifiers)
+}
+


the docs in environment.go say that "intra-phase data isolation is enforced through cloned snapshots", but this seems like it's supporting and allowing mutability? maps.Clone used all throughout environment.go is shallow so pointers are shared across all downstream components/phases.

is it possible to enforce true immutability? It's hard for me to tell where mutability is supported vs where it's not

Good point about the shallow copy. I wish go had better support for immutable data structures so that it was easier to enforce the design intent.

This iteration has gotten the business logic into components. The next few iterations are going to be about cleaning up the components in several ways:

Improved data types:

adding versions to component config

collapsing the multiple top-level CommitteeCCV types (environment_topology, aggregator, verifier)

better defaults (i.e. avoid 16 copies of a field called default-verifier-1, default-verifier-2, ...)

Reduced coupling across components (your other comment).

Removal of the "topology file", this is WIP on the changeset side.

ogtownsend · 2026-05-28T04:51:05Z

+		if useLegacyConfigureLane {
+			connectErr = ccdeploy.ConnectAllChainsLegacy(impls, blockchains, selectors, e, topology)
+		} else {
+			connectErr = ccdeploy.ConnectAllChainsCanonical(impls, blockchains, selectors, e, topology)
+		}


this feels like it should be in protocol_contracts but seems to be here just because it depends on the JD registration happening before it? curious if you tried to explore a DAG system for each step within a phase as opposed to bundled phases?

...I guess it could be considered CCV related 🤷 🤔

Agreed. "contracts" and "committee ccv" were very tightly coupled. The changeset still is. I'm planning to continue refining these components, this PR is a broad stroke to get things close.

There are two "levels" of lane connection in v2 - configuring source/dest chains in the ramps and then configuring the CCV's source/dest chains, the former is protocol level, latter is CCV level. I think the changeset right now does both together.

The changesets are evolving, we now have a "protocol" changeset (named something like "DeployProtocolContracts" to go along with the "protocol+committee" changeset (Named something like "Apply").

huangzhen1997 · 2026-05-28T23:24:55Z

+func validateVerifierNodeIndices(committeeName string, verifiers []*committeeverifier.Input) error {
+	seen := make(map[int]string, len(verifiers))
+	for _, ver := range verifiers {
+		if existing, dup := seen[ver.NodeIndex]; dup {


Seems like the old verifier/aggregator counts validation is dropped, is it intentional?

Yes, I think the logic was wrong before, there were additional gates prior to this call which would disable all checks in some conditions. This is a more general verification that will be expanded soon when I unify the aggregator/verifier configs into a single top-level [[committeeCCV]] section.

makramkd · 2026-06-02T06:41:10Z

+		if useLegacyConfigureLane {
+			connectErr = ccdeploy.ConnectAllChainsLegacy(impls, blockchains, selectors, e, topology)
+		} else {
+			connectErr = ccdeploy.ConnectAllChainsCanonical(impls, blockchains, selectors, e, topology)
+		}


There are two "levels" of lane connection in v2 - configuring source/dest chains in the ramps and then configuring the CCV's source/dest chains, the former is protocol level, latter is CCV level. I think the changeset right now does both together.

makramkd · 2026-06-02T06:42:13Z

+func stringSlicesEqual(a, b []string) bool {
+	if len(a) != len(b) {
+		return false
+	}
+	for i := range a {
+		if a[i] != b[i] {
+			return false
+		}
+	}
+	return true
+}


nit: prefer slices.Equal

makramkd · 2026-06-02T06:44:51Z

+}
+
+// decodeVerifiers round-trips the raw TOML []any into []*committeeverifier.Input.
+func decodeVerifiers(raw any) ([]*committeeverifier.Input, error) {


nit: decodeAggregators and decodeVerifiers seem identical apart from the return value, could there instead be a single generic helper function instead? Could be in some common package as I imagine this technique is used in many places.

makramkd · 2026-06-02T06:45:33Z

+}
+
+// decodeAggregators round-trips the raw TOML []any into []*services.AggregatorInput.
+func decodeAggregators(raw any) ([]*services.AggregatorInput, error) {


nit: this is missing the if raw == nil { return nil } guard.

makramkd · 2026-06-02T06:48:13Z

 		return map[string]any{}, nil, nil
 	}

+	jdInfra, ok := priorOutputs["jd"].(*jobs.JDInfrastructure)


question: Is there a plan to have constants for the keys that can be expected to be available in priorOutputs? I do see e.g. "_env" being used across the board a lot.

Constants for the identifier key probably make sense. I'd also like to find a way to make the types a bit easier to work with. I have a local issue to explore a special "bus" wrapper that supports types.

Feedback from #1133 Add devenvruntime.DecodeConfig[T] to replace the identical marshal/unmarshal boilerplate repeated across all 10 component decode functions; also replace stringSlicesEqual with slices.Equal per review feedback. Each component's decode function is now a single typed call plus its own version check, removing ~80 lines of duplicated code.

Feedback for #1133 Add an early nil check to DecodeConfig so that a missing config section (raw == nil) returns the zero value of T without error, matching the guard that was required on the original per-component decode functions.

winder added 3 commits May 21, 2026 18:14

winder changed the title ~~build/devenv: phased devenv~~ phased devenv: remove CL node support May 22, 2026

winder added 7 commits May 22, 2026 08:23

Temporarily enable phased devenv for all tests.

8cd7270

winder commented May 23, 2026

View reviewed changes

Comment thread .github/workflows/test-smoke.yaml

winder added 2 commits May 23, 2026 09:27

Add changelog.

236006d

Revert temporary CI modifications.

ff58ffe

winder marked this pull request as ready for review May 25, 2026 00:25

Copilot AI review requested due to automatic review settings May 25, 2026 00:25

winder requested review from a team as code owners May 25, 2026 00:25

Copilot started reviewing on behalf of winder May 25, 2026 00:25 View session

Copilot AI reviewed May 25, 2026

View reviewed changes

Comment thread build/devenv/components/committeeccv/component.go

Comment thread build/devenv/components/committeeccv/component.go

Comment thread build/devenv/cldf/cldf.go Outdated

winder commented May 25, 2026

View reviewed changes

winder added 2 commits May 26, 2026 09:47

Relax unclaimed key check.

46ad62b

winder force-pushed the will/committeeccv-component branch from f9e9ef7 to 46ad62b Compare May 26, 2026 14:17

Relax committeeccv error checking.

47764ea

winder changed the title ~~phased devenv: remove CL node support~~ phased devenv: finish components and remove fallback May 26, 2026

winder enabled auto-merge May 26, 2026 16:57

ogtownsend reviewed May 28, 2026

View reviewed changes

ogtownsend approved these changes May 28, 2026

View reviewed changes

huangzhen1997 reviewed May 28, 2026

View reviewed changes

makramkd reviewed Jun 2, 2026

View reviewed changes

makramkd approved these changes Jun 2, 2026

View reviewed changes

winder added this pull request to the merge queue Jun 2, 2026

makramkd approved these changes Jun 2, 2026

View reviewed changes

Merged via the queue into main with commit 49f4618 Jun 2, 2026
36 checks passed

winder deleted the will/committeeccv-component branch June 2, 2026 12:20

winder mentioned this pull request Jun 2, 2026

build/devenv: Review feedback from earlier pull requests. #1148

Draft

4 tasks

Conversation

winder commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Chain

Description

Testing

Checklist

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

winder commented May 21, 2026 •

edited

Loading