Skip to content

perf(ci): release-image first so arm64 unblocks during amd64 npm publish#23434

Draft
AztecBot wants to merge 1 commit into
nextfrom
cb/d8fbda83eab0
Draft

perf(ci): release-image first so arm64 unblocks during amd64 npm publish#23434
AztecBot wants to merge 1 commit into
nextfrom
cb/d8fbda83eab0

Conversation

@AztecBot
Copy link
Copy Markdown
Collaborator

Problem

ci-release-publish takes ~20 minutes. Breakdown of the amd64 instance for v5.0.0-nightly.20260518 (log):

Phase Wall Notes
EC2 spot + devbox + git fetch ~75s overhead
make release (build) ~3m all upstream artifacts cache-hit
npm/cargo/github publish loop ~11m 17s ~30 packages, serial
release-image push + manifest list ~45s

arm64 (log) does its release-image push at 03:29:23 (~3m in) and then sits idle for ~11m 39s in release-image/bootstrap.sh's Waiting for amd64 image to be pushed... poll loop, because the multi-arch manifest can only be stitched after amd64's docker push. amd64 doesn't get to its docker push until after the npm loop finishes.

Fix

Move release-image to the front of the projects array in bootstrap.sh:release. amd64 now docker-pushes its arch tag immediately after the build phase (~03:30 instead of ~03:41). arm64's poll loop terminates at the next 10s tick and the manifest list is created while amd64 is still publishing npm packages.

Expected effect:

  • Total ci-release-publish wall time roughly unchanged (amd64 still has ~11m of serial npm publishing to do — that's a separate optimization).
  • arm64 EC2 spot terminates ~11m earlier → ~11m of arm64 spot time saved per release.
  • Multi-arch docker tag is available to downstream consumers (anyone watching dockerhub for the release tag) ~11m earlier.

Risk / why this is safe

The 11 entries in the projects array are all independent at publish time — each calls its own <project>/bootstrap.sh release, writing to npm, cargo, github releases, or dockerhub. No cross-project ordering dependency exists. Concrete checks:

  • aztec-up/bootstrap.sh release uploads installer scripts to S3 (no docker dependency at publish time; the docker dependency is at build time and make release has already run).
  • release-image/bootstrap.sh release only needs the locally-built aztecprotocol/aztec:$COMMIT_HASH image, which make release has produced before this loop runs.
  • The arm64 branch (projects=(release-image)) is unchanged in behavior.

If anything in aztec-up did depend on the docker tag being pushed, the existing order had it running before release-image anyway, so moving release-image earlier strictly cannot regress that.

Test plan

  • CI green on this PR (no actual release runs, so the publish flow itself isn't exercised — verified by inspection + the next nightly tag).
  • After merge, watch the next nightly release pipeline and confirm arm64's Waiting for amd64... window shrinks to seconds.

Created by claudebox · group: slackbot

amd64's ci-release-publish loop spends ~11 minutes publishing ~30 npm
packages serially. arm64 finishes its docker push in ~3 minutes, then
sits idle in release-image/bootstrap.sh's 'Waiting for amd64 image to
be pushed...' poll until amd64 reaches the same step.

Reordering 'release-image' to the front of the projects array lets
arm64 stitch the multi-arch manifest while amd64 is still publishing
npm/cargo/github artifacts, eliminating the arm64 idle window.
@AztecBot AztecBot added the claudebox Owned by claudebox. it can push to this PR. label May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant