feat(makefile): extended-image build pipeline (Story 13.4b)#37
Merged
matthew-on-git merged 7 commits intomainfrom May 4, 2026
Merged
feat(makefile): extended-image build pipeline (Story 13.4b)#37matthew-on-git merged 7 commits intomainfrom
matthew-on-git merged 7 commits intomainfrom
Conversation
Completes Story 13.4 (Story 13.4a shipped foundation in PR #36; this PR wires the full build pipeline). When .devrail.yml declares one or more plugins, `make check` (and lint/format/fix/test/security) now auto-builds a project-local extended image (devrail-local:<hash>) that includes core dev-toolchain tools + each plugin's container fragment. Cache hits are free. Components: - scripts/plugin-extended-image.sh: HOST-side orchestrator. Stages install scripts from the host plugin cache into a build-context staging dir (.devrail-plugins-build/), runs `make _generate-dockerfile` inside a container to emit Dockerfile.devrail (which depends on _plugins-load to populate the loader cache), hashes the dockerfile for the tag, checks `docker image inspect` for cache hit, otherwise invokes `docker build`, and writes the resolved tag to .devrail/extended-image-tag. - Makefile: new _generate-dockerfile (in-container) + _extended-image (host) targets. _extended-image wired as a prereq of every public host target that uses DOCKER_RUN. New DEVRAIL_RESOLVED_IMAGE Make variable (recursively-expanded) reads .devrail/extended-image-tag when plugins are declared, else falls back to the core image. DOCKER_RUN switched from := to = expansion to support the swap-in. - tests/fixtures/plugin-repos/minimal-v1/: minimal hermetic fixture with no apt_packages and no copy_from_builder paths — fastest possible build for smoke testing the pipeline end-to-end. - tests/test-plugin-build-pipeline.sh: 9 cases (4 generator unit from 13.4a + 5 full-pipeline). New cases: - Case 5: real docker build → devrail-local:<hash> exists, install script ran (verified via marker file inside the image), env applied. - Case 6: cache hit on second invocation (< 30s end-to-end ceiling). - Case 7: no-plugins regression (no Dockerfile.devrail, no tag file). - Case 8: install_script that exits 1 → structured `error` event with stderr_tail + duration_ms. - Smoke tests bypass `make plugins-update` (which needs a network- reachable git source) by pre-populating the host cache directly and hand-crafting a matching .devrail.lock with the correct content_hash. This isolates the build pipeline as the system under test. Documentation: - CHANGELOG: [Unreleased] Added entry for the full pipeline. - STABILITY: row updated from "Plugin loader + resolver + lockfile" to "Plugin loader + resolver + lockfile + build pipeline" (still Preview; Story 13.5 ships plugin command execution). Test results (local, against freshly built image): - tests/test-plugin-build-pipeline.sh — 9/9 - tests/test-plugin-resolver.sh — 16/16 (regression-safe) - tests/test-plugin-loader.sh — 11/11 (regression-safe) - tests/smoke-rails.sh — 4/4 (regression-safe) - make check on dev-toolchain itself — pass Implementation notes: - DOCKER_RUN swap-in uses recursive expansion (=) so the tag file is re-read each invocation. Without this, immediate evaluation (:=) would capture the tag at make-time before _extended-image had a chance to produce it. - HAS_PLUGINS_DECLARED uses a `yq | awk` pipeline at make-time to detect non-empty plugins:; cheap. - Test fixtures in tests/fixtures/plugin-repos/minimal-v1/ are hermetic (no network, no apt) so the docker-build step is fast enough to run in CI without timeouts. Scope boundary: 13.4 closes here. Story 13.5 (next) implements the plugin command execution loop — the part that actually runs plugin- defined targets during _lint/_format/etc. inside the extended image. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
H1: probe `.devrail.yml` parse error vs no-plugins via DEVRAIL_PLUGIN_PROBE,
so a malformed config fails Make loudly instead of silently skipping the
extended-image build.
H2: source the host orchestrator from the dev-toolchain image when consumers
inherit only the Makefile (no `scripts/` dir locally) — extracted to
`.devrail/host-bin/`, cached and invalidated by image tag.
M1+L6: flock per-workspace `.devrail/.build.lock` so concurrent `make check`
invocations on the same checkout don't race on STAGING_DIR or
Dockerfile.devrail.
M2: source `lib/plugin-cache.sh` in the orchestrator and use the shared
`derive_slug` helper instead of duplicating the basename/.git logic.
M3: surface `make plugins-update` as the actionable hint when the host
cache is empty.
L1: document host-side requirements (yq v4+, sha256sum, flock, docker buildx)
in STABILITY.md.
L2: WORKSPACE override via DEVRAIL_WORKSPACE env for testability.
L3: BUILD_LOG cleanup folded into the EXIT trap.
L4: forward DEVRAIL_QUIET / DEVRAIL_DEBUG to the in-container generator for
consistent log behaviour.
L5: explicit `docker buildx version` precondition check with a clear error.
Tests:
- L7: two-plugin smoke (Case 10) — exercises the for-loop over plugin entries.
- L8: tighten Case 6 cache-hit ceiling from 30s to 10s (closer to the 1s AC).
- L9: Case 8 now asserts no tag file is written on build failure.
- L10: Case 9 transition test — removing plugins clears the stale tag file.
- L11: Case 11 end-to-end — plugins-update + _extended-image against a
file:// fixture, exercising the full resolver → loader → build path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous H1 fix used `$(error ...)` at Make parse time, which fired before `make _plugins-update` could invoke the resolver script — regressing the resolver-test Case 12 that expects the resolver to surface the "config could not be parsed by yq" event. Move the error to runtime inside `_extended-image` instead. Other targets (`_plugins-update`, `_check`) still parse Make successfully and let their in-container scripts handle malformed YAML with their existing structured events. `_extended-image` itself emits the same event format and exits 2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sition The `if [ -n "$(HAS_PLUGINS_DECLARED)" ]` gate skipped the orchestrator entirely when plugins were removed from `.devrail.yml`, so the orchestrator's tag-file cleanup never ran. DOCKER_RUN would keep referencing a phantom `devrail-local:<hash>` tag from a prior build. Add Makefile-level cleanup in the elif arm of `_extended-image`. Also gate `_devrail-host-bin` extraction on HAS_PLUGINS_DECLARED so consumer repos that don't declare plugins skip the docker-cp roundtrip on every `make check`. Caught by the L10 transition test (Case 9) added in the prior commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…event The transition cleanup now happens at the Makefile level (orchestrator is not invoked when plugins are removed). Assert the Makefile's "plugins removed; clearing stale extended-image tag" event instead of the orchestrator's "no plugins declared; using core image" message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The generator emits `# --- plugin: <name>@<rev> (source: <url>) ---`, not `# plugin:`. Update the count regex anchor accordingly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Case 11 used `make plugins-update` from the host, which goes through the standard DOCKER_RUN — that doesn't mount the fixture path inside the container, so the resolver can't reach `file:///tmp/...`. Mirror Case 4's harness: invoke `_plugins-update` directly via docker run with $WORKDIR bind-mounted at the same path inside the container. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes Story 13.4. PR #36 (Story 13.4a) shipped the foundation; this PR wires the full pipeline. When `.devrail.yml` declares plugins, `make check` (and lint/format/fix/test/security) now auto-builds a project-local extended image (`devrail-local:`) that includes core dev-toolchain tools + each plugin's container fragment. Cache hits are free.
What's in the PR
`scripts/plugin-extended-image.sh` (HOST orchestrator)
Stages install scripts from the host cache, runs the in-container generator (which depends on `_plugins-load`), hashes the resulting Dockerfile for the tag, checks `docker image inspect` for cache hit, otherwise invokes `docker build`, and writes the resolved tag to `.devrail/extended-image-tag`.
Makefile changes
Tests + Fixtures
Docs
Acceptance criteria from Story 13.4
Test plan
Recommended next step
After merge, cut v1.10.3 patch release. Then start Story 13.5 (plugin command execution loop) — the last big story before Story 13.6 packages everything as v1.11.0.
🤖 Generated with Claude Code