feat(makefile): extended-image build pipeline (Story 13.4b)

.gitignore

@@ -32,3 +32,8 @@ build/
 
 # DevRail generated output
 .devrail-output/
+
+# DevRail extended-image build pipeline (Story 13.4b)
+.devrail/
+.devrail-plugins-build/
+Dockerfile.devrail

CHANGELOG.md

@@ -9,6 +9,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Plugin build pipeline complete (Story 13.4b, Epic 13 / v1.10.x preview):
+  - **`make check` auto-builds a project-local extended image.** When
+    `.devrail.yml` declares one or more `plugins:`, the public host targets
+    (`check`, `lint`, `format`, `fix`, `test`, `security`) now run via
+    `_extended-image` first, which generates `Dockerfile.devrail`, builds
+    `devrail-local:<hash-of-dockerfile>` via BuildKit, and points
+    `DOCKER_RUN` at the new tag. Plugin tools are present alongside core
+    tools in a single container — preserving DevRail's "one container,
+    one make check" guarantee.
+  - **Cache hits are free.** `docker image inspect <tag>` is the
+    cache-detection mechanism — unchanged plugin sets reuse the existing
+    image. End-to-end overhead on a cache hit is ~3-5s (mostly the
+    in-container `_generate-dockerfile` step that re-confirms the cache
+    state); the build-vs-rebuild decision itself is instant.
+  - **Build failures surface structured errors.** Failed `docker build`
+    invocations emit a JSON `error` event with `tag`, `duration_ms`, and
+    the last 20 lines of build output as `stderr_tail`. Lockfile and
+    plugin caches are not touched on failure.
+  - **No-plugins regression-safe.** Projects without `plugins:` in
+    `.devrail.yml` see zero behavior change; `_extended-image` is a no-op
+    and `DOCKER_RUN` continues to use the core image.
+  - New `scripts/plugin-extended-image.sh` (host-side orchestrator) and
+    `_extended-image` / `_generate-dockerfile` Makefile targets.
+  - `DEVRAIL_RESOLVED_IMAGE` Make variable: recursively-expanded so it
+    re-evaluates each invocation, picking up
+    `.devrail/extended-image-tag` once `_extended-image` has run.
+  - `DOCKER_RUN` switched from immediate (`:=`) to recursive (`=`)
+    expansion to support the swap-in.
+
 - Plugin build pipeline foundations (Story 13.4a, Epic 13 / v1.10.x preview):
   - **Host-side persistent plugin cache.** `DEVRAIL_HOST_PLUGINS_CACHE`
     Makefile variable (defaults to `${HOME}/.cache/devrail/plugins`) is

Makefile

@@ -27,6 +27,35 @@ DEVRAIL_CONFIG     := .devrail.yml
 # invocations. Story 13.4. Override via env if you keep caches elsewhere.
 DEVRAIL_HOST_PLUGINS_CACHE ?= $(HOME)/.cache/devrail/plugins
 
+# Story 13.4b: when plugins are declared, `make check` etc. build a
+# project-local image (devrail-local:<hash-of-dockerfile.devrail>) and use it
+# for in-container targets. The tag is written to `.devrail/extended-image-tag`
+# by `_extended-image`. DEVRAIL_RESOLVED_IMAGE is recursively-expanded (=) so
+# it re-evaluates each time DOCKER_RUN expands — picking up the tag file once
+# `_extended-image` has run.
+DEVRAIL_RESOLVED_IMAGE = $(if $(and $(wildcard .devrail/extended-image-tag),$(HAS_PLUGINS_DECLARED)),$(shell cat .devrail/extended-image-tag),$(DEVRAIL_IMAGE):$(DEVRAIL_TAG))
+
+# Probe .devrail.yml in a single shell invocation that distinguishes missing
+# file, yq parse error, and valid + plugin-count. Without this, a malformed
+# `.devrail.yml` would silently fall through as "no plugins" and skip the
+# extended-image build (review finding H1). The "error" value is checked
+# explicitly inside `_extended-image` rather than via `$(error ...)` so other
+# targets (e.g., `_plugins-update`) can still run and let their scripts
+# surface the structured parse-error event for the user.
+DEVRAIL_PLUGIN_PROBE := $(shell \
+	if [ ! -r $(DEVRAIL_CONFIG) ]; then \
+		echo "missing"; \
+	elif count=$$(yq -r '.plugins // [] | length' $(DEVRAIL_CONFIG) 2>/dev/null); then \
+		echo "$$count"; \
+	else \
+		echo "error"; \
+	fi)
+
+# HAS_PLUGINS_DECLARED — set when .devrail.yml has a non-empty `plugins:` list.
+# Empty when file is missing, `plugins:` is `[]`/absent, OR yq could not parse
+# the file (the "error" case is caught explicitly inside `_extended-image`).
+HAS_PLUGINS_DECLARED := $(if $(filter-out missing error 0,$(DEVRAIL_PLUGIN_PROBE)),yes,)
+
 # Read project-specific env vars from .devrail.yml `env:` section and inject
 # them as `-e KEY=VALUE` into DOCKER_RUN. Empty/missing section is a no-op.
 DEVRAIL_ENV_FLAGS := $(shell yq -r '.env // {} | to_entries | .[] | "-e " + .key + "=" + .value' $(DEVRAIL_CONFIG) 2>/dev/null)
@@ -69,23 +98,23 @@ HAS_KOTLIN     := $(filter kotlin,$(LANGUAGES))
 # project's, and bundler can't find project-installed gems (issue #30 Gap A).
 RUBY_DOCKER_ENV := $(if $(HAS_RUBY),-e BUNDLE_APP_CONFIG=/workspace/.bundle,)
 
-DOCKER_RUN := docker run --rm \
+DOCKER_RUN = docker run --rm \
 	-v "$$(pwd):/workspace" \
 	-v "$(DEVRAIL_HOST_PLUGINS_CACHE):/opt/devrail/plugins" \
 	-w /workspace \
 	-e DEVRAIL_FAIL_FAST=$(DEVRAIL_FAIL_FAST) \
 	-e DEVRAIL_LOG_FORMAT=$(DEVRAIL_LOG_FORMAT) \
 	$(DEVRAIL_ENV_FLAGS) \
 	$(RUBY_DOCKER_ENV) \
-	$(DEVRAIL_IMAGE):$(DEVRAIL_TAG)
+	$(DEVRAIL_RESOLVED_IMAGE)
 
 .DEFAULT_GOAL := help
 
 # ---------------------------------------------------------------------------
 # .PHONY declarations
 # ---------------------------------------------------------------------------
 .PHONY: help build lint format fix test security scan docs changelog check install-hooks init release plugins-update
-.PHONY: _lint _format _fix _test _security _scan _docs _changelog _check _check-config _init _plugins-update _plugins-verify _ensure-host-cache
+.PHONY: _lint _format _fix _test _security _scan _docs _changelog _check _check-config _init _plugins-update _plugins-verify _ensure-host-cache _generate-dockerfile _extended-image _devrail-host-bin
 
 # ===========================================================================
 # Public targets (run on host, delegate to Docker container)
@@ -99,6 +128,64 @@ DOCKER_RUN := docker run --rm \
 _ensure-host-cache:
 	@mkdir -p "$(DEVRAIL_HOST_PLUGINS_CACHE)"
 
+# --- _devrail-host-bin: extract orchestrator script + libs from container ---
+# Story 13.4b/H2: consumer template repos inherit this Makefile but NOT
+# `scripts/`, so the host orchestrator must be sourced from the container.
+# When the dev-toolchain repo itself runs (scripts/ present locally) we use
+# the on-disk copy so changes take effect without a rebuild. Otherwise we
+# extract scripts + lib from the resolved core image to .devrail/host-bin/,
+# cached and invalidated by image tag (.devrail/host-bin/.image-tag).
+_devrail-host-bin:
+	@if [ -z "$(HAS_PLUGINS_DECLARED)" ]; then \
+		exit 0; \
+	fi; \
+	if [ -f scripts/plugin-extended-image.sh ]; then \
+		exit 0; \
+	fi; \
+	expected="$(DEVRAIL_IMAGE):$(DEVRAIL_TAG)"; \
+	cached=$$(cat .devrail/host-bin/.image-tag 2>/dev/null || true); \
+	if [ "$$cached" = "$$expected" ] && \
+	   [ -f .devrail/host-bin/scripts/plugin-extended-image.sh ] && \
+	   [ -f .devrail/host-bin/lib/log.sh ] && \
+	   [ -f .devrail/host-bin/lib/plugin-cache.sh ]; then \
+		exit 0; \
+	fi; \
+	mkdir -p .devrail/host-bin/scripts .devrail/host-bin/lib; \
+	echo '{"level":"info","msg":"extracting host orchestrator from container","image":"'"$$expected"'","language":"_plugins"}' >&2; \
+	cid=$$(docker create "$$expected" /bin/true) || { \
+		echo '{"level":"error","msg":"docker create failed for host-bin extraction","image":"'"$$expected"'","language":"_plugins"}' >&2; \
+		exit 2; \
+	}; \
+	trap 'docker rm "$$cid" >/dev/null 2>&1 || true' EXIT; \
+	docker cp "$$cid":/opt/devrail/scripts/plugin-extended-image.sh .devrail/host-bin/scripts/plugin-extended-image.sh && \
+	docker cp "$$cid":/opt/devrail/lib/log.sh             .devrail/host-bin/lib/log.sh && \
+	docker cp "$$cid":/opt/devrail/lib/plugin-cache.sh    .devrail/host-bin/lib/plugin-cache.sh && \
+	chmod +x .devrail/host-bin/scripts/plugin-extended-image.sh && \
+	printf '%s\n' "$$expected" > .devrail/host-bin/.image-tag
+
+# --- _extended-image: build the project-local image when plugins declared ---
+# Story 13.4b: HOST-side target. When .devrail.yml declares no plugins, this
+# is a no-op (DOCKER_RUN keeps using the core image). When plugins ARE
+# declared, the orchestrator script generates Dockerfile.devrail, builds
+# devrail-local:<hash>, and writes the tag to .devrail/extended-image-tag
+# so the recursive DOCKER_RUN picks it up. Cache hits are free.
+_extended-image: _ensure-host-cache _devrail-host-bin
+	@if [ "$(DEVRAIL_PLUGIN_PROBE)" = "error" ]; then \
+		echo '{"level":"error","msg":"config could not be parsed by yq","path":"$(DEVRAIL_CONFIG)","language":"_plugins","script":"_extended-image"}' >&2; \
+		exit 2; \
+	fi; \
+	if [ -n "$(HAS_PLUGINS_DECLARED)" ]; then \
+		if [ -f scripts/plugin-extended-image.sh ]; then \
+			bash scripts/plugin-extended-image.sh; \
+		else \
+			DEVRAIL_LIB="$$(pwd)/.devrail/host-bin/lib" \
+				bash .devrail/host-bin/scripts/plugin-extended-image.sh; \
+		fi; \
+	elif [ -f .devrail/extended-image-tag ]; then \
+		echo '{"level":"info","msg":"plugins removed; clearing stale extended-image tag","language":"_plugins","script":"_extended-image"}' >&2; \
+		rm -f .devrail/extended-image-tag; \
+	fi
+
 help: ## Show this help
 	@echo "DevRail dev-toolchain — container image build and validation"
 	@echo ""
@@ -111,16 +198,16 @@ build: ## Build the container image locally
 changelog: _ensure-host-cache ## Generate CHANGELOG.md from conventional commits
 	$(DOCKER_RUN) make _changelog
 
-check: _ensure-host-cache ## Run all checks (lint, format, test, security, scan, docs)
+check: _ensure-host-cache _extended-image ## Run all checks (lint, format, test, security, scan, docs)
 	$(DOCKER_RUN) make _check
 
 docs: _ensure-host-cache ## Generate documentation
 	$(DOCKER_RUN) make _docs
 
-fix: _ensure-host-cache ## Auto-fix formatting issues in-place
+fix: _ensure-host-cache _extended-image ## Auto-fix formatting issues in-place
 	$(DOCKER_RUN) make _fix
 
-format: _ensure-host-cache ## Run all formatters
+format: _ensure-host-cache _extended-image ## Run all formatters
 	$(DOCKER_RUN) make _format
 
 install-hooks: ## Install pre-commit hooks
@@ -148,7 +235,7 @@ install-hooks: ## Install pre-commit hooks
 init: _ensure-host-cache ## Scaffold config files for declared languages
 	$(DOCKER_RUN) make _init
 
-lint: _ensure-host-cache ## Run all linters
+lint: _ensure-host-cache _extended-image ## Run all linters
 	$(DOCKER_RUN) make _lint
 
 plugins-update: _ensure-host-cache ## Resolve plugin refs and write .devrail.lock
@@ -161,13 +248,13 @@ release: ## Cut a versioned release (usage: make release VERSION=1.6.0)
 	fi
 	@bash scripts/release.sh $(VERSION)
 
-scan: _ensure-host-cache ## Run universal scanners (trivy, gitleaks)
+scan: _ensure-host-cache _extended-image ## Run universal scanners (trivy, gitleaks)
 	$(DOCKER_RUN) make _scan
 
-security: _ensure-host-cache ## Run language-specific security scanners
+security: _ensure-host-cache _extended-image ## Run language-specific security scanners
 	$(DOCKER_RUN) make _security
 
-test: _ensure-host-cache ## Run validation tests
+test: _ensure-host-cache _extended-image ## Run validation tests
 	$(DOCKER_RUN) make _test
 
 # ===========================================================================
@@ -193,6 +280,15 @@ _check-config:
 		exit 2; \
 	fi
 
+# --- _generate-dockerfile: emit Dockerfile.devrail ---
+# Story 13.4b: in-container target. Depends on _plugins-load (which populates
+# the loader cache at /tmp/devrail-plugins-loaded.yaml in this container) and
+# then runs the generator from Story 13.4a. Writes Dockerfile.devrail to the
+# workspace root. No-op when no plugins declared.
+.PHONY: _generate-dockerfile
+_generate-dockerfile: _plugins-load
+	@bash /opt/devrail/scripts/plugin-build-extended-image.sh ./Dockerfile.devrail
+
 # --- _plugins-update: resolve plugin refs and write .devrail.lock ---
 # Story 13.3: invoked by `make plugins-update`. Reads `.devrail.yml`,
 # resolves each `rev:` to an immutable SHA via `git ls-remote`, fetches the

STABILITY.md

@@ -31,14 +31,15 @@ DevRail has reached **v1.0** across all repositories. The core standards, toolch
 | **CI workflow templates** | Stable | GitHub Actions workflows and GitLab CI pipeline shipped in template repos. |
 | **Pre-commit hooks** | Stable | Conventional commit hook and per-language hooks configured in template repos. |
 | **Documentation site** | Stable | [devrail.dev](https://devrail.dev) is live with full standards coverage. |
-| **Plugin loader + resolver + lockfile** | Preview (v1.10.x) | Validates `plugin.devrail.yml` manifests, resolves `rev:` to immutable SHAs via `make plugins-update`, records reproducibility metadata in `.devrail.lock`. Verifies lockfile + content_hash on every `make check`. No-op when `plugins:` is absent — v1.9.x behaviour unchanged. Execution loop ships in Story 13.5. |
+| **Plugin loader + resolver + lockfile + build pipeline** | Preview (v1.10.x) | Validates `plugin.devrail.yml` manifests, resolves `rev:` to immutable SHAs (`make plugins-update`), records reproducibility metadata in `.devrail.lock`, and auto-builds a project-local extended image (`devrail-local:<hash>`) when plugins are declared. Verifies lockfile + content_hash on every `make check`. No-op when `plugins:` is absent — v1.9.x behaviour unchanged. Plugin command execution (running plugin-defined `targets:`) ships in Story 13.5. |
 
 ## Consumer responsibilities
 
 These are services/data the dev-toolchain container does **not** provide; consumers must provide them when relevant:
 
 - **Database service** (Postgres, MySQL, etc.) — required for Rails projects whose specs touch the test database. The container runs `bundle exec rails db:test:prepare` before `rspec` (when `config/application.rb` + `Gemfile` are present), which needs a reachable database. Typical local pattern: `docker-compose up -d postgres` before `make test`. Typical CI pattern: a `services:` block.
 - **Project bundle install** — the container ships its own gems for `rubocop`/`reek`/etc. as defaults, but for Gemfile-pinned versions it expects the project's bundle to already be installed (`bundle install`) so `bundle exec <tool>` can find them.
+- **Host tooling for plugin builds** — when `.devrail.yml` declares `plugins:`, the host running `make check` must have Docker (with `buildx`), `yq` (v4+), `sha256sum` (coreutils), and `flock` (util-linux) available. No-op when `plugins:` is absent.
 
 ## Versioning