Add container build backend and build verify command#2525
Add container build backend and build verify command#2525leighmcculloch wants to merge 71 commits intomainfrom
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
Opened an issue about dbus creating problems with using the image for the build for the verification step: |
|
@leighmcculloch I just tried this, but I'm getting a warning, even though there are no unstaged files. $ git status
On branch main
nothing to commit, working tree clean
$ stellar contract build --backend docker
⚠️ git working tree has uncommitted changes; source_repo/source_rev/bldopt_* not embedded in contract metadata. Commit changes for a reproducible build. |
| } | ||
|
|
||
| let backend = match bldimg { | ||
| Some(image) => build::Backend::Docker { image }, |
There was a problem hiding this comment.
bldimg needs some form of allowlist, e.g. docker.io/stellar/stellar-cli@sha256:*, otherwise I can inject a docker image that bypasses verification
| } | ||
| }); | ||
|
|
||
| let build_cmd = build::Cmd { |
There was a problem hiding this comment.
Do we need feature flags here? While not extremely common, I have used them in the past to reduce code duplication between similar contracts. Ref -> https://github.com/script3/soroban-governor/tree/main/contracts/votes
| // - The official `stellar/stellar-cli` image's stock entrypoint is a | ||
| // wrapper script that launches dbus + gnome-keyring before exec-ing | ||
| // `stellar`; that setup is irrelevant for `contract build` and dbus | ||
| // refuses to start when the container runs as a host UID with no | ||
| // `/etc/passwd` entry. Skipping it keeps the host UID mapping intact. |
There was a problem hiding this comment.
This was cleaned up in a recent PR. Does this simplify anything?
| attach_stdout: Some(true), | ||
| attach_stderr: Some(true), | ||
| host_config: Some(HostConfig { | ||
| binds: Some(binds), |
There was a problem hiding this comment.
Have we considered removing the host bind mounts? Given verify will build untrusted code, it might be best if we keep the build artifacts within the container, then just extract the WASM file out.
We could consider having two configurations docker-build and docker-verify, where build keeps mounts to help speed up repeated builds and verify is more black-box to provide a bit more protection.
| let cliver = find_meta(&spec.meta, "cliver").ok_or(Error::MissingMeta("cliver"))?; | ||
| let bldimg = find_meta(&spec.meta, "bldimg"); | ||
| let rsver = find_meta(&spec.meta, "rsver").ok_or(Error::MissingMeta("rsver"))?; | ||
| let bldopt_manifest_path = find_meta(&spec.meta, "bldopt_manifest_path"); | ||
| let bldopt_package = find_meta(&spec.meta, "bldopt_package"); | ||
| let bldopt_profile = find_meta(&spec.meta, "bldopt_profile"); | ||
| let bldopt_optimize = find_meta(&spec.meta, "bldopt_optimize").is_some(); |
There was a problem hiding this comment.
we should enforce the regex here
What
Add
--backend docker[=<image>]tostellar contract build(anddeploy/upload) that runs the entire build pipeline inside a container whose entrypoint isstellar. Add astellar contract build verifysubcommand that reads everything it needs from the wasm's metadata, rebuilds, and reports which (if any) rebuilt artifact is byte-identical to the original. Add a mainnet warning onstellar contract deploywhen the wasm is missing the meta entries needed for independent verification.Why
Contract builds vary across host OS, architecture, and toolchain, preventing third parties from independently confirming a deployed contract was built from given source. Pinning the build to a docker image plus the rust toolchain version makes builds reproducible, recording the source repo + commit + per-package build options lets verifiers rebuild the exact same artifact, and the new
verifysubcommand automates the rebuild-and-compare check.Closes #2506.
How it works
Three parts: build-time recording, deploy-time warning, and verify-time reproduction.
Build
For all backends (including
local), the build:originremote, embedssource_repo(URL canonicalized tohttps://…),source_rev(full HEAD SHA), and per-package build options (bldopt_manifest_pathrelative to git root,bldopt_package,bldopt_profile, optionalbldopt_optimize). The manifest path is auto-inserted whether or not--manifest-pathwas passed on the CLI.source_repo/source_rev/bldopt_*.For
--backend docker, additionally:docker.io/stellar/stellar-cli@sha256:cb2fc3116a6ace37a77ca6bb88afb4bee57fc746cd556a4373f2c3ee95d4e917— pinned by digest so the recordedbldimgis reproducible from day one and we sidestep the longstanding Apple Silicon docker quirk where pulling a multi-arch tag with--platform=linux/amd64leavesRepoDigestsempty after pull.<git_root or workspace_root>→/source(rw, source — also where cargo writes its target dir, shared with the host)~/.cargo/registry→/usr/local/cargo/registry(rw, cached crate downloads)stellardirectly, bypassing the official image'sentrypoint.sh(which launchesdbus+gnome-keyringand trips when running under a host UID with no/etc/passwdentry — see Docker image's entrypoint dbus init fails when run as non-root UID #2543).contract builddoesn't use the keyring, so the wrapper is irrelevant here.stellar contract build --manifest-path /source/<rel> --profile <p> --locked --meta bldimg=<digest> [forwarded args]inside the container. The args use only flags that exist in publishedstellar/stellar-cliimages today; no new flags are added, and--backend localis deliberately not passed (it's a flag added in this PR and isn't recognized by published images).wasm-optitself; the host only orchestrates and copies outputs to--out-dirif requested.The wasm's
contractmetav0custom section is populated with up to nine entries:cliver26.0.0#abc1234…(CLI version + git rev)^\d+\.\d+\.\d+(-[A-Za-z0-9.+-]+)?#([0-9a-f]{40}(-dirty)?)?$bldimgdocker.io/stellar/stellar-cli@sha256:…^[^@\s]+@sha256:[0-9a-f]{64}$--backend docker)rsver1.83.0(resolved rustc version)^\d+\.\d+\.\d+(-[A-Za-z0-9.+-]+)?$source_repohttps://github.com/user/repo(clean repo's origin)^https?://\S+$source_rev^[0-9a-f]{40}$bldopt_manifest_pathcontracts/foo/Cargo.toml(relative to git)^([^/\s]+/)*Cargo\.toml$bldopt_package^[A-Za-z][A-Za-z0-9_-]*$bldopt_profilerelease)^[A-Za-z][A-Za-z0-9_-]*$bldopt_optimizetrue(only present when--optimizewas used)^true$The presence of
bldimgis what distinguishes a docker build from a local one — there's no separatebldbkdfield. For full reproducibility from day one, pin to a specific image with--backend docker=<name>@sha256:…and commit before building.--backendand--docker-hostare also exposed onstellar contract deployandstellar contract upload(which auto-build when no--wasm/--wasm-hashis given), so the same flags work end-to-end.Deploy
stellar contract deployagainst mainnet now warns when the wasm is missing any ofcliver,bldimg,rsver,source_repo,source_rev,bldopt_manifest_path,bldopt_package,bldopt_profile:The check is mainnet-only (matches network passphrase against
Public Global Stellar Network ; September 2015); on testnet/futurenet/local the wasm deploys silently.Verify
verifyis a subcommand ofbuild— it lives atstellar contract build verify, and works on multi-contract workspaces by rebuilding and finding the match.contract info).cliver,bldimg(optional),rsver, andbldopt_*(optional, best-effort) from the wasm's meta. Missingbldopt_*entries trigger a warning rather than an error and the build falls back to its defaults — verify still runs, just with the caveat that the rebuild may not be reproducible.bldimgpresent →Backend::Docker { image: bldimg }. The image's pinned digest pulls the same in-container cli that produced the original.bldimgabsent →Backend::Local. Best-effort rebuild on the host.rsverto the rebuild asRUSTUP_TOOLCHAIN(in-container) orcargo +<rsver>(local). For docker the toolchain inside the image is fixed by whoever built it; passingRUSTUP_TOOLCHAINlets rustup-managed cargo switch toolchains if the image carries multiple ones.bldopt_manifest_pathagainst the cwd's git top-level (viagit rev-parse --show-toplevel) so verify works from anywhere inside the checkout.The user is responsible for checking out the matching commit before running verify; verify rebuilds from the working tree. (
source_repoandsource_revare embedded in meta to help users find the right commit, but verify itself doesn't clone — that would add a separate trust path.)End-to-end example
The host CLI's version is irrelevant for verifying a docker-built wasm — whatever cli is in the image is what built (and rebuilds) the wasm.
Notes
/var/run/docker.sock, or whatever--docker-host/DOCKER_HOSTpoints at). Sameconnect_to_dockerhelper used bystellar container start/stop/logs, with the same Docker Desktop fallback ($HOME/.docker/run/docker.sock). No shell-out to thedockerCLI. A podman socket exposing the Docker API would also work (untested).--backend docker(no=...) defaults todocker.io/stellar/stellar-cli@sha256:cb2fc3..., notstellar/stellar-cli:latest. Recording a digest immediately makes builds reproducible day one and avoids the Apple SiliconRepoDigests-after-cross-platform-pull quirk. Bumping the default is a single-line const change inbuild.rs(see comments there for the recipe). Users who want a different image specify--backend docker=....stellar/stellar-cliimage's entrypoint runsentrypoint.sh, which launchesdbus+gnome-keyring. That setup fails when the container runs as a host UID without an/etc/passwdentry — see Docker image's entrypoint dbus init fails when run as non-root UID #2543. We override the entrypoint to point straight at thestellarbinary, which is fine becausecontract builddoesn't touch the keyring.~/.cargo/registrylets the container reuse crate downloads the host already has.wasm32v1-nonepre-installed for its default toolchain; ifRUSTUP_TOOLCHAINselects a different one (verify on a wasm built with another rust version), the cli/cargo handle target installation themselves.RUSTUP_TOOLCHAIN=<rsver>inside the container (andcargo +<rsver>for local rebuilds) so the rust version matches whatever the original build used.bldimgis normalized to<registry>/<path>@sha256:<digest>(e.g.stellar/stellar-cli:latest→docker.io/stellar/stellar-cli@sha256:…) so verify can resolve it without relying on the local registry config.source_repois normalized tohttps://…form (e.g.git@github.com:user/repo.git→https://github.com/user/repo).bldopt_manifest_pathis recorded relative to the git repo root regardless of whether--manifest-pathwas passed on the CLI. Verify resolves it against the cwd's git top-level so the command works from anywhere inside the checkout.stellar contract buildinside the image with only flags that exist in publishedstellar/stellar-cliimages today (--manifest-path,--profile,--locked,--meta,--package,--features,--all-features,--no-default-features,--optimize).bldimgis forwarded via--meta bldimg=<digest>, not a new flag.bldbkdfield: presence ofbldimgis the only signal needed to distinguish a docker build from a local one.docker container prune.Performance/runtime caveats
Building inside an
amd64container on a non-amd64 host (Apple Silicon, Linux/arm64) runs under emulation. For small contracts the difference is negligible; for workspaces with heavy dep trees the emulated build can be substantially slower than a native host build. Container runtimes that don't ship qemu/binfmt support won't run amd64 containers on arm64 hosts at all. See #2506 (comment).Related issues
stellar versionandclivermeta #2535 — normalizecliverrendering across install paths so it's a single shape.stellar/stellar-cliimage'sentrypoint.shfails under non-root UIDs, motivating the entrypoint override here.Status
This is an experiment in validating the ideas in #2506. May or may not be destined for merging — at this moment it's an experiment in validating the approach.