[BUG] `docker compose build` produces different image IDs on each rebuild with containerd image store, but works correctly with overlay2

[powerman]

Description

When Docker Engine uses the containerd image store (default for new installations since
Docker 28), docker compose build produces a new image ID on every rebuild even when no
files have changed. This causes docker compose up to unnecessarily recreate all containers
on every deploy.

Disabling the containerd image store on the same server fixes the issue — the second build
reuses the cache completely and the image ID stays the same.

Steps To Reproduce

Use Docker CE on Linux with the containerd image store enabled (default for fresh installs):

$ docker info -f '{{.Driver}}'
overlayfs

Create these three files:

Dockerfile:

FROM alpine:3.21
RUN echo "hello" > /hello.txt
COPY config.txt /config.txt
CMD ["cat", "/hello.txt", "/config.txt"]

config.txt:

static config file

docker-compose.yml:

services:
  app:
    build: .
    container_name: test-cache

Then run:

# Build twice with nothing changed in between:
docker compose build
docker images --format '{{.ID}}' --filter reference='*-app'

docker compose build
docker images --format '{{.ID}}' --filter reference='*-app'

Expected behavior

Second build reuses cache. Image ID is the same. docker compose up does not recreate the
container.

Actual behavior

Second build produces a different image ID. docker compose up sees a different image and
recreates the container.

Note: individual build steps show CACHED for RUN/COPY layers, but the final image still
gets a new ID. The exporting steps (exporting layers, writing image, naming) take
0.1–0.2s instead of 0.0s, suggesting they actually re-export rather than reuse.

Compose Version

Docker Compose version v5.1.0

Docker Environment

Client: Docker Engine - Community
 Version:    29.3.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.31.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 8
  Running: 7
  Paused: 0
  Stopped: 1
 Images: 8
 Server Version: 29.3.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: local
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 301b2dac98f15c27117da5c8af12118a041a31d9
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-106-generic
 Operating System: Ubuntu 24.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.57GiB
 Name: secondary
 ID: 091526cd-489b-4f47-acad-33efcea821cb
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Anything else?

Related issues

• docker compose build does not completely reuse cached layers on immediate rebuild, but docker build does for-mac#7341 — same symptom on macOS Docker Desktop
• Layer not cached under containerd image store for-mac#7359 — layer not cached under containerd image store (fixed by increasing
  defaultKeepStorage, but that does not help here — layers are small)

Workaround

Disable the containerd image store:

{
  "features": {
    "containerd-snapshotter": false
  }
}

After changing daemon.json and restarting Docker, existing images are no longer available
(format is incompatible) and need to be rebuilt/re-pulled. Volumes are preserved.

[ndeloof]

Can be reproduce with bake:

$ docker compose build --print > bake.json
$  docker buildx bake --progress=quiet -f bake.json && docker images --format '{{.ID}}' --filter reference='*-test'
3355a1162172
$ docker buildx bake --progress=quiet -f bake.json && docker images --format '{{.ID}}' --filter reference='*-test'
a935a9422cbf

Sounds like a Buildx Bake issue - cc @crazy-max