[BUG] Variables not interpolated in env_file in included subproject

[ianhinder]

Description

When I provide values for variable interpolation in a top-level compose file, and try to use them in an included compose file's service env_file, they are not interpolated. They are interpolated in the sub compose file itself. If I add the variable to a .env file in the subproject, it is interpolated in the env_file.

If the premise is that top-level variables are interpolated in the subproject, and this works in the compose file itself, then I think they should be interpolated in the env_files of the subproject's services as well.

Steps To Reproduce

$ tree
.
├── compose.yml
├── subproj
│   ├── subcompose.yml
│   └── subvalues.env
└── values.env

2 directories, 4 files

$ cat compose.yml 
include:
  - path: subproj/subcompose.yml
    env_file:
      - values.env

$ cat subproj/subcompose.yml 
services:
  app:
    env_file: subvalues.env
    image: helloworld

$ cat subproj/subvalues.env 
MYVAR=${VAR?}

$ cat values.env 
VAR=1

$ docker compose config
failed to read /Users/XXX/Documents/Projects/dc-bug/subproj/subvalues.env: required variable VAR is missing a value

Compose Version

Docker Compose version v5.0.1

Docker Environment

Client:
 Version:    29.1.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.17.1
    Path:     /Users/XXX/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.30.1-desktop.1
    Path:     /Users/XXX/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.1
    Path:     /Users/XXX/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.47
    Path:     /Users/XXX/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.2.0
    Path:     /Users/XXX/.docker/cli-plugins/docker-desktop
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.31
    Path:     /Users/XXX/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/XXX/.docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  v0.35.0
    Path:     /Users/XXX/.docker/cli-plugins/docker-mcp
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.0.6
    Path:     /Users/XXX/.docker/cli-plugins/docker-model
  offload: Docker Offload (Docker Inc.)
    Version:  v0.5.40
    Path:     /Users/XXX/.docker/cli-plugins/docker-offload
  pass: Docker Pass Secrets Manager Plugin (beta) (Docker Inc.)
    Version:  v0.0.22
    Path:     /Users/XXX/.docker/cli-plugins/docker-pass
  sandbox: Docker Sandbox (Docker Inc.)
    Version:  v0.6.0
    Path:     /Users/XXX/.docker/cli-plugins/docker-sandbox
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/XXX/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.19.0
    Path:     /Users/XXX/.docker/cli-plugins/docker-scout

Server:
 Containers: 24
  Running: 8
  Paused: 0
  Stopped: 16
 Images: 63
 Server Version: 29.1.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.54-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 14
 Total Memory: 15.6GiB
 Name: docker-desktop
 ID: 248e0de3-b470-4089-882a-ad50dd69405f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/XXX/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Anything else?

This seems similar to #10842, and might actually be an issue in compose-go, but I didn't want to make that assumption since compose is the implementation I can reproduce the issue with.

[MaheshThakur9152]

Hi @ianhinder @amyssnippet - I've been digging into this as well.

While the root cause is indeed in how compose-go handles recursive scope, I believe we can solve this in the CLI loader by pre-processing the top-level include env_files before the project load. This would fix the user issue immediately without requiring invasive changes to the core library's recursion logic.

I have a working implementation and a regression test ready. I'll open a PR shortly so we can discuss if this approach is preferred over moving the issue upstream.

[amyssnippet]

@ndeloof and @glours , if this isnt the right place for this issue, then close it with a perfect comment. and also we should place this issue to its right place and its origin https://github.com/compose-spec/compose-go

[ndeloof]

@amyssnippet docker/compose and compose-go are maintained in a coordinated way. Many compose issue are actually fixed on compose-go, without creating a dedicated issue

[glours]

@amyssnippet The Compose repository is the primary entry point for users to report issues.
As maintainers, we know where a given issue should ultimately be fixed, but we still need to keep the issue tracked here so others encountering the same problem can see that it’s already been reported.

[amyssnippet]

Ooh, as a new contributor, i guess this is a very obvious question, as i am new to the community, and thanks for the guidance!

[MaheshThakur9152]

The root cause is in compose-go include resolution, and the fix belongs in the core loader rather than the CLI shim.
I’ve opened an upstream PR with a proper fix and regression test here:
compose-spec/compose-go#848
Once merged, Docker Compose will automatically pick it up via dependency update.
Keeping this issue open here for tracking as recommended by maintainers.