GoogleContainerTools/distroless images fail for confidential containers in ACI

[dwhjames]

Bug Report: GoogleContainerTools/distroless images fail for confidential containers in ACI

Summary

When deploying gcr.io/distroless/static-debian12@sha256:2b0f5abab12e4d6a533b91a4796d10504a05d8c41a61d4969889efb66daafece to Azure Container Instances (ACI) with confidential computing enabled, the deployment fails with "deviceHash not found" errors. Other images (e.g., mcr.microsoft.com/aci/skr:2.12) work successfully with the same infrastructure.

We have conducted extensive investigation and are seeking feedback from the hcsshim maintainers on potential causes and debugging approaches.

Environment

• Failing Images:
  • gcr.io/distroless/static-debian12@sha256:2b0f5abab12e4d6a533b91a4796d10504a05d8c41a61d4969889efb66daafece
  • gcr.io/distroless/static-debian12:latest
  • gcr.io/distroless/cc-debian12:latest
  • any images layered on top of these
• Working Images:
  • mcr.microsoft.com/aci/skr:2.12
  • Custom images built from scratch with Bazel using rules_distroless and rules_oci
• Policy Generation: az confcom acipolicygen (Azure CLI confcom extension)
• Deployment Target: Azure Container Instances with confidential computing
• Dev Environment: macOS Darwin 26, Podman; Linux Debian Bookworm, Docker

Error Details

Full Error Message

Message: pulling image "<…>.azurecr.io/<…>/static-debian12@sha256:2b0f5abab12e4d6a533b91a4796d10504a05d8c41a61d4969889efb66daafece";
Successfully pulled image "<…>.azurecr.io/<…>/static-debian12@sha256:2b0f5abab12e4d6a533b91a4796d10504a05d8c41a61d4969889efb66daafece";
Error: Failed to start container distroless, Error response: to create containerd task: failed to create shim task: failed to mount container storage: failed to add LCOW layer: failed to add SCSI layer: mount scsi controller 0 lun 3 at /run/mounts/scsi/m2: guest RPC failure: mounting scsi device controller 0 lun 3 onto /run/mounts/scsi/m2 denied by policy: {"decision":"deny","input":{"deviceHash":"2a85d4c2663990ccd9fa75431898a43e4b6add34d68bc25288908b2946b9446d","mountPathRegex":"/run/mounts/scsi/m[0-9]+","rule":"mount_device","target":"/run/mounts/scsi/m2"},"reason":{"errors":["deviceHash not found"]}};
...
mount scsi controller 0 lun 7 at /run/mounts/scsi/m6: guest RPC failure: mounting scsi device controller 0 lun 7 onto /run/mounts/scsi/m6 denied by policy: {"decision":"deny","input":{"deviceHash":"56e7199031a6018c465f0fe348458e8f0c74f37547537f1f6f1beaba7bf7450b","mountPathRegex":"/run/mounts/scsi/m[0-9]+","rule":"mount_device","target":"/run/mounts/scsi/m6"},"reason":{"errors":["deviceHash not found"]}};
The container group provisioning has failed.

Investigation Performed

1. Local Hash Computation

We used a local build of dmverity-vhd from https://github.com/microsoft/integrity-vhd to compute root hashes locally:

./dmverity-vhd-darwin-amd64 --docker roothash \
  -i gcr.io/distroless/static-debian12@sha256:2b0f5abab12e4d6a533b91a4796d10504a05d8c41a61d4969889efb66daafece

Output:

WARN Errored while trying to parse OCI format: not able to parse in OCI format
Layer 0 root hash: fb8047aef725bf71a43923c23456270c74e74c92593aad752ef42ba9a1b9ce93
Layer 1 root hash: 59b808c419a9474a91c7c3d938405ed73a6149732c8c90fd5f541a17cff40441
Layer 2 root hash: f80e4f9375d4111eba47abc51d1a4a6ef82d215b824bc4642bc31f64fe0360f9
Layer 3 root hash: 8b4842f06982817534a75bcf71865213b09dfa8313229c384e5201dadbd75e25
Layer 4 root hash: f6b039e7dc0b28706659ff50d71330f8f63a4d17c729ca7bb375597e195501cd
Layer 5 root hash: 5b61d127749f53a0fe4898a6ae181470715093cea393e88327a59c3a68852998
Layer 6 root hash: 2c6f540907e3fd5b6182cded7492b68f8ce9332f30e89436afab59271c8592d8
Layer 7 root hash: ac3801059d83cfb321d565fb31653ae6e1a8741bc29c54fe6b1229b097e469e5
Layer 8 root hash: 7b4fbef988c8f7e20a7e469cf1109e74018bebac8221a5e27b0fdaac3b6f1772
Layer 9 root hash: 981cc31fc6ca57bb14cb155b43d45568859fdaf3dfb72b22dda3cdfb2e3a3da4
Layer 10 root hash: 2a85d4c2663990ccd9fa75431898a43e4b6add34d68bc25288908b2946b9446d

Of the various Confidential Computing Enforcement (CCE) policies we generated, via the
az confcom acipolicygen tooling (on Linux), we did see agreement with using the
dmverity-vhd tool directly (macOS).

2. Image Characteristics Analysis

We analyzed the distroless image layers and found:

Tar Format:

• All layers use standard POSIX tar format (GNU tar/ustar)
• Layer tar files themselves have epoch 0 timestamps (Dec 31 1969)
• Files within layers have fixed timestamps
• Extraction is consistent across macOS/BSD tar and GNU tar

Content:

• 11 layers total
• Contains 5 symlinks in base layer (e.g., etc/os-release -> ../usr/lib/os-release)
• All symlinks are under 59 bytes (fit in inode)
• Regular files: ~1400+
• Directories: ~130+
• No hardlinks, block devices, or character devices

Build Properties:

• Built with Bazel (hermetic builds)
• Uses rules_distroless and rules_oci Bazel rules

3. Code Analysis

We reviewed the hcsshim codebase (v0.13.0-rc.2; as pinned by integrity-vhd and thus azext_confcom) and observed:

ext4 Filesystem Creation (ext4/internal/compactext4/compact.go):

• Directory entries sorted by inode number (lines 986-999)
• Sequential inode allocation (line 395)
• Extended attributes sorted alphabetically (line 481)
• Zero-initialized UUID in ext4 superblock (lines 1315-1343)
• Timestamps from tar headers (not time.Now())
• Includes fix from commit 16dc8eb for parent directory timestamps

dm-verity (ext4/dmverity/dmverity.go):

• Fixed zero salt (line 30)
• SHA256 hash algorithm
• Sequential Merkle tree construction
• Superblock UUID is randomly generated (lines 137, 157-163)

Policy Validation (pkg/securitypolicy/framework.rego):

• deviceHash checked against data.policy.containers[].layers[] array (lines 15-18)
• deviceHash extracted from verityInfo.RootDigest (internal/guest/runtime/hcsv2/uvm.go:1188)

4. Comparison Tests

We compared the failing image with working images:

Working: mcr.microsoft.com/aci/skr:2.12

• Also contains symlinks (100+)
• Also uses epoch 0 timestamps in layer tars
• Deployment succeeds

Working: Custom Bazel Images

• Same Bazel rules used by GoogleContainerTools/distroless
  • github.com/bazel-contrib/rules_distroless
  • github.com/bazel-contrib/rules_oci
• Published to Azure Container Registry
• Deployment succeeds

Failing: gcr.io/distroless/*

• Multiple variants tested (static-debian12, cc-debian12)
• Policy generation appears to complete
• Deployment fails with "deviceHash not found"

Questions for Maintainers

1. Image Compatibility: Are there known compatibility issues with:

   • Images from GoogleContainerTools/distroless?
   • Images with specific layer characteristics?

2. Debugging Approach: What further debugging steps would you recommend?

We are happy to provide additional information, run tests, or assist with debugging. Thank you for your time and expertise.

[dwhjames]

Updates

Since filing this issue, we've hit another case of this issue for a different container image and resumed our investigation.

Our current hypothesis is that the issue is related to tar entry ordering in container image layers.

We analyzed gcr.io/distroless/static-debian12@sha256:2b0f5abab12e4d6a533b91a4796d10504a05d8c41a61d4969889efb66daafece:

11 layers with mixed ordering strategies:

• 8 layers: Alphabetically sorted tar entries
• 3 layers: Unsorted (creation order or type-based grouping)
  • Example: Layer 3 (tzdata) has 1,325 entries with all symlinks grouped at the end

Pattern observed: Some layers use file-type grouping (files first, then symlinks, then metadata) instead of alphabetical order.

Bazel-based builds (used by GoogleContainerTools/distroless):

• aspect_bazel_lib/mtree_spec generates tar entries in "Bazel order"
• Order is deterministic within a build but differs from alphabetical order
• Application layers built with Bazel may have different ordering than base image layers

Leading Hypothesis

The measurement inconsistency between local policy generation and Azure Container Instances deployment is caused by different tar entry processing orders at different stages:

1. Policy Generation (local/build time): Processes layers with original tar entry order (Bazel order, or as stored in image)
2. Runtime (Azure Container Instances): May process/extract layers in a different order (alphabetically? different tooling?)
3. Result: Different inode allocation → different ext4 filesystem → different dm-verity hash → "deviceHash not found" error

Observations

We manipulated the container image layers to sort the tarball contents. After doing so, the containers launched without this failure.

Questions

These questions require access to Azure Container Instances internals:

Tar Processing in ACI:

1. How does Azure Container Instances process OCI image layers for Confidential Containers?
2. Are tar entries processed in the order they appear in the tar file, or are they reordered (e.g., alphabetically, by type)?
3. What tooling is used to convert tar layers to ext4 filesystems? (Which version of hcsshim? Other tools?)

Policy Generation vs Runtime Consistency:

4. Is the same tar entry processing order used during:

• Policy generation (when generating the expected dm-verity hash)?
• Container runtime (when actually creating the ext4 filesystem)?

5. Are these two phases using the same version of hcsshim/tooling?

Layer Handling:

6. Does ACI decompress and recompress layers during processing?
7. Are layers ever 're-packaged' during the image pull/deployment process?

Reproducibility Guidance:

8. Is it already known or expected that all tar entries be alphabetically sorted?

Understanding Azure Container Instances' actual tar processing behavior would help us:

• Match the expected ordering instead of guessing
• Avoid unnecessary image rebuilding if the issue is elsewhere in the pipeline
• Implement a proper long-term solution