Skip to content

containerd: configure transfer service remote snapshot annotations#923

Merged
ginglis13 merged 2 commits into
bottlerocket-os:developfrom
ginglis13:fix-soci-conf
May 13, 2026
Merged

containerd: configure transfer service remote snapshot annotations#923
ginglis13 merged 2 commits into
bottlerocket-os:developfrom
ginglis13:fix-soci-conf

Conversation

@ginglis13
Copy link
Copy Markdown
Contributor

@ginglis13 ginglis13 commented May 12, 2026

Description of changes:

For containerd configurations making use of the transfer service, in
combination with remote snapshotters, the configuration of
enable_remote_snapshot_annotations = true must be set for pulls to route
through the containerd transfer service to the remote snapshotter.

containerd docs highlight this:
https://github.com/containerd/containerd/blob/2976f38ccbfcda5ef1364d63d60b0a304e4bf94a/docs/cri/config.md?plain=1#L191-L214

CRI Image Config Option Local Pull Transfer Service Pull
DisableSnapshotAnnotations ✅ Supported ⚠️ Must be configured in snapshotter plugin:
[proxy_plugins.stargz.exports]
enable_remote_snapshot_annotations = "true"

Testing done:

Tested on k8s-1.34 node with both soci (remote snapshotter) and image verifiers (routed through transfer service) enabled. (bottlerocket-os/bottlerocket#4832)

Before:

bash-5.2# journalctl -u containerd | grep warn
May 12 02:56:27 ip-192-168-9-180.us-west-2.compute.internal containerd[1718]: time="2026-05-12T02:56:27.075019003Z" level=info msg="loading plugin" id=io.containerd.warning.v1.deprecations type=io.containerd.warning.v1
May 12 02:56:27 ip-192-168-9-180.us-west-2.compute.internal containerd[1718]: time="2026-05-12T02:56:27.094178863Z" level=warning msg="Found 'DisableSnapshotAnnotations' in CRI config which is incompatible with transfer service (moved to snapshotter plugin when using transfer service). Falling back to local image pull mode."

And all images could be pulled without verification enforcement.

After:

bash-5.2# journalctl -u containerd | grep warn
May 12 20:37:42 ip-192-168-7-138.us-west-2.compute.internal containerd[1720]: time="2026-05-12T20:37:42.086816836Z" level=info msg="loading plugin" id=io.containerd.warning.v1.deprecations type=io.containerd.warning.v1
May 12 20:37:54 ip-192-168-7-138.us-west-2.compute.internal containerd[1720]: time="2026-05-12T20:37:54.438507669Z" level=warning msg="Image verifier blockedpull" digest="sha256:dbba6c54e51aaf079aa7c471f71bb63adf9eb828bdc64fde2bc8a3a256026c76" name="602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.34.6-eksbuild.2" ok=false reason="verifier digestion-image-verifier rejected image (exit code 1): image verification failed: digest not in allowlist" verifier=bindir

Testing backwards compat for a base k8s node to ensure pause image import does not regress:

# Node is ready
ip-192-168-31-133.us-west-2.compute.internal   Ready    <none>   20s   v1.34.4-eks-f69f56f

bash-5.2# systemctl status kubelet
● kubelet.service - Kubelet
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/service.d
             └─00-aws-config.conf, 10-requires-tmp.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d
             └─dockershim-symlink.conf
             /etc/systemd/system/kubelet.service.d
             └─exec-start.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d
             └─make-kubelet-dirs.conf, prestart-load-pause-ctr.conf
     Active: active (running) since Tue 2026-05-12 20:56:10 UTC; 48s ago

bash-5.2# systemctl status containerd
● containerd.service - containerd container runtime
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/containerd.service; enabled; preset: enabled)
    Drop-In: /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/service.d
             └─00-aws-config.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/containerd.service.d
             └─005-disable-pigz.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/service.d
             └─10-requires-tmp.conf
     Active: active (running) since Tue 2026-05-12 20:56:08 UTC; 1min 4s ago

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

For containerd configurations making use of the transfer service, in
combination with remote snapshotters, the configuration of
enable_remote_snapshot_annotations = true must be set for pulls to route
through the containerd transfer service to the remote snapshotter.

Signed-off-by: Gavin Inglis <giinglis@amazon.com>
@ginglis13 ginglis13 marked this pull request as ready for review May 12, 2026 20:58
@ginglis13 ginglis13 requested review from KCSesh and vigh-m and removed request for vigh-m May 12, 2026 20:58
@ginglis13
Copy link
Copy Markdown
Contributor Author

ginglis13 commented May 12, 2026

^ force push conditionalizes the --snapshotter value as there is possibility that downstreams of the core-kit don't package soci and snapshotter related settings and configuration files. Testing that case, overlayfs used:

bash-5.2# cat /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d/prestart-load-pause-ctr.conf
[Service]
Environment=ACTIVE_SNAPSHOTTER=overlayfs
EnvironmentFile=-/var/cache/containerd/active-snapshotter-NON_EXIST

# load the built-in pause image
ExecStartPre=/usr/bin/ctr \
    --namespace=k8s.io \
    image import \
    --snapshotter=${ACTIVE_SNAPSHOTTER} \
    --local \
    --all-platforms \
    /usr/libexec/kubernetes/kubernetes-pause.tar

# label it to prevent it from being removed
ExecStartPre=/usr/bin/ctr \
    --namespace=k8s.io \
    image label \
    localhost/kubernetes/pause:0.1.0 \
    io.cri-containerd.pinned=pinned
bash-5.2# cat /var/cache/containerd/active-snapshotter-NON_EXIST
cat: /var/cache/containerd/active-snapshotter-NON_EXIST: No such file or directory
bash-5.2# systemctl status kubelet
● kubelet.service - Kubelet
     Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/service.d
             └─00-aws-config.conf, 10-requires-tmp.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d
             └─dockershim-symlink.conf
             /etc/systemd/system/kubelet.service.d
             └─exec-start.conf
             /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d
             └─make-kubelet-dirs.conf, prestart-load-pause-ctr.conf
     Active: active (running) since Tue 2026-05-12 23:25:59 UTC; 1min 16s ago

For supporting a remote snapshotter with transfer service like
soci-snapshotter, explicitly specify snapshotter performing an image
import. Make this conditional, falling back to existing behavior
(overlayfs) for backwards compat and for compatility with downstream
consumers of k8s packages who do not package soci or its related
settings and configuration files.

Signed-off-by: Gavin Inglis <giinglis@amazon.com>
Comment thread packages/kubernetes-1.36/prestart-load-pause-ctr.conf
@ginglis13 ginglis13 merged commit be4d19c into bottlerocket-os:develop May 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants