Image I'm using:

• Bottlerocket OS 1.61.0 (aws-k8s-1.35)
• kernel 6.12.83
• containerd 2.1.7+bottlerocket
• SOCI snapshotter active (settings.container-runtime.snapshotter = "soci").

AMI alias: bottlerocket@v1.61.0 (region eu-west-1):
- ami-0890f01013ea0ec68 - bottlerocket-aws-k8s-1.35-aarch64-v1.61.0-8ef015e0
- ami-066a1a9eec5802bd0 - bottlerocket-aws-k8s-1.35-x86_64-v1.61.0-8ef015e0

What I expected to happen:

Pods with user namespaces (hostUsers: false) should start when SOCI is the active CRI snapshotter.

What actually happened:

Pods with hostUsers: false get stuck and fail with container ID cannot be mapped to a host ID, like so:

Events:
  Warning  FailedCreatePodSandBox  kubelet  Failed to create pod sandbox: rpc error: code = Unknown
    desc = failed to start sandbox "f223a179226e5ba450b0b490016d8346a5a9c7216e840dc7776da81627e8f793":
    failed to create containerd container: mount callback failed on /tmp/containerd-mount2726205542:
    container ID 498008064 cannot be mapped to a host ID

Also see awslabs/soci-snapshotter#1888

How to reproduce the problem:

1. Bottlerocket node with snapshotter = "soci".
2. Schedule a pod with spec.hostUsers: false.
3. Pod fails with container ID cannot be mapped to a host ID.
4. Add capabilities = ["remap-ids"] to proxy_plugins.soci, restart containerd -> pod starts.

Fix?

Manually modifying the containerd configuration on a node like so, resolves the issue:

# /etc/containerd/config.d/001-snapshotter.toml
[proxy_plugins.soci]
+ capabilities = ["remap-ids"]
    type = "snapshot"
    address = "/run/soci-snapshotter/soci-snapshotter.sock"