The workers' sandbox on the nodes¶
Every command a session runs on a worker pod runs in bubblewrap, which builds its sandbox from an unprivileged user namespace, and three things on the nodes decide whether it can start there: the worker container's seccomp profile, its AppArmor profile, and a user namespace of the pod's own. This page is step 2 of installing in full: what each needs and what it allows, how to check a cluster before installing, and what bubblewrap's refusals mean. A pod that cannot start bubblewrap refuses the command with a sentence; it never runs one unconfined (Decision 832).
The workers' seccomp profile¶
A container runtime's default seccomp profile refuses an unprivileged user namespace, so
a worker under it refuses every command ("bubblewrap could not start one here (bwrap: No
permissions to create new namespace ...)"). worker.seccomp chooses the worker
container's profile, and the default needs nothing from you:
worker:
seccomp:
type: Localhost # or RuntimeDefault, or Unconfined
localhostProfile: troupe/worker.json # under the kubelet's seccomp directory
install:
enabled: true # the DaemonSet below
kubeletSeccompDir: /var/lib/kubelet/seccomp
Localhost, the default, is the profile the chart ships: the runtime's default plus the calls bubblewrap makes its namespaces with (what it allows and why, below). The kubelet reads it from the node, so it must be on every node that runs workers. Theseccompcontainer of the chart's DaemonSettroupe-worker-profiles, one per node, copies it there from a ConfigMap and waits, ready while the file is in place. It mounts that one host directory (/var/lib/kubelet/seccomp/troupe) and runs as root with every capability dropped, because the directory is root's, sotroupe-systemneeds theprivilegedPod Security level, which the chart labels it with while the DaemonSet runs (podSecurity). Where the kubelet runs with another--root-dir, setkubeletSeccompDirto<root-dir>/seccomp. A worker pod that reaches a node before the file waits inCreateContainerError("cannot load seccomp profile") and starts once it is there:kubectl -n troupe-system rollout status daemonset/troupe-worker-profiles.- With the Security Profiles Operator, or nodes built with the file, set
install.enabled: falseand pointlocalhostProfileat where it is. For the operator, apply aSeccompProfileintroupe-systemwhose spec is the rules ofcharts/troupe/files/worker-seccomp.json(defaultAction,architectures,syscalls), and setlocalhostProfile: operator/troupe-system/<its name>.json. RuntimeDefaultkeeps the runtime's own profile: workers start and run no command.Unconfinedis no filter at all, for a development cluster (dev/kind/values.yaml). ThebaselineandrestrictedPod Security levels refuse it.
What the chart's profile allows (Decision 890). It is containerd's default profile for a container with no capabilities, with three changes:
cloneandunsharemay make a user, mount, PID, IPC or UTS namespace, the ones the sandbox is made of;mount,umount2andpivot_rootare allowed, which a process with no capabilities can use only inside a user namespace it made, where bubblewrap builds the sandbox's root;- nothing else. A network, cgroup or time namespace is still refused, so a command never
owns a network namespace and the netfilter it would bring (
CAP_NET_ADMINthere is where many kernel exploits through user namespaces start), andsetns,bpf,keyctl,perf_event_open, io_uring and the new mount API stay refused as in the default.
What this adds is the kernel's user-namespace code, reachable from a command a session
runs. That is the price of a sandbox built without privileges, and less than the
alternatives: CAP_SYS_ADMIN would allow all of this and mounting in the pod's own
namespace besides, and Unconfined removes the filter. Only the worker container has this
profile; the pod keeps the runtime's default.
The workers' AppArmor profile¶
Where a node enforces AppArmor, as Ubuntu's do, two things of AppArmor's stop bubblewrap:
- The runtime's default AppArmor profile (containerd's
cri-containerd.apparmor.d) hasdeny mount, so bubblewrap stops at its first mount: "bwrap: Failed to make / slave: Permission denied". - Ubuntu's restriction on unprivileged user namespaces
(
kernel.apparmor_restrict_unprivileged_userns=1, the default from Ubuntu 23.10, so on 24.04 node pools). An unconfined process that makes a user namespace is moved into a profile with no capabilities, so a container with no AppArmor profile at all (Unconfined) gets one step further and stops at the namespace's uid map: "bwrap: setting up uid map: Permission denied".
worker.appArmor gives the worker container a profile of its own that answers both
without touching the node's sysctl: the runtime's default with bubblewrap's mounts and
pivot_root allowed, under which a worker is confined when it makes the namespace, so the
restriction does not apply (what it allows and denies, below). The default needs nothing
from you:
worker:
appArmor:
enabled: true # false on nodes without AppArmor
profile: troupe-worker # the name it is loaded under
install:
enabled: true # the DaemonSet's `apparmor` container
- Loaded on every node by the
apparmorcontainer oftroupe-worker-profiles, withapparmor_parser --replacefrom the operator's image; it then waits, ready while the kernel has the profile in enforce mode. Loading takes root in the node's user namespace withMAC_ADMIN, no AppArmor profile of its own, and the host's/sys/kernel/security, and the container has those and nothing else (every other capability dropped, a read-only root, no API token). A worker pod that reaches a node before the profile waits inCreateContainerError("apparmor profile not found") and starts once it is loaded; it never runs without it. - A node without AppArmor: the container loads nothing and says so in its log. The
kubelet refuses a pod that asks for a profile there ("Cannot enforce AppArmor: AppArmor
is not enabled on the host"), so a cluster whose nodes run no AppArmor sets
enabled: false, and the worker container keeps the runtime's default. - With the Security Profiles Operator, or nodes built with the profile
(
charts/troupe/files/worker-apparmor), setinstall.enabled: falseandprofileto the name it is loaded under. - Not the sysctl.
kernel.apparmor_restrict_unprivileged_userns=0would lift the restriction for every process on the node, and a managed node pool often cannot set it. The chart never sets it, and the worker container is neverUnconfined.
What the chart's profile allows and denies (Decision 898). It is containerd's default profile with these changes:
mountandpivot_rootare allowed, where the default denies every mount. As with seccomp, a process with no capabilities can mount only inside a user and mount namespace it made itself, so these reach the sandbox's mounts, never the pod's;- a worker is confined by it, so the kernel's restriction on unprivileged user namespaces, which applies to unconfined processes, does not move it into a profile with no capabilities. The profile is written for the AppArmor 3.0 policy ABI, which the parser that loads it speaks and which does not mediate user namespaces, so a worker may make one. The node's sysctl is left as it is, so every other process keeps the restriction;
- denied beyond the default: mounting sysfs, cgroup and cgroup2, mqueue, fuse, overlay,
bpf, ramfs and binder filesystems even inside the sandbox; the capabilities that mean
something only in the node's own namespace (
mac_admin,mac_override,sys_module,sys_rawio,sys_boot,sys_time,syslog,audit_control); and raw IP and packet sockets (netlink's stay, which listing the network interfaces needs). The default's denials stay: no writes to the kernel's files in/procor to sysfs outside cgroups, nothing of/proc/kcore,/proc/sysrq-trigger, firmware or/sys/kernel/security.
The loader is the one new privilege the sandbox takes: a container per node, root in the
node's user namespace with MAC_ADMIN alone, unconfined by AppArmor (the default profile
would refuse it /sys/kernel/security, the only part of the host it mounts).
The workers' user namespaces¶
Bubblewrap also mounts a fresh /proc for the sandbox, and the kernel refuses that in a
user namespace while the runtime's masks hide part of the container's own (/proc/kcore
and others mounted over, /proc/sys read-only): "bwrap: Can't mount proc on /newroot/proc:
Operation not permitted". Kubernetes unmasks /proc only for a pod in a user namespace of
its own, so with worker.userNamespaces: true, the default, every worker pod gets
hostUsers: false and its worker container procMount: Unmasked, always together. Every
user in the container, root included, is then an unprivileged user on the node, mapped to
a range no other pod has: a process that escaped the container would be nobody there.
Because the operator writes the two together or neither, what the masks hid
(/proc/kcore, /proc/sys, ...) is visible only to a root with no privilege over the node.
What the cluster needs for it:
| Kubernetes | 1.33 or later, where user namespaces are on by default |
| Container runtime | containerd 2.0 or later with runc 1.2 or later (CRI-O 1.25 or later with crun or runc should do; not tried) |
| Nodes | Linux 6.3 or later: the projected tokens and ConfigMaps are tmpfs, and idmapped tmpfs came in 6.3 |
| Storage | filesystems that take idmapped mounts. A block volume with ext4 or xfs does, and so do kind's and local-path's directories. NFS does not, and most RWX file storage is NFS: a team volume there is expected to keep its pod in ContainerCreating or CreateContainerError, with the idmapped mount refused in the pod's events (not tried here) |
| Pod Security | nothing to do: privileged in the worker namespaces, since baseline and restricted refuse procMount: Unmasked even in a user namespace, and the operator labels each troupe-w-<profile> so (enforce, audit and warn), at every pass, a namespace made before it included. With worker.userNamespaces: false and a seccomp profile other than Unconfined the level is restricted. podSecurity.labels: false leaves the labels to you; a worker pod is then refused at creation where the cluster's default is higher, in a FailedCreate event on its StatefulSet |
A cluster that cannot have them sets worker.userNamespaces: false. Its workers start and
refuse every command, with bubblewrap's /proc sentence, until it can: a pod never runs a
command outside the sandbox.
Check your cluster before installing, with no secret and nothing of Troupe's but the
worker image. This makes a namespace, a volume on the storage class your profiles use, and
one pod as a worker pod would be, then runs a sandbox the way a worker does. Where the
nodes enforce AppArmor, run it once the chart's DaemonSet has loaded the profile
(kubectl -n troupe-system rollout status daemonset/troupe-worker-profiles), or install
with the check first and come back:
kubectl version | grep Server # 1.33 or later
kubectl get nodes -o custom-columns=NODE:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion
kubectl create namespace troupe-check
kubectl label namespace troupe-check pod-security.kubernetes.io/enforce=privileged # as the operator labels a worker namespace
kubectl apply -n troupe-check -f - <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: data}
spec:
accessModes: [ReadWriteOnce] # ReadWriteMany and the class your team volumes use, to check those
storageClassName: standard # the class your profiles name
resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Pod
metadata: {name: worker-check}
spec:
hostUsers: false
securityContext: {runAsNonRoot: true, runAsUser: 1000, fsGroup: 1000}
containers:
- name: worker
image: ghcr.io/it-minds/troupe-worker:X.Y.Z-beta # the release you install
command: ["sleep", "600"]
securityContext:
allowPrivilegeEscalation: false
capabilities: {drop: ["ALL"]}
seccompProfile: {type: Localhost, localhostProfile: troupe/worker.json} # once the chart's DaemonSet has run; else Unconfined, for the check only
appArmorProfile: {type: Localhost, localhostProfile: troupe-worker} # where nodes enforce AppArmor, once the DaemonSet has loaded it; leave it out on nodes without
procMount: Unmasked
volumeMounts: [{name: data, mountPath: /var/lib/troupe}]
volumes: [{name: data, persistentVolumeClaim: {claimName: data}}]
YAML
kubectl -n troupe-check wait --for=condition=Ready pod/worker-check --timeout=120s
kubectl -n troupe-check exec worker-check -- sh -c '
head -1 /proc/self/uid_map # not "0 0 4294967295"
grep -c " /proc/" /proc/self/mountinfo # 0: nothing masked
touch /var/lib/troupe/ok && echo volume ok
bwrap --unshare-pid --proc /proc --dev /dev --ro-bind /usr /usr --symlink usr/bin /bin \
--symlink usr/lib /lib --symlink usr/lib64 /lib64 -- /bin/sh -c "cat /proc/1/comm"' # bwrap
kubectl delete namespace troupe-check
A pod that never becomes ready says why in kubectl -n troupe-check describe pod
worker-check: hostUsers dropped or refused (Kubernetes or the runtime too old), an
idmapped mount refused (the storage), the seccomp profile missing from the node, the
AppArmor profile not loaded ("apparmor profile not found"), or AppArmor absent from the node
("Cannot enforce AppArmor": leave appArmorProfile out, and set
worker.appArmor.enabled: false).
What the last line, bubblewrap's, says when it does not print bwrap:
| bubblewrap says | what refused it | what fixes it |
|---|---|---|
No permissions to create new namespace |
the seccomp profile: the runtime's default refuses a user namespace | worker.seccomp.type: Localhost, with the profile on the node |
Failed to make / slave: Permission denied |
AppArmor: the container runs under the runtime's default profile, which refuses every mount | worker.appArmor.enabled: true, with the profile loaded on the node, and appArmorProfile as above in the check |
setting up uid map: Permission denied |
AppArmor's restriction on unprivileged user namespaces (kernel.apparmor_restrict_unprivileged_userns=1): the container runs with no AppArmor profile (Unconfined), and an unconfined process that makes a user namespace is moved into a profile with no capabilities |
the same: under the chart's profile the worker is confined, and the restriction does not apply. Never Unconfined, and the sysctl can stay as it is |
Can't mount proc on /newroot/proc: Operation not permitted |
the container's /proc is masked |
hostUsers: false and procMount: Unmasked (worker.userNamespaces: true) |