--systemd-cgroup silently bypasses Kubernetes CPU limits when CRI emits cgroupfs cgroupsPath (contradicts #12392 guidance)
## Summary
On a host where containerd CRI emits `linux.cgroupsPath` in **cgroupfs path form** (`/kubepods/<qos>/pod<uid>/<id>`), enabling `--systemd-cgroup` causes runsc to land in a **parallel** systemd slice tree that kubelet never writes limits to. Pod CPU/memory limits are **silently bypassed**. The "Enable `systemd-cgroup = "true"`" advice in #12392 inverts the outcome on these clusters.
## Environment
- runsc: `release-20260520.0`
- containerd: `2.2.2`
- kubelet: `1.35.2`, `cgroupDriver: systemd`
- OS: Ubuntu 22.04, kernel 6.8, cgroup v2 unified (`stat -fc %T /sys/fs/cgroup` → `cgroup2fs`)
- systemd: 249
## How cgroups diverge
```mermaid
flowchart TD
KL[kubelet 1.35<br/>cgroupDriver: systemd] -->|writes cpu.max here| KFS["/sys/fs/cgroup/kubepods/burstable/<br/>pod<uid-dashes>/<br/>cpu.max = 50000 100000"]
KL -->|cgroupParent = /kubepods/burstable| CRI[containerd CRI 2.2]
CRI -->|"linux.cgroupsPath = /kubepods/burstable/pod<uid>/<id>"| RUNC[runc]
CRI --> RUNSC[runsc]
RUNC -->|joins| KFS
RUNSC -->|systemd-cgroup=false| KFS
RUNSC -->|systemd-cgroup=true<br/>rejects path,<br/>OR wrapper rewrites| SD["/sys/fs/cgroup/kubepods.slice/<br/>kubepods-burstable.slice/<br/>kubepods-burstable-pod<uid_underscores>.slice/<br/>cri-containerd-<id>.scope<br/>cpu.max = max 100000"]
SD -.->|no limit set here| X[CPU NOT enforced]
KFS -.->|kernel parent inheritance| OK[CPU enforced]
style KFS fill:#d4f5d4
style SD fill:#f5d4d4
style OK fill:#d4f5d4
style X fill:#f5d4d4
```
Two distinct cgroup directories exist on the host:
- `/sys/fs/cgroup/kubepods/burstable/pod<uid-with-dashes>/cpu.max` — written by kubelet
- `/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid_with_underscores>.slice/cpu.max` — NOT written by kubelet on this distro
`containerd-2.2` CRI's [`getCgroupsPath`](https://github.com/containerd/containerd/blob/main/internal/cri/server/podsandbox/helpers.go) only converts to `slice:cri-containerd:id` when the cgroup parent ends in `.slice`. Kubelet here passes `/kubepods/burstable`, so the OCI spec gets the path form even though every component is "configured for systemd".
## Reproduction
```
$ cat /etc/kubernetes/kubelet.yaml | grep cgroup
cgroupDriver: systemd
$ grep SystemdCgroup /etc/containerd/conf.d/*.toml
runc.options: SystemdCgroup = true
runsc.options: SystemdCgroup = true # tried both true and false
```
Pod:
```yaml
spec:
runtimeClassName: gvisor
containers:
- name: stress
image: polinux/stress
command: ["sh","-c","stress --cpu 4 --timeout 120s"]
resources:
limits: { cpu: 500m }
```
OCI spec containerd hands to runsc (`crictl inspectp`):
```
"cgroupsPath": "/kubepods/burstable/pod<uid>/<sandbox-id>"
```
### Mode A — `systemd-cgroup = "true"` in `runsc.toml`
Sandbox create fails immediately:
```
failed to create shim task: OCI runtime create failed: creating container:
cannot set up cgroup for root: invalid systemd path:
"/kubepods/burstable/pod<uid>"
```
(`runsc/cgroup/cgroup.go::TransformSystemdPath` requires `slice:prefix:name` form.)
### Mode B — same `systemd-cgroup = "true"` + a shim wrapper that rewrites the OCI bundle to slice form before runsc starts
Sandbox starts. Lives at:
```
/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid_>.slice/cri-containerd-<id>.scope
```
But kubelet writes the limit elsewhere:
```
$ cat /sys/fs/cgroup/kubepods.slice/.../kubepods-burstable-pod<uid_>.slice/cpu.max
max 100000 # ← UNLIMITED
$ cat /sys/fs/cgroup/kubepods/burstable/pod<uid-dashes>/cpu.max
50000 100000 # ← actual kubelet limit (500m)
```
CPU stress consumes the host:
```
# 30s window, pod cgroup cpu.stat:
usage_usec: +106,313,470 µs / 30s = 3.54 CPUs ← wanted 0.5
nr_throttled: 0
throttled_usec: 0
```
### Mode C — `systemd-cgroup = "false"`
Sandbox joins `/sys/fs/cgroup/kubepods/burstable/pod<uid-dashes>/<id>`. Kernel enforces parent `cpu.max`:
```
# Same 30s window, same pod, same node:
cpu.max: 50000 100000
usage_usec: +15,001,282 µs / 30s = 0.50 CPUs ← correct
nr_throttled: +300 ← throttling fires
throttled_usec: +103,727,954 µs
```
## Comparison table
| `systemd-cgroup` | Pod cgroup runsc joined | `cpu.max` there | CPU used (limit 500m) | `nr_throttled` |
|---|---|---|---|---|
| `true` (no wrapper) | sandbox create fails | n/a | n/a — pod never runs | n/a |
| `true` (with shim wrapper) | `kubepods.slice/.../pod<uid_>.slice` | `max 100000` | **3.54 CPU** | **0** |
| `false` | `/kubepods/burstable/pod<uid>` | `50000 100000` | **0.50 CPU** | **300/30s** |
Only `systemd-cgroup=false` enforces the limit. Maintainer @fvoznika's earlier comment in #9580 — *"cgroupv2 should still be enforcing the proper memory/cpu limit on the sandbox process externally since the parent slice has those limits set"* — is correct, but only when runsc is in **the same** cgroup subtree where the parent limits live.
## Why #12392's fix doesn't generalize
#12392 was closed Dec 17 2025 with:
> The fix was to enable `systemd-cgroup = "true"` in the runsc configuration via `/etc/containerd/runsc.toml` ...
That worked on the reporter's environment (Amazon Linux 2023 / EKS), presumably because **kubelet there really does populate `kubepods.slice/.../pod<uid>.slice/cpu.max`**. On Ubuntu 22.04 + kubelet 1.35 + containerd 2.2, kubelet writes the cgroupfs tree. **Following #12392's recommendation here silently disables CPU enforcement.**
## Proposed fixes (any of)
1. **Detect-and-fail-fast.** When `--systemd-cgroup` is set and `Linux.CgroupsPath` is in cgroupfs path form, log a loud error and refuse to start (instead of `invalid systemd path` which sounds like a transient parser issue). At minimum, document this as a hard prerequisite.
2. **Detect-and-fall-back.** Recognize cgroupfs path form even when `--systemd-cgroup=true` is set, and join the cgroupfs tree directly with `fs` driver semantics (warn once). This preserves limits regardless of mode.
3. **Documentation.** [Containerd setup doc](https://gvisor.dev/docs/user_guide/containerd/configuration/) and [Systemd cgroup driver doc](https://gvisor.dev/docs/user_guide/systemd/) should add a verification step:
```bash
# On a node, after creating a gVisor pod:
PID=$(sudo crictl inspectp <sandbox-id> | jq -r .info.pid)
POD_CG="/sys/fs/cgroup$(dirname $(awk -F'::' '{print $2}' /proc/$PID/cgroup))"
cat $POD_CG/cpu.max # must show '<quota> <period>', NOT 'max 100000'
```
And note that on cgroup-v2 hosts where kubelet writes to `/sys/fs/cgroup/kubepods/...`, `systemd-cgroup` **must remain false** for CPU enforcement to work.
4. **Walk the cgroup tree upward when `max`.** Less invasive than the above; same idea as #12391. If runsc reads `max` on the immediate cgroup, walk up to find the effective limit. This won't fully fix enforcement (the kernel still won't throttle a `max` cgroup), but at least gets gVisor's internal accounting correct.
## Related
- #12392 — closing comment is the source of the misleading guidance
- #9580 — same root cause described, host inherits parent limits
- #8047 — design rationale: gVisor enforces pod limits by joining the pod cgroup
- #13067 — separate cAdvisor regression in `systemd-cgroup=true` mode
- #7671 — initial `invalid systemd path` report on Ubuntu 22.04 + cgroup v2
- containerd `getCgroupsPath`: only converts to slice form when parent ends in `.slice`
I'm happy to send a PR for option (1) or (3) if there's interest. Option (2) likely needs design input from the cgroup maintainers.
1 条评论