Kubernetes cgroup v2 Migration: Audit Nodes Before Upgrades

Kubernetes operators still running Linux nodes on cgroup v1 now have an upgrade risk that can become a node-availability problem. Kubernetes deprecated cgroup v1 in version 1.35, where the kubelet refuses to start on a v1 node by default, and the project is preparing for eventual code removal. This guide explains how to inventory that exposure, align the kubelet and container runtime, validate monitoring agents, and test a migration plan without changing a production cluster.

This is a practical migration guide, not a breaking-vulnerability alert. The current upstream guidance was published on October 6, 2026. Release targets can move, so confirm the documentation for the exact Kubernetes minor version and distribution you operate before scheduling changes.

What changed in Kubernetes

The Kubernetes project moved cgroup v1 into maintenance mode in version 1.31. Cgroup v2 has been stable since version 1.25. Starting with Kubernetes 1.35, failCgroupV1 defaults to true, which prevents the kubelet from starting on a cgroup v1 node unless an administrator deliberately applies the temporary override. The October 6 project guidance says the fallback remains available in version 1.36 and is scheduled for removal in version 1.38. The associated KEP states that removal must respect the Kubernetes deprecation policy.

Do not treat failCgroupV1: false as the migration. It is a short-term compatibility control that keeps a legacy node running while you remove the dependencies that block cgroup v2.

Why cgroup v2 matters to security and reliability

Cgroups enforce CPU, memory, process and I/O resource policy underneath Kubernetes. Version 2 replaces the separate v1 hierarchies with a unified hierarchy and gives the kernel a more consistent resource-control model. Kubernetes highlights safer subtree delegation, unified accounting and Pressure Stall Information as important improvements. The Linux kernel documentation also explains that delegated child hierarchies remain constrained by limits imposed by their parent.

• Resource isolation is easier to reason about. CPU and memory controls live in one hierarchy instead of potentially diverging across several trees.
• Delegation has stronger containment rules. A delegated workload can organize its children but cannot move processes outside the delegated subtree or escape parent limits.
• Pressure becomes observable. PSI can expose CPU, memory and I/O contention before it turns into sustained application failure.
• OOM behavior is more coherent. On v2 nodes, Kubernetes can use memory.oom.group so a container’s processes are terminated together rather than leaving a partially functioning multi-process container.

Cgroup v2 is not a substitute for seccomp, AppArmor or SELinux, Pod Security Admission, user namespaces, network policy, or a hardened runtime. It strengthens resource management and delegation; it does not turn an untrusted container into a complete security boundary.

Inventory the fleet before changing it

Start with evidence. Identify every Linux node pool, its operating-system image, kernel, Kubernetes version, runtime, kubelet driver and runtime driver. Include DaemonSets and host agents that read /sys/fs/cgroup directly. Those consumers are often the hidden dependency that turns a straightforward node-image change into a monitoring blind spot.

Read-only node checks

# Run on a Linux node
stat -fc %T /sys/fs/cgroup/
uname -r
systemctl show kubelet -p ActiveState -p SubState

# Useful when crictl is configured for the node runtime
crictl info | jq '.config.cgroupDriver // .config.containerd.runtimes'

# Cluster-side inventory
kubectl get nodes -o custom-columns='NAME:.metadata.name,OS:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion,KUBELET:.status.nodeInfo.kubeletVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion'

The filesystem type should be cgroup2fs for cgroup v2. A result of tmpfs indicates cgroup v1. Treat command output as an inventory input, not as permission to edit boot arguments on a live node.

Compatibility checks that are easy to miss

• Standalone cAdvisor should be version 0.43.0 or later.
• Monitoring, EDR and security DaemonSets that parse the cgroup filesystem need explicit v2 support.
• Older language runtimes can misread container limits. Kubernetes specifically notes that Node.js 18 does not reliably detect cgroup v2 memory limits, while Node.js support starts with version 20.3.0.
• The kubelet and container runtime must agree on the cgroup driver. For kubeadm systems, Kubernetes recommends systemd.
• Capacity management must account for eviction thresholds, reserved resources and the behavior of memory-sensitive workloads after the change.

Use a staged migration sequence

1. Classify dependencies. Mark direct cgroup readers, old runtimes, unsupported kernels and driver mismatches as blockers.
2. Build a replacement node image. Use an OS that enables cgroup v2 by default, a supported kernel and a compatible CRI runtime.
3. Align drivers. Configure both the kubelet and runtime for systemd, or verify supported automatic driver detection where your runtime and Kubernetes version provide it.
4. Canary one node pool. Cordon and drain according to your disruption budgets, join replacement capacity, then place representative workloads on the canary nodes.
5. Validate enforcement and telemetry. Check resource limits, eviction behavior, OOM events, PSI metrics, DaemonSet health and application memory sizing.
6. Expand gradually. Replace or reimage nodes in batches small enough to preserve capacity and quorum.

Changing the cgroup driver in place is possible in some kubeadm environments, but it affects both the kubelet and runtime and requires node-by-node draining. Replacing nodes with a tested image is often easier to audit and roll back. Follow the procedure supported by your Kubernetes distribution or managed service.

Validate the kernel-enforced result

A healthy node status does not prove that every limit reached the kernel. For a controlled test Pod, compare the requested resources in the Kubernetes object with the runtime specification and the corresponding v2 files. The upstream guide documents cpu.weight, cpu.max and memory.max as key inspection points. Avoid assuming that an OCI systemd cgroup path is a literal filesystem path.

kubectl get pod <pod> -n <namespace> 
  -o jsonpath='{.spec.nodeName}{"n"}{range .spec.containers[*]}{.name}{"t"}{.resources}{"n"}{end}'

# On the selected node, pin the exact container by namespace, Pod and name.
CONTAINER_ID=$(crictl ps 
  --label io.kubernetes.pod.namespace=<namespace> 
  --label io.kubernetes.pod.name=<pod> 
  --name <container> -q | head -n1)

PID=$(crictl inspect "$CONTAINER_ID" | jq -r '.info.pid')
CGROUP="/sys/fs/cgroup$(awk -F: '$1=="0"{print $3}' /proc/$PID/cgroup)"
cat "$CGROUP/cpu.weight" "$CGROUP/cpu.max" "$CGROUP/memory.max"

Run these commands only on systems you administer. They are read-only, but access to the runtime socket and other containers’ process metadata is privileged. Record the expected values for the test workload before deciding whether the observed mapping is correct.

Hands-on lab: audit a synthetic node inventory

This offline lab checks a CSV inventory. It does not connect to Kubernetes, read a host cgroup filesystem, change a node, or prove that a production workload is safe to migrate.

Prerequisites and isolated environment

• Python 3.10 or later
• An empty working directory
• No cluster credentials and no network access are required

Create fleet.csv with representative, non-sensitive values from your inventory system:

node,os,kernel,kubernetes,cgroup_fs,kubelet_driver,runtime,runtime_driver,cadvisor,nodejs,direct_cgroup_reader
worker-a,Ubuntu 24.04,6.8.0,1.37,cgroup2fs,systemd,containerd 2.0.4,systemd,0.49.1,22.12.0,no
worker-b,Ubuntu 20.04,5.4.0,1.34,tmpfs,cgroupfs,containerd 1.6.33,cgroupfs,0.42.0,18.20.4,yes
worker-c,RHEL 9,5.14.0,1.37,cgroup2fs,systemd,CRI-O 1.32.2,systemd,0.49.1,20.18.1,no
worker-d,Debian 12,6.1.0,1.36,cgroup2fs,systemd,containerd 2.0.4,cgroupfs,0.49.1,20.18.1,no

Save the following as audit_cgroup_v2.py:

#!/usr/bin/env python3
import csv, re, sys

def ver(value):
    match = re.search(r"(d+(?:.d+)+)", value)
    return tuple(map(int, match.group(1).split("."))) if match else ()

def older(value, minimum):
    current = ver(value)
    return bool(current) and current < minimum

def audit(row):
    findings = []
    if row["cgroup_fs"] != "cgroup2fs": findings.append("BLOCK:cgroup-v1")
    if older(row["kernel"], (5, 8)): findings.append("BLOCK:kernel<5.8")
    if row["kubelet_driver"] != "systemd": findings.append("WARN:kubelet-driver")
    if row["runtime_driver"] != row["kubelet_driver"]: findings.append("BLOCK:driver-mismatch")
    runtime = row["runtime"].lower()
    if runtime.startswith("containerd") and older(runtime, (1, 4)): findings.append("BLOCK:containerd<1.4")
    if runtime.startswith("cri-o") and older(runtime, (1, 20)): findings.append("BLOCK:cri-o<1.20")
    if row["cadvisor"] and older(row["cadvisor"], (0, 43, 0)): findings.append("WARN:cadvisor<0.43")
    if row["nodejs"] and ver(row["nodejs"])[:1] == (18,): findings.append("WARN:node18-cgroup-limits")
    if row["direct_cgroup_reader"].lower() in {"yes", "true", "1"}: findings.append("REVIEW:direct-cgroup-reader")
    return findings

with open(sys.argv[1], newline="", encoding="utf-8") as handle:
    rows = list(csv.DictReader(handle))

attention = 0
for row in rows:
    findings = audit(row)
    status = "READY" if not findings else "ATTENTION"
    attention += bool(findings)
    print(f"{status:9} {row['node']:<10} {','.join(findings) if findings else 'none'}")
print(f"summary total={len(rows)} ready={len(rows)-attention} attention={attention}")
raise SystemExit(1 if attention else 0)

Run the audit:

python3 audit_cgroup_v2.py fleet.csv

Expected output

READY     worker-a   none
ATTENTION worker-b   BLOCK:cgroup-v1,BLOCK:kernel<5.8,WARN:kubelet-driver,WARN:cadvisor<0.43,WARN:node18-cgroup-limits,REVIEW:direct-cgroup-reader
READY     worker-c   none
ATTENTION worker-d   BLOCK:driver-mismatch
summary total=4 ready=2 attention=2

The script exits with code 1 when any node needs attention, which makes it suitable for a CI inventory gate. BLOCK identifies an upstream requirement or a configuration conflict; WARN identifies a known compatibility concern; REVIEW requires human confirmation.

Troubleshooting and cleanup

• If the script reports a missing column, compare the CSV header with the example exactly.
• If a runtime version is unknown, leave the field accurate rather than inventing a value, then review that row manually.
• Delete the lab directory when finished. It creates no services, packages, users or network connections.

Detection and rollback criteria

During the canary, alert on kubelet startup failures, NotReady nodes, unexpected evictions, container OOM changes, missing DaemonSet telemetry, altered CPU throttling and sustained PSI pressure. Compare the canary with a similar v1 node pool under the same workload. A single green node is not enough; validate application behavior and security visibility.

• Pause expansion if the kubelet and runtime disagree on their driver.
• Roll back the node image if critical monitoring or EDR telemetry disappears.
• Stop if resource limits do not map to the expected cgroup v2 files.
• Restore capacity before draining more nodes if workloads cannot reschedule within disruption budgets.

Conclusion

The safe Kubernetes cgroup v2 migration is an inventory and validation project, not a boot-flag change. Find v1 nodes before the next upgrade, align the kubelet and runtime, remove direct-filesystem dependencies, canary a supported node image, and verify that resource policy reaches the kernel. That approach improves readiness for Kubernetes’ cgroup v1 retirement while protecting workload availability and defensive telemetry.

Related VigilSecureInfo guides

• Ubuntu security update auditing and staged patching
• NIST O-RAN RIC zero-trust checklist
• Shared filesystem risk in containerized AI workloads

Primary references

• Kubernetes: The Shift to cgroup v2 in Kubernetes, October 6, 2026
• Kubernetes documentation: About cgroup v2
• KEP-5573: Remove cgroup v1 support
• Kubernetes container runtime configuration
• Linux kernel cgroup v2 documentation