nomadLab

Search nomadLab

Infrastructure8 min read

Kubernetes 1.35 and cgroup v1: What Broke, and What Did Not

The kubelet stopped starting on cgroup v1 in 1.35, and one line of config still turns that off. The code is not gone, no removal release has been named, and the deadline that actually costs you money is somewhere else.

Updated

A node comes up, the kubelet exits, the node never goes Ready, and the log line says something about cgroup v1 being deprecated. That is Kubernetes 1.35 doing exactly what it said it would do.

What 1.35 changed is one default. The kubelet config field failCgroupV1 flipped from false to true, so a kubelet that finds a cgroup v1 hierarchy at /sys/fs/cgroup refuses to start instead of warning and continuing. The 1.35 changelog files it under ACTION REQUIRED: “nodes will not start on a cgroup v1 by default. This puts cgroup v1 into a deprecated state.”

Then there is the part that most write-ups get wrong, including the one I am rewriting here.

Versions, dates and prices below were checked against vendor documentation on 22 August 2026.

The cgroup v1 code is still in the kubelet

The common claim is that the v1 code paths disappear in the next release, so the override buys you one cycle. That is not what happened. KEP-5573 graduated to beta in 1.35, and its own removal section is still marked unresolved: “Once all supported releases of Kubernetes have FailCgroupV1 set to true, we can begin the removal of the cgroup v1 support.” No release is named. Upstream still runs a cgroup v1 test lane and still treats regressions in it as cherry-pick candidates.

Kubernetes 1.36 shipped on 22 April 2026 and did not remove it either. So if you flipped the flag back in January, your nodes are still booting today, and nothing on the calendar is going to stop them.

I would treat that as the bad news rather than the good news. A hard cutoff at least schedules the work for you. What you have instead is a node that boots fine while quietly falling out of the feature set everyone else is getting.

What a kubelet running version 1.35 or newer does with each cgroup layout. On cgroup2fs it boots and every cgroup v2 feature works. On a cgroup v1 hierarchy, reported by stat as tmpfs, it refuses to start, which is the default from 1.35 onward. On cgroup v1 with failCgroupV1 set to false it boots and keeps booting, but every cgroup v2 feature stays off. kubelet 1.35 or newer, by what it finds at /sys/fs/cgroup cgroup2fs → Boots. MemoryQoS and PSI work, and so does everything 1.36 added on top of them. tmpfs → Refuses to start. This is the 1.35 default. Node never reaches Ready. Nothing schedules. tmpfs, and failCgroupV1: false → Boots, and keeps booting. Every cgroup v2 feature stays off for as long as you leave it there. The third row has no expiry date. KEP-5573 has not named a release that removes the v1 code, so this is a state a cluster can sit in indefinitely, which is the actual risk.
Only the middle row announces itself. The bottom row is the one that lasts for years.

Find out what you actually have

One command on a node settles it:

stat -fc %T /sys/fs/cgroup

cgroup2fs means v2. tmpfs means v1, including the hybrid layout, which counts as v1 as far as the kubelet is concerned. That is the check in the Kubernetes docs, not a folk remedy.

For a whole cluster, a DaemonSet that mounts the host path and prints one line per node is enough:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: cgroup-audit
  namespace: kube-system
spec:
  selector:
    matchLabels: { app: cgroup-audit }
  template:
    metadata:
      labels: { app: cgroup-audit }
    spec:
      containers:
      - name: probe
        image: busybox:1.37
        command: ["sh", "-c"]
        args:
        - |
          echo "$(hostname) cgroup=$(stat -fc %T /host/sys/fs/cgroup) kernel=$(uname -r)"
          sleep 3600
        volumeMounts:
        - { name: cgroup, mountPath: /host/sys/fs/cgroup, readOnly: true }
      volumes:
      - name: cgroup
        hostPath: { path: /sys/fs/cgroup }

Then kubectl logs -l app=cgroup-audit --tail=1 --prefix | sort. On GKE there is a shortcut, because the cgroup mode is a field on the node pool: gcloud container clusters describe CLUSTER --format='value(nodePools[0].config.effectiveCgroupMode)'.

The reason to probe rather than read your own Terraform is that launch templates lie by omission. A pinned AMI ID from two years ago still says AL2023 in the variable name.

Where cgroup v1 still lives in 2026

Managed platforms mostly moved, and the exceptions are specific enough to check by name.

PlatformState as of August 2026
EKS on AL2023cgroup v2, matches upstream behavior
EKS on Bottlerocketcgroup v2, but AWS ships failCgroupV1: false in the kubelet config
EKS Fargatestill cgroup v1
GKEv2 default for new nodes from 1.26, older nodes keep their mode until migrated
AKSUbuntu 22.04 and Azure Linux 3.0 node images have been v2 for several releases

Those first three rows come straight out of the EKS 1.35 release notes, and the Fargate line is the one worth reading twice. If part of your workload runs on Fargate profiles, that part is on cgroup v1 right now and you are not the one who decides when it moves.

The GKE behavior is the other quiet one. New nodes from 1.26 onward are v2, but existing nodes keep whatever mode they were created with. A long-lived node pool that has been auto-upgrading its Kubernetes version this whole time can still be sitting on v1.

What the override actually costs

The argument for migrating is not that the flag will stop working. It is that the flag freezes a node while the memory and pressure features keep moving.

The Kubernetes docs name MemoryQoS as cgroup v2 only, and list Pressure Stall Information among the things v2 buys you. Both got real work in 1.36. KubeletPSI graduated to GA and is on by default, exposing per-pod CPU, memory and IO contention through the Summary API. MemoryQoS gained tiered protection in the same release, writing memory.min for Guaranteed pods and memory.low for Burstable ones, plus a MemoryReservationPolicy field to control it.

None of that reaches a v1 node. You get coarse node-pressure heuristics and no per-pod contention signal, on the same cluster where your other node groups have both. That gap widens every release, and it does so without an error message.

The clock that costs money

For EKS specifically, the cgroup deadline is not the expensive one. The AL2 node group behind it is.

Kubernetes 1.32 was the last EKS version with AL2 AMIs. Its standard support ended on 23 March 2026 and its extended support ends on 23 March 2027, which leaves about seven months. Extended support is not free: an EKS cluster costs $0.10 per hour on standard support and $0.60 per hour in extended support, so every cluster parked on 1.32 has been paying an extra $0.50 an hour, roughly $365 a month, since March. At the end of extended support the control plane is auto-upgraded whether or not the nodes are ready.

There is no in-place AL2 to AL2023 upgrade. You stand up a new node group, cordon, drain in small waves while watching PodDisruptionBudgets, and delete the old one. AL2023 is a different distribution rather than a newer AL2, and the specific pieces that break are their own subject, which I wrote up separately in the AL2023 migration post.

The runtime deadline rides along with it. AWS states that Kubernetes 1.35 is the last release supporting containerd 1.x, so 1.36 requires 2.0 or later. EKS 1.34 already ships containerd 2.1, and AKS held Ubuntu 22.04 with containerd 1.7.29 through 1.35 before jumping to Ubuntu 24.04 with containerd 2.3.1 in 1.36. If you bake your own AMIs, the registry config format changed in a way that fails silently, which is covered in the containerd 2.x post.

Closing the ticket

Once the new nodes are in and the old group is gone, three checks are worth the minutes.

Confirm every node reports cgroup2fs by rerunning the audit DaemonSet rather than trusting the node group config. Grep your kubelet configuration for failCgroupV1 and delete the line, because a leftover false on a v2 node does nothing today and hides the next v1 node that sneaks in. Then look at a workload with a tight memory limit for a few days: PSI metrics are now available through the Summary API, so you can see whether a container is stalling on memory instead of guessing from restart counts.

If you have no idea where your fleet stands, the DaemonSet takes five minutes and answers the only question that matters first. On EKS, run aws eks describe-cluster --name CLUSTER --query 'cluster.version' in the same sitting. If it says 1.32, the cgroup problem is the smaller half of what you found.

Keep reading