Kernfs patch ends multi-minute cgroupfs stalls under OOM

Meta is still seeing multi-minute cgroupfs stalls on production hosts, and a new kernfs change targets the remaining cause: allocating inodes while holding the shared kernfs semaphore.

Shakeel Butt reported that on Meta's fleet, with a 6.16-based kernel and later kernfs work backported, oomd and related processes waited 72 seconds in readdir and 15 minutes in lookup while an out-of-memory container created cgroups. In one incident PID 1 blocked on the semaphore inside rmdir while already holding the cgroup mutex. A lock profiler sampling part of the fleet caught the lookup path holding the semaphore across memcg reclaim on 26 hosts over 29 days, for as long as 625 ms.

Lookups and related export paths allocate a new inode under the semaphore for read and charge it to the caller's memory cgroup. When that cgroup is already at memory.max, the allocation can sit in reclaim for a long time, sometimes until the OOM killer runs. A writer then queues on the lock and every new reader queues behind the writer. On cgroup2 that blocks every cgroupfs user on the machine. The pattern is common under pressure: reclaim drops kernfs dentries quickly, so files of a new cgroup have no inodes yet and every lookup tries to create one.

The patch allocates the inode before taking the semaphore and frees it if it is unused, the same approach earlier kernfs work used for name strings and open nodes. All other work stays under the lock, so a new inode is still attached only to an active node and removal still finds every inode it must clear. A related tweak stops holding the lock across dentry allocation when resolving a parent. Missing-name lookups cost about 0.35 µs more on cgroupfs; cold lookups of existing sysfs and cgroupfs files are not slower.

Cgroupfs is the control plane for containers. A single thrashing workload should not be able to freeze init, the OOM daemon, and every other cgroupfs client on the host by coupling reclaim to a machine-wide lock.