Isolcpus FIFO spinners freeze systemd during page migration
A production hang lasting minutes on large nohz_full systems prompted kernel fixes so isolated CPUs no longer block LRU drains and cpuset memory moves.
Low-latency Linux deployments that pin SCHED_FIFO busy loops to isolated CPUs can leave systemd stuck uninterruptible for minutes whenever large NUMA page migrations run, blocking SSH logins, session creation, service restarts, and container lifecycle operations.
Caron Wills hit the failure in production on a 192-CPU dual-socket EPYC machine using isolcpus and nohz_full for most cores. Matthew Ruffell helped reproduce and ship a six-part series after seeing multi-minute hangs under synthetic load. When cpuset memory node changes migrate tens of gigabytes of pages, the migration path must drain per-CPU folio batches and sometimes wait on expedited RCU. Drain work is queued onto every CPU that still holds batched pages. On isolated cores, real-time spinners never yield to ordinary kworkers, so the drain never finishes. PID 1, which flushes the migration workqueue before returning to userspace, waits behind it. The same class of systems also allowed RCU expedited workers onto nohz_full CPUs when those CPUs stayed in scheduler domains, so FIFO tasks could starve grace-period progress system-wide.
The fixes stop batching folios on isolated CPUs, drain any leftover batches when isolation is applied at runtime, keep housekeeping kthreads off kernel-noise CPUs even when domain isolation is incomplete, skip the synchronous flush for non-blocking cpuset.mems writers (matching how memory controller limits already behave), and stop forcing parent tasks to flush unrelated migrations on CLONE_INTO_CGROUP forks. Migrations still complete in the background; init is no longer held hostage by them.
Michal Hocko noted that isolated CPUs should not run unrelated work, yet their cached pages should not stall the rest of the machine, and asked about earlier ideas to flush per-CPU caches on return to userspace as a broader alternative. The series targets the concrete hangs seen with cpuset-driven migration and systemd.