Kernel series cuts CPU hotplug time 21x on large Xeon systems

Pasha Tatashin has posted an eight-part series that shortens the time to offline and online all application processors by 95.2 percent (21x) on a two-socket, 480-CPU Intel Xeon 6985P-C and by 59.8 percent (2.5x) on a two-socket, 512-CPU AMD EPYC 9B45.

The motivation is live update, which needs fast CPU hotplug. On modern high-core x86 servers, sequential offline and online of every application processor exposed hardware and algorithmic bottlenecks in architecture-specific bring-up, the scheduler topology builder, deadline bandwidth accounting, and vmstat. Under a standard expedited-RCU test loop, the Intel box fell from about 106 seconds total to about 5 seconds; the AMD box fell from about 8.4 seconds to about 3.4 seconds.

On Intel parts with very large shared last-level caches, full coherence-domain flushes dominated. Each WBINVD cost roughly 85 to 115 ms and stalled active cores on the socket. The series drops an obsolete WBINVD from the secondary-CPU trampoline, a leftover from 1990s Sequent NUMA-Q hardware whose platform support was removed years ago. It also skips WBINVD in the MWAIT path used when a CPU goes offline at runtime, retaining the flush only when tasks are frozen for system sleep, on the grounds that hardware C-states already keep the package coherent while it stays powered. Related changes stop broadcasting machine-check interrupt rediscovery to every online CPU and instead probe a single topology peer for orphaned banks, and they skip a stop-machine MTRR rendezvous when the incoming CPU's hardware ranges already match the saved state.

Scheduler and memory-management work attacks costs that grow badly with CPU count. Domain rebuilds no longer recompute shared group weights and capacities repeatedly, allocate large temporary structures that are immediately discarded, or walk covered CPUs one bit at a time after whole child groups are accounted for. Deadline server bandwidth updates lose redundant remote runqueue passes that previously scaled as O(N^2) per hotplug step (and O(N^3) across a full sequential sweep). Vmstat stops zeroing and rewriting identical per-CPU zone and node thresholds on every transition, writing only when the computed value actually changes.

Early testing on a smaller two-socket Ice Lake system by Tony Luck showed clear offline and online gains, with a further offline improvement when CPUs were taken down from high to low numbers so fewer shared machine-check banks were orphaned. Review feedback from Tim Chen and K Prateek Nayak has already flagged locking documentation and a narrow topology-group reuse check that still need tightening before the series is ready.

Taken together, the changes turn multi-minute CPU drains on large Intel servers into a few seconds, which is the difference between practical and painful live-update and maintenance flows on high-core machines.