Linux caps pagecache folio refs to stop large-page overflow

The Linux kernel is about to refuse extra references to pagecache folios once their refcount gets too close to wrapping, after developers concluded that large file pages make an overflow that was long treated as theoretical actually reachable.

David Hildenbrand (Arm) posted the series as urgent work. A folio's reference count is a signed integer. If it wraps negative, later code can treat a live page as unreferenced or, worse, keep using a page whose accounting no longer makes sense. Kernel developers had assumed that kind of wrap would be loud. Hildenbrand wrote that he is no longer sure that is always true, and that leaving the hole open is irresponsible.

The practical case is large folios in the page cache, including transparent huge pages backed by files and shmem. Unlike anonymous huge pages, a file folio can be mapped many times from one process, so a single address space can pile up references cheaply. With enough memory and processes the same trick works on ordinary pages, but that is much less realistic. Anonymous folios are treated as not urgent: overflowing them would take more processes than a normal system allows, and they already have a separate pin count.

The fix stops ordinary callers short of the point where the count would go negative, and keeps a higher reserve for processes with CAP_SYS_RESOURCE so an administrator can still recover a wedged machine. There is also deliberate headroom so concurrent increments and short-lived references do not have to be exact. Mapping a pagecache folio that is already at the limit raises SIGBUS. On fork, the kernel drops the page-table entry rather than failing the fork. If a migration entry cannot be restored under the limit, the entry is zapped.

The same ceiling has to be honored everywhere an unbounded number of references can appear. That includes splicing folios into pipes (including FUSE and tee of pipe buffers), get-user-pages and memfd pins, and deferred reference drops while page tables are being torn down. Chris Mason reported the last of those: after MADV_DONTNEED, the kernel can drop the page-table lock and hold folio references until a later flush, while faults remap the same folios and add more. The series flushes those pending references early. A related iommufd cleanup bug, easier to hit once pins can fail, is fixed in the same set.

Hildenbrand has already applied the first five patches to the mm tree's fixes branch, so they are headed for linux-next and are marked for stable. He asked networking maintainers to look at the rest. Copying socket-buffer fragments (packet sockets trimming a clone, nftables dup, NFQUEUE), software segmentation, TCP receive coalescing, SACK shifting, and splicing pages into or out of sockets (TCP, AF_UNIX, KCM, kTLS, and in-kernel users such as sunrpc) can multiply references a packet already holds. Those paths are being taught to use a fallible page grab and return an error instead of wrapping the count. One split path is left alone for now, on the argument that reaching the limit that way would already have consumed a large amount of socket memory.

Hildenbrand called the approach a short-term measure that backports more comfortably than the alternatives, and wrote that overflowing the refcount "is not really big news, but it's about time that we do something about it because some modern technology might find ways to abuse it." Pedro Falcato and Jose A. Perez de Azpillaga reviewed parts of the series.