Kernel fix corrects wrong TLB flush in page table reclaim

Andy Lutomirski has fixed a bug in the Linux kernel's direct page table reclaim path that passed the wrong address when freeing a reclaimed page table, so architectures that use that address would flush the wrong TLB (paging-structure cache) entry.

When zap code reclaims an empty page table, it calls the free helper with an address one past the end of the range the table covers. The free itself was correct; the address is only used to drop cached references to the table about to be freed. Any architecture that honors the parameter therefore flushed the wrong location.

The bad case is not ordinary munmap or exit. It shows up when a process issues something like MADV_DONTNEED over a range large enough to reclaim a full page table. Under a narrow set of conditions the upper-level entry can be cleared after the last flush of the range, leaving a CPU free to cache a reference to the now-empty table, including via speculation. The subsequent flush hits the wrong address, then the table is freed and may be reused while still cached.

On x86 the bug was often masked. Non-KPTI Intel systems flush all paging-structure caches on INVLPG regardless of address, but INVPCID does not, and the kernel uses INVPCID when it is available. AMD systems that set EFER.TCE are more exposed because even INVLPG becomes address-specific there.

Linus Torvalds called the fix obviously correct. He said the existing code looked so broken he half expected some subtle reason it still worked, but concluded systems probably survived by accident because other paths through the zap code flush via mmu_gather anyway.

David Hildenbrand noted the failure is limited to direct reclaim of that kind and suggested selftests that populate tables, reclaim them with MADV_DONTNEED, pressure the allocator, and re-fault the range. Qi Zheng reported that an earlier form of page-table reclaim had run stably in production without the rework that introduced the bad address.