freenode
Kernel & Low-Level

Linus calls EPOLLEXCLUSIVE broken after pipe readers hang on data

A wakeup bug in Linux pipes shows exclusive epoll watchers can swallow a wake meant for a blocked reader and never pass it on.

A Linux pipe can hold unread data while a task blocked in read() sleeps on it, if an epoll watcher registered with EPOLLEXCLUSIVE was added to the read end first. Mani Tofigh reported the hang and a small fix; kernel developer Oleg Nesterov then showed the failure is older than the recent optimization Tofigh blamed, and Linus Torvalds said the flag itself violates the only rule exclusive waits exist to enforce.

Pipes wake one exclusive waiter when a write fills an empty pipe. The kernel assumes those exclusive waiters are blocked readers, and that a woken reader will either consume the data or wake the next waiter. An EPOLLEXCLUSIVE watcher is also an exclusive waiter, and it can sit ahead of a real reader. A write then wakes only the watcher. If that watcher reads without blocking, it never hands the wakeup on, and the reader stays asleep with bytes still in the pipe.

A recent change made this worse for later writes. Extra per-write wakeups, which used to run after any poll of the pipe, were limited to edge-triggered consumers. Subsequent writes into a pipe that was no longer empty stopped waking anyone. Tofigh's patch would turn those wakeups back on for EPOLLEXCLUSIVE as well, restoring the older behavior for the case he hit. He found it with a deterministic hypervisor still under development; a small C program reproduces it in about two seconds. He also noted related gaps, including the write side with EPOLLOUT and EPOLLEXCLUSIVE, and left those alone.

Nesterov agreed that change matters, then posted a simpler program that hangs even with the patch, and even without the earlier optimization: one child waits in epoll and reads a single byte, another blocks in read(), and a two-byte write wakes only the first. "EPOLLEXCLUSIVE breaks this logic," he wrote.

Torvalds rejected that framing. "This literally violates THE WHOLE AND ONLY POINT of exclusive waits," he wrote. An exclusive waiter must act on the wakeup, either by consuming the data or by waking the next waiter if it cannot. "The logic is fine. EPOLLEXCLUSIVE is broken beyond words." He said the kernel should start de-emphasizing epoll, beginning with refusing EPOLLEXCLUSIVE because callers do not honor it.