lib: fix crash in thread_process_io_inner_loop on stale epoll event

[Manpreet-k0]

When do_event_cancel() processes a pending cancellation at the top of event_fetch_inner_loop(), it removes the fd from the epoll_event_hash and calls EPOLL_CTL_DEL. However, epoll_wait() can still deliver events that were already queued in the kernel's ready list before the EPOLL_CTL_DEL took effect.

When thread_process_io_inner_loop() processes such a stale event, the hash lookup returns NULL and the assert(hash_ev) fires, crashing the daemon.

Fix:

Replace the assert with a graceful return

Crash:

Thread 4 (Thread 0x7f9ea41e89c0 (LWP 8531)):
0  0x00007f9ea449112b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
1  0x00007f9ea4497482 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libc.so.6
2  0x00007f9ea471eaa5 in _frr_mtx_lock (mutex=0x5629ffcf86a8) at ../lib/frr_pthread.h:253
3  event_cancel_async (master=0x5629ffcf8580, thread=0x562a024559f8, eventobj=0x0) at ../lib/event.c:2040
4  0x00005629e3b4b15e in bgp_writes_off (connection=0x562a02455990) at ../bgpd/bgp_io.c:83
5  0x00005629e3b430ad in bgp_stop (connection=0x562a02455990) at ../bgpd/bgp_fsm.c:2065
6  0x00005629e3b48a9d in bgp_event_update (connection=connection@entry=0x562a02455990, event=TCP_connection_closed) at ../bgpd/bgp_fsm.c:3191
7  0x00005629e3c14828 in bgp_process_conn_error (event=<optimized out>) at ../bgpd/bgpd.c:9707
8  0x00007f9ea471f59a in event_call (event=event@entry=0x7fff59c62f90) at ../lib/event.c:2730
9  0x00007f9ea46bfed0 in frr_run (loop=0x5629ffb4fcd0) at ../lib/libfrr.c:1258
10 0x00005629e3af650e in main (argc=<optimized out>, argv=0x7fff59c63248) at ../bgpd/bgp_main.c:549
<SNIP>
Thread 1 (Thread 0x7f9ea2be06c0 (LWP 8534)):
0  0x00007f9ea4495eec in ?? () from /lib/x86_64-linux-gnu/libc.so.6
1  0x00007f9ea4446fb2 in raise () from /lib/x86_64-linux-gnu/libc.so.6
2  0x00007f9ea470bf86 in core_handler (signo=6, siginfo=0x7f9ea2bdeb30, context=<optimized out>) at ../lib/sigevent.c:268
3  <signal handler called>
4  0x00007f9ea4495eec in ?? () from /lib/x86_64-linux-gnu/libc.so.6
5  0x00007f9ea4446fb2 in raise () from /lib/x86_64-linux-gnu/libc.so.6
6  0x00007f9ea4431472 in abort () from /lib/x86_64-linux-gnu/libc.so.6
7  0x00007f9ea473def9 in _zlog_assert_failed (xref=xref@entry=0x7f9ea4803080 <_xref.24>, extra=extra@entry=0x0) at ../lib/zlog.c:779
8  0x00007f9ea471b979 in thread_process_io_inner_loop (m=m@entry=0x5629ffcf8580, revent=0x5629ffd00740) at ../lib/event.c:2170
9  0x00007f9ea471f3c3 in thread_process_io (num=1, m=0x5629ffcf8580) at ../lib/event.c:2247
10 event_fetch_inner_loop (event=0x0, continued=<synthetic pointer>, broken=<synthetic pointer>, fetch=0x7f9ea2bdfb80, m=0x5629ffcf8580) at ../lib/event.c:2544
11 event_fetch (m=0x5629ffcf8580, fetch=fetch@entry=0x7f9ea2bdfb80) at ../lib/event.c:2559
12 0x00007f9ea46ae9e9 in fpt_run (arg=0x5629ffcf8490) at ../lib/frr_pthread.c:368
13 0x00007f9ea44941f5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
14 0x00007f9ea45148dc in ?? () from /lib/x86_64-linux-gnu/libc.so.6

[greptile-apps]

Greptile Summary

This PR fixes a crash in thread_process_io_inner_loop() caused by a race condition between do_event_cancel() and the kernel's epoll ready-event queue. The single-line assert(hash_ev) is replaced with a graceful early return and a diagnostic zlog_debug message, correctly handling the case where the kernel delivers a stale event for an fd that was already removed from the epoll_event_hash and deregistered via EPOLL_CTL_DEL.

• Root cause: do_event_cancel() removes the fd from epoll_event_hash and calls EPOLL_CTL_DEL at the top of event_fetch_inner_loop(), but the kernel may have already queued a ready event for that fd before EPOLL_CTL_DEL took effect; subsequent epoll_wait() then delivers the stale event.
• Fix: hash_ev == NULL is now treated as a known-benign condition (stale event) with a zlog_debug and early return instead of a fatal assert.
• Side note: m->handler.fd_poll_counter[fd] is still incremented (line 2162) for stale events before the early return. This is a pre-existing counter used only for VTY diagnostics, so the slight over-count is harmless but worth noting.
• Correctness: do_event_cancel() already zeroes m->read[fd] / m->write[fd] via thread_array[event->u.fd] = NULL before returning, so no dangling event pointers remain when the stale-event path returns early.

Confidence Score: 4/5

• This PR is safe to merge — it replaces a crashing assert with a well-understood graceful no-op return for a documented kernel race condition.
• The change is minimal (12 lines), correctly identified and well-explained, and the stale-event path has been verified to be a clean dead-end with no dangling state (do_event_cancel already nulls the read/write arrays). No new logic is introduced beyond the null guard and a debug log. Score is 4 rather than 5 because the race involves multi-threaded epoll subtleties and no regression test is included.
• No files require special attention beyond lib/event.c.

Important Files Changed

Filename	Overview
lib/event.c	Replaces a crashing assert(hash_ev) with a graceful early-return and a zlog_debug message to handle the race where do_event_cancel() removes an fd from epoll_event_hash before epoll_wait flushes its already-queued ready events for that fd.

Sequence Diagram

sequenceDiagram
    participant OT as Other Thread
    participant EL as Event Loop
    participant K as Kernel (epoll)

    Note over EL: event_fetch_inner_loop() begins
    OT->>EL: event_cancel_async(fd)
    EL->>EL: do_event_cancel(): remove fd from epoll_event_hash
    EL->>K: EPOLL_CTL_DEL(fd)
    Note over K: fd removed from interest list,<br/>but stale event already in ready list
    EL->>K: fd_poll() / epoll_wait()
    K-->>EL: returns stale event for fd
    EL->>EL: thread_process_io_inner_loop(revent)
    EL->>EL: hash_ev = epoll_event_hash_find(fd) → NULL
    Note over EL: BEFORE fix: assert(hash_ev) → CRASH
    Note over EL: AFTER fix: zlog_debug + graceful return ✓

Loading

_{Last reviewed commit: cbb00ba}

[mjstapp]

yes, thanks, this assert was not really safe

[donaldsharp]

@Mergifyio backport stable/10.6

[mergify]

backport stable/10.6

✅ Backports have been created

Details

• #21156 lib: fix crash in thread_process_io_inner_loop on stale epoll event (backport #21124) has been created for branch stable/10.6

[mjstapp]

thanks, looks good