[Triage] reuse_port cause connection error on worker reload

[methane]

Type

Bug Report

Description

In the current implementation, when reuse_port is enabled, each worker process creates its own new socket.

The backlog exists independently for each socket, and connections in the backlog have completed the TCP handshake, so the client may have already sent an (unretryable) HTTP request.

Therefore, when reloading a worker due to a HUP signal or max_requests, connections remaining in the backlog will return "Connection reset by peer" to the client. This is not the expected behavior for a graceful reload.

To avoid this problem, the following methods can be considered:

• Even when using SO_REUSEPORT, the master process creates the socket, and when reloading, it reuses the socket passed to the terminating worker. (In this case, connections remaining in the backlog will also be disconnected when the worker terminates due to the TTOU signal.)
• Document the limitations of reuse_port: In sync workers, reuse_port has no benefits at all. It reduces load balancing performance and makes graceful reload incomplete. For other worker classes, it achieves better load balancing at the cost of incomplete graceful reload.

ref: https://lwn.net/Articles/853637/

Steps to Reproduce (for bugs)

1. Start a WSGI application that just sleeps for 1 second with worker_class=2, threads=2, worker_connections = 2, and reuse_port=True.
2. Send requests with 5 or more concurrent connections to create a state where connections remain in the backlog.
3. Send a HUP signal to the master.

client and app code to reproduce: https://gist.github.com/methane/81c517feb5bd5aad59c2fefd31eff6e5

Configuration

bind = "127.0.0.1:5000"
workers = 2
threads = 2
worker_connections = 2
worker_class = "gthread"
reuse_port = True
backlog = 128
graceful_timeout = 10
pidfile = "gunicorn.pid"
accesslog = "-"
errorlog = "-"
loglevel = "info"

Logs / Error Output

$ .venv/bin/python load_client.py --master-pid "$(cat ./gunicorn.pid)"  --concurrency 8
[client] elapsed=1.0s total=5 ok=5 errors=0
[client] elapsed=2.0s total=13 ok=13 errors=0
[client] elapsed=3.0s total=21 ok=21 errors=0
[client] sent SIGHUP to master pid 531703
[client] elapsed=4.0s total=26 ok=26 errors=0
[client] elapsed=5.0s total=31 ok=31 errors=0
[Errno 104] Connection reset by peer
[client] error:ConnectionResetError: [Errno 104] Connection reset by peer
[Errno 104] Connection reset by peer
[Errno 104] Connection reset by peer
[Errno 104] Connection reset by peer
[client] error:ConnectionResetError: [Errno 104] Connection reset by peer
[client] error:ConnectionResetError: [Errno 104] Connection reset by peer
[client] error:ConnectionResetError: [Errno 104] Connection reset by peer
[client] elapsed=6.0s total=44 ok=40 errors=4
[client] elapsed=7.1s total=52 ok=48 errors=4
[client] elapsed=8.1s total=60 ok=56 errors=4
[client] elapsed=9.1s total=67 ok=63 errors=4
[client] elapsed=11.1s total=75 ok=71 errors=4

=== summary ===
requests: 82
results: {'ok': 78, 'error:ConnectionResetError': 4}
errors: {'error:ConnectionResetError': 4}
before signal: {'ok': 31}
after signal: {'ok': 47, 'error:ConnectionResetError': 4}
worker PIDs: {'531704': 22, '531705': 15, '531778': 19, '531781': 22}

### Gunicorn Version

gunicorn (version 26.0.0)

### Python Version

Python 3.14.2

### Worker Class

gthread

### Operating System

Ubuntu 24.04

### Additional Context

_No response_

### Checklist

- [X] I have searched existing discussions and issues for duplicates
- [X] I have checked the documentation and FAQ

[methane]

• Even when using SO_REUSEPORT, the master process creates the socket, and when reloading, it reuses the socket passed to the terminating worker. (In this case, connections remaining in the backlog will also be disconnected when the worker terminates due to the TTOU signal.)

A little out of topic, but if you implement this feature, it will also be easy to implement separating the ports that each worker process listens to. This is a load balancing technique used in uWSGI.

https://uwsgi-docs.readthedocs.io/en/latest/articles/SerializingAccept.html#bonus-chapter-using-the-zeeg-approach-in-a-uwsgi-friendly-way

This method, like SO_REUSEPORT, solves both the thundering herd problem of epoll and the load balancing problem at the same time. The difference from SO_REUSEPORT is that load balancing is performed by the reverse proxy rather than the OS kernel. Compared to load balancing using Linux's addr/port 4-tuple, nginx's round robin or least conn algorithm provides a better load balancing and significantly improves 99%-tile latency.

[methane]

one-to-one port/worker mapping PR.
methane#3

[methane]

This is draft PR via Codex.
https://github.com/methane/gunicorn/pull/1/changes