Load balancing via reverse proxy

[methane]

Current status

Granian does not provide a mechanism for load balancing between workers. Since a thread separate from the application thread performs accept in the worker, there is a possibility that a busy worker will accept instead of an idle worker until it reaches backpressure, resulting in an uneven load.

#544

This PR enables SO_REUSEPORT and allows each worker to have its own backlog. Linux distributes connections based on the hash value of the socket pair, which achieves a distribution close to random.
In addition, this also avoids the thundering herd problem where multiple workers wake up simultaneously for new connections.

On the other hand, because the backlog is distributed, there was a problem that the contents of the backlog held by the worker that terminates during a graceful restart are discarded.

#567
#579

In this PR, to fix graceful restart, workers share the listening socket again. SO_REUSEPORT remains enabled, but it no longer has any effect.

Idea

Each worker listens on its own independent port. This allows a reverse proxy like nginx to distribute connections using algorithms such as least conn or round robin.

When performing a graceful restart, the old worker and the new worker share the same listening socket, so requests remaining in the backlog are not discarded.

This idea is not my original. It is described in the uWSGI documentation, and we are currently using it in our production environment.

https://uwsgi-docs.readthedocs.io/en/latest/articles/SerializingAccept.html#bonus-chapter-using-the-zeeg-approach-in-a-uwsgi-friendly-way

Before considering whether this idea will be accepted and how to support it in the CLI, I first verified whether this idea is valid for granian.

Prooof of concept

application

This sample application is about 50% CPU bound and 50% I/O bound.
If the load is well balanced, using 2 threads should yield performance close to double.

# wsgiapp.py
import time
import threading

def fib(n):
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

def app(environ, start_response):
    for _ in range(5):
        time.sleep(0.01) # 10ms
        fib(25)  # about 10ms
    start_response("200 OK", [("Content-type", "text/plain; charset=utf-8")])
    return [b"Hello, world"]

nginx configuration

# nginx.conf
worker_processes 1;
pid nginx.pid;
error_log error.log;

events {
    worker_connections 768;
}

http {
    access_log access.log;

    # single port backend
    upstream backend {
        server 127.0.0.1:5000;
    }
    # multi port backend
    upstream backend2 {
        server 127.0.0.1:5001;
        server 127.0.0.1:5002;
        server 127.0.0.1:5003;
        server 127.0.0.1:5004;
        # default balancing algorithm is round-robin. You can use least_conn; for least connection algorithm.
        # https://nginx.org/en/docs/http/ngx_http_upstream_module.html
    }

    server {
        listen 8000 default_server;
        server_name _;

        location / {
            proxy_pass http://backend;
            # proxy_pass http://backend2;
        }
    }
}

8 workers, 1 thread, backpressure=1

server:

.venv/bin/granian --port 5000 --interface wsgi --http 1 --workers 8 --blocking-threads 1 --backpressure 1 --no-http1-keep-alive wsgiapp:app

load test:

$ wrk -t1 -c8 --latency -d20s http://127.0.0.1:8000
Running 20s test @ http://127.0.0.1:8000
  1 threads and 8 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    75.76ms    3.18ms 114.76ms   88.63%
    Req/Sec   105.79     18.00   151.00     77.00%
  Latency Distribution
     50%   74.61ms
     75%   76.82ms
     90%   79.33ms
     99%   86.01ms
  2110 requests in 20.02s, 379.14KB read
Requests/sec:    105.38
Transfer/sec:     18.94KB

This result is a reference value when the load balance is close to ideal.

4 workers, 4 threads, backpressure=4

$ .venv/bin/granian --port 5000 --interface wsgi --http 1 --workers 4 --blocking-threads 4 --backpressure 4 --no-http1-keep-alive wsgiapp:app

load test:

$ wrk -t1 -c8 --latency -d20s http://127.0.0.1:8000
Running 20s test @ http://127.0.0.1:8000
  1 threads and 8 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    89.34ms   13.48ms 160.28ms   70.79%
    Req/Sec    89.47     12.59   121.00     83.00%
  Latency Distribution
     50%   86.95ms
     75%   96.64ms
     90%  107.88ms
     99%  132.28ms
  1787 requests in 20.02s, 321.10KB read
Requests/sec:     89.26
Transfer/sec:     16.04KB

You can see that the values of Avg, Stdev, Max have deteriorated significantly.

4 workers, 4 threads, backpressure=4, dedicated port for each worker

poc patch for multiport support: https://gist.github.com/methane/205485901684a66f4e77a399a49c4cb1

# serve.py
from granian.server import MultiPortServer
from granian.constants import Interfaces

if __name__ == "__main__":
    server = MultiPortServer(
        "wsgiapp:app",
        address="127.0.0.1",
        ports=[*range(5001, 5005)],
        workers=4,
        blocking_threads=4,
        backpressure=4,
        interface=Interfaces.WSGI,
    )

    server.serve()

load test:

$ wrk -t1 -c8 --latency -d20s http://127.0.0.1:8000
Running 20s test @ http://127.0.0.1:8000
  1 threads and 8 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    79.56ms    5.82ms 119.78ms   74.49%
    Req/Sec   100.65     12.25   130.00     78.00%
  Latency Distribution
     50%   78.15ms
     75%   82.59ms
     90%   87.18ms
     99%   98.09ms
  2007 requests in 20.01s, 360.63KB read
Requests/sec:    100.29
Transfer/sec:     18.02KB

You can see that the values of Avg, Stdev, and Max have come quite close to those of 8process/1thread.

[gi0baro]

Each worker listens on its own independent port.

No. Absolutely not.

[methane]

Can you tell me the reason?

It can be seen from past issues and other threads that there are people interested in 99%tile and load balancing.
Do you plan to add a load balancing mechanism to granian itself without relying on a reverse proxy?

Or do you want to recommend starting multiple granian if you use multiple ports? That configuration increases the management burden. But if that is the design philosophy, it is OK.

[gi0baro]

See #897

[methane]

Understood. Multiport not only enables better load balancing but also allows for safe worker restarts without relying on net.ipv4.tcp_migrate_req. However, until you express interest in that, I think it would be beneficial to provide a CLI that allows users to start granian on multiple ports as a third-party tool. This way, other users who are interested in the same issue can try this method.

• PyPI: https://pypi.org/project/granian-multiport/
• Repository: https://github.com/methane/granian-multiport