Console Login

Tuning Nginx and Kong as an API Gateway: Limits, Keepalive, Kernel

Symptoms

An API gateway built on Nginx, or on Kong (which runs on OpenResty, itself Nginx-based), can show one or more of these symptoms under sustained load, even when the backend services are fast:

  • Error-log lines containing Too many open files, or a start-up warning that worker_connections exceeds the open file limit.
  • Error-log lines containing Cannot assign requested address on connections to the backends, together with a very large number of sockets in TIME-WAIT between the gateway and the upstreams.
  • A latency floor on every request that is higher than the backend's own response time, because each call pays for a fresh TCP connection to the upstream.
  • Occasional 502 responses on otherwise idle periods, caused by a connection the backend closed while the gateway still considered it reusable.
  • Latency jitter that no configuration change explains, accompanied by a non-zero st (steal) value in top or vmstat.

Root cause

Several independent defaults add up:

  • File descriptor limit. Every client connection and every upstream connection is a file descriptor. A proxied request therefore holds at least two descriptors in a worker. A per-process limit of 1024 caps a worker far below a typical worker_connections value.
  • No upstream keepalive. Nginx talks HTTP/1.0 to upstreams by default and sends Connection: close. Each request opens and tears down a TCP connection, which costs a handshake, a slow-start phase and a socket left in TIME-WAIT.
  • Ephemeral port pressure. Connections from the gateway to one backend address and port are distinguished only by the local port. The default range is roughly 28000 ports, and sockets in TIME-WAIT hold ports for 60 seconds.
  • Small accept queues. Nginx's listen backlog is 511 on Linux unless set, and the kernel clamps it to net.core.somaxconn. Bursts of new connections overflow the queue and clients see retransmitted SYNs.
  • Synchronous work in the worker. Unbuffered access logging, full TLS handshakes without session reuse, and a stolen CPU all delay the single-threaded event loop of a worker, which delays every connection it serves.

Source of truth

The fixes below rely on the following references. Each statement is what the page documents.

  • ngx_http_upstream_module: the keepalive directive sets the maximum number of idle upstream connections kept in the cache of each worker process. For HTTP, the documented recipe is proxy_http_version 1.1 plus clearing the Connection header. The keepalive_requests and keepalive_timeout directives are available inside upstream in recent releases.
  • ngx_core_module: worker_rlimit_nofile raises the open file limit for worker processes. worker_connections counts all connections, including those to proxied servers, and cannot exceed the open file limit.
  • ngx_http_core_module: the backlog parameter of listen defaults to 511 on Linux.
  • ngx_http_log_module: access_log accepts buffer= and flush= parameters for buffered writes.
  • Kernel ip-sysctl: documents tcp_tw_reuse (it affects outgoing connections only) and ip_local_port_range. The net sysctl page documents somaxconn and netdev_max_backlog.
  • systemd.exec: LimitNOFILE= sets the descriptor limit of a service. Limits in /etc/security/limits.conf are applied by PAM at login and do not apply to services started by systemd.
  • Kong configuration reference: kong.conf properties prefixed with nginx_ are injected into the generated Nginx configuration, and dedicated upstream_keepalive_* properties control connection reuse to upstreams. The reference for the installed version lists the exact names and defaults.

Ways to fix it

1. Raise limits and kernel queues only

Increase the descriptor limit, somaxconn and the port range. This removes hard errors, but each request still opens a new upstream connection, so latency and TIME-WAIT churn remain.

2. Enable upstream keepalive

Reuse connections to the backends. This removes the handshake from the request path and the port pressure at its source. It needs HTTP/1.1 towards the upstream and a matching idle timeout.

3. Reduce per-request work in the worker

Buffer access logs, enable TLS session reuse and OCSP stapling, and verify that the CPU has AES instructions.

4. Change the capacity or host

If st is consistently above zero, or disk write latency is high, software tuning does not remove the cause. The options are a different instance type or host, or fewer co-tenants.

5. Tune tcp_tw_recycle

Not an option. The setting was removed in Linux 4.12 because it broke clients behind NAT. Use tcp_tw_reuse instead.

Why this fix

Options 1 to 3 are combined in a fixed order, and option 4 is only a reaction to measurements.

  • Keepalive first. It removes a per-request cost, not just a ceiling. It also makes tcp_tw_reuse and a wider port range a safety margin instead of a dependency.
  • Limits together with keepalive. Idle upstream connections and client connections all consume descriptors, so the limit must be raised in the same change, and through systemd, because that is what applies to a service.
  • Only well-founded kernel settings. tcp_tw_reuse helps because a gateway mostly opens outgoing connections. Large TCP buffers and TCP Fast Open are left out: short API payloads on low-latency links rarely benefit, and Fast Open also requires fastopen= on the listen line and client support.
  • Measure before the rest. Log buffering and TLS settings are cheap and safe, but their effect depends on traffic.

Applying the fix

Step 1: inspect the current state

ulimit -n
ss -s
sysctl net.core.somaxconn net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse
grep 'open files' /proc/$(pgrep -o -f 'nginx: master')/limits
vmstat 1 5

The last vmstat column, st, is the CPU time stolen by the hypervisor.

Step 2: raise the descriptor limit for the service

Create a drop-in with systemctl edit nginx (for Kong, edit the Kong unit, or set ulimit -n in its launch environment) and add:

[Service]
LimitNOFILE=65535
systemctl daemon-reload
systemctl restart nginx

Editing /etc/security/limits.conf is only useful for interactive shells and similar PAM sessions, not for the service.

Step 3: kernel settings

Put these in /etc/sysctl.d/90-gateway.conf:

net.core.somaxconn = 65535
net.core.netdev_max_backlog = 16384
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_tw_reuse = 1

The port range starts at 10240 so that it does not overlap ports used by local services. Apply and verify:

sysctl --system
sysctl net.core.somaxconn net.ipv4.tcp_tw_reuse

Step 4: Nginx main and events context

worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 16384;
}

A proxied request uses two connections in a worker, so 16384 here leaves headroom below the descriptor limit.

Step 5: upstream keepalive and listen backlog

upstream backend_api {
    server 10.0.0.5:8080;
    server 10.0.0.6:8080;

    keepalive 64;
    keepalive_requests 1000;
    keepalive_timeout 30s;
}

server {
    listen 443 ssl backlog=4096;

    location /api/ {
        proxy_pass http://backend_api;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

The keepalive number is per worker and limits idle connections only, not total connections. Keep keepalive_timeout below the idle timeout of the backend, otherwise the backend may close a connection at the moment Nginx reuses it, which produces sporadic 502 responses. The backlog parameter can be set once per address and port; it should not exceed somaxconn.

Step 6: the Kong equivalent

Kong already uses HTTP/1.1 towards upstreams. Control reuse with the dedicated properties in kong.conf (check the reference for the installed version):

upstream_keepalive_pool_size = 128
upstream_keepalive_max_requests = 1000
upstream_keepalive_idle_timeout = 30

The prefix selects the target block: nginx_http_ injects a directive into the http block, where keepalive is not valid, so a property such as nginx_http_upstream_keepalive is not the right way to set it. Worker limits can be set with nginx_main_worker_rlimit_nofile and nginx_events_worker_connections. Restart Kong afterwards:

kong restart

Step 7: logging and TLS

access_log /var/log/nginx/access.log combined buffer=32k flush=5s;

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/ssl/certs/ca-chain.pem;
resolver 127.0.0.53 valid=300s;
resolver_timeout 5s;

The log buffer is written when 32 KB accumulate or after five seconds, so the last seconds of a crash can be missing from the log. ssl_stapling_verify needs the issuer chain in ssl_trusted_certificate, and the resolver must be one the host can reach. Check for AES instructions:

grep -m1 -o aes /proc/cpuinfo

No output means the virtual CPU does not expose AES-NI, and TLS handshakes and bulk encryption cost more CPU.

Step 8: validate and watch

nginx -t
systemctl reload nginx
grep 'open files' /proc/$(pgrep -o -f 'nginx: master')/limits
ss -tan state time-wait | wc -l
ss -tn dst 10.0.0.5:8080 | wc -l

With keepalive working, the count of TIME-WAIT sockets towards the backends stays low under steady load and the number of established upstream connections stays roughly flat instead of growing with request rate. A load generator such as wrk can compare before and after:

wrk -t4 -c200 -d30s https://gateway.example.com/api/health

Limits and alternatives

  • What this addresses. Descriptor exhaustion, port exhaustion, accept-queue overflow and per-request upstream handshakes. These are the causes of the symptoms listed at the top, but the size of the gain depends on request rate, payload size and backend behaviour. No specific percentage can be promised.
  • Keepalive requires cooperation. Backends that close idle connections quickly, or that only speak HTTP/1.0, reduce or cancel the effect. For gRPC upstreams use grpc_pass, which has its own handling.
  • Per-worker pools. With many workers and many upstream servers, a small keepalive value is shared poorly. Raise it if connection churn persists, and watch the descriptor budget.
  • Steal time and storage. If vmstat shows steady st values, or log writes stall despite buffering, move the gateway to a host with dedicated CPU or faster storage. Configuration does not fix a contended CPU.
  • Buffer settings. Custom proxy_buffers sizes are not recommended without measuring the actual response sizes, so they are left at defaults here.
  • When to choose another method. If the gateway needs to scale beyond one host, add more gateway instances behind a load balancer instead of raising limits further. If TLS dominates CPU, terminate it on a dedicated tier or use session reuse more aggressively.