Scaling NGINX as an API Gateway: Survival Guide for High-Load Systems
There is nothing quite like the silence of a server room (or a Slack channel) right before the storm hits. I recently audited a payment API for a mid-sized Norwegian retailer preparing for the summer sales. Their Ruby code was optimized, their database queries were indexed, yet under load testing, their 95th percentile latency spiked to 2 seconds. The culprits weren't in the application layer. They were buried in the Linux kernel and default NGINX configurations.
Most VPS providers hand you a vanilla OS image that assumes you are running a low-traffic blog. If you are building an API Gateway to handle thousands of concurrent connections, those defaults are a death sentence. Here is how to tune your stack to handle the load without melting down.
1. The Kernel is Choking Your Connections
Before requests even hit NGINX, they have to traverse the TCP stack. By default, Linux is conservative. In a high-throughput API scenario, you will run out of file descriptors and ephemeral ports fast.
I see this error constantly in /var/log/messages: "nf_conntrack: table full, dropping packet". This is unacceptable. You need to open the floodgates in /etc/sysctl.conf.
# /etc/sysctl.conf optimizations for CentOS 7 / Ubuntu 14.04
# Increase system-wide file descriptors
fs.file-max = 2097152
# Widen the port range for outgoing connections
net.ipv4.ip_local_port_range = 1024 65535
# Allow reuse of sockets in TIME_WAIT state for new connections
net.ipv4.tcp_tw_reuse = 1
# Increase backlog for incoming connections
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 262144
Run sysctl -p to apply these. If you don't tweak somaxconn, NGINX can't accept connections faster than the kernel queues them, no matter how many worker processes you spawn.
2. NGINX Upstream Keepalives
Using NGINX as a reverse proxy/gateway often means terminating SSL at the edge and passing unencrypted HTTP to backend services (like Node.js or PHP-FPM). The mistake everyone makes? Opening a new TCP connection for every single request to the upstream.
TCP handshakes are expensive. In a microservices architecture, this overhead adds significant latency. You must configure keepalives to the upstream.
Here is the configuration pattern I use on production gateways:
upstream backend_api {
server 10.0.0.5:8080;
# Keep 64 idle connections open per worker
keepalive 64;
}
server {
location /api/ {
proxy_pass http://backend_api;
# Required for keepalive to work
proxy_http_version 1.1;
proxy_set_header Connection "";
# Pass real IP to backend
proxy_set_header X-Real-IP $remote_addr;
}
}
Pro Tip: Withoutproxy_set_header Connection "";, NGINX defaults toConnection: close, rendering your keepalive directive useless. I've seen senior engineers lose days debugging this.
3. The "Noisy Neighbor" Problem
Software tuning only gets you so far. If you are hosting your API on a budget VPS that uses OpenVZ or LXC, you are sharing the kernel with every other customer on that physical host. If another tenant decides to mine Bitcoin or compile a massive kernel, your CPU steal time goes up, and your API latency fluctuates unpredictably.
For consistent API performance, you need hardware isolation. This is why we built CoolVDS on top of KVM (Kernel-based Virtual Machine). With KVM, your RAM and CPU are allocated to you. No overselling. If you have a 4-core instance, those cycles are yours.
Furthermore, disk I/O is the silent killer of API throughput, especially if you are logging heavily or caching to disk. While many providers in 2015 are still rotating spinning rust (HDD) or standard SATA SSDs, CoolVDS has moved aggressively to high-performance SSD storage arrays. When your database needs to flush to disk, it happens instantly, keeping your threads free to accept new API calls.
4. Local Latency and Compliance
If your primary user base is in Norway, hosting in Frankfurt or London adds 20-40ms of round-trip time (RTT) purely due to physics. That doesn't sound like much, but for an API making multiple sequential calls, it compounds.
Hosting locally isn't just about speed; it's about trust. With the Norwegian Personal Data Act (Personopplysningsloven) and strict oversight from Datatilsynet, keeping data within national borders simplifies your legal posture significantly compared to navigating the murky waters of US Safe Harbor (especially with the recent scrutiny on data exports).
CoolVDS infrastructure is peered directly at NIX (Norwegian Internet Exchange). Your packets stay local, your latency stays low, and your data stays under Norwegian jurisdiction.
Summary
Building a high-performance API gateway in 2015 requires a holistic approach:
- Kernel: Open up file descriptors and enable TCP reuse.
- NGINX: Enforce upstream keepalives to reduce handshake overhead.
- Infrastructure: Avoid container-based VPS overselling; insist on KVM and fast storage.
You can spend weeks optimizing your Ruby or Python code, or you can fix the foundation today. Don't let IOwait kill your application.
Ready to test your configuration? Deploy a KVM-backed CoolVDS instance in Oslo and see the difference raw I/O power makes.