Microservices Gateway and Circuit Breaker Patterns on Nginx and…

The problem

Splitting a monolith into services replaces in-process function calls with network calls. A typical early symptom is a system that works at low traffic and then degrades as a whole when one part slows down:

• A checkout service waits on an inventory service that is slow or down, and every checkout request hangs until a long timeout expires.
• A burst of requests from a single client, or a misbehaving mobile app, reaches internal services directly and saturates them.
• The gateway or reverse proxy opens and closes a new TCP connection for every request to a backend, so connection setup becomes a measurable share of request time and the backend accumulates sockets in TIME_WAIT.
• Dashboards show high latency everywhere, even though only one service is actually unhealthy.

This is a cascading failure: a local fault spreads because callers keep waiting on, and keep sending traffic to, something that cannot answer.

Why it happens

Several independent causes combine:

• No admission control at the edge. Without a limit, the number of concurrent requests is decided by the clients, not by what the backend can handle.
• No connection reuse to the upstream. Nginx talks to proxied servers using HTTP/1.0 by default and closes the connection after each request. Every proxied request then pays for a new TCP handshake.
• Long default timeouts. The Nginx proxy_read_timeout default is 60 seconds. A caller that waits that long holds a socket, a worker slot and often a database connection for the whole time.
• No fail-fast logic in the caller. A service that retries or waits on a dead dependency turns one failure into a queue of blocked requests.
• Misleading health settings. The upstream parameters max_fails and fail_timeout only have effect when the upstream group has more than one server. In a one-server group they are ignored and the server is never marked unavailable.

Shared resources worsen the effect: several containers flushing logs and database commits to the same disk compete for I/O, so a high iowait value in top or iostat can look like a service bug. Check it before blaming application code.

Source of truth

The fix relies on the documented behaviour of the following references:

• Nginx ngx_http_limit_req_module confirms that limit_req_zone defines a shared-memory zone and a rate, that limit_req applies it with an optional burst, that the default burst is zero, and that nodelay stops excess requests inside the burst from being delayed. Requests beyond the burst are rejected, and limit_req_status sets the status code returned.
• Nginx ngx_http_upstream_module documents the keepalive directive (a cache of idle connections to upstream servers, counted per worker process), the requirement to use HTTP/1.1 and clear the Connection header for HTTP proxying, and the rule that max_fails and fail_timeout are ignored for a single-server group.
• The opossum repository is the project's own reference for the Node.js circuit breaker: the action passed to CircuitBreaker must return a promise, fire() forwards its arguments to that action, and the options timeout, errorThresholdPercentage, resetTimeout and volumeThreshold control when the circuit opens and when it is probed again.

Ways to fix it

Gateway-level controls

Put one entry point in front of all internal services. It terminates TLS, applies rate limits, sets short upstream timeouts and reuses connections. Nginx does this with core modules, and Kong or Traefik are alternatives with more built-in features.

Circuit breaker in the calling service

Wrap each remote call in a breaker that counts failures and timeouts. When the failure ratio passes a threshold, the breaker opens and calls fail immediately or return a fallback. After a reset period it lets a trial call through. In Node.js, opossum implements this.

Sidecar or service mesh

A companion container handles cross-cutting work such as log shipping, TLS or metrics. A log-shipping sidecar in Docker Compose is a small step. A full mesh such as Istio gives retries, circuit breaking and mutual TLS outside application code, at the price of a much larger operational footprint.

Data isolation

Giving each service its own data store removes shared-table coupling. It does not fix the cascading-failure problem by itself, and it multiplies the number of database processes that share one disk.

Why this fix

The chosen combination is an Nginx gateway with rate limiting, short timeouts and upstream keepalive, plus an opossum breaker around the one call that can hang. The reasons:

• The two layers address different faults. The gateway limits what enters the system; the breaker limits what a service sends to a failing dependency.
• Both use components that are well documented and need no new infrastructure.
• Both are reversible: remove the limit_req line or the wrapper and the previous behaviour returns.
• A mesh is justified when many services in several languages need identical policies. For a handful of services it adds more moving parts than it removes.

Several details of commonly circulated versions of this setup need correcting, and the configuration below applies them: a TLS server block needs ssl_certificate and ssl_certificate_key; a single-server upstream makes max_fails pointless, so the example uses two servers; and passing inventoryClient.checkStock to the breaker without binding it loses this inside the method.

Applying the fix

No command output is reproduced here. The steps below are instructions, and the expected results are described in words. Replace the addresses, host name and certificate paths with real values.

1. Configure the gateway

Edit the Nginx configuration, for example in /etc/nginx/conf.d/api.conf (the http context is provided by nginx.conf when the file is included there, so the directives below are shown at that level):

limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
limit_req_status 429;

upstream order_service {
    server 10.10.0.5:8080 max_fails=3 fail_timeout=30s;
    server 10.10.0.6:8080 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name api.example.com;

    ssl_certificate     /etc/ssl/certs/api.example.com.pem;
    ssl_certificate_key /etc/ssl/private/api.example.com.key;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 10m;

    location /orders/ {
        limit_req zone=api_limit burst=20 nodelay;

        proxy_connect_timeout 2s;
        proxy_read_timeout    5s;

        proxy_pass http://order_service;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Notes on the settings: the rate is per client address, so many users behind one NAT share a single budget; raise the burst or choose another key if that matters. If another proxy or CDN sits in front, $binary_remote_addr is that proxy's address unless the real-IP module is configured. The keepalive value is the number of idle connections kept per worker process, not a total.

2. Validate and reload

sudo nginx -t
sudo systemctl reload nginx

The test must report a successful syntax check before the reload is run.

3. Check rate limiting

for i in $(seq 1 60); do curl -s -o /dev/null -w '%{http_code}\n' https://api.example.com/orders/; done | sort | uniq -c

With a burst of 20 and a rate of 10 requests per second, a rapid loop should produce a block of normal responses followed by 429 responses once the burst is used up. A slow loop should produce none.

4. Check connection reuse

While a load test runs, count sockets from the gateway to one backend:

ss -tan state established '( dport = :8080 )' | wc -l
ss -tan state time-wait '( dport = :8080 )' | wc -l

With keepalive active, the number of TIME-WAIT sockets towards the backend should stay low compared with a run where the keepalive line is removed.

5. Add the circuit breaker

npm install opossum

const CircuitBreaker = require('opossum');

const options = {
  timeout: 3000,                // fail a call that takes longer than 3 s
  errorThresholdPercentage: 50, // open when half of the calls in the window fail
  resetTimeout: 10000,          // probe again after 10 s
  volumeThreshold: 10           // ignore the ratio until 10 calls were seen
};

// The action must return a promise and keep its own 'this'.
const breaker = new CircuitBreaker(
  (sku) => inventoryClient.checkStock(sku),
  options
);

breaker.fallback(() => ({ stock: 'unknown', status: 'degraded' }));

breaker.on('open', () => console.warn('inventory circuit open'));
breaker.on('close', () => console.info('inventory circuit closed'));

breaker.fire('SKU-12345').then(console.log).catch(console.error);

The volumeThreshold line is an addition to the commonly shown example. Without it the default is zero, so a single early failure can open the circuit. To test, stop the inventory service and send repeated calls: after the threshold is passed, calls should return the fallback immediately instead of waiting for the timeout, and the open message should be logged. After the reset time and a restart of the dependency, the circuit should close again.

6. Optional: a log-shipping sidecar

version: '3.8'
services:
  payment-service:
    image: my-payment-app:v1.4
    volumes:
      - shared-logs:/var/log/app

  log-shipper:
    image: fluentd:v1.14
    volumes:
      - shared-logs:/var/log/app:ro
      - ./fluentd.conf:/fluentd/etc/fluent.conf:ro
    environment:
      - FLUENTD_CONF=fluent.conf

volumes:
  shared-logs:

The shipper only needs to read the logs, so the volume is mounted read-only. The depends_on entry from the usual example is dropped: it only orders container start and expresses no real dependency here.

Looking back

What the fix covers

• The gateway turns unbounded client traffic into a bounded, predictable load and reduces per-request connection overhead.
• Short upstream timeouts and a breaker stop callers from queueing behind a dependency that cannot answer.
• Passive health checks (max_fails) work only with several upstream servers.

Limits

• Per-IP limits are coarse. They do not distinguish users behind shared addresses and do not replace authentication-based quotas.
• The open-source Nginx version offers only passive upstream health checks; active checks need other software or a commercial edition.
• A breaker lives inside one process. Each instance of a service keeps its own state, so instances may open at different moments.
• A fallback value is only acceptable when stale or unknown data is safe for the business. For payments and similar operations, fail with a clear error instead.
• Disk contention from many containers is a separate issue: measure iowait and device latency, and place write-heavy databases on fast storage rather than tuning timeouts.

When to choose another method

Use a service mesh when many services in several languages need the same retry, breaker and mutual-TLS policy and there is staff to operate it. Use Kong or Traefik when plugins, dynamic service discovery or an admin API are needed at the gateway. Keep the sidecar approach for peripheral tasks such as log shipping, where it keeps application containers free of that code.