Console Login

Offloading Slow Work from Web Requests with RabbitMQ and Celery

The problem

A web application sends a confirmation email, resizes an uploaded image or starts a video conversion inside the request that triggered it. The visitor waits for the whole job. Under load the symptoms are typical:

  • pages hang until the browser or the proxy gives up, and the frontend returns 502 or 504 errors;
  • the pool of Apache, PHP-FPM, uWSGI or Gunicorn workers is fully occupied while the machine is mostly waiting on SMTP or disk;
  • a job that dies halfway (a worker restart, an out-of-memory kill) is simply gone, because nothing recorded that it was pending.

Adding web servers helps only until the next burst. The slow work still sits on the request path.

Why it happens

Most classic application servers handle one request per worker process at a time. The pool is fixed. Each request that spends several seconds on a slow task occupies one slot for that whole time. When the number of simultaneous slow requests reaches the pool size, new requests queue up in the kernel's listen backlog, and then time out.

Two further effects make it worse:

  • CPU-heavy work such as transcoding competes with the processes that serve pages, so even fast requests slow down.
  • There is no buffer between the part that receives work and the part that does it. Producer and consumer are the same process, so a burst cannot be absorbed and drained later.

The remedy is architectural. The frontend accepts the request, records a unit of work in a durable queue and returns immediately. Separate worker processes, possibly on other machines, consume the queue at their own pace.

Source of truth

The fix relies on three behaviours that are documented by the projects themselves:

  • RabbitMQ: consumer acknowledgements. A broker keeps a delivered message until the consumer acknowledges it. If the consumer's connection closes without an acknowledgement, the message is made available again. This is what prevents a crashed worker from losing a task, provided the worker acknowledges after the work is finished.
  • Nginx core module reference. It lists the defaults and semantics of client_header_buffer_size, large_client_header_buffers, client_body_buffer_size and the client timeouts. A request body larger than the body buffer is written to a temporary file. A request line or header that does not fit the large header buffers is rejected with an error.
  • Nginx connection processing methods. On Linux 2.6 and later, epoll is the efficient method, and Nginx selects the most efficient available method by itself. The use directive is therefore only needed to force a specific one.

Celery 3.0 is configured through its own settings (CELERY_ACKS_LATE, CELERYD_PREFETCH_MULTIPLIER) and the amqp:// broker URL. The settings and their defaults should be checked against the documentation of the installed version.

Fix options

MethodStrengthsWeaknesses
Threads or forked processes inside the web appNo new componentsJobs die with the web process; CPU load stays on the frontend; no retry or visibility
Cron job polling a database tableSimple, uses existing infrastructurePolling delay; row locking and cleanup must be written by hand; adds write load to the main database
Redis list as a queueVery fast and simple to runPersistence is weaker by default, and a task taken by a crashed worker is lost unless extra bookkeeping is added
RabbitMQ (AMQP) with Celery workersDurable queues, acknowledgements, routing, workers on any number of hostsOne more service to run, secure and monitor

Why this fix

RabbitMQ with Celery is the choice when tasks must not be lost and may be expensive. The broker implements AMQP, supports durable queues and persistent messages, and re-delivers unacknowledged work. Celery turns a plain Python function into such a task and handles serialization, worker processes and retries.

Workers hold no state. A worker node can be added or removed without touching the web tier, which gives elasticity on ordinary virtual machines. Nginx stays in front as a thin gateway: it serves static files and passes dynamic requests to the application server.

The price is operational: a broker, a worker supervisor and a queue-length check. Those are small compared with an unbounded request path.

Step by step

The commands below target Debian 7 (Wheezy) with Nginx 1.2.x, RabbitMQ 3.0.x and Celery 3.0. They are instructions to apply and adapt; no program output is reproduced here. The addresses (10.0.0.5 for the broker, 10.0.0.0/24 for the private network) are examples.

1. Install the broker

The distribution package can be older than the current stable release, so the project's own APT repository is used. The key fingerprint should be checked against the project's download page before trusting it.

echo 'deb http://www.rabbitmq.com/debian/ testing main' > /etc/apt/sources.list.d/rabbitmq.list
wget https://www.rabbitmq.com/rabbitmq-signing-key-public.asc
apt-key add rabbitmq-signing-key-public.asc
apt-get update
apt-get install rabbitmq-server

2. Bind to the private address and create a dedicated user

A default guest account with a well-known password must not be reachable from a network. The broker is bound to the private interface and a dedicated user is created.

echo 'NODE_IP_ADDRESS=10.0.0.5' > /etc/rabbitmq/rabbitmq-env.conf
service rabbitmq-server restart
rabbitmqctl add_user appuser 'a-long-random-password'
rabbitmqctl set_permissions -p / appuser '.*' '.*' '.*'
rabbitmqctl delete_user guest

3. Restrict access to the AMQP port

iptables -A INPUT -p tcp --dport 5672 -s 10.0.0.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 5672 -j DROP

4. Define the task

Three details differ from a naive version. The URL carries credentials. The task acknowledges its message only after it finishes (acks_late), so a crashed worker causes a redelivery instead of a lost job; this requires the task to be safe to run twice. check_call raises an exception when ffmpeg fails, so a failure is not reported as success.

pip install 'celery>=3.0,<3.1'

# tasks.py
import os
import subprocess
from celery import Celery

app = Celery('tasks', broker='amqp://appuser:[email protected]//')
app.conf.update(
    CELERY_ACKS_LATE=True,
    CELERYD_PREFETCH_MULTIPLIER=1,
)

@app.task
def transcode_video(filename):
    target = os.path.splitext(filename)[0] + '.avi'
    subprocess.check_call(['ffmpeg', '-y', '-i', filename, target])
    return target

A prefetch multiplier of 1 stops a worker from reserving a batch of long jobs while other workers sit idle.

5. Start workers and enqueue work

celery -A tasks worker --loglevel=info --concurrency=2

# from the web application
from tasks import transcode_video
transcode_video.delay('/srv/uploads/clip42.mov')

The delay call returns as soon as the message is published. Each additional worker host needs only tasks.py, its dependencies and the same broker URL. A process supervisor such as supervisord should restart the worker if it exits.

6. Watch the queue

rabbitmqctl list_queues name messages consumers

A monitoring system can run this command and alert when the message count keeps growing, which means the workers are too slow or have stopped.

7. Configure Nginx as the gateway

The header buffers and body buffer of the common copy-and-paste snippet need correcting. Header buffers of 1 KB for large headers reject requests with ordinary cookies. A 10 KB body buffer does not avoid disk writes: bodies larger than the buffer are spooled to a temporary file. The defaults for the large header buffers are used below, and use epoll is dropped because Nginx picks it on Linux anyway.

worker_processes 4;

events {
    worker_connections 4096;
}

http {
    client_header_buffer_size 1k;
    large_client_header_buffers 4 8k;
    client_body_buffer_size 16k;
    client_max_body_size 8m;

    client_body_timeout 12;
    client_header_timeout 12;
    keepalive_timeout 15;
    send_timeout 10;

    gzip on;
    gzip_comp_level 2;
    gzip_min_length 1000;
    gzip_types text/plain application/x-javascript text/xml text/css;

    server {
        listen 80;
        server_name example.com;

        location /static/ {
            root /srv/app;
        }

        location / {
            proxy_pass http://127.0.0.1:8000;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}
nginx -t
service nginx reload

The short client timeouts limit how long a slow client can hold a connection, which reduces the effect of slow-request attacks. They do not replace a firewall or rate limits.

Review

What works. The web tier answers quickly because it only publishes a message. Bursts accumulate in the queue instead of exhausting the worker pool. A crashed worker's job returns to the queue, as long as the task is idempotent and acknowledged late. Capacity grows by starting more workers against the same broker.

Limits.

  • The broker is a single point of failure. RabbitMQ can mirror queues across nodes, but that adds complexity; for a single node, backups of the broker's data directory and a tested restart procedure are the minimum.
  • RabbitMQ writes messages to disk when queues are persistent and moves them to disk when memory use is high. On a slow or shared disk, a large backlog slows the broker. The broker's data directory should not share a disk with the main database.
  • Late acknowledgement gives at-least-once delivery, so a task can run twice. Sending an email twice is a real consequence and needs a guard, such as a unique key stored when the mail is sent.
  • Results and progress are not visible to the user unless the application stores them, for example in a status row or a Celery result backend.
  • Container-style virtualization shares the host kernel and its I/O scheduling. Full virtualization such as KVM gives a dedicated kernel and block device, which makes disk latency for the broker more predictable. That is a property of the platform to verify, not something this setup guarantees.

When to choose another method. For low volumes of non-critical jobs, a cron job and a database table are easier to run. If losing a task is acceptable and speed matters most, a Redis list is lighter than a full AMQP broker. For very short tasks the broker round trip may cost more than the task. Handle those in the request or with a small in-process pool.

Whatever the method, the setup should be automated with a configuration management tool such as Puppet or Chef, and the queue length should be monitored. Root access remains useful for tuning kernel parameters and the broker.