Console Login

Apache and Lighttpd: Cutting Idle-Connection Memory on CentOS 5

The problem

A LAMP server on CentOS 5 with Apache 2.2 and PHP runs out of memory long before the CPU is busy. The symptoms are a growing number of httpd processes, heavy swapping, and new visitors waiting or timing out even though the load comes mostly from small requests: favicon.ico, spacer GIFs, stylesheets and scripts requested by browsers on slow links.

This is often called the C10k problem, but strictly that term describes handling ten thousand simultaneous connections. A typical forum or portal on a small VPS hits the same architectural limit at a few hundred connections, because every connection ties up a whole Apache process.

Typical signs:

  • The count of httpd processes sits at or near MaxClients.
  • Free memory falls and swap usage rises under moderate traffic.
  • The error log reports that MaxClients was reached.
  • Most worker processes are idle or sending data to slow clients, not running PHP.

Why it happens

PHP is usually run inside Apache through mod_php, and many PHP extensions are not guaranteed to be thread-safe. For that reason the non-threaded prefork MPM is the usual choice. In prefork, one child process handles one connection at a time, and that process carries the full PHP interpreter and its modules in memory, whether the request is a PHP page or a tiny GIF.

Three things make this expensive:

  1. Per-process memory. Every child is a full copy of the server with mod_php loaded. Memory use scales with the number of simultaneous connections, not with the amount of work done.
  2. Slow clients. A child stays occupied while the response is written to a client on a slow DSL or mobile link. The process spends its time waiting on the network.
  3. Keep-alive. With KeepAlive On, a child stays attached to a finished connection for KeepAliveTimeout seconds, waiting for a next request that may never come. Pages with dozens of embedded files multiply this effect.

The ceiling is MaxClients. Set it too low and visitors queue once that many children are busy. Set it too high and the combined size of the children exceeds physical RAM, so the box swaps and slows down for everyone. MaxRequestsPerChild 0 additionally means children are never recycled, so a leaking PHP script keeps its memory until the process dies.

An event-driven server such as lighttpd 1.4 works differently. It waits on many sockets in one process using the Linux 2.6 epoll interface and only does work for sockets that are ready. An idle or slow connection costs a small data structure and a file descriptor, not a process.

Source of truth

The fix relies on the following references.

  • Apache 2.2 documentation, prefork MPM. It confirms that prefork is a non-threaded, pre-forking design, and that MaxClients must be large enough for the expected simultaneous requests but small enough that all processes fit in physical RAM. It also says that sites with limited memory should lower MaxClients to avoid thrashing.
  • Apache 2.2 documentation, common MPM directives. It describes MaxClients and the related ServerLimit hard ceiling for the prefork MPM.
  • lighttpd wiki, mod_proxy. It confirms that mod_proxy is a reverse proxy, that the backend is set with proxy.server using a host and port, and that backend connections are plain HTTP without TLS. This page describes current releases. Several details in it, such as DNS names for the backend host, Unix socket backends and HTTP/1.1 to the backend, are newer than 2010 and must not be relied on in a lighttpd 1.4 installation of that era. Use an IP address and TCP port.
  • lighttpd 1.4 repository, original mod_proxy document. This old text shows the original form of proxy.server, where the key is a file extension or a path prefix, and an empty key matches all requests.

Ways to fix it

1. Tune Apache prefork

Lower KeepAliveTimeout to a few seconds, size MaxClients from measured process size, and recycle children with a non-zero MaxRequestsPerChild. This costs nothing and keeps every feature, but the process-per-connection model remains.

2. Put a lightweight front end in front of Apache

Run lighttpd on port 80. It serves static files itself and forwards everything else to Apache on a local port. Slow clients are absorbed by the event loop, and Apache children are only used for dynamic requests. .htaccess and Apache modules keep working for the dynamic part.

3. Replace Apache with lighttpd and run PHP over FastCGI

This removes Apache entirely. It gives the smallest footprint, but .htaccess files are not read, rewrite rules must be translated to mod_rewrite in lighttpd syntax, and Apache-only modules are lost.

4. Keep Apache but change how PHP runs

Run PHP as FastCGI so that the threaded worker MPM becomes usable and PHP processes are decoupled from connections. This reduces memory but is a bigger change to the PHP setup and to the way existing sites are deployed.

MethodKeeps .htaccessEffortIdle connections cost
Tune preforkYesLowStill a process each
lighttpd front, Apache backYesMediumSmall for static and slow clients
lighttpd only, PHP via FastCGINoHighSmall
Apache worker MPM with FastCGI PHPYesHighThread, not process

Why this fix

Method 2, combined with the tuning from method 1, is the best balance for a site whose developers depend on .htaccess rules or Apache modules. It changes no application code and no PHP settings. It removes the main source of wasted memory, since static files and slow downloads no longer occupy Apache children. It is also easy to undo: moving Apache back to port 80 and stopping lighttpd restores the original setup.

Tuning is applied first because it is needed in every case. Even with a front end, the dynamic requests still reach prefork children, so MaxClients has to match the RAM available.

A side effect must be planned for. Apache sees every request as coming from 127.0.0.1. lighttpd's mod_proxy passes the client address in an X-Forwarded-For header, and the Apache side must be told to use it for logging and access rules, for example with the third-party mod_rpaf module.

Step by step

The commands below target CentOS 5 with Apache 2.2 from the distribution and lighttpd from a third-party repository such as EPEL. Paths follow the CentOS layout. Expected results are described in prose; check them on the actual system before and after each change.

Step 1: Measure

Find the average resident size of an Apache child and the state of the port 80 connections. The average size, multiplied by the process count, shows how much RAM Apache holds. The state counts show how many connections are idle (ESTABLISHED with no work, or TIME_WAIT).

ps -C httpd -o rss= | awk '{s+=$1; n++} END {print n, s/n/1024 " MB average"}'
free -m
netstat -ant | grep ':80 ' | awk '{print $6}' | sort | uniq -c

Part of each child's resident size is shared with the parent, so the average slightly overstates the real cost per child. Use it as a conservative figure.

Step 2: Tune Apache and move it to a local port

Edit /etc/httpd/conf/httpd.conf. Replace Listen 80 and change any NameVirtualHost and <VirtualHost> entries from port 80 to 8080. The values below are placeholders. Set MaxClients to roughly the RAM reserved for Apache divided by the average child size from step 1.

Listen 127.0.0.1:8080
NameVirtualHost *:8080

KeepAlive Off

<IfModule prefork.c>
    StartServers          4
    MinSpareServers       4
    MaxSpareServers       8
    ServerLimit          40
    MaxClients           40
    MaxRequestsPerChild 1000
</IfModule>

Keep-alive is switched off here because the front end is on the same host, so the connection setup to the backend is cheap and no child has to wait for a follow-up request. Each virtual host must now be declared as <VirtualHost *:8080>.

Step 3: Install and configure lighttpd

yum install lighttpd
chkconfig lighttpd on

Edit /etc/lighttpd/lighttpd.conf. The document root must be the same directory tree Apache serves, so static files are found on disk. A file that is missing there yields a 404 from lighttpd.

server.modules = ( "mod_access", "mod_proxy", "mod_accesslog" )
server.document-root = "/var/www/html"
server.port = 80
server.event-handler = "linux-sysepoll"
server.max-fds = 2048
server.max-keep-alive-idle = 5
accesslog.filename = "/var/log/lighttpd/access.log"

mimetype.assign = (
  ".html" => "text/html",
  ".txt"  => "text/plain",
  ".css"  => "text/css",
  ".js"   => "application/x-javascript",
  ".gif"  => "image/gif",
  ".jpg"  => "image/jpeg",
  ".png"  => "image/png",
  ".ico"  => "image/x-icon"
)

$HTTP["url"] !~ "\.(css|js|gif|jpe?g|png|ico)$" {
  proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => 8080 ) ) )
}

The condition forwards every request whose path does not end in a static extension, so PHP pages, rewritten URLs and directory requests all reach Apache. The empty key in proxy.server matches all requests inside the condition.

Step 4: Check syntax, then switch over

apachectl configtest
lighttpd -t -f /etc/lighttpd/lighttpd.conf
service httpd restart
service lighttpd start

Both syntax checks must report no errors before the restarts. Stop at the first error and fix it, because Apache and lighttpd cannot both hold port 80.

Step 5: Verify

curl -I http://127.0.0.1:8080/index.php
curl -I http://localhost/index.php
curl -I http://localhost/favicon.ico
netstat -tlnp | grep -E ':(80|8080) '
ps -C httpd -o rss= | wc -l

The first request goes straight to Apache and the second goes through lighttpd; both should return the same status and headers for the dynamic page. The icon should be answered by lighttpd, and the server header of that response should name lighttpd. Port 80 must be owned by lighttpd and port 8080 by httpd. Under normal browsing, the number of httpd processes should now stay close to the dynamic request rate instead of the connection count.

Step 6: Restore the client address in Apache

Without this step, Apache logs show only 127.0.0.1. After installing mod_rpaf from source or a repository, a typical configuration is:

RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1

Reload Apache with service httpd reload and confirm that the access log shows real client addresses.

Review

The front-end setup attacks the actual cause: connections that need no PHP no longer hold a prefork child. Apache memory then depends on the dynamic request rate and on the MaxClients cap, and the cap can be sized from measurement rather than guesswork.

Limits to keep in mind:

  • Rules in .htaccess that apply to static files, such as hotlink protection, password protection or per-directory headers, are bypassed because lighttpd answers those requests. Move them into the lighttpd configuration or exclude those paths from the static condition.
  • Dynamic requests are still bound by prefork. A slow client downloading a large generated page still holds a child until the response is delivered to lighttpd, but not until the client has received it, since the proxy buffers the response in the common case. Very large generated downloads may need separate handling.
  • Two servers mean two configurations, two sets of logs and an extra hop for dynamic pages. Client address handling needs the extra step above.
  • lighttpd 1.4 has a default file-descriptor limit, controlled by server.max-fds. High connection counts need this raised together with the process limit for open files.
  • The gain depends on the page mix. A site made of mostly dynamic pages gains little from offloading static files.

Choose another method when the situation differs. With a small, static-heavy site and no dependence on Apache features, a lighttpd-only setup with PHP over FastCGI is leaner. With plenty of RAM and low concurrency, tuning prefork alone is enough and avoids a second daemon. When the PHP code and its extensions are known to be thread-safe or can run under FastCGI, the worker MPM is an option that lowers memory use without a second web server.