The problem
A LAMP server on CentOS 5 with Apache 2.2 and PHP runs out of memory long before the CPU is busy. The symptoms are a growing number of httpd processes, heavy swapping, and new visitors waiting or timing out even though the load comes mostly from small requests: favicon.ico, spacer GIFs, stylesheets and scripts requested by browsers on slow links.
This is often called the C10k problem, but strictly that term describes handling ten thousand simultaneous connections. A typical forum or portal on a small VPS hits the same architectural limit at a few hundred connections, because every connection ties up a whole Apache process.
Typical signs:
- The count of
httpdprocesses sits at or nearMaxClients. - Free memory falls and swap usage rises under moderate traffic.
- The error log reports that
MaxClientswas reached. - Most worker processes are idle or sending data to slow clients, not running PHP.
Why it happens
PHP is usually run inside Apache through mod_php, and many PHP extensions are not guaranteed to be thread-safe. For that reason the non-threaded prefork MPM is the usual choice. In prefork, one child process handles one connection at a time, and that process carries the full PHP interpreter and its modules in memory, whether the request is a PHP page or a tiny GIF.
Three things make this expensive:
- Per-process memory. Every child is a full copy of the server with
mod_phploaded. Memory use scales with the number of simultaneous connections, not with the amount of work done. - Slow clients. A child stays occupied while the response is written to a client on a slow DSL or mobile link. The process spends its time waiting on the network.
- Keep-alive. With
KeepAlive On, a child stays attached to a finished connection forKeepAliveTimeoutseconds, waiting for a next request that may never come. Pages with dozens of embedded files multiply this effect.
The ceiling is MaxClients. Set it too low and visitors queue once that many children are busy. Set it too high and the combined size of the children exceeds physical RAM, so the box swaps and slows down for everyone. MaxRequestsPerChild 0 additionally means children are never recycled, so a leaking PHP script keeps its memory until the process dies.
An event-driven server such as lighttpd 1.4 works differently. It waits on many sockets in one process using the Linux 2.6 epoll interface and only does work for sockets that are ready. An idle or slow connection costs a small data structure and a file descriptor, not a process.
Source of truth
The fix relies on the following references.
- Apache 2.2 documentation, prefork MPM. It confirms that prefork is a non-threaded, pre-forking design, and that
MaxClientsmust be large enough for the expected simultaneous requests but small enough that all processes fit in physical RAM. It also says that sites with limited memory should lowerMaxClientsto avoid thrashing. - Apache 2.2 documentation, common MPM directives. It describes
MaxClientsand the relatedServerLimithard ceiling for the prefork MPM. - lighttpd wiki, mod_proxy. It confirms that mod_proxy is a reverse proxy, that the backend is set with
proxy.serverusing ahostandport, and that backend connections are plain HTTP without TLS. This page describes current releases. Several details in it, such as DNS names for the backend host, Unix socket backends and HTTP/1.1 to the backend, are newer than 2010 and must not be relied on in a lighttpd 1.4 installation of that era. Use an IP address and TCP port. - lighttpd 1.4 repository, original mod_proxy document. This old text shows the original form of
proxy.server, where the key is a file extension or a path prefix, and an empty key matches all requests.
Ways to fix it
1. Tune Apache prefork
Lower KeepAliveTimeout to a few seconds, size MaxClients from measured process size, and recycle children with a non-zero MaxRequestsPerChild. This costs nothing and keeps every feature, but the process-per-connection model remains.
2. Put a lightweight front end in front of Apache
Run lighttpd on port 80. It serves static files itself and forwards everything else to Apache on a local port. Slow clients are absorbed by the event loop, and Apache children are only used for dynamic requests. .htaccess and Apache modules keep working for the dynamic part.
3. Replace Apache with lighttpd and run PHP over FastCGI
This removes Apache entirely. It gives the smallest footprint, but .htaccess files are not read, rewrite rules must be translated to mod_rewrite in lighttpd syntax, and Apache-only modules are lost.
4. Keep Apache but change how PHP runs
Run PHP as FastCGI so that the threaded worker MPM becomes usable and PHP processes are decoupled from connections. This reduces memory but is a bigger change to the PHP setup and to the way existing sites are deployed.
| Method | Keeps .htaccess | Effort | Idle connections cost |
|---|---|---|---|
| Tune prefork | Yes | Low | Still a process each |
| lighttpd front, Apache back | Yes | Medium | Small for static and slow clients |
| lighttpd only, PHP via FastCGI | No | High | Small |
| Apache worker MPM with FastCGI PHP | Yes | High | Thread, not process |
Why this fix
Method 2, combined with the tuning from method 1, is the best balance for a site whose developers depend on .htaccess rules or Apache modules. It changes no application code and no PHP settings. It removes the main source of wasted memory, since static files and slow downloads no longer occupy Apache children. It is also easy to undo: moving Apache back to port 80 and stopping lighttpd restores the original setup.
Tuning is applied first because it is needed in every case. Even with a front end, the dynamic requests still reach prefork children, so MaxClients has to match the RAM available.
A side effect must be planned for. Apache sees every request as coming from 127.0.0.1. lighttpd's mod_proxy passes the client address in an X-Forwarded-For header, and the Apache side must be told to use it for logging and access rules, for example with the third-party mod_rpaf module.
Step by step
The commands below target CentOS 5 with Apache 2.2 from the distribution and lighttpd from a third-party repository such as EPEL. Paths follow the CentOS layout. Expected results are described in prose; check them on the actual system before and after each change.
Step 1: Measure
Find the average resident size of an Apache child and the state of the port 80 connections. The average size, multiplied by the process count, shows how much RAM Apache holds. The state counts show how many connections are idle (ESTABLISHED with no work, or TIME_WAIT).
ps -C httpd -o rss= | awk '{s+=$1; n++} END {print n, s/n/1024 " MB average"}'
free -m
netstat -ant | grep ':80 ' | awk '{print $6}' | sort | uniq -c
Part of each child's resident size is shared with the parent, so the average slightly overstates the real cost per child. Use it as a conservative figure.
Step 2: Tune Apache and move it to a local port
Edit /etc/httpd/conf/httpd.conf. Replace Listen 80 and change any NameVirtualHost and <VirtualHost> entries from port 80 to 8080. The values below are placeholders. Set MaxClients to roughly the RAM reserved for Apache divided by the average child size from step 1.
Listen 127.0.0.1:8080
NameVirtualHost *:8080
KeepAlive Off
<IfModule prefork.c>
StartServers 4
MinSpareServers 4
MaxSpareServers 8
ServerLimit 40
MaxClients 40
MaxRequestsPerChild 1000
</IfModule>
Keep-alive is switched off here because the front end is on the same host, so the connection setup to the backend is cheap and no child has to wait for a follow-up request. Each virtual host must now be declared as <VirtualHost *:8080>.
Step 3: Install and configure lighttpd
yum install lighttpd
chkconfig lighttpd on
Edit /etc/lighttpd/lighttpd.conf. The document root must be the same directory tree Apache serves, so static files are found on disk. A file that is missing there yields a 404 from lighttpd.
server.modules = ( "mod_access", "mod_proxy", "mod_accesslog" )
server.document-root = "/var/www/html"
server.port = 80
server.event-handler = "linux-sysepoll"
server.max-fds = 2048
server.max-keep-alive-idle = 5
accesslog.filename = "/var/log/lighttpd/access.log"
mimetype.assign = (
".html" => "text/html",
".txt" => "text/plain",
".css" => "text/css",
".js" => "application/x-javascript",
".gif" => "image/gif",
".jpg" => "image/jpeg",
".png" => "image/png",
".ico" => "image/x-icon"
)
$HTTP["url"] !~ "\.(css|js|gif|jpe?g|png|ico)$" {
proxy.server = ( "" => ( ( "host" => "127.0.0.1", "port" => 8080 ) ) )
}
The condition forwards every request whose path does not end in a static extension, so PHP pages, rewritten URLs and directory requests all reach Apache. The empty key in proxy.server matches all requests inside the condition.
Step 4: Check syntax, then switch over
apachectl configtest
lighttpd -t -f /etc/lighttpd/lighttpd.conf
service httpd restart
service lighttpd start
Both syntax checks must report no errors before the restarts. Stop at the first error and fix it, because Apache and lighttpd cannot both hold port 80.
Step 5: Verify
curl -I http://127.0.0.1:8080/index.php
curl -I http://localhost/index.php
curl -I http://localhost/favicon.ico
netstat -tlnp | grep -E ':(80|8080) '
ps -C httpd -o rss= | wc -l
The first request goes straight to Apache and the second goes through lighttpd; both should return the same status and headers for the dynamic page. The icon should be answered by lighttpd, and the server header of that response should name lighttpd. Port 80 must be owned by lighttpd and port 8080 by httpd. Under normal browsing, the number of httpd processes should now stay close to the dynamic request rate instead of the connection count.
Step 6: Restore the client address in Apache
Without this step, Apache logs show only 127.0.0.1. After installing mod_rpaf from source or a repository, a typical configuration is:
RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1
Reload Apache with service httpd reload and confirm that the access log shows real client addresses.
Review
The front-end setup attacks the actual cause: connections that need no PHP no longer hold a prefork child. Apache memory then depends on the dynamic request rate and on the MaxClients cap, and the cap can be sized from measurement rather than guesswork.
Limits to keep in mind:
- Rules in
.htaccessthat apply to static files, such as hotlink protection, password protection or per-directory headers, are bypassed because lighttpd answers those requests. Move them into the lighttpd configuration or exclude those paths from the static condition. - Dynamic requests are still bound by prefork. A slow client downloading a large generated page still holds a child until the response is delivered to lighttpd, but not until the client has received it, since the proxy buffers the response in the common case. Very large generated downloads may need separate handling.
- Two servers mean two configurations, two sets of logs and an extra hop for dynamic pages. Client address handling needs the extra step above.
- lighttpd 1.4 has a default file-descriptor limit, controlled by
server.max-fds. High connection counts need this raised together with the process limit for open files. - The gain depends on the page mix. A site made of mostly dynamic pages gains little from offloading static files.
Choose another method when the situation differs. With a small, static-heavy site and no dependence on Apache features, a lighttpd-only setup with PHP over FastCGI is leaner. With plenty of RAM and low concurrency, tuning prefork alone is enough and avoids a second daemon. When the PHP code and its extensions are known to be thread-safe or can run under FastCGI, the worker MPM is an option that lowers memory use without a second web server.