The problem
A common move for a small web business in 2009 is to leave shared hosting for a virtual dedicated server (VDS, used here as a synonym for VPS) running CentOS 5. Shared hosting suffers from noisy neighbours, and a dedicated server costs more than a young project can justify. A VPS sits between the two.
Many first-time VPS users then hit a failure that does not occur on a physical machine. The service works for days. Then a traffic peak, a cron job or a backup run arrives, and one of these happens:
- Apache logs errors containing
Cannot allocate memory, or cannot create child processes. - MySQL refuses to start or dies during a large query.
- A shell reports
fork: Cannot allocate memoryorResource temporarily unavailable. - Meanwhile
free -minside the VPS still shows memory that looks available.
The sales description of a VDS ("your own server with guaranteed RAM") does not explain this. The failure is a resource limit, and it can be diagnosed.
Why it happens
"VDS" is a marketing term, not a technology. Underneath it can be one of several different designs:
- Container-based virtualization (OpenVZ, and its commercial sibling Virtuozzo). All containers share one host kernel. Each container has its own file system, processes and IP address, but no kernel of its own. Resources are controlled by per-container limits called User Beancounters (UBC).
- Hypervisor-based virtualization (Xen, VMware, and KVM, which has been in the mainline Linux kernel since 2.6.20). Each guest runs its own kernel and is given a fixed amount of memory.
Container-based VPS plans are common because they allow high density at a low price. In a container, memory is not a single number. It is a set of counters, each with a barrier and a limit. A container can have free memory by the free output and still fail an allocation, because a different counter is exhausted. Typical counters:
privvmpages: allocated address space, counted in 4 KB pages. This is the closest thing to a memory cap. Many processes reserve more address space than they touch, so it can run out before real RAM does.vmguarpagesandoomguarpages: the amount of memory the container is guaranteed. Allocations up to the guarantee should succeed, which is what "guaranteed RAM" means in this model.numproc,numfile,numtcpsock: limits on processes, open files and TCP sockets.kmemsize: kernel memory used on behalf of the container, such as socket buffers and process structures.
When a request would exceed a limit, the kernel refuses it and increments that counter's fail count. The application sees an ordinary out-of-memory or fork error, not a message naming the limit.
The marketing claim that a VDS "acts exactly like a dedicated server" is therefore only partly true. On a container platform you cannot compile or load your own kernel or kernel modules. Features that depend on the kernel, such as some firewall modules, depend on what the host offers.
Source of truth
The OpenVZ project documents the mechanism on its wiki: UBC failcnt reset. That page confirms the following:
- A container has a set of resource limits called User Beancounters (UBC).
- When a resource hits its limit, the matching fail counter increases. The counter is the last column of
/proc/user_beancounters. - Fail counters are kept for the lifetime (uptime) of the container. A non-zero value can therefore be old, so compare values over time rather than reading one snapshot.
- Resetting the counters is possible but not recommended, because other applications read the file.
The page does not describe every parameter. For the exact meaning of each one on the installed kernel, consult the OpenVZ user documentation for that kernel branch. Provider-specific values (barriers and limits) are set by the host and can only be changed by the provider.
Fix options
1. Measure, then shrink the application's memory footprint
Find which counter fails, then reduce what the services request. On a typical LAMP stack, the main levers are Apache prefork process count, PHP memory_limit, and MySQL buffer sizes.
2. Ask the provider to raise the limit or move to a larger plan
If the counters show that a properly tuned workload still needs more, the limit must be raised on the host. A container user cannot change it. Plan prices usually reflect these limits.
3. Move to a hypervisor-based guest
A Xen or KVM guest has a fixed memory size and its own kernel, with normal swap and normal free semantics. This also allows custom kernels and modules. The cost is usually a lower density per host and therefore a higher price.
4. Rent a dedicated server
The whole machine is the resource limit. This is the most predictable option and the most expensive. It also makes hardware faults the renter's problem.
5. Replace Apache prefork with a lighter front end
The Nginx 0.7 series can serve static files and proxy to a small number of back-end processes with far less memory per connection. It is a larger change than tuning, but it works under any of the virtualization types.
Why this fix
Options 1 and 2 are combined. Measuring first is free, needs no migration, and produces evidence. If the evidence shows a tuned service fails the same counter, the request to the provider is specific ("raise privvmpages"), not a vague complaint. Moving platform (option 3) or hardware (option 4) is justified only when tuning is exhausted or when the kernel itself must be controlled. Option 5 is a larger redesign and is left for cases where Apache's per-connection cost is the main problem.
Step by step
The commands below are instructions for a CentOS 5 container with root access. None of them is shown with output here, because the output depends entirely on the host's limits.
Step 1: confirm the platform
uname -r
ls /proc/user_beancounters
ls /proc/vz
If /proc/user_beancounters exists, the VPS is an OpenVZ or Virtuozzo container and the counters apply. If it does not exist, the platform is probably a hypervisor guest, and ordinary memory tools such as free and vmstat are reliable.
Step 2: take a baseline and list non-zero fail counters
cp /proc/user_beancounters /root/ubc.before
awk 'NR>2 && $NF>0 {print $(NF-5), $NF}' /proc/user_beancounters
The first two lines of the file are a version line and a header, so the filter starts at line 3. For each counter, the resource name is printed together with its fail count. Run the workload that normally causes the failure, then compare:
diff /root/ubc.before /proc/user_beancounters
A counter whose fail count rises between the two snapshots is the one that is actually limiting the container.
Step 3: find the biggest memory users
ps aux --sort=-rss | head -n 10
ps -C httpd -o rss= | awk '{s+=$1; n++} END {print s/n/1024, n}'
The second command prints the average resident size of an Apache process in MB and the number of processes. RSS counts shared pages in every process, so the result overstates real use. That is a safe direction for a limit calculation.
Step 4: cap Apache by what the budget allows
Divide the memory budget for Apache by the average process size and use that number as MaxClients. For example, a 200 MB budget and 20 MB per process gives 10. Apply it in /etc/httpd/conf/httpd.conf. The values below are an example, not a recommendation:
<IfModule prefork.c>
StartServers 3
MinSpareServers 3
MaxSpareServers 6
ServerLimit 20
MaxClients 20
MaxRequestsPerChild 1000
</IfModule>
service httpd configtest
service httpd restart
Step 5: cap MySQL buffers and connections
In /etc/my.cnf, under [mysqld]. These are example values for MySQL 5.0 on a small container:
[mysqld]
key_buffer = 32M
max_connections = 50
table_cache = 128
query_cache_size = 16M
innodb_buffer_pool_size = 64M
service mysqld restart
PHP's memory_limit in /etc/php.ini multiplies with the number of Apache processes, so lower it as well if scripts do not need more.
Step 6: repeat the comparison
cp /proc/user_beancounters /root/ubc.after
diff /root/ubc.before /root/ubc.after
If no fail count rises under the same workload, the tuning is sufficient. If the same counter still rises, send the provider the counter name and the two snapshots and ask for a higher limit, or choose a different plan type.
Limits and alternatives
This method finds the limiting counter and gives evidence for a decision. It does not create resources. If a service needs more than the container's limits allow, tuning only postpones the problem.
- Counters are cumulative. Fail counts persist for the uptime of the container, so always compare snapshots taken before and after a test.
- Do not reset counters to make a graph look clean. Monitoring tools may depend on them.
- Choose a hypervisor guest when a custom kernel, kernel modules, or ordinary swap and memory behaviour are required.
- Choose dedicated hardware when the load is steady and high enough that a whole machine is used well, or when no neighbour effects can be tolerated.
A few other claims in typical VDS marketing from this period also need checking before they influence a decision:
- "Upgrade with a reboot." Changing limits on a container is a host-side operation. Whether it needs a restart depends on the provider's tooling.
- "Burst" memory. On OpenVZ this usually refers to usage between a barrier and a limit. It is not guaranteed capacity.
- Dedicated IP and SSL. Each HTTPS site generally needs its own IP address on current software, because name-based virtual hosting for SSL is not widely supported by clients.
- Location. A server close to the users reduces latency. Treat any claim of a large search-ranking effect from server location alone with caution.
- Isolation. Containers separate files and processes. The kernel is still shared, so a kernel-level problem on the host affects every container on it.
Whatever the plan is called, ask the provider which virtualization technology is used and what the memory limits are. Then verify the answer from inside the VPS with the commands above.