Console Login

OpenVZ failcnt: Diagnosing Memory Denials on a Shared-Kernel VPS

Symptoms

A typical report on a container-based VPS looks like this: a PHP application (for example a Magento shop on Apache 2.2, MySQL 5.5 and PHP 5.3) returns intermittent 500 errors or goes down completely. Over SSH the shell may be sluggish. top shows an idle CPU and a low load average, so nothing looks wrong.

Other signs seen in this situation:

  • MySQL or Apache stops, or logs 'Cannot allocate memory' or 'Out of memory' even though free -m shows free memory inside the container.
  • Forking fails ('fork: Cannot allocate memory') or sockets cannot be opened, although no process is obviously using many resources.
  • Errors recur at the same time each day, which points to a periodic load elsewhere on the physical node, such as a neighbour's backup job competing for the same disks.

An idle CPU does not mean the service is healthy. The limit that was hit may not be CPU at all.

Root cause

OpenVZ is operating-system-level virtualization. Every container on a physical node runs on the same Linux kernel, which on CentOS 6 era hosts is a patched Red Hat Enterprise Linux 6 kernel. Two consequences follow.

Resource accounting by beancounters

The host limits each container with user beancounters. These cover more than memory: process count, open files, sockets, kernel memory, and the sizes of TCP and socket buffers. When an application asks for a resource and the counter would pass its barrier or limit, the kernel refuses the request and increments that counter's failcnt. The application sees an ordinary error such as a failed malloc, fork or socket call. Whether it logs the error, retries or exits depends on the program. A refusal therefore does not always kill MySQL immediately, but a database that cannot allocate a buffer will often fail the query or abort.

The memory counter that matters most is privvmpages. It counts memory that has been allocated, whether or not it has been touched. A process can therefore be refused memory while the resident memory shown by free still looks modest. Values are counted in pages (4 KB on x86).

Shared kernel, shared disks

Containers share one kernel, one I/O scheduler and the same disk array. A neighbour that saturates the disks raises I/O wait for every container on the node, and this shows up as slow queries and timeouts. This cannot be fixed from inside a container. A container also cannot load its own kernel modules or choose its own kernel, which limits some firewall and security setups.

Source of truth

The fix relies on the OpenVZ project's own documentation:

  • /proc/user_beancounters confirms that failcnt counts refused resource allocations over the lifetime of the container. It is a cumulative counter, so a non-zero value is only a problem if it keeps growing.
  • UBC secondary parameters confirms that privvmpages limits the total allocated memory, including memory not yet used. Normal-priority allocations fail when the barrier is reached. The page also says a safety gap between barrier and limit reduces failures the application cannot handle, and that the sum of privvmpages across containers may exceed the node's RAM plus swap, so a node can be oversold.
  • User Guide: Managing Resources confirms that the failcnt column shows unsuccessful attempts to allocate a particular resource, and lists the parameters that cover memory, sockets and processes.

Fix options

OptionWhat it doesDrawback
Shrink the application footprintLower MySQL buffers and connection limits, and cap Apache workers, so peak allocation stays under the privvmpages barrier.Less cache and fewer concurrent requests. Does nothing for I/O contention.
Ask the provider to raise the limitsThe host administrator can change beancounters for the container.Not possible from inside the container. The provider may refuse, and the node may already be oversold.
Move to a smaller or leaner stackUse a lighter web server setup or fewer PHP processes.Requires application changes and testing.
Move to full virtualization (KVM) or a dedicated serverThe guest has its own kernel, its own swap and memory sized by the guest OS. Custom kernels and modules become possible.Migration effort, and usually a higher price per unit of memory.

Why this fix

The first step is to confirm that beancounter refusals really are the cause, and then to fit the services inside the existing limits. This is the only option that can be carried out entirely from inside the container, and it needs no change of provider. The counters give direct evidence, so the effort goes into tuning rather than guessing. Memory limits are only half of the problem, though. If the failcnt values stay at zero and the slowdowns continue, the cause is I/O contention from other tenants, and tuning will not help. In that case the correct decision is to move to a platform with isolation (see the last section).

Step by step

The procedure below is a recipe for a CentOS 6 container. The commands are shown as instructions to run, and the results depend on the host.

1. Read the counters

Inside the container, print the table. Each row shows held (current use), maxheld (peak), barrier, limit and failcnt.

cat /proc/user_beancounters

List only the rows that have ever refused a request. The last field of every data row is failcnt, whether or not the row starts with the container ID.

awk '$NF ~ /^[0-9]+$/ && $NF > 0' /proc/user_beancounters

2. Record the counters over time

Because the counter is cumulative, a single reading proves little. Append a reading every five minutes and compare entries around the time of an outage.

*/5 * * * * root (date; awk '$NF ~ /^[0-9]+$/ && $NF > 0' /proc/user_beancounters) >> /var/log/ubc-failcnt.log

Save that line in /etc/cron.d/ubc-failcnt. A monitoring system such as Nagios can alert on a rising value in the same way.

3. Look for matching errors in the application logs

grep -i -E 'cannot allocate|out of memory' /var/log/mysqld.log /var/log/httpd/error_log

4. Measure the real per-process footprint

Apache with the prefork MPM usually dominates memory. Print the average resident size of a worker (in KB) to work out how many workers fit.

ps -C httpd -o rss= | awk '{s+=$1; n++} END {if (n) print s/n}'
free -m

Divide the memory budget reserved for Apache by that average to get a MaxClients value, and cap it in /etc/httpd/conf/httpd.conf. The numbers below are a starting point to adjust.

<IfModule prefork.c>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
MaxClients 20
MaxRequestsPerChild 1000
</IfModule>

5. Make MySQL conservative

MySQL's memory use is roughly the global buffers plus the per-connection buffers multiplied by the number of connections. Lowering both reduces the peak. For a container with about 512 MB, a conservative /etc/my.cnf is:

[mysqld]
# Leave room for Apache and PHP
innodb_buffer_pool_size = 64M

# Disable the query cache to save memory
query_cache_size = 0
query_cache_type = 0

# Lower the connection limit
max_connections = 50

# Per-thread buffers: keep them small
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K

The per-thread buffers are allocated for each connection that needs them, so large values multiplied by max_connections can exceed the barrier under load. A smaller InnoDB buffer pool means more disk reads, which matters on shared disks, so set it as high as the memory budget allows rather than as low as possible.

6. Apply and verify

service mysqld restart
service httpd configtest
service httpd graceful
mysql -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size'"

After a day of normal traffic, compare the entries in /var/log/ubc-failcnt.log. If the failcnt values no longer grow, the footprint now fits. The maxheld value of privvmpages shows the peak and how much room is left below the barrier.

Limits and alternatives

Tuning works only when the cause is a beancounter limit that the application can stay under. It has several limits:

  • It cannot fix disk contention from other containers. If the failcnt values stay flat and slowdowns still appear at fixed times, the shared disks are the likely cause, and the options are to ask the provider for per-container I/O priority or to move.
  • Barriers and limits are set by the host. If the plan's limits are too small for the workload even after tuning, only the provider can raise them.
  • Some hosts run newer kernels with a different memory accounting model (the OpenVZ wiki documents one called VSwap), where the visible beancounter set differs. Check which model the host uses before interpreting the table.
  • A container cannot load its own kernel modules or run its own kernel, so firewall features that need extra modules, or SELinux-based hardening, may be unavailable.

Choose full virtualization such as KVM, or a dedicated server, when the workload is business-critical and needs predictable memory and disk behaviour, its own swap, a custom kernel, or modules the host does not provide. Containers remain a reasonable choice for testing, small personal sites and workloads that stay well inside their limits.