OpenVZ Containers: Diagnosing and Fixing User Beancounter Limits

The problem

A service inside an OpenVZ container starts failing in ways that do not match the load. Typical signs are Apache children that cannot be forked, MySQL that refuses to start or drops connections, Cannot allocate memory errors, or fork: retry: Resource temporarily unavailable in a shell. Inside the container, free and top often still show unused memory, and the host node is not under visible pressure.

The failures are intermittent and hard to reproduce, and the application logs usually name no specific limit. A web server may run for days and then fail during a traffic peak, a backup job or a package upgrade.

Why it happens

OpenVZ containers share one kernel. The kernel limits each container with a set of counters called User Beancounters (UBC). Each counter covers one resource: number of processes, kernel memory, allocated private address space, open sockets, socket buffers and so on. Each has a barrier and a limit, and the kernel counts every refused allocation in a fail counter.

Several of these counters do not behave like the numbers a normal Linux guest shows:

• privvmpages counts allocated private virtual memory in 4 KB pages, not resident memory. Programs that reserve a lot of address space (prefork Apache with many children, Java, some database settings) can hit the limit while real memory use is low.
• numproc caps processes and threads together. Threaded servers can reach it sooner than expected.
• kmemsize covers unswappable kernel memory used on behalf of the container, such as process structures and socket buffers. A busy network service consumes it.
• numtcpsock, tcpsndbuf and tcprcvbuf limit connections and their buffers.

When an allocation would cross the limit, the kernel refuses it and the application sees an ordinary out-of-memory or fork error. Nothing else is reported, which is why the cause is easy to miss. The "burstable RAM" idea from the original article follows from the two-level design. Usage between barrier and limit is tolerated, and usage past the limit is refused. It is not a promise that extra memory will always be available.

Source of truth

The OpenVZ wiki is the project's own documentation, and the fix relies on these pages:

• /proc/user_beancounters confirms the file layout (held, maxheld, barrier, limit, failcnt). It also confirms that failcnt counts refused resource allocations over the lifetime of the container's process group, and it shows how to extract a single container's block with sed or egrep.
• UBC Monitoring confirms that containers hitting limits can make applications malfunction in strange ways. It says such faults can only be diagnosed by recognising a UBC configuration problem, and it recommends watching the fail counters periodically from the host.
• UBC failcnt reset confirms that each counter increases when its resource hits its limit. It also warns that resetting the counters is not recommended, because other tools read the same file.

Per-parameter semantics and the vzctl set options are in the vzctl(8) manual page that ships with the tools and in the wiki's "Setting UBC parameters" page. Check them against the installed vzctl version before applying values.

Ways to fix it

1. Raise the exhausted limit with vzctl

Run vzctl set CTID --parameter barrier:limit --save on the host node. The kernel applies the new value to the running container, and --save writes it to /etc/vz/conf/CTID.conf so it survives a restart.

2. Edit the container configuration file

Change the values in /etc/vz/conf/CTID.conf and restart the container. This is useful for scripted or templated changes, but it needs a restart. It also makes typos possible, such as a barrier above the limit.

3. Reduce what the application asks for

Lower Apache MaxClients, shrink MySQL buffers, or cut thread stack sizes. This works with any limits, and it also helps on any other platform. It does not help when a workload legitimately needs more than the plan provides.

4. Move to a different isolation model

Move to a larger container plan, a dedicated server, or a hardware-virtualized guest (Xen or the emerging KVM), where the guest kernel has its own memory accounting. This costs more and is only worth it when the workload does not fit shared-kernel limits.

Why this fix

The chosen method combines options 1 and 3. First identify the counter that has a non-zero fail count. Then raise only that counter with vzctl set --save. Finally, bring the application's own limits below the new ceiling so it fails gracefully instead of hitting the kernel's refusal.

Raising limits live avoids downtime, the saved configuration keeps the change after a restart, and changing only the failing counter keeps the node's overcommit under control. Blindly setting every limit to a huge value defeats the isolation that makes a shared-kernel node safe for its neighbours. A configuration validator should check the result, because the barrier must not exceed the limit. Option 2 is the fallback when many containers need the same change. Option 4 is a decision about capacity, not a fix.

Step by step

All commands run as root on the host node. Container ID 101 and the numbers below are examples. Scale them to the node's real capacity. No program output is reproduced here, so compare results with the layout described in the source section.

1. Find the affected container and read its counters

vzlist -a
sed -nr "1,2p;/101:/,+23p" /proc/user_beancounters

The block for the container lists one line per resource. Look at the last column, failcnt.

2. List only counters that have refused allocations

awk '$NF ~ /^[0-9]+$/ && $NF > 0' /proc/user_beancounters

Every line printed has a non-zero fail count. Note the container UID, the resource name and whether maxheld has reached the barrier or the limit. Save a baseline so later runs can be compared with it:

awk '$NF ~ /^[0-9]+$/ && $NF > 0' /proc/user_beancounters > /root/ubc-before.txt

3. Raise only the exhausted counters

Example for a container that ran out of private address space and processes. privvmpages is counted in 4 KB pages, so 393216 pages is about 1.5 GB:

vzctl set 101 --privvmpages 393216:419430 --save
vzctl set 101 --numproc 400:400 --save

If kmemsize is the counter that fails, it is set in bytes:

vzctl set 101 --kmemsize 25165824:27262976 --save

4. Validate the configuration

vzcfgvalidate /etc/vz/conf/101.conf

The validator reports barrier and limit combinations that are inconsistent or unsafe for the node. Fix anything it flags before moving on. To make the change by editing the file instead (option 2), the entries have this form, and the container needs vzctl restart 101 afterwards:

PRIVVMPAGES="393216:419430"
NUMPROC="400:400"

5. Bring the application below the new ceiling

For Apache 2.2 with the prefork module, size MaxClients to what the container can hold, and recycle children so address space does not grow without bound:

<IfModule prefork.c>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients           50
    MaxRequestsPerChild 1000
</IfModule>

vzctl exec 101 service httpd restart

6. Verify and keep watching

Re-run the awk filter from step 2 under normal load and compare with the baseline. Fail counters are cumulative since the container started, so a stable value means the problem is fixed, and a growing value means it is not. Avoid resetting the counters. Instead, check them periodically from the host, as the UBC monitoring page recommends. A minimal hourly job (/etc/cron.hourly/ubc-failcnt, mode 755) is:

#!/bin/sh
awk '$NF ~ /^[0-9]+$/ && $NF > 0' /proc/user_beancounters | mail -E -s "UBC failures on $(hostname)" root

This mails the list whenever any counter is non-zero. It does not detect new failures, because old counts stay until the container restarts. To report only changes, store the previous output and compare it with diff.

Looking back

What worked

Reading the fail counters turns a vague "out of memory" symptom into a named resource. A live change with vzctl set --save then fixes it without a restart. Checking counters from the host regularly catches the problem before users report it.

Limits

• Limits protect the other containers, so a node cannot hand out unlimited increases. The sum of the barriers in use must fit the node's real RAM and kernel memory.
• Raising a limit does not fix a leak or an oversized application configuration. If maxheld keeps climbing after a raise, look at the application.
• A container cannot load its own kernel modules, change kernel parameters freely, or run a different operating system, because the kernel is shared. This affects some VPN and firewall setups.
• A shared kernel is a shared failure domain. A kernel panic or kernel vulnerability affects every container on the node, so the host kernel must be kept patched. Hosting in Norway fixes where the data physically sits, but it does not by itself make a setup compliant with any data protection rules. That depends on contracts, access control and processing practices.

When to choose another method

The overhead of container virtualization is low because there is no emulated hardware, but figures such as "under 2%" depend on the workload and should be measured, not assumed. OpenVZ also supports checkpointing a running container and restoring it on another node, which is how its live migration works. It still needs shared planning of limits between the source and destination nodes.