Cutting CI Build Time by Fixing Disk I/O, Swap and Heap on Linux

The problem

A CI server is idle on CPU yet builds take far longer than they should, queue up, or fail with timeouts at random. Typical jobs are a Git checkout, npm install, a Composer install, a Java compile or an asset build. Rerunning the same commit sometimes gives a very different duration. Reading the build script or blaming the network does not explain it.

Typical signs on the build host:

• CPU usage is low while the load average is high.
• The wa (I/O wait) column in top or vmstat stays well above zero during builds.
• The Jenkins JVM is slow or unresponsive when several jobs run at once, and swap is in use.

Why it happens

Build workloads are dominated by many small file operations rather than long sequential transfers. Unpacking dependency trees, writing class files and checking out thousands of source files all generate large numbers of small reads and writes. Three things commonly make this worse.

Storage latency

When the block device is slow or shared with busy neighbours, each small operation waits. Processes sit in uninterruptible sleep and the CPU looks idle. Rotating disks and heavily shared storage suffer most.

Access-time updates

Every file read can cause a metadata write if the filesystem updates access times. Modern kernels (2.6.30 and later, including the 2.6.32 kernel in CentOS 6) mount with relatime by default. That limits the updates, so the cost is smaller than often claimed, but not zero. Explicit noatime removes it completely.

Memory pressure on the JVM host

Jenkins runs on a JVM. If heap size and the kernel's swapping behaviour are left at defaults, the host can start paging heap memory to disk. That adds disk I/O on top of the build I/O. Without an explicit -Xmx, the maximum heap is derived from the machine's RAM, which may not match what is actually left for Jenkins once build tools and the OS take their share.

Source of truth

The fix relies on the kernel and tool documentation, not on folklore:

• mount(8) documents the noatime, nodiratime and relatime options. It confirms that noatime applies to all inode types including directories, so it already implies nodiratime, and that relatime has been the default since Linux 2.6.30. The page also describes remount.
• The kernel's vm sysctl documentation describes vm.swappiness (how aggressively the kernel swaps pages out, default 60) and vm.vfs_cache_pressure (default 100; lower values make the kernel keep directory and inode caches longer).
• The Capistrano 3 deployment flow documents the release directory layout and the deploy:publishing step, in which the current symlink is switched to the new release. This is what makes the cutover atomic.

The iostat man page from the sysstat package defines the await and %util columns used for diagnosis.

Fix options

Why this fix

The sensible order is to measure first, then apply the cheap, reversible changes that address the measured cause, and only then change the infrastructure.

1. Measure with iostat and vmstat to separate a slow device from swapping.
2. Mount with noatime. It is one fstab word, reversible and safe for a build host. nodiratime is redundant and can be dropped.
3. Set an explicit heap and lower swappiness so the JVM stays in RAM.
4. Use tmpfs only for disposable workspaces that fit comfortably in memory.
5. Move the build server to faster storage only if the device itself is still the bottleneck.

Kernel-level tuning needs a kernel the administrator controls. On container-based virtualization such as OpenVZ, most sysctls cannot be changed from inside the guest. The kernel is also the old 2.6.32 branch, which is older than the 3.8 kernel Docker requires. A KVM guest does not have these restrictions.

Step by step

The commands below are instructions to apply on a Linux build host, as root. They were not run for this article, so no output is shown. Compare your own output before and after each change.

1. Diagnose

Install the tools. On Debian or Ubuntu:

apt-get install sysstat

On CentOS 6:

yum install sysstat

Watch the device while a build runs:

iostat -x 1
vmstat 1

In iostat, a high await (average time per request, in milliseconds) together with %util close to 100 on a single disk means the device is saturated. In vmstat, a high wa means CPUs are waiting on I/O. Non-zero si and so columns mean the host is actively swapping.

2. Mount with noatime

Edit the options field in /etc/fstab. Keep the device identifier and filesystem type from the existing line:

# /etc/fstab
UUID=xxxx-xxxx   /   ext4   errors=remount-ro,noatime   0   1

Apply it without a reboot. Naming the option explicitly avoids depending on how the mount tool merges fstab options on a remount:

mount -o remount,noatime /
grep ' / ' /proc/mounts

3. Optional: tmpfs for throwaway workspaces

Size the RAM disk well below free memory, and use it only for data that can be rebuilt. Add it to /etc/fstab:

tmpfs   /var/lib/jenkins/workspace   tmpfs   size=2g,uid=jenkins,gid=jenkins   0   0

Then mount it:

mount /var/lib/jenkins/workspace

Mounting over an existing directory hides what was in it, so stop Jenkins and move any existing workspace data first.

4. Reduce swapping

Apply the values immediately:

sysctl -w vm.swappiness=10
sysctl -w vm.vfs_cache_pressure=50

Make them persistent in /etc/sysctl.conf:

vm.swappiness = 10
vm.vfs_cache_pressure = 50

Reload and check:

sysctl -p
sysctl vm.swappiness

5. Set the Jenkins heap

On Debian and Ubuntu packages the JVM options live in /etc/default/jenkins. Choose values that leave room for the OS and the build tools that run beside Jenkins. -XX:MaxPermSize applies only to Java 7 and earlier. Java 8 ignores it and prints a warning, so drop it there.

# /etc/default/jenkins
JAVA_ARGS="-Xms512m -Xmx1024m -XX:MaxPermSize=256m -Djava.awt.headless=true"

On CentOS the equivalent is JENKINS_JAVA_OPTIONS in /etc/sysconfig/jenkins. Restart the service afterwards:

service jenkins restart

6. Atomic deployment with Capistrano 3

Capistrano clones the repository into a timestamped directory under releases/, runs the deploy tasks, and switches the current symlink only in the publishing step. If an earlier step fails, current keeps pointing at the previous release. Capistrano does not compile or test anything by itself. Add such steps as tasks. This example restarts PHP-FPM after publishing:

# config/deploy.rb
lock '3.2.1'

set :application, 'my_app'
set :repo_url, 'git@example.com:username/my_app.git'
set :keep_releases, 5

namespace :deploy do
  desc 'Reload PHP-FPM'
  task :restart do
    on roles(:app), in: :sequence, wait: 5 do
      execute :sudo, :service, 'php5-fpm', 'reload'
    end
  end

  after :publishing, :restart
end

The deploy user needs a sudoers entry allowing exactly that command without a password. Deploy with:

bundle exec cap production deploy

Limits and alternatives

• Tuning cannot beat a saturated device. If await stays high on an idle system after the changes, the storage is the problem. Move the build host to faster or less contended storage.
• noatime gains are modest on a kernel with relatime. The default already avoids most atime writes, so the change matters most for read-heavy trees.
• tmpfs competes with the JVM for RAM. A workspace that grows past its size limit fails builds. Under memory pressure its pages can be swapped out as well.
• Swappiness is not a heap size. A swappiness of 10 does not prevent swapping. It only biases the kernel. The reliable fix is a heap that fits in RAM.
• Virtualization type matters for control, not for magic. KVM allows kernel and sysctl changes and custom modules. The physical disks are still shared between guests, so measure rather than assume.
• Network distance counts too. A build server far from the Git remote and artifact stores adds time to every fetch. Place it near both.
• Alternatives to Capistrano. Fabric (Python) or a plain script that builds into a new directory and swaps a symlink give the same atomic switch. Database migrations still need separate handling in every case.