Stop Tolerating Slow Builds: Optimizing CI/CD Pipelines with Private NVMe Runners

The 15-Minute Build is a Productivity Killer

There is a silent killer in modern software development, and it isn't legacy code or technical debt. It is the "coffee break" build pipeline. When a developer pushes a commit and has to wait 15 to 20 minutes for the CI/CD pipeline to turn green, context is lost. They switch tasks, check Slack, or grab coffee. When they return, the mental flow is broken.

In 2021, with the maturity of Docker and Kubernetes, there is no excuse for sluggish pipelines. Yet, I constantly see engineering teams in Oslo and Bergen relying on oversubscribed shared runners provided by SaaS platforms. These shared environments are fundamentally inconsistent. One build takes 4 minutes; the next takes 12 because a noisy neighbor is monopolizing the I/O.

If you are serious about DevOps, you need to own your infrastructure. This guide covers how to transition from sluggish shared runners to high-performance, self-hosted pipelines using NVMe-backed VPS instances. We will focus on I/O bottlenecks, Docker caching, and the legal implications of data residency under GDPR post-Schrems II.

The I/O Bottleneck: Why NVMe Matters

Most developers assume CPU is the primary constraint for CI jobs. While true for compilation (C++, Rust, Go), the vast majority of web pipelines (Node.js, PHP, Python) are I/O bound. Consider npm install, composer install, or extracting Docker images. These operations involve writing thousands of small files to the disk.

On a standard SATA SSD or, worse, a network-attached block storage solution with capped IOPS, your build hangs while waiting for the disk subsystem. To prove this, run a rigid fio test on your current runner. If you aren't seeing random write speeds above 200MB/s, your storage is the bottleneck.

Here is a benchmark command to test your current environment's random write performance:

fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=512M --numjobs=2 --runtime=240 --group_reporting

At CoolVDS, we utilize local NVMe storage passed through via KVM. We typically see IOPS values 5x to 10x higher than standard cloud block storage. This difference directly translates to reducing an npm ci step from 4 minutes to 45 seconds.

Configuring a High-Performance GitLab Runner

GitLab CI is the standard for many European dev teams due to its robust self-hosted options. Moving off shared runners to a CoolVDS instance allows you to cache dependencies persistently and control the concurrency.

When setting up your /etc/gitlab-runner/config.toml, do not rely on defaults. You need to tune the concurrent and limit parameters based on your vCPUs. For a 4 vCPU CoolVDS instance, I recommend allowing slight oversubscription if your jobs are I/O heavy, or strict 1:1 mapping for CPU-heavy compilation.

Below is a production-hardened configuration for a dedicated runner:

concurrent = 4
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "CoolVDS-NVMe-Runner-01"
  url = "https://gitlab.com/"
  token = "YOUR_TOKEN_HERE"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03.12"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0
    # Use overlay2 for best performance
    storage_driver = "overlay2"

Critical Detail: Notice the mounting of /var/run/docker.sock. While this has security implications (see below), it allows the runner to spawn sibling containers rather than using the slower Docker-in-Docker (dind) approach. This enables massive layer caching benefits.

Optimizing Docker Builds with Layer Caching

Hardware is only half the battle. Your Dockerfile architecture determines how effectively the cache is utilized. A common mistake is copying the entire source code before installing dependencies.

Bad Practice:

FROM node:14
WORKDIR /app
COPY . .
RUN npm install
CMD ["node", "server.js"]

Optimized Practice:

FROM node:14-alpine
WORKDIR /app

# Copy only package files first to leverage cache
COPY package*.json ./

# This layer is cached unless package.json changes
RUN npm ci --only=production

# Then copy the rest of the code
COPY . .

CMD ["node", "server.js"]

By using npm ci (clean install) and ordering the layers correctly, you ensure that changing a single line of code in server.js does not trigger a re-download of the entire internet. Combined with the fast I/O of our NVMe infrastructure, this results in near-instant build phases for minor commits.

Pro Tip: Enable Docker BuildKit. In 2021, this is becoming the new standard. Set DOCKER_BUILDKIT=1 in your CI environment variables to unlock parallel build steps and advanced caching mechanisms.

Data Sovereignty: The Schrems II Reality

Technical performance is not the only reason to bring your CI/CD runners home. In July 2020, the CJEU's Schrems II ruling invalidated the Privacy Shield framework. If your code contains personal data (PII) or production database dumps used for testing, processing that data on US-owned shared runners creates a compliance risk.

By hosting your runners on CoolVDS instances located physically in Norway (outside the EU but within the EEA framework and strict local laws), you gain greater control over data residency. You can demonstrate to the Datatilsynet (Norwegian Data Protection Authority) that your testing data never leaves the region.

Network Latency: The NIX Advantage

Finally, let's talk about the network. If your production servers are hosted in the Nordics, but your CI/CD pipeline runs in us-east-1, you are introducing unnecessary latency during the deployment phase (rsync, scp, or kubectl apply).

Hosting your build agents close to your target infrastructure minimizes transfer times. CoolVDS peers directly at NIX (Norwegian Internet Exchange). This ensures that once your artifact is built, the push to production happens over a low-latency, high-bandwidth path, often under 5ms.

Comparison: Shared vs. CoolVDS Private

Implementation Strategy

Don't migrate everything at once. Start by identifying your slowest pipeline. Spin up a CoolVDS KVM instance (I recommend the 4GB RAM / 2 vCPU plan as a starter). Install the runner agent, tag it specifically (e.g., tags: ["nvme-oslo"]), and route that slow job to the new runner.

Measure the difference. In almost every case we have analyzed, the combination of raw NVMe throughput and persistent caching reduces build times by at least 50%.

Your developers' time is the most expensive resource you have. Don't waste it on loading bars. Deploy a high-performance runner today and get back to shipping code.