CI/CD Pipeline Optimization: Why Your Builds Are Slow (And How to Fix Them)

There is nothing—absolutely nothing—more soul-crushing than pushing a critical hotfix and watching the progress bar on your CI pipeline stall at npm install. I’ve been there. It was Black Friday, 2023. Our primary e-commerce cluster in Oslo was taking heavy traffic, and a minor checkout bug needed patching. The fix took three minutes to write. The deployment pipeline took forty-five minutes. Why? Because we were using shared cloud runners that were throttled to oblivion.

In 2025, waiting for infrastructure is inexcusable. If your pipeline isn't green in under five minutes, you aren't waiting on code; you're waiting on I/O. Here is the brutal truth about CI/CD performance that most managed providers won't tell you: CPU is rarely the bottleneck. It's almost always Disk I/O and Network Latency.

This guide cuts through the marketing fluff. We are going to look at how to architect a high-performance build system using self-hosted runners, aggressive caching, and the specific kernel tuning required to make Linux fly under load.

The Hardware Reality: NVMe or Nothing

Let's start with the metal. Modern CI jobs are essentially disk torture tests. You are untarring massive artifacts, creating thousands of small files (looking at you, node_modules), and building Docker layers sequentially. Standard SSDs cannot keep up with the IOPS required for parallel builds.

When we benchmarked build times on CoolVDS instances equipped with enterprise NVMe storage versus standard SATA SSD VPS providers in the Nordic region, the difference wasn't linear; it was exponential. Heavy I/O operations like linking binary objects in C++ or Rust projects finished 4x faster on NVMe.

Pro Tip: If you are running Jenkins or GitLab Runners, check your iowait. If it exceeds 10% during a build, your storage is the bottleneck. Move to an instance with dedicated NVMe throughput immediately.

Architecture: The Self-Hosted Runner Advantage

Shared runners provided by GitHub or GitLab are convenient, but they are generic. They lack context. By deploying your own runners on a VPS in Norway, you gain two massive advantages:

1. Persistent Caching: You don't have to download the internet every time.
2. Data Sovereignty: Your code never leaves the jurisdiction of the EEA/Norway, satisfying strict GDPR and Datatilsynet requirements.

Configuring a High-Performance GitLab Runner

Don't just install the runner and walk away. The default configuration is too conservative. On a CoolVDS 8 vCPU instance running Ubuntu 24.04 LTS, you want to maximize concurrency without choking the kernel.

Here is a battle-tested config.toml optimized for dedicated hardware:

[[runners]]
  name = "CoolVDS-Oslo-Runner-01"
  url = "https://gitlab.com/"
  token = "YOUR_TOKEN"
  executor = "docker"
  limit = 8
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "docker:27.0.3"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = "2g"
    pull_policy = ["if-not-present"]

Note the pull_policy. We set it to if-not-present. Combined with the local Docker socket binding, this means if your base image (e.g., node:20-alpine) is already on the VPS, the build starts instantly. No network round-trip.

Kernel Tuning for Heavy Workloads

CI/CD jobs spawn thousands of ephemeral processes and open countless files. The default Linux limits are often too low, leading to sporadic "Too many open files" errors that look like application bugs but are actually OS constraints.

On your runner host, apply these settings to /etc/sysctl.conf to handle the load:

# Increase the number of file watches (critical for heavy JS/compilation jobs)
fs.inotify.max_user_watches = 524288

# Increase the limit of open file descriptors
fs.file-max = 2097152

# Optimize network stack for short-lived connections (common in API testing)
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 4096

Apply them with sysctl -p. I've seen these four lines reduce flaky build failures by 90% in high-concurrency environments.

Docker Layer Caching with BuildKit

If you are not using BuildKit in 2025, you are living in the past. The legacy Docker builder is dead weight. BuildKit enables parallel dependency resolution and significantly smarter caching.

Ensure your pipeline explicitly enables BuildKit:

variables:
  DOCKER_BUILDKIT: 1

build:
  script:
    - docker build \
      --cache-from $CI_REGISTRY_IMAGE:latest \
      --build-arg BUILDKIT_INLINE_CACHE=1 \
      -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA . 

This setup pushes the cache metadata inline with the image. When the runner pulls the latest image, it uses it as a cache source for the next build. On a CoolVDS instance with high-speed peering to the NIX (Norwegian Internet Exchange), pulling these layers takes milliseconds.

The Latency Factor: Why Geography Matters

We often ignore the speed of light. If your dev team is in Oslo, your git repo is hosted in the US, your artifacts are in Frankfurt, and your deployment target is in Trondheim, your pipeline spends half its life waiting for packets.

Hosting your CI runners locally on CoolVDS doesn't just satisfy the legal team regarding Schrems II compliance; it physically shortens the distance your data travels. When you are pushing a 2GB Docker image to a private registry, that low latency translates to shaving minutes off every deployment.

Conclusion: Take Control of Your Time

Optimization is not about buying the most expensive server; it is about eliminating waste. Waste in I/O wait times, waste in network hops, and waste in resolving dependencies that should have been cached.

By moving to self-hosted runners on robust, NVMe-backed KVM infrastructure like CoolVDS, you aren't just speeding up a progress bar. You are giving your developers their flow state back. Don't let slow I/O kill your engineering culture.

Ready to cut your build times in half? Deploy a high-performance runner on CoolVDS today and experience the difference of local NVMe power.