Deep Learning Bottlenecks: Why Fast NVMe and KVM Matter More Than Your GPU
It is March 2017, and the release of TensorFlow 1.0 just a few weeks ago has fundamentally shifted the server landscape. We are no longer just serving PHP scripts or caching database queries. We are training Convolutional Neural Networks (CNNs) and LSTMs. Whether you are running image recognition on the ImageNet dataset or crunching seismic data from the North Sea, the demand on infrastructure has spiked.
But here is the brutal truth most hosting providers will not tell you: Your expensive GPU is sitting idle 40% of the time.
I have spent the last week auditing a client's setup in Oslo. They were paying for a massive dedicated GPU cluster to train a ResNet-50 model, yet their training times were abysmal. The culprit wasn't the CUDA cores; it was the I/O wait. In this article, we are going to dissect the anatomy of a high-performance deep learning stack and why the underlying virtualization technology—specifically KVM and NVMe—is the only viable path for serious machine learning in Norway.
The "Starving GPU" Problem
Deep Learning frameworks like TensorFlow, Theano, or Caffe are voracious. They consume data in massive batches. If you are training a vision model, your CPU needs to read JPEGs from the disk, decode them, apply augmentations (crops, flips, rotations), and pack them into a tensor to push to the GPU VRAM.
If your storage subsystem is built on standard SATA SSDs (or worse, spinning HDDs in a RAID array), your disk cannot feed the CPU fast enough. The result? Your GPU computes the batch in 50ms, then waits 200ms for the next batch. You are paying for compute you aren't using.
Diagnosing the Bottleneck
To verify this, we use nvidia-smi in watch mode combined with iostat. Here is what a starving GPU looks like on a legacy VPS provider using OpenVZ and shared storage:
watch -n 1 nvidia-smi
# Output:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26 Driver Version: 375.26 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 0000:06:00.0 Off | 0 |
| N/A 42C P0 65W / 149W | 0MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+See that 0% GPU-Util? That is the sound of money burning. The GPU is waiting for the hard drive.
The Architecture of Speed: NVMe and KVM
To solve this, we need to move from SATA (600 MB/s theoretical max) to NVMe (Non-Volatile Memory express), which connects directly to the PCIe bus. On CoolVDS, we deploy high-frequency compute instances backed by local NVMe storage. We don't use network-attached storage (SAN) for these workloads because the latency penalty over the network is unacceptable for high-throughput training loops.
Benchmarking I/O for ML Workloads
Before you deploy your TensorFlow job, test your disk. We use fio to simulate the random read patterns typical of training data loaders.
# Install fio on Ubuntu 16.04
apt-get update && apt-get install -y fio
# Run a random read test (simulating image fetching)
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 \
--name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randreadOn a standard SATA SSD, you might see 40k IOPS. On CoolVDS NVMe instances, we regularly clock significantly higher numbers. This difference is what allows the CPU to keep the pre-fetching queues full.
The Software Stack: CUDA 8.0 and TensorFlow 1.0
Setting up the environment in 2017 is still known as "dependency hell," but we can mitigate it. If you are running on our KVM instances, you have full kernel control, which is mandatory for installing the NVIDIA kernel modules. Container-based virtualization like LXC often prohibits this level of access.
Here is the battle-tested setup script for Ubuntu 16.04 LTS to get your environment ready for Deep Learning:
#!/bin/bash
# 1. Update and install build tools
apt-get update
apt-get install -y build-essential git python-pip python-dev
# 2. Install CUDA 8.0 (The current standard)
wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64-deb
dpkg -i cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64-deb
apt-get update
apt-get install -y cuda
# 3. Add to path
echo 'export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc
source ~/.bashrc
# 4. Install cuDNN v5.1 (Required for TF 1.0)
tar -xzvf cudnn-8.0-linux-x64-v5.1.tgz
cp cuda/include/cudnn.h /usr/local/cuda/include
cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
Pro Tip: Avoid installing TensorFlow via simple `pip install tensorflow-gpu` if you have specific CPU instruction sets like AVX2 or FMA. Compiling from source using Bazel takes time, but it can yield a 15-20% speedup in inference on CoolVDS vCPUs.
Data Sovereignty: Why Norway?
Technical performance isn't the only metric. If you are processing personal data—faces, financial records, or medical imaging—you are bound by the Norwegian Personal Data Act and the upcoming GDPR regulations from the EU.
Using US-based cloud providers introduces legal friction regarding Safe Harbor (which is effectively dead) and Privacy Shield. By hosting your data preprocessing and training pipelines on CoolVDS in Norway, you ensure that the data remains under the jurisdiction of Datatilsynet. Furthermore, the latency from our Oslo datacenter to local fiber networks is typically under 3ms. This is critical if you are serving predictions via a REST API to end-users in Scandinavia.
Optimizing the Input Pipeline
Even with NVMe, your Python code can be the bottleneck. In Python 2.7 (still the default, though Python 3.5 is safer for unicode), the Global Interpreter Lock (GIL) is a pain. In TensorFlow 1.0, you must use tf.train.QueueRunner and Coordinators to decouple reading from training.
Here is a snippet of how to efficiently load data without blocking the training loop:
import tensorflow as tf
# Create a filename queue
filename_queue = tf.train.string_input_producer(["data_01.csv", "data_02.csv"])
reader = tf.TextLineReader()
key, value = reader.read(filename_queue)
# Defaults force type inference
record_defaults = [[1], [1], [1], [1], [1]]
col1, col2, col3, col4, label = tf.decode_csv(value, record_defaults=record_defaults)
# Shuffle batch - CRITICAL for stochastic gradient descent
# This requires memory, which CoolVDS provides in abundance
features, labels = tf.train.shuffle_batch(
[col1, col2, col3, col4],
batch_size=128,
capacity=50000,
min_after_dequeue=10000
)
with tf.Session() as sess:
# Start populating the filename queue.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
# Now your training loop runs without disk wait
# ... training logic ...
Conclusion
Building a Deep Learning rig in 2017 is about balance. A fast GPU with slow storage is a Ferrari engine in a tractor. You need high IOPS, low latency, and a virtualization layer that doesn't fight you for resources.
CoolVDS offers the NVMe storage and KVM isolation required to keep your queues full and your training epochs short. Plus, with our datacenters located strictly in Norway, your compliance strategy is as solid as your infrastructure.
Do not let your data pipeline be the reason your model fails to converge. Spin up a high-performance instance today.