Stop Leaking Data to OpenAI: High-Performance Local LLM Deployment with Ollama

Date: October 11, 2023

Everyone is talking about Generative AI. But if you are a CTO or a Lead Dev in Norway, you are likely screaming about something else: Data Privacy. Sending customer PII or proprietary code to OpenAI’s API is a GDPR nightmare waiting to happen. The solution isn’t to ban AI; it’s to bring it in-house.

Until recently, running Large Language Models (LLMs) meant wrestling with Python dependencies, compiling C++ binaries for llama.cpp, or spending $10,000/month on AWS GPU instances. Not anymore.

Enter Ollama. This tool has simplified local inference to the point where it feels like Docker for LLMs. Combined with the release of Mistral 7B just two weeks ago, we now have open-weights models that punch way above their weight class, running on standard CPU architecture.

I’m going to show you how to build a private, legally compliant inference server on a standard Linux VPS. No A100s required.

The Architecture: Why CPU Inference is Viable Now

Let’s clear up a misconception: "You need a GPU for LLMs."

For training? Yes. For inference (running the model)? Not necessarily. Thanks to 4-bit quantization (GGUF format), a 7-billion parameter model like Llama 2 or Mistral can run comfortably in 8GB of RAM. The bottleneck is no longer raw compute; it is memory bandwidth and storage I/O.

This is where most hosting providers fail. They oversell RAM and give you spinning rust (HDD) or cheap SATA SSDs. When you load a 5GB model file into memory, slow storage hangs your application. You need NVMe.

Pro Tip: On a CoolVDS instance, we map NVMe storage directly via KVM. This drastically reduces the "cold start" time when swapping models, a critical metric if you are switching between codellama for coding and mistral for chat.

Step 1: The Environment

We will use a CoolVDS High-Freq 4-Core instance with Ubuntu 22.04 LTS. You need AVX2 support on the CPU (standard on our nodes) to accelerate matrix multiplications.

First, secure the box. We don't want the world querying our API.

# Standard hygiene
apt-get update && apt-get upgrade -y
ufw allow 22/tcp
ufw allow 11434/tcp  # Ollama's default port
ufw enable

Step 2: Installing Ollama

Ollama abstracts away the complexity of the underlying inference engine. As of this week (early October), they’ve added better support for Linux distributions. Install it with one command:

curl https://ollama.ai/install.sh | sh

Once installed, start the service. I recommend running it via systemd (which the script sets up), but for debugging, you can run:

ollama serve

Step 3: Deploying Mistral 7B (The Game Changer)

Llama 2 was the king until September. But Mistral 7B, released under Apache 2.0, is currently outperforming Llama 2 13B in many benchmarks. It is efficient, fast, and perfect for CPU-based VPS hosting.

Pull the model:

ollama pull mistral

This will download the 4.1GB quantized file. On CoolVDS NVMe infrastructure, this write operation completes in seconds. Now, run it interactively:

ollama run mistral "Explain the concept of GDPR data sovereignty to a 5-year-old."

You should see tokens streaming immediately. On a 4-core dedicated slice, you can expect 8-12 tokens per second. That is faster than human reading speed.

Step 4: Customizing the System Prompt

The real power comes from the Modelfile. This allows you to bake a persona or strict rules into the model, similar to a Dockerfile. Let's create a specialized "Norwegian Legal Assistant".

Create a file named Modelfile:

FROM mistral

# Set parameters to reduce hallucination
PARAMETER temperature 0.2
PARAMETER stop "User:"

# System Context
SYSTEM """
You are a helpful assistant for a Norwegian Systems Administrator.
You answer briefly and technically.
If asked about data locations, always emphasize that data stored on CoolVDS remains physically in Norway.
"""

Build your custom model:

ollama create cool-admin -f Modelfile
ollama run cool-admin "Where is my data?"

Performance Tuning: Avoiding "Steal Time"

If you run this on a cheap $5 VPS from a budget provider, you will experience "CPU Steal". This happens when the hypervisor forces your VM to wait while other neighbors use the physical CPU. For a web server, this causes a 50ms delay. For an LLM, it causes the generation to hang for seconds, breaking the user experience.

This is why Dedicated KVM resources are non-negotiable. You need guaranteed CPU cycles. We monitor CPU Steal closely; if it goes above 0.1%, we move noisy neighbors. Your inference latency must be predictable.

Integration: The API

Ollama provides a clean REST API. You can replace your OpenAI calls with this local endpoint. Here is a simple Python 3.10 example to integrate into your backend:

import requests
import json

url = "http://localhost:11434/api/generate"

data = {
    "model": "mistral",
    "prompt": "Write a Nginx config block for a reverse proxy.",
    "stream": False
}

response = requests.post(url, json=data)
print(response.json()['response'])

Conclusion: Own Your Intelligence

The era of relying on US-based APIs for every text generation task is ending. With tools like Ollama and efficient models like Mistral, you can build powerful, private, and fast AI applications right here in Europe.

Don't let latency or privacy concerns dictate your architecture. Spin up a CoolVDS instance today, deploy Mistral in under 5 minutes, and keep your data where it belongs: under your control.