Surviving the Inevitable: A DevOps Guide to Disaster Recovery in Norway

Let’s be honest with ourselves. Hardware fails. Humans delete the wrong table. Power grids fluctuate, even here in the Nordics. If your disaster recovery (DR) plan consists of a cron job running tar -czf once a week, you are not an administrator; you are a gambler.

In my decade managing infrastructure across Europe, I've seen entire businesses evaporate because their "backups" were corrupted archives sitting on the same physical disk as the production database. It happens more often than you think.

Today, we are going to architect a disaster recovery strategy that actually works. We will focus on the technical implementation using tools available right now in 2017, specifically for environments running on Linux VPS infrastructure within Norway. We have to consider the Datatilsynet (Norwegian Data Protection Authority) guidelines and the looming GDPR enforcement next year.

The Geometry of Failure: RPO and RTO

Before we touch the terminal, you need to define two metrics. If you can't answer these, stop reading and go ask your CTO.

• RPO (Recovery Point Objective): How much data can you afford to lose? One hour? One transaction?
• RTO (Recovery Time Objective): How long can you stay offline before the CEO starts screaming?

If you need an RPO of zero (no data loss), you need synchronous replication. If you need an RTO of minutes, you need high-performance I/O. This is where the underlying hardware matters. We built CoolVDS on pure NVMe storage because restoring 500GB of data on spinning rust (HDD) takes hours. On NVMe, it takes minutes. That difference is your RTO.

1. Hot Database Backups (Without Locking)

The most common mistake I see is using mysqldump on a live, high-traffic database. It locks tables. It kills performance. It causes downtime just to create the backup.

For MySQL or MariaDB (the standard stack in 2017), you should be using Percona XtraBackup. It performs hot backups of InnoDB tables without locking the database.

Installing XtraBackup on Ubuntu 16.04

The Backup Script

Do not run this manually. Automate it. Here is a robust snippet that creates a timestamped backup and prepares it for immediate restoration.

#!/bin/bash

# Configuration
BACKUP_DIR="/mnt/backups/mysql"
DATE=$(date +%Y-%m-%d_%H-%M-%S)
TARGET_DIR="$BACKUP_DIR/$DATE"
LOG_FILE="/var/log/db_backup.log"

# Ensure directory exists
mkdir -p $TARGET_DIR

echo "Starting backup at $(date)" >> $LOG_FILE

# Execute Backup
# --compress: saves space, but uses CPU. Good for network transfer.
# --slave-info: specific if you are backing up a replication slave.
innobackupex --user=root --password=YOUR_SECURE_PASSWORD --compress --no-timestamp $TARGET_DIR 2>> $LOG_FILE

if [ $? -eq 0 ]; then
    echo "Backup successful: $TARGET_DIR" >> $LOG_FILE
else
    echo "CRITICAL: Backup failed!" >> $LOG_FILE
    # Insert your alert logic here (mail, nagios, etc)
fi

Pro Tip: A raw backup isn't ready to run. You must apply the transaction logs. If you are aiming for the lowest RTO, run the "prepare" step immediately after the backup finishes, not when you are panicking during a crash.

2. Off-Site Replication (The "NIX" Factor)

Keeping backups on the same server is suicide. Keeping them in the same datacenter is risky (fire, flood, routing issues). You need to move data off-site.

In Norway, latency is low. Pinging from a CoolVDS instance in Oslo to a backup server in another Norwegian city usually stays under 10ms. This allows for aggressive rsync usage without stalling the link.

Secure Synchronization

We use rsync over SSH. It's standard, it's reliable, and it's verifiable.

rsync -avz -e "ssh -p 22" --bwlimit=5000 --delete /mnt/backups/ user@backup-server.no:/remote/backup/dir/

Note on Privacy: If you are handling Norwegian citizen data, ensure your off-site location is within the EEA. Sending unencrypted backups to a cheap bucket in the US is a violation of current privacy standards and will be illegal under GDPR next year. Keep it local. Keep it safe.

3. Encryption at Rest

If someone physically steals the hard drive from the datacenter, is your data safe? Datatilsynet requires reasonable measures to protect personal data. Encrypt your backups before they leave the server.

GPG is your friend here. It’s old, it’s clunky, and it works perfectly.

4. The "CoolVDS" Advantage: KVM Snapshots

While file-level backups are essential, sometimes you break the OS configuration, not the data. You upgraded OpenSSL and now Apache won't start. This is where virtualization saves you.

Because CoolVDS runs on KVM (Kernel-based Virtual Machine), you have access to block-level snapshots. Before performing a risky upgrade (`do-release-upgrade` or a major kernel patch), take a snapshot via the panel.

5. Verify or Die

Schrödinger's Backup: The condition of any backup is unknown until a restore is attempted. I have seen companies diligently backup zero-byte files for months.

The Fire Drill:

1. Spin up a fresh CoolVDS instance (it takes about 55 seconds).
2. Transfer your encrypted backup.
3. Decrypt and restore the database.
4. Point your application DNS (via hosts file) to this new IP.
5. Does the site load?

If you don't do this monthly, you don't have a DR plan. You have a hope and a prayer.

Conclusion

Disaster recovery isn't about buying expensive software. It's about discipline. It's about using standard, battle-tested tools like innobackupex and rsync, and deploying them on infrastructure that supports rapid recovery.

At CoolVDS, we don't just sell VPS hosting; we sell the infrastructure we would want to use ourselves. High-performance NVMe storage, strict KVM isolation, and a network optimized for the Nordic region.

Don't wait for the crash. Deploy a test instance on CoolVDS today and see how fast your restore script actually runs.