The problem
A typical multi-server setup treats the private network as a safe zone. The application server talks to the database in cleartext, internal HTTP APIs accept any caller on the LAN, and administrators log in with long-lived SSH keys copied to several laptops. Anything that gets a foothold inside the network, such as a compromised workstation, a vulnerable web app or a stolen key, can reach every service that was only ever protected by its address.
Remote work makes this worse. Administrators connect from home networks, so the boundary that was supposed to separate trusted from untrusted traffic no longer matches where people and machines actually are.
Typical symptoms
- The database listens on a LAN address, or on all interfaces, and accepts password or even
trustauthentication from a whole subnet. - Packet captures between nodes show readable SQL or HTTP.
- An internal API has no authentication because it is "not reachable from outside".
- One SSH private key with no expiry grants access to many hosts.
Why it happens
The root cause is implicit trust based on network location. A source IP address says where a packet claims to come from. It does not say who or what sent it. Firewalls enforce rules on addresses and ports. Once a single host inside the perimeter is compromised, lateral movement meets no further checks.
Several common defaults reinforce this:
- Services are configured to be reachable first and restricted later.
- Encryption is seen as a cost, so it is skipped on internal links.
- Credentials such as SSH keys and certificates are issued once and never rotated.
Zero trust, in this context, means three things: every connection is encrypted, every connection is authenticated at the service or data layer, and every credential is scoped and short-lived. It is a configuration state, not a product.
Jurisdiction is a separate question. Where a provider is legally established, and which legal regime applies to the disks, cannot be changed by software. Since the 2020 Schrems II judgment invalidated the EU-US Privacy Shield, that choice belongs in the legal risk assessment. The technical controls below reduce exposure, but they do not replace legal advice.
Source of truth
The fix relies on the behaviour documented by the upstream projects:
- WireGuard quick start confirms the key generation commands (
wg genkey,wg pubkey) and the interface model. Each peer is identified by its public key, andAllowedIPsacts as both a routing table and a source-address filter for that peer. WireGuard has been part of the mainline Linux kernel since version 5.6. Older kernels need an out-of-tree module package. - nginx ngx_http_ssl_module documents
ssl_client_certificate(the CA certificates used to verify clients) andssl_verify_client. With the valueon, a client certificate is required, and a missing or invalid one produces an HTTP error response (such as 400 with code 496) instead of reaching the backend. It also documents the embedded variables$ssl_client_s_dnand$ssl_client_verify. - sshd_config(5) confirms
AuthenticationMethods(for examplepublickey,keyboard-interactiveto require both factors). It also confirms thatClientAliveCountMax 0disables connection termination instead of disconnecting at once, and thatClientAlive*probes test whether the client responds, not whether the user is idle. - PostgreSQL pg_hba.conf confirms that
hostsslrecords match only TLS connections, thathostnosslmatches only non-TLS ones, and that the first matching record wins.scram-sha-256is the recommended password method there.
Ways to fix it
Encrypting east-west traffic
| Method | Strengths | Weaknesses |
|---|---|---|
| WireGuard | Small code base, in-kernel since 5.6, simple key-based peer config, no application changes | Static peer list to manage; identity is per host, not per service |
| IPsec | Standardised, broad interoperability | Complex configuration and troubleshooting |
| OpenVPN | Mature, flexible, runs in userspace on any kernel | Larger code base, more configuration, userspace data path |
| TLS in each service | Per-service identity, no tunnel needed | Every service must support it and have its certificates managed |
Authenticating HTTP callers
- API keys or tokens: simple, but they are bearer secrets, and the application must validate them.
- Mutual TLS at nginx: the proxy rejects callers without a certificate issued by the internal CA before the backend sees the request.
Authenticating administrators
- Static keys plus a PAM second factor (TOTP): quick to add on a single host.
- SSH certificates signed by a CA: short validity, central issuance, no key distribution to each host.
- The two can be combined.
Why this fix
The chosen fix is layered, because each layer closes a different gap.
- WireGuard between app and database nodes. It encrypts the link without touching the applications. The database is bound only to the tunnel address, so it is not reachable on any other interface.
- mTLS on nginx for internal HTTP APIs. It adds per-caller identity without changing the backend, and the subject DN is passed to it for auditing.
- SSH with public key plus TOTP. It is available on any distribution that packages the PAM module. SSH certificates are noted as the next step once more than a handful of hosts and users exist.
WireGuard is preferred over IPsec and OpenVPN here for its simpler configuration on Linux-to-Linux links. It requires kernel 5.6 or later, or the module package on older systems. Encryption overhead on CPUs with AES instructions is small for typical business workloads, but it should be measured on the actual hardware before assuming that.
Step by step
The commands below are instructions to apply. No program output is reproduced, so check the result on the real systems with the verification commands at the end of each part. Addresses, host names and the example domain are placeholders.
1. WireGuard between two nodes
On each node, generate a key pair with restrictive permissions:
umask 077
mkdir -p /etc/wireguard
wg genkey | tee /etc/wireguard/private.key | wg pubkey > /etc/wireguard/public.key
On the app node, create /etc/wireguard/wg0.conf:
[Interface]
PrivateKey = <app_node_private_key>
Address = 10.0.0.1/24
ListenPort = 51820
[Peer]
PublicKey = <db_node_public_key>
AllowedIPs = 10.0.0.2/32
Endpoint = 192.168.1.5:51820
On the database node, mirror it:
[Interface]
PrivateKey = <db_node_private_key>
Address = 10.0.0.2/24
ListenPort = 51820
[Peer]
PublicKey = <app_node_public_key>
AllowedIPs = 10.0.0.1/32
Endpoint = 192.168.1.4:51820
Start the interface on both nodes, enable it at boot and verify the handshake:
chmod 600 /etc/wireguard/wg0.conf
systemctl enable --now wg-quick@wg0
wg show
ping -c 3 10.0.0.2
In wg show, a "latest handshake" line for the peer indicates that the tunnel works.
2. Bind PostgreSQL to the tunnel and require TLS
In postgresql.conf (TLS needs a server certificate and key configured as well):
listen_addresses = '10.0.0.2'
ssl = on
In pg_hba.conf, rules are evaluated top to bottom, so the specific allow rule comes first:
hostssl all all 10.0.0.1/32 scram-sha-256
hostnossl all all 0.0.0.0/0 reject
The database must start only after wg0 exists, otherwise it cannot bind to 10.0.0.2. Add a systemd drop-in (the unit name may differ by distribution, for example postgresql-13):
systemctl edit postgresql
with this content:
[Unit]
[email protected]
[email protected]
Then restart the service and check the listener. It should show only 10.0.0.2:5432.
systemctl restart postgresql
ss -ltnp | grep 5432
Also allow UDP 51820 in the host firewall only from the peer's address. That keeps the tunnel endpoint itself from being open to the world.
3. Mutual TLS on nginx
Create a small internal CA, a server certificate with a subject alternative name, and a short-lived client certificate:
openssl genrsa -out ca.key 4096
openssl req -x509 -new -key ca.key -sha256 -days 365 -subj '/CN=Internal CA' -out ca.crt
openssl genrsa -out server.key 2048
openssl req -new -key server.key -subj '/CN=api.internal.example' -out server.csr
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -sha256 -days 90 -extfile <(printf 'subjectAltName=DNS:api.internal.example') -out server.crt
openssl genrsa -out client.key 2048
openssl req -new -key client.key -subj '/CN=frontend-1' -out client.csr
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -sha256 -days 30 -out client.crt
The process substitution needs bash. Then configure the virtual host:
server {
listen 443 ssl;
server_name api.internal.example;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_certificate /etc/nginx/certs/server.crt;
ssl_certificate_key /etc/nginx/certs/server.key;
ssl_client_certificate /etc/nginx/certs/ca.crt;
ssl_verify_client on;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-Client-Verify $ssl_client_verify;
}
}
Because nginx sets X-Client-DN itself, a client cannot spoof it. The backend must listen on 127.0.0.1 only. Test both cases and compare the status codes:
nginx -t && systemctl reload nginx
curl -sS -o /dev/null -w '%{http_code}\n' --cacert ca.crt https://api.internal.example/
curl -sS -o /dev/null -w '%{http_code}\n' --cacert ca.crt --cert client.crt --key client.key https://api.internal.example/
The first request, without a certificate, should be refused with a 4xx status. The second should return what the backend answers. Note that with ssl_verify_client on the TLS handshake can complete, and nginx then returns the error response. The connection is not silently dropped at the TCP level.
4. SSH with key plus TOTP
Install the PAM module (the Debian and Ubuntu package name is shown) and enrol each admin user:
apt install libpam-google-authenticator
google-authenticator
In /etc/pam.d/sshd, comment out the @include common-auth line so that the password prompt is not used, and add:
auth required pam_google_authenticator.so
Then set /etc/ssh/sshd_config:
UsePAM yes
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
ClientAliveInterval 300
ClientAliveCountMax 2
AllowUsers admin deploy
The original idea of using ClientAliveCountMax 0 as an idle timeout is wrong, because the value 0 disables termination. With the values above, an unresponsive client is disconnected after about 15 minutes. Idle but responsive sessions stay open. An idle limit for shells needs a separate mechanism, such as TMOUT in the shell profile.
Validate the file and reload, keeping the current session open until a second login works:
sshd -t && systemctl reload sshd
5. Optional: SSH certificates
To replace static keys, create a CA key, sign each user key with a short validity and trust the CA on the servers:
ssh-keygen -t ed25519 -f ssh_ca
ssh-keygen -s ssh_ca -I admin-2021 -n admin -V +1d user_key.pub
echo 'TrustedUserCAKeys /etc/ssh/ssh_ca.pub' >> /etc/ssh/sshd_config
Limits and alternatives
What this achieves: database traffic is encrypted and reachable only through an authenticated tunnel, internal HTTP callers need a certificate from the internal CA, and administrators need both a key and a one-time code.
The limits are as follows:
- WireGuard authenticates hosts, not services. Any process on an authorised host can use the tunnel. Per-service identity still needs TLS or mTLS at the application level.
- The static peer lists do not scale well beyond a small number of nodes. Beyond that point, use configuration management or a mesh tool that distributes keys.
- The manual openssl commands show the mechanism, not a rotation process. Short-lived certificates need automation, for example a tool such as cfssl or a PKI service, and nginx can only reject revoked certificates with a configured CRL (
ssl_crl) or by relying on very short lifetimes. - TOTP on a phone protects against stolen keys but not against a compromised admin machine or phone.
Choose another method when interoperability with existing gateways matters (IPsec), when a kernel older than 5.6 cannot take the module (OpenVPN or TLS in the service), or when many services call each other (a service mesh with automatic mTLS). Hosting location and the legal basis for data transfers remain decisions outside these configurations.