Console Login

Self-Hosted Latency Monitoring with Prometheus, Grafana and Nginx

The problem

A site that feels fast when clicked through from a good connection says little about what users see. Without per-request timing, a slow page cannot be traced to its cause. Possible causes are the application, the network, the database, or a starved virtual machine. Typical symptoms:

  • Users report sluggishness, but the average response time on a dashboard looks fine.
  • Graphs have gaps, or the monitoring server itself becomes slow when the disk is busy.
  • Response times jitter although the application code has not changed.
  • The usual hosted APM agents send request data, including client IP addresses, to a third-party service outside the operator's control.

The last point matters in Europe. The Court of Justice of the EU invalidated the EU-US Privacy Shield in the Schrems II judgment of July 2020. Since then, sending personal data to US-hosted services needs a careful legal assessment. IP addresses are generally treated as personal data. Keeping the monitoring stack on infrastructure the operator controls removes one transfer to assess. It does not make a setup compliant by itself, and this article is not legal advice.

Why it happens

Several independent causes tend to overlap:

  • Default logs lack timing. The default Nginx access log does not record how long a request took or how long the upstream application needed. Without both numbers, the application cannot be separated from slow clients.
  • Averages hide tails. A mean latency can look healthy while the 99th percentile is poor. Percentiles require histogram data, not a single gauge.
  • Virtualisation adds a hidden variable. On a virtual machine, CPU steal time is the share of time a virtual CPU was ready to run but the hypervisor ran something else. High steal produces latency jitter that no application change fixes.
  • Storage is chosen by folklore. Prometheus is often blamed for being slow when the disk is network-mounted, throttled or shared. Prometheus keeps recent samples in memory with a write-ahead log. It periodically compacts them into on-disk blocks, so it needs a reliable local filesystem. Fast storage helps with large ingestion rates, but an NVMe drive is not a hard requirement. The sizing depends on the number of series and the sample rate.
  • Telemetry itself contains personal data. Raw client IPs and full request URIs, with their query strings, end up in logs and, through them, in monitoring systems.

Source of truth

The storage behaviour that the sizing and retention advice relies on is documented in the Prometheus manual: Prometheus storage documentation. It confirms the following:

  • Prometheus includes a local on-disk time series database, configured with --storage.tsdb.path.
  • Retention is set with --storage.tsdb.retention.time, which defaults to 15 days when no time or size retention is given. --storage.tsdb.retention.size caps the block size instead. If both are set, whichever triggers first applies.
  • A sample takes on average only about 1-2 bytes. Disk capacity can therefore be estimated as retention time in seconds, times ingested samples per second, times bytes per sample.
  • Non-POSIX-compliant filesystems are not supported for local storage, because unrecoverable corruption may happen. A local filesystem is strongly recommended.
  • Retention size should be at most about 80-85% of the allocated disk. Compaction can briefly push on-disk usage above the configured size.

The meaning of the Nginx timing variables ($request_time, $upstream_response_time), PromQL functions such as predict_linear and histogram_quantile, and the node exporter CPU metrics is defined in each project's own manual. Read those pages before relying on the queries below.

Ways to fix it

MethodStrengthsWeaknesses
Hosted SaaS APM agentLittle setup, traces includedData leaves the operator's control, sampling is common, and cost grows with volume
Prometheus + Grafana + node exporter, plus Nginx timing logsSelf-hosted, open source, flexible queries and alert rulesThe operator runs it, and application-level histograms need instrumentation
Log-centric stack (a log shipper plus a search store)Full per-request detailHeavy on RAM and disk, and the logs contain personal data that must be managed
Nginx logs only, analysed with command-line toolsNo new servicesNo history, graphs or alerting

Within the self-hosted option, alerting can live in Prometheus rules with Alertmanager. It can also live in Grafana 8, whose new alerting system is opt-in behind a feature toggle in 8.0. Legacy dashboard alerts remain the default there.

Why this fix

The chosen combination is Nginx timing logs, Prometheus for metrics, node exporter for host metrics, and Grafana for dashboards. Alert rules are kept in Prometheus.

  • It separates the three questions: how long the app took ($upstream_response_time), how long the whole request took ($request_time), and whether the host is healthy (steal, I/O wait, disk).
  • All data stays on a server the operator chooses, so no third party receives request data. Client IPs are truncated before they reach the log.
  • Rules in Prometheus are plain files that can be validated with promtool and kept in version control. This avoids depending on a feature that is still opt-in in Grafana 8.0.
  • Pinned container image tags give reproducible, portable deployment. Host networking skips the Docker bridge, and every listener is bound to localhost so that nothing is exposed by accident.

Step by step

1. Add timing and privacy-aware logging to Nginx

In the http block, truncate the client address and define a JSON log format. $uri is used instead of $request_uri so that query strings, which may carry personal data, are not logged. Values stay quoted because $upstream_response_time can be - or hold several comma-separated values, which would break unquoted JSON numbers.

map $remote_addr $ip_anonymized {
    default 0.0.0.0;
    "~(?P<ip>(\d+)\.(\d+)\.(\d+))\.\d+" "$ip.0";
    "~(?P<ip>[^:]+:[^:]+):" "$ip::";
}

log_format apm_json escape=json
  '{"time":"$time_iso8601",'
  '"remote_addr":"$ip_anonymized",'
  '"request_time":"$request_time",'
  '"upstream_response_time":"$upstream_response_time",'
  '"status":"$status",'
  '"method":"$request_method",'
  '"uri":"$uri"}';

access_log /var/log/nginx/access_apm.log apm_json;
sudo nginx -t && sudo systemctl reload nginx

When request_time is high but upstream_response_time is low, the time went into the client's network or into sending the response. When both are high, the application or its backends are slow. The slowest requests can be listed with:

jq -r '[.request_time, .upstream_response_time, .uri] | @tsv' /var/log/nginx/access_apm.log | sort -rn | head -20

2. Configure Prometheus

sudo mkdir -p /etc/prometheus /var/lib/prometheus_data
sudo chown 65534:65534 /var/lib/prometheus_data

The official image runs as the unprivileged nobody user (UID 65534), so the data directory must be writable by it. Then create /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/alerts.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'my_app_production'
    static_configs:
      - targets: ['10.8.0.5:3000']

A 15-second interval is a sound default. A 1-second interval is possible but multiplies the samples and the disk use by fifteen for every target. It should be reserved for a few series that need it.

3. Add alert rules

Create /etc/prometheus/alerts.yml. The latency rule assumes the application exposes a histogram named http_request_duration_seconds, so adjust the metric name to the client library in use.

groups:
  - name: apm
    rules:
      - alert: DiskFullIn4Hours
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[1h], 4 * 3600) < 0
        for: 15m
        labels:
          severity: warning
      - alert: HighCpuSteal
        expr: avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m])) > 0.02
        for: 10m
        labels:
          severity: warning
      - alert: HighP99Latency
        expr: histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > 0.5
        for: 5m
        labels:
          severity: critical

The 2% steal threshold and the 500 ms latency limit are starting points, not universal values. Tune them to the workload. Validate both files before starting:

docker run --rm --entrypoint promtool -v /etc/prometheus:/etc/prometheus:ro prom/prometheus:v2.28.1 check config /etc/prometheus/prometheus.yml

4. Start the services

docker run -d --name prometheus --network host \
  -v /etc/prometheus:/etc/prometheus:ro \
  -v /var/lib/prometheus_data:/prometheus \
  prom/prometheus:v2.28.1 \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.listen-address=127.0.0.1:9090
docker run -d --name node_exporter --network host --pid host \
  -v /:/host:ro,rslave \
  quay.io/prometheus/node-exporter:v1.1.2 \
  --path.rootfs=/host \
  --web.listen-address=127.0.0.1:9100
docker run -d --name grafana --network host \
  -e GF_SERVER_HTTP_ADDR=127.0.0.1 \
  -v grafana_data:/var/lib/grafana \
  grafana/grafana:8.0.3

After a configuration change, reload Prometheus with docker kill --signal=HUP prometheus. The Targets page should show every job as UP.

5. Publish Grafana through Nginx and connect the data source

server {
    listen 443 ssl;
    server_name grafana.example.com;
    ssl_certificate     /etc/ssl/grafana.example.com.crt;
    ssl_certificate_key /etc/ssl/grafana.example.com.key;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;
    }
}

In Grafana, add a Prometheus data source pointing at http://localhost:9090, then import dashboard ID 1860 (Node Exporter Full) for a first overview of the host.

6. Check steal and disk latency on the host

iostat -c 1 5
iostat -x 1 5

The first command prints CPU shares including %steal. The second shows per-device wait times; persistently high write waits point at the storage, not at Prometheus. Both come from the sysstat package. Sustained steal of a few percent is a sign that the host is oversubscribed. The same signal is available as a graph through the node_cpu_seconds_total{mode="steal"} metric.

Limits and alternatives

  • Metrics are not traces. Prometheus shows rates and percentiles. It does not show where one specific request spent its time inside the application. For that, distributed tracing needs separate tooling.
  • Nginx logs are not metrics. The log file stays on disk for ad-hoc analysis. To graph it in Prometheus, either instrument the application with a client library that exposes histograms, or add a log-to-metrics exporter.
  • Single server, single point of failure. One Prometheus instance on local disk has no replication. Run a second instance scraping the same targets, or use remote write, when monitoring must survive the loss of the host. A monitoring server should also not be the only copy of the data it protects.
  • Storage still needs sizing. Estimate disk from series count and scrape interval, using the formula in the storage documentation. Keep retention size below about 85% of the volume. Do not place the data directory on a network filesystem.
  • Location does not equal compliance. Hosting in Norway or elsewhere in the EEA keeps data under a known jurisdiction, but the lawful basis, retention period and IP truncation still need a decision.
  • When to choose another method. A hosted APM may be reasonable when no personal data is involved and nobody can run the stack. A log-centric stack suits cases that need full per-request search despite the cost.