Console Login

Keeping Prometheus Monitoring Alive During Traffic Spikes

The problem

A monitoring stack is usually the first thing to lose data during a traffic spike, which is when the data is needed most. A typical setup is a single Prometheus server scraping many hosts, with Grafana dashboards on top. Under load it shows a recognisable pattern:

  • Scrapes start to time out and graphs show holes exactly during the incident.
  • Dashboards that were fast on a quiet day take many seconds to render, because every refresh recomputes expensive queries over long ranges.
  • Disk wait on the monitoring host climbs while the server compacts data.
  • Exporters publish far more series than anyone looks at, so storage and network traffic grow without any benefit.
  • Exporter ports such as 9100 and 9090 answer on every interface and reveal kernel versions, load and internal layout to anyone who can reach them.

These are separate faults that tend to appear together. The sections below treat them as one reliability problem: the monitoring system must keep ingesting, answering queries and staying private while the monitored systems are under stress.

Why it happens

Write-heavy storage

Prometheus keeps recent samples in memory and appends them to a write-ahead log (WAL) on disk. Periodically the in-memory head is compacted into immutable blocks, and old blocks are merged and deleted according to retention settings. Every sample therefore causes sequential WAL writes, and compaction adds bursts of read and write activity. If the disk has high or unpredictable latency, which is common with shared or network-attached volumes, ingestion stalls and scrapes are missed.

One server doing everything

A single Prometheus instance scrapes, stores, evaluates rules and serves every dashboard query. Memory and disk use grow with the number of active series, and the instance has no built-in clustering. Once it is saturated, nothing else in the stack can compensate.

Default exporters collect everything enabled

node_exporter enables a broad default set of collectors. Some of them (for example ARP or hardware-specific ones) produce series nobody queries. Collectors that are off by default, such as the systemd collector, are off precisely because of cardinality or cost, so enabling them blindly is the opposite mistake.

Queries recomputed on every refresh

A panel that evaluates rate(http_requests_total[5m]) summed across many series does that work again on every dashboard refresh and for every viewer. During an incident the number of viewers goes up, so the read load on the disk goes up exactly when the write load does.

Listening on all interfaces

A listen address of :9100 binds every interface. Without a host firewall or a private bind address, the metrics endpoint is open to whatever network the host is attached to.

Source of truth

The fix relies on three upstream documents.

  • Prometheus storage documentation. It confirms that local storage is protected against crashes by a write-ahead log that is replayed on restart. It also confirms that compaction keeps the source blocks and the new block on disk at the same time, so disk usage temporarily exceeds the steady state. It documents the --storage.tsdb.wal-compression flag and the retention flags (--storage.tsdb.retention.time and --storage.tsdb.retention.size), and notes that after enabling WAL compression, downgrading below 2.11.0 requires deleting the WAL.
  • Prometheus federation documentation. It confirms that hierarchical federation lets Prometheus scale to many data centers by having higher-level servers collect aggregated series from lower-level ones. It also provides the reference scrape configuration with honor_labels: true, metrics_path: '/federate' and match[] selectors.
  • node_exporter README. It confirms that collectors are enabled with --collector.<name>, that default collectors can be turned off with --no-collector.<name>, and that --collector.disable-defaults followed by explicit --collector.<name> flags enables only the chosen set. It also says collectors that are off by default may be high-cardinality or expensive, and recommends watching scrape_duration_seconds and scrape_samples_post_metric_relabeling after any change.

Ways to fix it

MethodHelps withCost or risk
Faster, local, low-latency storage for the Prometheus data directoryWAL writes and compaction stallsNeeds a suitable host or volume; does not reduce series count
Reduce what is collected (--collector.disable-defaults, relabeling, dropping unused series)Disk growth, payload size, query costDashboards and alerts that use a removed series stop working
Split scraping across several Prometheus servers and federate aggregatesSingle-server saturationMore servers to operate; the global server sees only what is federated
Recording rules for expensive queriesSlow dashboards and read loadRules must be maintained; new queries need new rules
Bind exporters to private addresses and add host firewall rulesInformation exposureA wrong rule can block scraping or lock out SSH
Remote storage or a long-term layer such as Thanos or CortexRetention beyond one server, global queries over full dataConsiderably more moving parts

Why this fix

The chosen combination is: local fast storage with WAL compression, a lean exporter, zone-level Prometheus servers with a federating global server, recording rules for dashboard queries, and a firewall plus private bind addresses. The reasons:

  • Each part addresses one cause listed above and can be applied independently, so a failure in one change does not hide the effect of the others.
  • All of it uses stock Prometheus, node_exporter and the host firewall. There is no new component to learn during an incident.
  • Federation matches the documented use: the global server gets aggregated job:* series, not every raw series, so its load stays small.
  • Heavier options (remote storage, Thanos, Cortex) are worth adding only when retention or global raw-data queries demand them.

Two corrections to common claims are worth stating. Federation does not buffer data for the global server: if the global server cannot reach a zone server it simply has a gap in its own copy, while the zone server keeps its full data locally. Recording rules are also not computed at ingestion; they are evaluated at the rule evaluation interval and stored as new series.

Step by step

The commands below are instructions to adapt to the host; addresses are placeholders. Compare the output on your own system before and after each step. No output is reproduced here because it depends entirely on hardware and workload.

1. Measure storage behaviour first

Watch disk latency and wait while the server is under load, so the effect of later changes can be judged.

iostat -x 5

2. Enable WAL compression and set retention

Add flags to the Prometheus service and restart it. Keep the data directory on local, POSIX-compliant storage.

/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --storage.tsdb.wal-compression

Leave free space for compaction, because the old and new blocks coexist briefly. If --storage.tsdb.retention.size is used, set it below the capacity of the volume.

3. Run a lean node_exporter bound to a private address

Replace the address with the host's private IP. The collector list is an example; keep any collector that a dashboard or alert uses.

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/node_exporter \
    --collector.disable-defaults \
    --collector.cpu \
    --collector.meminfo \
    --collector.loadavg \
    --collector.filesystem \
    --collector.netdev \
    --collector.diskstats \
    --web.listen-address=10.10.1.5:9100

[Install]
WantedBy=multi-user.target

Count the exposed node series before and after the change to see the reduction, then reload the unit.

curl -s http://10.10.1.5:9100/metrics | grep -c '^node_'
systemctl daemon-reload
systemctl restart node_exporter
curl -s http://10.10.1.5:9100/metrics | grep -c '^node_'

4. Add recording rules on each zone server

Reference a rule file from prometheus.yml:

rule_files:
  - /etc/prometheus/rules/*.yml

The rule file below uses the level:metric:operation naming convention, so the job: prefix is matched by the federation selector.

groups:
  - name: http_aggregates
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_inprogress_requests:sum
        expr: sum by (job) (http_inprogress_requests)

Validate both files, then reload or restart Prometheus.

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml
systemctl restart prometheus

Point Grafana panels at the recorded series (for example job:http_requests:rate5m) instead of the raw expression.

5. Federate the aggregates into the global server

On the global server, scrape the /federate endpoint of each zone server, as in the upstream example:

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'
    static_configs:
      - targets:
        - '10.10.1.5:9090'
        - '10.10.1.6:9090'

Test what a zone server returns for the selector before relying on it:

curl -sG 'http://10.10.1.5:9090/federate' --data-urlencode 'match[]={__name__=~"job:.*"}' | head

6. Restrict access with a host firewall

Only the scraping server needs to reach the exporter port, and only the global server needs to reach a zone server's port 9090. Allow SSH first so enabling the firewall does not cut off the session. Use the scraper's real address in place of the placeholder below.

ufw default deny incoming
ufw allow 22/tcp
ufw allow from 192.168.10.50 to any port 9100 proto tcp
ufw enable
ufw status verbose

On a zone Prometheus host, add a similar rule for port 9090 with the global server's address. Confirm the listeners are bound as intended:

ss -tlnp | grep -E ':(9090|9100)'

Limits and alternatives

What this achieves: the write path is less likely to stall, each server holds fewer series, dashboards read precomputed series, and the metric ports are reachable only from known addresses. What it does not achieve:

  • Not high availability. Federation is a scaling pattern, not redundancy. A zone server that dies loses its scraping for the duration. For redundancy, run two identical servers that scrape the same targets and alert from both.
  • Aggregates only. The global server sees only the series matched by match[]. Federating all raw series defeats the purpose and overloads it. For queries across all raw data, or retention beyond one server, use remote write or a long-term layer such as Thanos or Cortex.
  • Collector trimming is a trade-off. Removing collectors removes series that someone may need later. Check dashboards and alert rules before trimming, and watch scrape_duration_seconds and scrape_samples_post_metric_relabeling after enabling or disabling anything.
  • Firewall rules are not authentication. For access across untrusted networks, add TLS and authentication in front of the endpoints, or use a VPN.
  • Personal data in metrics. Labels such as client IPs or user IDs can carry personal data and also inflate cardinality. Keep them out of metric labels, and decide where metric data is stored with that in mind.