The problem
A monitoring stack is usually the first thing to lose data during a traffic spike, which is when the data is needed most. A typical setup is a single Prometheus server scraping many hosts, with Grafana dashboards on top. Under load it shows a recognisable pattern:
- Scrapes start to time out and graphs show holes exactly during the incident.
- Dashboards that were fast on a quiet day take many seconds to render, because every refresh recomputes expensive queries over long ranges.
- Disk wait on the monitoring host climbs while the server compacts data.
- Exporters publish far more series than anyone looks at, so storage and network traffic grow without any benefit.
- Exporter ports such as 9100 and 9090 answer on every interface and reveal kernel versions, load and internal layout to anyone who can reach them.
These are separate faults that tend to appear together. The sections below treat them as one reliability problem: the monitoring system must keep ingesting, answering queries and staying private while the monitored systems are under stress.
Why it happens
Write-heavy storage
Prometheus keeps recent samples in memory and appends them to a write-ahead log (WAL) on disk. Periodically the in-memory head is compacted into immutable blocks, and old blocks are merged and deleted according to retention settings. Every sample therefore causes sequential WAL writes, and compaction adds bursts of read and write activity. If the disk has high or unpredictable latency, which is common with shared or network-attached volumes, ingestion stalls and scrapes are missed.
One server doing everything
A single Prometheus instance scrapes, stores, evaluates rules and serves every dashboard query. Memory and disk use grow with the number of active series, and the instance has no built-in clustering. Once it is saturated, nothing else in the stack can compensate.
Default exporters collect everything enabled
node_exporter enables a broad default set of collectors. Some of them (for example ARP or hardware-specific ones) produce series nobody queries. Collectors that are off by default, such as the systemd collector, are off precisely because of cardinality or cost, so enabling them blindly is the opposite mistake.
Queries recomputed on every refresh
A panel that evaluates rate(http_requests_total[5m]) summed across many series does that work again on every dashboard refresh and for every viewer. During an incident the number of viewers goes up, so the read load on the disk goes up exactly when the write load does.
Listening on all interfaces
A listen address of :9100 binds every interface. Without a host firewall or a private bind address, the metrics endpoint is open to whatever network the host is attached to.
Source of truth
The fix relies on three upstream documents.
- Prometheus storage documentation. It confirms that local storage is protected against crashes by a write-ahead log that is replayed on restart. It also confirms that compaction keeps the source blocks and the new block on disk at the same time, so disk usage temporarily exceeds the steady state. It documents the
--storage.tsdb.wal-compressionflag and the retention flags (--storage.tsdb.retention.timeand--storage.tsdb.retention.size), and notes that after enabling WAL compression, downgrading below 2.11.0 requires deleting the WAL. - Prometheus federation documentation. It confirms that hierarchical federation lets Prometheus scale to many data centers by having higher-level servers collect aggregated series from lower-level ones. It also provides the reference scrape configuration with
honor_labels: true,metrics_path: '/federate'andmatch[]selectors. - node_exporter README. It confirms that collectors are enabled with
--collector.<name>, that default collectors can be turned off with--no-collector.<name>, and that--collector.disable-defaultsfollowed by explicit--collector.<name>flags enables only the chosen set. It also says collectors that are off by default may be high-cardinality or expensive, and recommends watchingscrape_duration_secondsandscrape_samples_post_metric_relabelingafter any change.
Ways to fix it
| Method | Helps with | Cost or risk |
|---|---|---|
| Faster, local, low-latency storage for the Prometheus data directory | WAL writes and compaction stalls | Needs a suitable host or volume; does not reduce series count |
Reduce what is collected (--collector.disable-defaults, relabeling, dropping unused series) | Disk growth, payload size, query cost | Dashboards and alerts that use a removed series stop working |
| Split scraping across several Prometheus servers and federate aggregates | Single-server saturation | More servers to operate; the global server sees only what is federated |
| Recording rules for expensive queries | Slow dashboards and read load | Rules must be maintained; new queries need new rules |
| Bind exporters to private addresses and add host firewall rules | Information exposure | A wrong rule can block scraping or lock out SSH |
| Remote storage or a long-term layer such as Thanos or Cortex | Retention beyond one server, global queries over full data | Considerably more moving parts |
Why this fix
The chosen combination is: local fast storage with WAL compression, a lean exporter, zone-level Prometheus servers with a federating global server, recording rules for dashboard queries, and a firewall plus private bind addresses. The reasons:
- Each part addresses one cause listed above and can be applied independently, so a failure in one change does not hide the effect of the others.
- All of it uses stock Prometheus,
node_exporterand the host firewall. There is no new component to learn during an incident. - Federation matches the documented use: the global server gets aggregated
job:*series, not every raw series, so its load stays small. - Heavier options (remote storage, Thanos, Cortex) are worth adding only when retention or global raw-data queries demand them.
Two corrections to common claims are worth stating. Federation does not buffer data for the global server: if the global server cannot reach a zone server it simply has a gap in its own copy, while the zone server keeps its full data locally. Recording rules are also not computed at ingestion; they are evaluated at the rule evaluation interval and stored as new series.
Step by step
The commands below are instructions to adapt to the host; addresses are placeholders. Compare the output on your own system before and after each step. No output is reproduced here because it depends entirely on hardware and workload.
1. Measure storage behaviour first
Watch disk latency and wait while the server is under load, so the effect of later changes can be judged.
iostat -x 52. Enable WAL compression and set retention
Add flags to the Prometheus service and restart it. Keep the data directory on local, POSIX-compliant storage.
/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=15d \
--storage.tsdb.wal-compressionLeave free space for compaction, because the old and new blocks coexist briefly. If --storage.tsdb.retention.size is used, set it below the capacity of the volume.
3. Run a lean node_exporter bound to a private address
Replace the address with the host's private IP. The collector list is an example; keep any collector that a dashboard or alert uses.
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/node_exporter \
--collector.disable-defaults \
--collector.cpu \
--collector.meminfo \
--collector.loadavg \
--collector.filesystem \
--collector.netdev \
--collector.diskstats \
--web.listen-address=10.10.1.5:9100
[Install]
WantedBy=multi-user.targetCount the exposed node series before and after the change to see the reduction, then reload the unit.
curl -s http://10.10.1.5:9100/metrics | grep -c '^node_'
systemctl daemon-reload
systemctl restart node_exporter
curl -s http://10.10.1.5:9100/metrics | grep -c '^node_'4. Add recording rules on each zone server
Reference a rule file from prometheus.yml:
rule_files:
- /etc/prometheus/rules/*.ymlThe rule file below uses the level:metric:operation naming convention, so the job: prefix is matched by the federation selector.
groups:
- name: http_aggregates
rules:
- record: job:http_requests:rate5m
expr: sum by (job) (rate(http_requests_total[5m]))
- record: job:http_inprogress_requests:sum
expr: sum by (job) (http_inprogress_requests)Validate both files, then reload or restart Prometheus.
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml
systemctl restart prometheusPoint Grafana panels at the recorded series (for example job:http_requests:rate5m) instead of the raw expression.
5. Federate the aggregates into the global server
On the global server, scrape the /federate endpoint of each zone server, as in the upstream example:
scrape_configs:
- job_name: 'federate'
scrape_interval: 15s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
static_configs:
- targets:
- '10.10.1.5:9090'
- '10.10.1.6:9090'Test what a zone server returns for the selector before relying on it:
curl -sG 'http://10.10.1.5:9090/federate' --data-urlencode 'match[]={__name__=~"job:.*"}' | head6. Restrict access with a host firewall
Only the scraping server needs to reach the exporter port, and only the global server needs to reach a zone server's port 9090. Allow SSH first so enabling the firewall does not cut off the session. Use the scraper's real address in place of the placeholder below.
ufw default deny incoming
ufw allow 22/tcp
ufw allow from 192.168.10.50 to any port 9100 proto tcp
ufw enable
ufw status verboseOn a zone Prometheus host, add a similar rule for port 9090 with the global server's address. Confirm the listeners are bound as intended:
ss -tlnp | grep -E ':(9090|9100)'Limits and alternatives
What this achieves: the write path is less likely to stall, each server holds fewer series, dashboards read precomputed series, and the metric ports are reachable only from known addresses. What it does not achieve:
- Not high availability. Federation is a scaling pattern, not redundancy. A zone server that dies loses its scraping for the duration. For redundancy, run two identical servers that scrape the same targets and alert from both.
- Aggregates only. The global server sees only the series matched by
match[]. Federating all raw series defeats the purpose and overloads it. For queries across all raw data, or retention beyond one server, use remote write or a long-term layer such as Thanos or Cortex. - Collector trimming is a trade-off. Removing collectors removes series that someone may need later. Check dashboards and alert rules before trimming, and watch
scrape_duration_secondsandscrape_samples_post_metric_relabelingafter enabling or disabling anything. - Firewall rules are not authentication. For access across untrusted networks, add TLS and authentication in front of the endpoints, or use a VPN.
- Personal data in metrics. Labels such as client IPs or user IDs can carry personal data and also inflate cardinality. Keep them out of metric labels, and decide where metric data is stored with that in mind.