The problem
A microservice system often needs a broker that stores messages on disk and delivers them at least once. It should not bring a JVM and a separate coordination service along. NATS with JetStream fits that need, because persistence is a feature of the same nats-server binary. In 2021 that meant server 2.x and the Go client nats.go 1.13.
Setups built from copied snippets tend to show the same symptoms:
- A three-node cluster is running, but a stream becomes unavailable when one particular node goes down.
- Nodes refuse to join the cluster, or the cluster accepts route connections from anyone who can reach port 6222.
- Messages that were processed and acknowledged are still stored in the stream, so the stream keeps growing.
- Unacknowledged messages come back repeatedly.
- The monitoring URL on port 8222 does not answer.
- Clients drop connections under load, or leader elections repeat on slow disks.
Why it happens
Each symptom traces back to a misunderstanding of how JetStream works.
- Replica count is a property of the stream. A stream created with
Replicas: 1lives on exactly one server, whatever the size of the cluster. Losing that server makes the stream unavailable. Clustering the servers does not replicate the data. - Consensus needs a majority. JetStream replicates stream and consumer state with Raft. Three servers tolerate the loss of one. Two servers tolerate none, because a majority of two is two. A stream cannot be created with more replicas than there are servers available.
- Identity must be unique. Every server needs its own
server_name, and all members share one clustername. Copying one configuration file to all nodes unchanged breaks this. - Client authentication does not protect routes. The top-level
authorizationblock applies to clients. Route connections on the cluster port are configured separately, insidecluster. - An ack is not a delete. The default retention policy of a stream is limits-based, so messages stay until a limit (age, size, count) removes them. An ack only tells the consumer state that the message was handled. A message that is not acknowledged within the consumer's ack wait is redelivered.
- The monitoring port is off unless enabled. Exposing port 8222 in a container does nothing if the server was not started with a monitoring port.
- Raft and the stream store write to disk. High disk latency delays acknowledgements and can cause missed heartbeats and elections. File descriptor limits cap the number of client connections.
Some claims commonly repeated around this topic need correcting:
- JetStream is the successor to NATS Streaming, but the two are separate systems. Moving between them means adopting the JetStream API.
- Kafka releases of this era rely on ZooKeeper for cluster metadata. That is an operational difference, not a measured ranking, so no size or start-up figures are given here.
- A configuration file that mentions TLS 1.3 in a comment but has no
tlsblock does not encrypt anything.
Source of truth
The NATS documentation describes how JetStream is enabled and limited:
- Configuring JetStream (docs.nats.io) confirms that JetStream is enabled with a top-level
jetstreamsetting or the-jsand-sd(store directory) command-line flags. It also confirms that ajetstreamblock carriesstore_dirand the memory and file limits, and that account-level limits can restrict what a tenant may use. - The jetstream configuration reference lists
store_dir,max_memory_storeandmax_file_storewith their meaning. The memory limit defaults to 75% of available memory, and the file store limit is derived from available disk space when unset. - The nats.docs repository holds the same page as source, so changes can be traced in version control.
These pages describe current releases. Check option names against the documentation that matches the server version in use. The options in this article are the ones used by the 2.x line of 2021.
The remaining statements in this article rest on general NATS and Raft behaviour: stream replicas, route authorization, retention policies and ack semantics. They are not tied to a single source above, so check them against the documentation of the installed version.
Ways to fix it
| Approach | Strengths | Weaknesses |
|---|---|---|
| Single node with JetStream, file storage, replica 1 | Simplest to run, a single binary and one data directory. | The node is a single point of failure. Backups of the store directory are the only protection. |
| Three-node cluster, streams with 3 replicas | Survives the loss of one node. Raft handles leader election. | Needs three hosts and fast, low-latency local disks. Writes wait for a majority. |
| NATS core only, with application-level retries | Minimal moving parts and very light. | Delivery is at most once. Nothing is stored for offline consumers. |
| Another broker (Kafka, RabbitMQ) | Mature ecosystems. Kafka suits large replayable logs and RabbitMQ suits rich routing. | More components to run (Kafka needs ZooKeeper in this era), or a different delivery model. |
Within JetStream there are two further choices:
- Retention can be limits-based (a replayable log) or work-queue style (messages removed once acknowledged).
- Consumers can be push-based (the server delivers) or pull-based (the client asks for batches).
The chosen fix
The chosen setup is a three-node cluster with these properties:
- Streams are created with 3 replicas, on three separate hosts.
- Each node has a unique
server_name, a shared cluster name, separate credentials for routes and for clients, and atlsblock. - Stream retention is work-queue style, so acknowledged messages are removed.
- Consumers are durable pull consumers with explicit acks.
A cluster gives real tolerance of a single node failure, which a single node cannot. A work-queue stream matches the common pattern of one logical consumer per message, such as order processing. A pull consumer lets the application decide how many messages to take at a time, so a slow worker is not flooded. If several independent services must each see every message, use a limits-based stream with one durable consumer per service instead.
A single node remains the right choice for development and small non-critical setups. It differs only in that it omits the cluster block and uses replica 1.
Step by step
This is a procedure to follow, not a recording of a run. No command output is reproduced. Each verification command says what to look for.
1. Prepare the hosts
Use three hosts named nats-01, nats-02 and nats-03 that resolve to each other. Allow port 4222 from the application hosts and port 6222 only between the three nodes. Keep 8222 on loopback. Put the JetStream store directory on local SSD or NVMe storage. Set max_file_store below the real size of that volume.
2. Write the server configuration
On each node, change only server_name. Create the certificate files first. Replace the placeholder passwords.
# /etc/nats/nats-server.conf (nats-01; change server_name on the others)
server_name: nats-01
listen: 0.0.0.0:4222
http: 127.0.0.1:8222
jetstream {
store_dir: /data/nats/jetstream
max_memory_store: 1G
max_file_store: 90G
}
tls {
cert_file: /etc/nats/tls/server.crt
key_file: /etc/nats/tls/server.key
}
authorization {
user: app
password: CHANGE_ME_client
timeout: 2
}
cluster {
name: orders-cluster
listen: 0.0.0.0:6222
authorization {
user: route
password: CHANGE_ME_route
timeout: 2
}
routes: [
nats-route://route:CHANGE_ME_route@nats-01:6222
nats-route://route:CHANGE_ME_route@nats-02:6222
nats-route://route:CHANGE_ME_route@nats-03:6222
]
}
Route traffic can also be encrypted with a tls block inside cluster. The certificates must then be valid for the node host names.
3. Run the server with enough file descriptors
With Docker, host networking lets the loopback-only monitoring port be reached from the host:
docker run -d --name nats --restart always --network host \
--ulimit nofile=65535:65535 \
-v /etc/nats/nats-server.conf:/etc/nats/nats-server.conf:ro \
-v /etc/nats/tls:/etc/nats/tls:ro \
-v /data/nats/jetstream:/data/nats/jetstream \
nats:2.6-alpine -c /etc/nats/nats-server.conf
For a quick single-node test the flags are enough, with no configuration file:
docker run -d --name nats-dev -p 4222:4222 -p 8222:8222 -v nats-dev-data:/data nats:2.6-alpine -js -sd /data -m 8222
When the server runs under systemd instead, raise the limit in a drop-in created with systemctl edit nats-server:
[Service]
LimitNOFILE=65535
4. Check the cluster
On each node, the routes endpoint should list the two other servers. The JetStream endpoint should show JetStream enabled with the configured limits.
curl -s http://127.0.0.1:8222/routez
curl -s http://127.0.0.1:8222/jsz
5. Create the stream and publish
The stream is created from the Go client. The call is safe to repeat as long as the configuration is identical. The MsgId option lets the server drop duplicates inside the stream's duplicate window, which makes retried publishes safe.
package main
import (
"log"
"github.com/nats-io/nats.go"
)
func main() {
nc, err := nats.Connect(
"tls://nats-01:4222,tls://nats-02:4222,tls://nats-03:4222",
nats.UserInfo("app", "CHANGE_ME_client"),
nats.RootCAs("/etc/nats/tls/ca.crt"),
)
if err != nil {
log.Fatal(err)
}
defer nc.Close()
js, err := nc.JetStream()
if err != nil {
log.Fatal(err)
}
_, err = js.AddStream(&nats.StreamConfig{
Name: "ORDERS",
Subjects: []string{"ORDERS.*"},
Storage: nats.FileStorage,
Retention: nats.WorkQueuePolicy,
Replicas: 3,
})
if err != nil {
log.Fatal(err)
}
ack, err := js.Publish("ORDERS.created", []byte(`{"id": 101}`), nats.MsgId("order-101"))
if err != nil {
log.Fatal(err)
}
log.Printf("stored in stream %s at sequence %d", ack.Stream, ack.Sequence)
}
6. Consume with a durable pull consumer
Acknowledge only after the work has succeeded. A timeout from Fetch means no message arrived in the wait period, which is normal on an idle stream.
sub, err := js.PullSubscribe("ORDERS.created", "order-processor")
if err != nil {
log.Fatal(err)
}
for {
msgs, err := sub.Fetch(10, nats.MaxWait(2*time.Second))
if err != nil {
if err == nats.ErrTimeout {
continue
}
log.Fatal(err)
}
for _, m := range msgs {
log.Printf("processing %s", string(m.Data))
m.Ack()
}
}
This loop also needs time in the import list. Use m.Nak() to request an immediate redelivery after a failure.
7. Inspect state from the command line
The nats CLI shows the replica placement and the consumer's pending and redelivered counts:
nats --server tls://nats-01:4222 --user app --password CHANGE_ME_client --tlscacert /etc/nats/tls/ca.crt stream info ORDERS
nats --server tls://nats-01:4222 --user app --password CHANGE_ME_client --tlscacert /etc/nats/tls/ca.crt consumer info ORDERS order-processor
The stream information should list a leader and two followers. After stopping the container on one node, a new leader should be elected and publishing should continue.
8. Optional host tuning
The kernel file limit should not be lower than the per-process limit. BBR needs kernel 4.9 or newer, and the buffer values only matter for high-bandwidth links. Test any network change before keeping it.
# /etc/sysctl.d/90-nats.conf
fs.file-max = 100000
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
sysctl --system
sysctl net.ipv4.tcp_congestion_control
Limits and alternatives
What this setup provides:
- Streams with three replicas survive the loss of any single node while two remain.
- Route and client credentials are separate.
- Acked messages leave a work-queue stream.
- Retried publishes can be de-duplicated.
Its limits:
- The cluster does not tolerate two simultaneous node losses, and a replicated write waits for a majority. Keep the nodes in one low-latency network, and do not stretch one cluster across distant sites.
- Delivery is at least once, so consumers must be idempotent. De-duplication on publish does not remove duplicates caused by redelivery.
- Token or password authentication is the simplest model. Larger deployments with many tenants should look at accounts and per-account JetStream limits, described in the configuration page linked above.
- Data location follows the hosts. Where the servers run decides where persisted messages are stored, which matters for data-residency requirements. This article makes no legal claim beyond that.
Choose a single node when downtime is acceptable and backups are enough. Choose a limits-based stream with multiple durable consumers when several services need the same events. Choose Kafka when very large retained logs and an existing stream-processing ecosystem are the main requirement. Choose RabbitMQ when complex routing between queues dominates.