Console Login

NATS JetStream cluster for microservices: setup and pitfalls (2021)

The problem

A microservice system often needs a broker that stores messages on disk and delivers them at least once. It should not bring a JVM and a separate coordination service along. NATS with JetStream fits that need, because persistence is a feature of the same nats-server binary. In 2021 that meant server 2.x and the Go client nats.go 1.13.

Setups built from copied snippets tend to show the same symptoms:

  • A three-node cluster is running, but a stream becomes unavailable when one particular node goes down.
  • Nodes refuse to join the cluster, or the cluster accepts route connections from anyone who can reach port 6222.
  • Messages that were processed and acknowledged are still stored in the stream, so the stream keeps growing.
  • Unacknowledged messages come back repeatedly.
  • The monitoring URL on port 8222 does not answer.
  • Clients drop connections under load, or leader elections repeat on slow disks.

Why it happens

Each symptom traces back to a misunderstanding of how JetStream works.

  • Replica count is a property of the stream. A stream created with Replicas: 1 lives on exactly one server, whatever the size of the cluster. Losing that server makes the stream unavailable. Clustering the servers does not replicate the data.
  • Consensus needs a majority. JetStream replicates stream and consumer state with Raft. Three servers tolerate the loss of one. Two servers tolerate none, because a majority of two is two. A stream cannot be created with more replicas than there are servers available.
  • Identity must be unique. Every server needs its own server_name, and all members share one cluster name. Copying one configuration file to all nodes unchanged breaks this.
  • Client authentication does not protect routes. The top-level authorization block applies to clients. Route connections on the cluster port are configured separately, inside cluster.
  • An ack is not a delete. The default retention policy of a stream is limits-based, so messages stay until a limit (age, size, count) removes them. An ack only tells the consumer state that the message was handled. A message that is not acknowledged within the consumer's ack wait is redelivered.
  • The monitoring port is off unless enabled. Exposing port 8222 in a container does nothing if the server was not started with a monitoring port.
  • Raft and the stream store write to disk. High disk latency delays acknowledgements and can cause missed heartbeats and elections. File descriptor limits cap the number of client connections.

Some claims commonly repeated around this topic need correcting:

  • JetStream is the successor to NATS Streaming, but the two are separate systems. Moving between them means adopting the JetStream API.
  • Kafka releases of this era rely on ZooKeeper for cluster metadata. That is an operational difference, not a measured ranking, so no size or start-up figures are given here.
  • A configuration file that mentions TLS 1.3 in a comment but has no tls block does not encrypt anything.

Source of truth

The NATS documentation describes how JetStream is enabled and limited:

  • Configuring JetStream (docs.nats.io) confirms that JetStream is enabled with a top-level jetstream setting or the -js and -sd (store directory) command-line flags. It also confirms that a jetstream block carries store_dir and the memory and file limits, and that account-level limits can restrict what a tenant may use.
  • The jetstream configuration reference lists store_dir, max_memory_store and max_file_store with their meaning. The memory limit defaults to 75% of available memory, and the file store limit is derived from available disk space when unset.
  • The nats.docs repository holds the same page as source, so changes can be traced in version control.

These pages describe current releases. Check option names against the documentation that matches the server version in use. The options in this article are the ones used by the 2.x line of 2021.

The remaining statements in this article rest on general NATS and Raft behaviour: stream replicas, route authorization, retention policies and ack semantics. They are not tied to a single source above, so check them against the documentation of the installed version.

Ways to fix it

ApproachStrengthsWeaknesses
Single node with JetStream, file storage, replica 1Simplest to run, a single binary and one data directory.The node is a single point of failure. Backups of the store directory are the only protection.
Three-node cluster, streams with 3 replicasSurvives the loss of one node. Raft handles leader election.Needs three hosts and fast, low-latency local disks. Writes wait for a majority.
NATS core only, with application-level retriesMinimal moving parts and very light.Delivery is at most once. Nothing is stored for offline consumers.
Another broker (Kafka, RabbitMQ)Mature ecosystems. Kafka suits large replayable logs and RabbitMQ suits rich routing.More components to run (Kafka needs ZooKeeper in this era), or a different delivery model.

Within JetStream there are two further choices:

  • Retention can be limits-based (a replayable log) or work-queue style (messages removed once acknowledged).
  • Consumers can be push-based (the server delivers) or pull-based (the client asks for batches).

The chosen fix

The chosen setup is a three-node cluster with these properties:

  • Streams are created with 3 replicas, on three separate hosts.
  • Each node has a unique server_name, a shared cluster name, separate credentials for routes and for clients, and a tls block.
  • Stream retention is work-queue style, so acknowledged messages are removed.
  • Consumers are durable pull consumers with explicit acks.

A cluster gives real tolerance of a single node failure, which a single node cannot. A work-queue stream matches the common pattern of one logical consumer per message, such as order processing. A pull consumer lets the application decide how many messages to take at a time, so a slow worker is not flooded. If several independent services must each see every message, use a limits-based stream with one durable consumer per service instead.

A single node remains the right choice for development and small non-critical setups. It differs only in that it omits the cluster block and uses replica 1.

Step by step

This is a procedure to follow, not a recording of a run. No command output is reproduced. Each verification command says what to look for.

1. Prepare the hosts

Use three hosts named nats-01, nats-02 and nats-03 that resolve to each other. Allow port 4222 from the application hosts and port 6222 only between the three nodes. Keep 8222 on loopback. Put the JetStream store directory on local SSD or NVMe storage. Set max_file_store below the real size of that volume.

2. Write the server configuration

On each node, change only server_name. Create the certificate files first. Replace the placeholder passwords.

# /etc/nats/nats-server.conf  (nats-01; change server_name on the others)
server_name: nats-01
listen: 0.0.0.0:4222
http: 127.0.0.1:8222

jetstream {
  store_dir: /data/nats/jetstream
  max_memory_store: 1G
  max_file_store: 90G
}

tls {
  cert_file: /etc/nats/tls/server.crt
  key_file: /etc/nats/tls/server.key
}

authorization {
  user: app
  password: CHANGE_ME_client
  timeout: 2
}

cluster {
  name: orders-cluster
  listen: 0.0.0.0:6222
  authorization {
    user: route
    password: CHANGE_ME_route
    timeout: 2
  }
  routes: [
    nats-route://route:CHANGE_ME_route@nats-01:6222
    nats-route://route:CHANGE_ME_route@nats-02:6222
    nats-route://route:CHANGE_ME_route@nats-03:6222
  ]
}

Route traffic can also be encrypted with a tls block inside cluster. The certificates must then be valid for the node host names.

3. Run the server with enough file descriptors

With Docker, host networking lets the loopback-only monitoring port be reached from the host:

docker run -d --name nats --restart always --network host \
  --ulimit nofile=65535:65535 \
  -v /etc/nats/nats-server.conf:/etc/nats/nats-server.conf:ro \
  -v /etc/nats/tls:/etc/nats/tls:ro \
  -v /data/nats/jetstream:/data/nats/jetstream \
  nats:2.6-alpine -c /etc/nats/nats-server.conf

For a quick single-node test the flags are enough, with no configuration file:

docker run -d --name nats-dev -p 4222:4222 -p 8222:8222 -v nats-dev-data:/data nats:2.6-alpine -js -sd /data -m 8222

When the server runs under systemd instead, raise the limit in a drop-in created with systemctl edit nats-server:

[Service]
LimitNOFILE=65535

4. Check the cluster

On each node, the routes endpoint should list the two other servers. The JetStream endpoint should show JetStream enabled with the configured limits.

curl -s http://127.0.0.1:8222/routez
curl -s http://127.0.0.1:8222/jsz

5. Create the stream and publish

The stream is created from the Go client. The call is safe to repeat as long as the configuration is identical. The MsgId option lets the server drop duplicates inside the stream's duplicate window, which makes retried publishes safe.

package main

import (
	"log"

	"github.com/nats-io/nats.go"
)

func main() {
	nc, err := nats.Connect(
		"tls://nats-01:4222,tls://nats-02:4222,tls://nats-03:4222",
		nats.UserInfo("app", "CHANGE_ME_client"),
		nats.RootCAs("/etc/nats/tls/ca.crt"),
	)
	if err != nil {
		log.Fatal(err)
	}
	defer nc.Close()

	js, err := nc.JetStream()
	if err != nil {
		log.Fatal(err)
	}

	_, err = js.AddStream(&nats.StreamConfig{
		Name:      "ORDERS",
		Subjects:  []string{"ORDERS.*"},
		Storage:   nats.FileStorage,
		Retention: nats.WorkQueuePolicy,
		Replicas:  3,
	})
	if err != nil {
		log.Fatal(err)
	}

	ack, err := js.Publish("ORDERS.created", []byte(`{"id": 101}`), nats.MsgId("order-101"))
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("stored in stream %s at sequence %d", ack.Stream, ack.Sequence)
}

6. Consume with a durable pull consumer

Acknowledge only after the work has succeeded. A timeout from Fetch means no message arrived in the wait period, which is normal on an idle stream.

sub, err := js.PullSubscribe("ORDERS.created", "order-processor")
if err != nil {
	log.Fatal(err)
}

for {
	msgs, err := sub.Fetch(10, nats.MaxWait(2*time.Second))
	if err != nil {
		if err == nats.ErrTimeout {
			continue
		}
		log.Fatal(err)
	}
	for _, m := range msgs {
		log.Printf("processing %s", string(m.Data))
		m.Ack()
	}
}

This loop also needs time in the import list. Use m.Nak() to request an immediate redelivery after a failure.

7. Inspect state from the command line

The nats CLI shows the replica placement and the consumer's pending and redelivered counts:

nats --server tls://nats-01:4222 --user app --password CHANGE_ME_client --tlscacert /etc/nats/tls/ca.crt stream info ORDERS
nats --server tls://nats-01:4222 --user app --password CHANGE_ME_client --tlscacert /etc/nats/tls/ca.crt consumer info ORDERS order-processor

The stream information should list a leader and two followers. After stopping the container on one node, a new leader should be elected and publishing should continue.

8. Optional host tuning

The kernel file limit should not be lower than the per-process limit. BBR needs kernel 4.9 or newer, and the buffer values only matter for high-bandwidth links. Test any network change before keeping it.

# /etc/sysctl.d/90-nats.conf
fs.file-max = 100000
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
sysctl --system
sysctl net.ipv4.tcp_congestion_control

Limits and alternatives

What this setup provides:

  • Streams with three replicas survive the loss of any single node while two remain.
  • Route and client credentials are separate.
  • Acked messages leave a work-queue stream.
  • Retried publishes can be de-duplicated.

Its limits:

  • The cluster does not tolerate two simultaneous node losses, and a replicated write waits for a majority. Keep the nodes in one low-latency network, and do not stretch one cluster across distant sites.
  • Delivery is at least once, so consumers must be idempotent. De-duplication on publish does not remove duplicates caused by redelivery.
  • Token or password authentication is the simplest model. Larger deployments with many tenants should look at accounts and per-account JetStream limits, described in the configuration page linked above.
  • Data location follows the hosts. Where the servers run decides where persisted messages are stored, which matters for data-residency requirements. This article makes no legal claim beyond that.

Choose a single node when downtime is acceptable and backups are enough. Choose a limits-based stream with multiple durable consumers when several services need the same events. Choose Kafka when very large retained logs and an existing stream-processing ecosystem are the main requirement. Choose RabbitMQ when complex routing between queues dominates.