Production Checklist

A pre-launch checklist covering NATS tuning, monitoring, backup, scaling, and health checks.

NATS Configuration

JetStream Storage

Set max_store_bytes based on your expected event volume. Left unset, it is half of the free disk, capped at 10 GiB. For high-throughput deployments, raise it:

# dagnats.yaml
max_store_bytes: 53687091200  # 50 GiB

JetStream data lives in {data_dir}/jetstream/. Use fast storage (SSD or NVMe) for this directory. Spinning disks create latency spikes during fsync.

Stream Retention

Each file-backed stream has a max age and a byte ceiling that is a share of max_store_bytes (see NATS Infrastructure). Raising max_store_bytes raises every ceiling with it.

• WORKFLOW_HISTORY: 30-day max age, 25% of the store budget. Events older than your recovery window are safe to discard.
• EVENTS: 14-day max age, 15% of the budget.
• DEAD_LETTERS: 30-day max age, 5% of the budget. Raise max_store_bytes if you need longer post-mortem windows.
• TELEMETRY: 7-day max age, 10% of the budget. Raise max_store_bytes if you export slowly or want deeper history.
• BUILD_LOGS: DAGNATS_BUILD_LOGS_TTL, default 7 days, 5% of the budget.
• TASK_QUEUES and SLEEP_TIMERS have no max age, because their messages are pending work.

Dedup Window

WORKFLOW_HISTORY and TELEMETRY use a 5-second dedup window via Nats-Msg-Id (DEAD_LETTERS uses 24 hours and BUILD_LOGS 2 minutes). This prevents duplicate events during retries. Do not reduce this below 5 seconds unless you understand the implications for at-least-once delivery.

Monitoring

Health Endpoints

Wire /health to your container orchestrator’s liveness check and /ready to the readiness check. In a cluster, point each node’s load-balancer check at that node’s own /health/cluster. See Clustering.

NATS Monitoring

Enable the NATS monitoring port for direct server metrics:

monitor_port: 8222

This exposes the standard NATS monitoring endpoints (/varz, /jsz, /connz, /subsz) on the specified port. Use these for:

• Stream health (/jsz): message counts, consumer lag, storage usage
• Connection health (/connz): connected workers, subscription counts
• Server health (/varz): memory, CPU, goroutines

Key Metrics to Watch

Status Command

dagnats status

Shows connection state, stream info, active workers, and pending tasks at a glance.

Backup

JetStream Snapshots

NATS JetStream supports stream snapshots for backup:

nats stream backup WORKFLOW_HISTORY /backups/history-$(date +%Y%m%d).tar
nats stream backup TASK_QUEUES /backups/tasks-$(date +%Y%m%d).tar

Back up at minimum:

• WORKFLOW_HISTORY – your source of truth for all workflow state
• KV buckets (workflow_defs, workflow_runs) – workflow_runs holds the current state of every run

KV buckets are stored as JetStream streams internally (prefixed KV_), so you can back them up the same way:

nats stream backup KV_workflow_defs /backups/defs-$(date +%Y%m%d).tar
nats stream backup KV_workflow_runs /backups/runs-$(date +%Y%m%d).tar

Recovery

Restore from snapshots:

nats stream restore WORKFLOW_HISTORY /backups/history-20260401.tar

The orchestrator keeps no state of its own between events. Run state is the snapshot in workflow_runs, and the engine’s durable orchestrator consumer on WORKFLOW_HISTORY remembers how far it has read. Restore WORKFLOW_HISTORY and KV_workflow_runs from the same point in time, or the snapshots and the event log will disagree.

Scaling Workers

Workers scale horizontally. Each worker instance creates a pull consumer on the TASK_QUEUES stream filtered to its task types. NATS distributes work automatically.

Guidelines:

• Start with 1 worker per task type, add more when consumer pending count rises
• Each worker process handles one task at a time per task type by default
• For CPU-bound tasks, match worker count to available cores
• For I/O-bound tasks (API calls, LLM inference), run more workers than cores

Worker Affinity

For stateful workers (large model caches, local file context), use sticky bindings. The sticky_bindings KV bucket maps runs to specific workers. Tasks for that run route to the same worker via the STICKY_TASKS stream.

Memory and Disk Sizing

Memory

• Engine: ~2 KiB per active workflow actor. 10,000 concurrent runs needs ~20 MiB for actor state alone, plus overhead.
• Workers: depends on task payload size. Budget for the largest payload you expect times MaxAckPending.
• NATS server: JetStream uses memory-mapped files. Budget at least 256 MiB for the NATS process itself.

Disk

• WORKFLOW_HISTORY: ~500 bytes per event. A workflow with 10 steps generates ~12 events. 1 million completed workflows = ~6 GiB.
• TASK_QUEUES: WorkQueue retention means completed tasks are deleted. Disk usage is proportional to in-flight tasks, not total tasks.
• TELEMETRY: 7-day retention, capped at 10% of max_store_bytes.
• KV buckets: typically small. workflow_runs is the largest, proportional to active + recently completed runs.

Recommended Minimums

High Availability

A single dagnats serve node is a single point of failure for its data. To survive losing a node, run three or five nodes as a self-contained cluster, or connect leaf nodes to an external NATS cluster. See Clustering and Deployment Models.

Agent Journal

Steps that call Journal() write to the AGENT_JOURNAL stream, and their inputs go to the submissions bucket. Both are file-backed and use the cluster’s replica count, so a single node loses them with its disk. The journal holds model inputs, model outputs and tool results: give journal.> the same access control as run history, and keep secrets out of records. Roll out engines before workers, then turn legacy_step_events off once every worker is upgraded. See Durable Agents.

Security

Network Isolation

In standalone mode, the embedded NATS server binds to 127.0.0.1 – only local connections accepted. In leaf node mode, it binds to 0.0.0.0 for hub communication.

For production:

• Use NATS credentials (leaf_credentials config key) for leaf-to-hub authentication
• Place the HTTP API behind a reverse proxy with TLS
• Use DAGNATS_WEBHOOK_SECRET for webhook trigger authentication
• Set DAGNATS_BRIDGE_TOKEN for production: no env token means an open bridge (dev mode, unauthenticated); set it and every worker needs either the env token or a minted one. It doubles as the admin/root credential – mint scoped, revocable worker tokens from it (POST /v1/tokens) to hand to individual machines rather than distributing the admin credential itself

Data Directory Permissions

The data_dir contains JetStream storage including workflow definitions, run state, and task payloads. Restrict access:

chmod 700 /var/lib/dagnats
chown dagnats:dagnats /var/lib/dagnats