Platform: State Sync Setup

State Synchronization Setup (NATS + JetStream)

This guide walks you through enabling global state synchronization for health and membership across multiple gslbd instances using NATS with JetStream.

Overview

  • Each node publishes local health and heartbeats to NATS subjects and KV buckets.

  • Health state changes (healthy→unhealthy or unhealthy→healthy) are pushed immediately to NATS peers — peer convergence latency is under 200 ms regardless of checkInterval. A 30 s catchup ticker republishes the full snapshot in the background to recover from any missed signals.

  • Each node subscribes to the cluster subjects and KV snapshots to maintain a global view (GlobalHealthView).

  • The load balancer uses a composite provider to merge local and global health according to policy.

When to enable

  • You run multiple GSLB nodes across regions and want them to share health state.

  • You want quorum-based decisions or fast propagation of health changes.

Prerequisites

  • NATS 2.x with JetStream enabled.

  • TLS/mTLS credentials for clients (recommended).

  • A cluster ID configured in gslbd.

Recommended topology (super-cluster)

  • 3+ NATS servers per region (odd number for availability).

  • Connect regions via NATS gateways to form a super-cluster.

  • Enable JetStream in each cluster; optionally set a JetStream domain (e.g., gslb).

Minimal NATS server (single region) example

# nats-server.conf (excerpt)
jetstream: {
  domain: gslb
}
server_name: n1
port: 4222
cluster: {
  name: gslb
  port: 6222
}
# TLS, accounts, and gateways omitted for brevity; see NATS docs

Configure gslbd Add to /etc/gslb/config.yaml:

cluster:
  id: "prod-global"
 
state:
  healthPolicy: "prefer-local"   # or global-any-healthy | global-quorum
  quorumMinPercent: 51            # used for global-quorum
  heartbeatInterval: "10s"
  heartbeatTTL: "30s"
  nats:
    servers: ["nats://n1.example.com:4222","nats://n2.example.com:4222","nats://n3.example.com:4222"]
    tls:
      caFile: "/etc/gslb/pki/ca.crt"
      certFile: "/etc/gslb/pki/client.crt"
      keyFile: "/etc/gslb/pki/client.key"
    jetStream:
      domain: "gslb"

!!! note "NATS is ephemeral-only" State sync over NATS covers health and membership only — ephemeral, TTL'd data. Configuration and data-plane objects are not synced over NATS: config flows from git via GitOps self-restart, and pools/members/services/users are replicated by the rqlite datastore (Raft). The old JetStream config-sync (enableConfigSync, NEXUS_CFG) has been removed. See State Sync reference.

Policies explained

  • local-or-global-fallback (default): use local health, falling back to the global view.

  • prefer-local: local result is authoritative; global used mainly for hints.

  • local-only: ignore global state (useful during incidents or testing).

  • global-any-healthy: consider healthy if any site reports healthy.

  • global-quorum: healthy only if a percent of active members report healthy within staleness TTL.

How quorum works

  • Each node publishes periodic heartbeats.

  • Active membership = nodes whose latest heartbeat is within heartbeatTTL.

  • For an endpoint, only reports from active members within heartbeatTTL are counted.

  • quorumMinPercent computes healthy threshold: healthy*100/total >= quorumMinPercent.

Verifying connectivity and state

  • Metrics to watch:

    • gslbd_state_nats_connected (1 when connected)

    • gslbd_state_active_members

    • gslbd_state_nats_{published,received}_total{type}

    • gslbd_state_kv_{put,get}_total{bucket,result}

    • gslbd_state_merge_lag_ms — histogram of NATS transit latency from publish to peer receipt; p95 > 500 ms indicates network issues between nodes

    • gslbd_health_failover_detection_seconds{direction} — Layer 1 detection latency per state change on this node; direction="down" is healthy→unhealthy

  • When Configuration Sync is enabled, logs will include messages like: config snapshot loaded and config event: cluster=... version=....

  • Logs will show connect errors or KV bucket creation errors.

Failure modes

  • If NATS is unreachable: gslbd continues with local health only (policy prefer-local).

  • If clocks skew: events include timestamps; quorum logic tolerates within TTL but keep NTP enabled.

  • If membership is zero: no quorum can be established; provider falls back to local (for global-quorum).

Security

  • Use mTLS for client/server auth or NATS accounts/JWT with TLS.

  • Restrict subjects and KV permissions to the cluster namespace (e.g., gslb.<cluster>.*).

  • Rotate client certs periodically; see Security Guide.

Was this article helpful?