Three-Node Cluster (Cloud/VM)

Three-Node Test Cluster

This guide walks through deploying a Nexus GSLB cluster across three geographically distributed nodes using WireGuard for overlay networking and a three-peer NATS JetStream cluster for state synchronisation. It is written for the specific topology of two public VPS nodes and one home-lab node behind NAT, but the pattern applies to any mix of public and private nodes.

Topology

Node	Location	WireGuard IP	DNS public?
`lon-01`	London VPS	10.100.0.1	Yes
`eu-01`	EU VPS	10.100.0.2	Yes
`lab-01`	Home lab VM	10.100.0.3	No (internal)

Each node runs:

gslbd with its own SQLite database
A NATS server (cluster peer)

NATS traffic stays inside the WireGuard overlay. The home lab node connects outbound through WireGuard so NAT traversal is transparent.

Why three full NATS peers and not a leaf node? Three peers give JetStream a 2-of-3 quorum, which means the cluster survives a single node failure while still persisting health events and membership state. A leaf node only relays messages — it does not participate in consensus.

Home lab: VM or Kubernetes? Start with a VM (Docker Compose or bare binary) for the initial cluster test. The Kubernetes manifests in deploy/kubernetes/ can be validated as a second pass once the NATS path is confirmed working.

Phase 1 — WireGuard Mesh

Install WireGuard on all three nodes.

Generate a key pair on each node:

wg genkey | tee /etc/wireguard/privatekey | wg pubkey > /etc/wireguard/publickeychmod 600 /etc/wireguard/privatekeycat /etc/wireguard/publickey

Exchange the three public keys, then write the config files.

lon-01 — /etc/wireguard/wg0.conf

[Interface]Address    = 10.100.0.1/24ListenPort = 51820PrivateKey = <LON_PRIVATE_KEY>[Peer]# eu-01PublicKey           = <EU_PUBLIC_KEY>AllowedIPs          = 10.100.0.2/32Endpoint            = <EU_VPS_PUBLIC_IP>:51820PersistentKeepalive = 25[Peer]# lab-01 — no Endpoint; lab connects to usPublicKey           = <LAB_PUBLIC_KEY>AllowedIPs          = 10.100.0.3/32PersistentKeepalive = 25

eu-01 — /etc/wireguard/wg0.conf

[Interface]Address    = 10.100.0.2/24ListenPort = 51820PrivateKey = <EU_PRIVATE_KEY>[Peer]# lon-01PublicKey           = <LON_PUBLIC_KEY>AllowedIPs          = 10.100.0.1/32Endpoint            = <LON_VPS_PUBLIC_IP>:51820PersistentKeepalive = 25[Peer]# lab-01 — no Endpoint; lab connects to usPublicKey           = <LAB_PUBLIC_KEY>AllowedIPs          = 10.100.0.3/32PersistentKeepalive = 25

lab-01 — /etc/wireguard/wg0.conf

[Interface]Address    = 10.100.0.3/24ListenPort = 51820PrivateKey = <LAB_PRIVATE_KEY># No ListenPort — we initiate all connections outbound[Peer]# lon-01PublicKey           = <LON_PUBLIC_KEY>Endpoint            = <LON_VPS_PUBLIC_IP>:51820PersistentKeepalive = 25[Peer]# eu-01PublicKey           = <EU_PUBLIC_KEY>AllowedIPs          = 10.100.0.2/32Endpoint            = <EU_VPS_PUBLIC_IP>:51820PersistentKeepalive = 25

Enable and start on each node:

systemctl enable --now wg-quick@wg0

Verify the mesh before continuing — all three nodes must reach each other:

# from lon-01ping -c 3 10.100.0.2   # eu-01ping -c 3 10.100.0.3   # lab-01# check peer statuswg show

Firewall: open WireGuard on VPS nodes

# lon-01 and eu-01ufw allow 51820/udp

The home lab node opens no inbound ports — it initiates all WireGuard handshakes outbound.

Phase 2 — NATS Cluster

Download the nats-server binary on each node from https://nats.io/download/. Install to /usr/local/bin/nats-server.

Each server listens on its WireGuard IP. The cluster routing port (6222) is never exposed publicly — it is only reachable over the overlay.

lon-01 — /etc/nats/nats.conf

server_name: lon-01
listen:      10.100.0.1:4222
http:        10.100.0.1:8222
 
cluster {
  name:   nexus-gslb
  listen: 10.100.0.1:6222
  routes: [
    nats-route://10.100.0.2:6222
    nats-route://10.100.0.3:6222
  ]
}
 
jetstream {
  store_dir:       /var/lib/nats/jetstream
  max_memory_store: 256MB
  max_file_store:   1GB
}

eu-01 — /etc/nats/nats.conf

server_name: eu-01
listen:      10.100.0.2:4222
http:        10.100.0.2:8222
 
cluster {
  name:   nexus-gslb
  listen: 10.100.0.2:6222
  routes: [
    nats-route://10.100.0.1:6222
    nats-route://10.100.0.3:6222
  ]
}
 
jetstream {
  store_dir:       /var/lib/nats/jetstream
  max_memory_store: 256MB
  max_file_store:   1GB
}

lab-01 — /etc/nats/nats.conf

server_name: lab-01
listen:      10.100.0.3:4222
http:        10.100.0.3:8222
 
cluster {
  name:   nexus-gslb
  listen: 10.100.0.3:6222
  routes: [
    nats-route://10.100.0.1:6222
    nats-route://10.100.0.2:6222
  ]
}
 
jetstream {
  store_dir:       /var/lib/nats/jetstream
  max_memory_store: 256MB
  max_file_store:   1GB
}

systemd unit — all nodes

# /etc/systemd/system/nats.service[Unit]Description=NATS ServerAfter=network.target wg-quick@wg0.serviceWants=wg-quick@wg0.service[Service]ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats.confRestart=on-failureUser=natsStateDirectory=nats[Install]WantedBy=multi-user.target

useradd --system --no-create-home --shell /usr/sbin/nologin natsmkdir -p /var/lib/nats/jetstreamchown -R nats:nats /var/lib/natssystemctl enable --now nats

Verify NATS cluster

Install the nats CLI (https://nats.io/download/), then:

nats --server nats://10.100.0.1:4222 server listnats --server nats://10.100.0.1:4222 server report jetstream

All three servers (lon-01, eu-01, lab-01) should appear. JetStream should show a single meta leader with two followers.

Phase 3 — gslbd Deployment

Use Docker Compose on all three nodes (including the home lab VM). network_mode: host is required so the container can reach the WireGuard IPs without any special routing.

Directory layout (each node)

/opt/nexus-gslb/
├── docker-compose.yml
├── config.yaml
├── .env
└── data/              ← SQLite volume (created by Docker)

docker-compose.yml (same on all nodes)

services:  gslbd:    image: registry.starstorm.dev/nexus-gslb/gslbd:latest    restart: unless-stopped    network_mode: host    volumes:      - ./config.yaml:/etc/gslb/config.yaml:ro      - ./data:/var/lib/gslbd    env_file: .env    healthcheck=:      test: ["CMD", "wget", "-qO-", "http://localhost:8080/api/v1/health"]      interval: 30s      timeout: 5s      retries: 3      start_period: 15s

.env (each node, do not commit)

GSLB_API_KEY=<strong-random-secret>GSLB_WS_SECRET=<strong-random-secret># License (apply the same floating token to all nodes — see docs/reference/Licensing.md)GSLB_LICENSE_PUBLIC_KEY=<ed25519-public-key-from-licensectl>GSLB_LICENSE_KEY=<floating-token-from-licensectl>

Use the same GSLB_API_KEY on all nodes so any node can authenticate API requests from your workstation. Use the same floating license token on all nodes — do not include an Install ID when generating the token for a cluster.

lon-01 — config.yaml

dns:  listenAddr: "0.0.0.0"  port: 5353  domain: "gslb.example.com"cluster:  id: "nexus-test"node:  id: "lon-01"rqlite:  httpAddr: "localhost:4001"api:  enabled=: true  listenAddr: "0.0.0.0"  port: 8080  webuiPath: "/opt/gslb/webui"metrics:  enablePrometheus: true  listenAddr: "0.0.0.0"  port: 9090health:  enabled: true  type: "tcp"  port: 80  checkInterval: "10s"  timeout: "2s"state:  nats:    servers:      - "nats://10.100.0.1:4222"      - "nats://10.100.0.2:4222"      - "nats://10.100.0.3:4222"  healthPolicy: "prefer-local"  heartbeatInterval: "10s"  heartbeatTTL: "30s"

eu-01 — config.yaml

Identical to lon-01 except:

node:  id: "eu-01"

lab-01 — config.yaml

Identical to lon-01 except:

node:  id: "lab-01"dns:  listenAddr: "127.0.0.1"   # not exposed publicly  port: "5353"

Start gslbd on each node

cd /opt/nexus-gslbdocker compose pulldocker compose up -ddocker compose logs -f

Phase 4 — Firewall RulesVPS nodes (lon-01, eu-01)

# WireGuard (already opened in Phase 1)ufw allow 51820/udp# DNS — publicufw allow 5353/udpufw allow 5353/tcp# API — restrict to your own IP or put behind a reverse proxyufw allow from <YOUR_IP> to any port 8080# Metrics — internal only (or Prometheus scraper IP)ufw allow from 10.100.0.0/24 to any port 9090# NATS — WireGuard only, no public rule needed# (already scoped by WireGuard AllowedIPs)

DNS on port 53

The default port is 5353 (no capabilities required). To accept queries on port 53, add a redirect on each VPS:

iptables -t nat -A PREROUTING -p udp --dport 53 -j REDIRECT --to-port 5353iptables -t nat -A PREROUTING -p tcp --dport 53 -j REDIRECT --to-port 5353# Persist with iptables-save or nftables equivalent

Phase 5 — Verification1. Confirm all nodes are healthy

curl http://10.100.0.1:8080/api/v1/health   # lon-01curl http://10.100.0.2:8080/api/v1/health   # eu-01curl http://10.100.0.3:8080/api/v1/health   # lab-01

All should return {"status":"ok"}.

2. Check NATS cluster state

nats --server nats://10.100.0.1:4222 server listnats --server nats://10.100.0.1:4222 server report jetstream

Three peers, one meta leader, JetStream enabled on all.

3. Create a test pool and members on one node

API=http://10.100.0.1:8080KEY=<your-api-key># Create poolPOOL=$(curl -sX POST $API/api/v1/pools \  -H "Authorization: Bearer $KEY" \  -H "Content-Type: application/json" \  -d '{"name":"test-pool"}' | jq -r .id)# Add two members — use IPs you can control firewall rules oncurl -sX POST $API/api/v1/pools/$POOL/members \  -H "Authorization: Bearer $KEY" \  -H "Content-Type: application/json" \  -d '{"name":"host-a","ipAddress":"1.1.1.1","port":80,"weight":1}'curl -sX POST $API/api/v1/pools/$POOL/members \  -H "Authorization: Bearer $KEY" \  -H "Content-Type: application/json" \  -d '{"name":"host-b","ipAddress":"1.0.0.1","port":80,"weight":1}'# Add a health checkcurl -sX PUT $API/api/v1/pools/$POOL/healthcheck \  -H "Authorization: Bearer $KEY" \  -H "Content-Type: application/json" \  -d '{"type":"tcp","port":80,"intervalMs":5000,"timeoutMs":2000}'

4. Test health propagation across nodes

Open a WebSocket listener on two different nodes simultaneously:

# Terminal 1 — watch lon-01wscat -c "ws://10.100.0.1:8080/api/v1/ws?token=$(  curl -s http://10.100.0.1:8080/api/v1/ws/token \    -H "Authorization: Bearer $KEY" | jq -r .token)"# Terminal 2 — watch eu-01wscat -c "ws://10.100.0.2:8080/api/v1/ws?token=$(  curl -s http://10.100.0.2:8080/api/v1/ws/token \    -H "Authorization: Bearer $KEY" | jq -r .token)"

Block 1.0.0.1 on the firewall of lon-01 (e.g., iptables -A OUTPUT -d 1.0.0.1 -j DROP). Within two health intervals both WebSocket connections should receive a health_update event showing 1.0.0.1 as unhealthy. This confirms:

lon-01's health checker detected the failure
dbSink published to NATS
HealthBridge on both eu-01 and lab-01 received the event and pushed it to their WebSocket clients

Remove the block and verify recovery:

iptables -D OUTPUT -d 1.0.0.1 -j DROP

5. Test global-quorum policy

Update one node's config to use global-quorum:

state:  healthPolicy: "global-quorum"  quorumMinPercent: 67     # 2 of 3 nodes must agree

With the block on lon-01 only, 1.0.0.1 is unhealthy from lon-01's view but healthy from eu-01 and lab-01. With 67% quorum (2 of 3), the member stays healthy. Block it on two nodes — quorum fails and DNS responses on all nodes stop including that member.

Health Policy Reference

Policy	Behaviour
`prefer-local`	Uses local checker result; global is informational only
`local-only`	Ignores NATS entirely; each node decides independently
`global-any-healthy`	Healthy if local or any active node reports healthy
`global-quorum`	Healthy if ≥ `quorumMinPercent` of active nodes report healthy

prefer-local is the right default for a split-horizon setup where each node checks its nearest backends. Use global-quorum when you want the cluster to agree before withdrawing an endpoint from DNS.

What Is Not Covered Here

Feature	Where to look
Config sync (pool/member replication across nodes)	`docs/StateSync.md` → Configuration Sync section
Geo-IP routing	`docs/Configuration.md` → `algorithm: geo-ip`
DNSSEC	`docs/Security.md`
DNS delegation (pointing a real subdomain at both VPS IPs)	Your registrar + two NS/A records
Kubernetes deployment for the lab node	`docs/user/Kubernetes.md`
Prometheus + Grafana setup	`docs/user/MetricsObservability.md`
TLS on the REST API	Put nginx or Caddy in front; see `docs/Security.md`