Core: Troubleshooting Reference

Troubleshooting Guide

Common failure modes and step-by-step playbooks. Each section starts with symptoms, then diagnosis commands, then fixes.

Quick sanity checks

Run these first for any issue:

# Is the daemon running?systemctl status gslbd          # systemddocker compose ps               # Dockerkubectl get pods -l app=gslbd   # Kubernetes# Is it healthy?curl -s http://localhost:8080/api/v1/health | jq .# Recent log lines (structured JSON)journalctl -u gslbd -n 50 --output=json | jq -r '[.SYSLOG_TIMESTAMP, .MESSAGE] | @tsv'

DNS not responding

Symptoms: dig times out; no DNS response at all.

Diagnosis:

# Is the listener up?dig @127.0.0.1 -p 5353 A example.com +time=2# Is port 5353 open?ss -ulnp | grep 5353   # UDPss -tlnp | grep 5353   # TCP# Any bind errors in logs?journalctl -u gslbd | grep -i "listen\|bind\|address in use"

Fixes:

Cause	Fix
Port already in use	Another process holds 5353. `ss -lnp sport = :5353` to find it. Change `dns.port` in config or stop the conflicting process.
Port 53 permission denied	Port 53 requires `CAP_NET_BIND_SERVICE`. Uncomment `AmbientCapabilities=CAP_NET_BIND_SERVICE` in the systemd unit, or use port 5353 and redirect with iptables: `iptables -t nat -A PREROUTING -p udp --dport 53 -j REDIRECT --to-port 5353`.
Database path wrong	If `database.path` does not exist or is unreadable, the DNS server starts but returns SERVFAIL for all DB-backed services. Check: `ls -la /var/lib/gslbd/` and ensure the `gslbd` user owns it.
Config parse failure	The daemon exits immediately. Check exit code: `systemctl show gslbd --property=ExecMainStatus`. Revalidate: `gslbd -config /etc/gslb/config.yaml -validate`.

DNS returns NXDOMAIN or SERVFAIL for a known service

Symptoms: dig gets a response but the domain does not resolve to any IP.

Diagnosis:

# Enable query logging in config: dns.queryLog: true, then:journalctl -u gslbd | grep "dns query"# Check the service exists in DBcurl -s http://localhost:8080/api/v1/services | jq '.[] | select(.domain == "app.example.com.")'# Check pool members exist and are healthycurl -s http://localhost:8080/api/v1/pools/<pool-id>/status | jq .

Common causes:

Domain not terminated with . — Nexus GSLB matches on the fully-qualified name including the trailing dot. Ensure services.domain is app.example.com. not app.example.com.
No healthy members — If all pool members are unhealthy (health checks failing), the service returns no records. Check pool status above; see All members unhealthy.
Service not linked to pool — A service exists but has no pool assigned. Update via API: PUT /api/v1/services/<id> with poolID.
Algorithm mismatch — If the service uses geo-ip but no GeoIP database is configured, resolution falls back to no candidates. Check logs for geo-ip watcher could not start.

All members unhealthy

Symptoms: Pool status shows 0 healthy members; DNS returns no records.

Diagnosis:

# Pool health overviewcurl -s http://localhost:8080/api/v1/pools | jq '.[] | {id, name}'curl -s http://localhost:8080/api/v1/pools/<pool-id>/status# Metrics (if enabled)curl -s http://localhost:9090/metrics | grep gslbd_health_endpoints_healthy# Health checker logsjournalctl -u gslbd | grep -E "health|probe|unhealthy"

Fixes:

Cause	Fix
Health check type wrong	`tcp` check on an HTTP-only port returns unhealthy. Change health check `type` to `http` or `https`.
Timeout too short	Under load, probes time out before the backend responds. Increase `healthCheck.timeout` (default 5 s).
TLS mismatch with HTTPS check	If the backend uses a self-signed cert and `insecureSkipVerify: false`, probes fail. Either fix the cert or set `insecureSkipVerify: true` (logs a warning).
Backend port wrong	`healthCheck.port` does not match the actual service port. Verify with `curl -v http://<backend>:<port>/`.
Health checks disabled	`healthCheck.enabled: false` treats all endpoints as healthy. If probes were just enabled, wait one full `interval` for the first probe to complete.
Firewall blocking probes	The `gslbd` host cannot reach backend ports. Test: `nc -zv <backend-ip> <port>` from the gslbd host.

Failover is slow

Symptoms: After a backend goes down, DNS continues returning it for longer than expected.

Failover has three independent layers — diagnose each separately:

Layer 1 — Detection (gslbd still returning the failed IP)

# Check the pool's current health statuscurl -s http://localhost:8080/api/v1/pools/<pool-id>/status | jq .# Is the detection latency in line with your checkInterval?curl -s http://localhost:9090/metrics | grep failover_detection# gslbd_health_failover_detection_seconds_bucket{direction="down",...}# What is the configured check interval and timeout?curl -s http://localhost:8080/api/v1/pools/<pool-id>/health-check | jq '{intervalMs, timeoutMs}'

Cause	Fix
`checkInterval` too long	Lower `health.checkInterval` (e.g. `5s` or `2s`). Tradeoff: higher probe volume.
`timeout` too long relative to interval	`timeout` should be ≤ 50% of `checkInterval`. A timed-out probe blocks the next tick.
`scoreWindow` delaying confirmation	With `scoreWindow: N`, the state only flips when the rolling success rate crosses a threshold — up to N probe intervals. Lower `scoreWindow` or set to 0 for immediate flip.
Health check type too slow	`script` and `webhook` probes have subprocess/network overhead. `tcp` is fastest.

Layer 2 — Peer convergence (other cluster nodes still returning the failed IP)

# Is NATS connected?curl -s http://localhost:9090/metrics | grep gslbd_state_nats_connected# What is the NATS transit lag?curl -s http://localhost:9090/metrics | grep merge_lag_ms

Peer convergence should be under 200 ms with the event-driven publisher. If it is slow:

Cause	Fix
NATS not connected	Fix NATS connectivity (see NATS state sync not working). Until reconnected, peers use stale state.
High `gslbd_state_merge_lag_ms` p95	High network latency between nodes or NATS servers. Check cross-region RTT; confirm NATS super-cluster gateway links are up.

Layer 3 — Client cache (clients still hitting the failed IP)

This is a DNS protocol property, not a gslbd bug. Clients cache responses for the TTL configured on the service.

# What TTL are you serving?dig +noall +answer @<gslbd-host> -p <port> <service-domain> A | awk '{print $2}'

Cause	Fix
TTL too high	Lower per-service `ttl` (API: `PUT /api/v1/services/<id>` with `"ttl": 10`). Tradeoff: more DNS queries.
Clients ignoring TTL	Some HTTP clients and OS caches ignore short TTLs. Nothing gslbd can do — this is client-side behaviour.
Load balancer / CDN in front	Intermediate load balancers may cache DNS results independently. Check their DNS TTL settings.

See Performance — Failover latency for a full breakdown and recommended configurations by SLA target.

High DNS latency

Symptoms:gslbd_dns_query_duration_seconds p99 > 10 ms; queries are slow under load.

Diagnosis:

# Latency breakdown from metricscurl -s http://localhost:9090/metrics | grep gslbd_dns_query_duration# Database slow queries (SQLite WAL mode should be fast, but check)# Add -v to see timing:time curl -s http://localhost:8080/api/v1/services > /dev/null# Is the DB index present? (query the rqlite datastore over its HTTP API)curl -sG http://localhost:4001/db/query --data-urlencode \  "q=SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='services'"# Should include: idx_services_domain

Fixes:

Cause	Fix
Missing domain index	Run via the rqlite CLI: `rqlite -H localhost -p 4001 "CREATE UNIQUE INDEX IF NOT EXISTS idx_services_domain ON services(lower(rtrim(domain,'.')));"` The daemon creates this on startup — if missing, the datastore may predate this version.
No datastore reachable	If `rqlite.httpAddr` is unset or `rqlited` is down, gslbd cannot resolve DB-backed services. Check `systemctl status rqlited` and `curl -s localhost:4001/status`.
GeoIP DB too large	MaxMind City databases are 60–70 MB. Cold lookups (first query after reload) may spike. Confirm with the `gslbd_dns_query_duration_seconds{algorithm="geo-ip"}` bucket. Switch to Country DB if City granularity is not needed.
DNS query timeout context	The DNS server uses a 200 ms timeout per SQLite query. If the DB is under heavy write load (backup running), reads may queue. Schedule backups off-peak.

NATS state sync not working

Symptoms: Nodes do not reflect each other's health state; gslbd_state_nats_connected == 0.

Diagnosis:

# Check NATS connection statuscurl -s http://localhost:9090/metrics | grep gslbd_state_nats# Logsjournalctl -u gslbd | grep -i "nats\|jetstream\|state"# Can the node reach NATS?nc -zv <nats-host> 4222

Fixes:

Cause	Fix
NATS server unreachable	Firewall or wrong `state.nats.servers` list. Check `telnet <nats-host> 4222`.
JetStream not enabled	Connect to NATS CLI: `nats server info`. If JetStream is disabled, `jetstream: enabled: true` in the NATS server config and restart.
TLS cert mismatch	If `state.nats.tls.caFile` is set, the CA must match the NATS server's cert. Verify: `openssl s_client -connect <nats-host>:4222 -CAfile <caFile>`.
Auth credentials wrong	`state.nats.auth.user/password` or NKey seed mismatch. Check NATS server logs for auth errors.
Cluster ID mismatch	All nodes sharing state must have the same `cluster.id`. Nodes with different cluster IDs publish to different JetStream buckets and will not see each other.
JetStream stream missing	On first connect, `gslbd` creates the stream automatically. If it fails (quota, permission), check NATS account limits.

GitOps sync failures

Symptoms: Config changes pushed to the repo are not applied; gslbd_gitops_apply_total{result="error"} increases.

Diagnosis:

curl -s http://localhost:9090/metrics | grep gslbd_gitopsjournalctl -u gslbd | grep -i "gitops\|reconcile\|signature\|verify"

Fixes:

Cause	Fix
Signature verification failed	`requireSignature: true` but the commit is not signed, or the signing key is not in `allowedSigners`. Verify: `git log --show-signature HEAD`. Add the signer's fingerprint to config.
Repo URL unreachable	The daemon cannot clone/fetch. Test: `git ls-remote <repoURL>` as the `gslbd` user. Check SSH keys or token auth.
Config file parse error	The new config has a syntax error. Fix the YAML and push a new commit. The daemon reverts to last-good config on delete events.
`pathPrefix` wrong	The config file is not at `<pathPrefix>/config.yaml` in the repo. Check `gitops.pathPrefix`.
Clock drift	Signature timestamps outside tolerance. Sync clocks: `chronyc tracking`.

DNSSEC validation failures

Symptoms: Resolvers return SERVFAIL for signed zones; DNSSEC-aware clients cannot validate responses.

Diagnosis:

# Test DNSSEC validation end-to-enddig @127.0.0.1 -p 5353 +dnssec A app.example.com.# Check key expirycurl -s http://localhost:9090/metrics | grep gslbd_dnssec_key_days_remaining# Verify DS record is published at registrardig DS example.com. @8.8.8.8

Fixes:

Cause	Fix
KSK/ZSK not loaded	Check logs for `DNSSEC disabled (key load error)`. Verify PEM file paths and permissions (`gslbd` user must be able to read them).
DS record not published	After generating keys, export the DS record and submit to the registrar: `gslbctl dnssec ds --zone example.com. --ksk-file /etc/gslb/ksk.pem`.
Key expired	`gslbd_dnssec_key_days_remaining` < 0. Generate new ZSK, publish alongside old ZSK (double-signing window), then remove old. See `docs/Security.md`.
Clock skew	RRSIG inception is `now − 1h` to tolerate drift. If the resolver's clock is more than 1 h ahead, signatures appear invalid. Sync clocks.
Response truncation causing TC=1 loop	Large signed responses (> 1232 bytes) set TC=1 to force TCP retry. If the client does not retry over TCP, it will see no answer. Ensure TCP port 5353 is open and reachable.

API returns 401 Unauthorized

Symptoms: curl, browser, or API client calls return HTTP 401.

Causes and fixes:

Cause	Fix
Not logged in / session expired	Log in again: `POST /api/v1/auth/login` or navigate to `/ui/login` in the browser.
Sending an old or revoked session token	Obtain a new token by logging in again.
Sending the system API key but it doesn't match the server	Check `GSLB_API_KEY` in the server's `/etc/gslb/env`.
`GSLB_API_KEY` unset on server (dev mode) but client sends a Bearer	In dev mode the server accepts all requests; a 401 suggests the server does have `GSLB_API_KEY` set.
Cookie not sent by browser	Ensure the browser is hitting the same origin as the API (served via the same Caddy/nginx reverse proxy). SameSite=Strict cookies are not sent cross-origin.

Diagnosis:

# Test the system API key directlycurl -s -o /dev/null -w '%{http_code}' \  -H "Authorization: Bearer $GSLB_API_KEY" \  http://localhost:8080/api/v1/pools# Test a session tokencurl -s -H "Authorization: Bearer <token>" http://localhost:8080/api/v1/auth/me# Check whether the daemon has GSLB_API_KEY set (dev mode check)sudo cat /etc/gslb/env | grep GSLB_API_KEY

Symptoms: POST /api/v1/auth/login returns {"error": "invalid credentials"}.

Diagnosis and fixes:

Wrong email or password — double-check the email address (case-insensitive) and password.
Password was set via direct DB manipulation with Python bcrypt — ensure the $2b$ hash prefix is correct. Python's bcrypt library produces $2b$ hashes, which Go's golang.org/x/crypto/bcrypt accepts.
The user exists in the database with the wrong tenant_id — check:

# Query the rqlite datastore over its HTTP APIcurl -sG http://localhost:4001/db/query --data-urlencode \  "q=SELECT email, tenant_id, password_hash IS NOT NULL FROM users"curl -sG http://localhost:4001/db/query --data-urlencode \  "q=SELECT id, name FROM tenants"

The tenant_id in users must match an id in tenants. On upgraded deployments, run the latest binary once — startup automatically migrates any legacy UUID tenant id to the literal string "default".

Symptoms: Login succeeds for password but TOTP code is required.

The account has TOTP enrolled. Include the 6-digit code from your authenticator app:

curl -s -X POST http://localhost:8080/api/v1/auth/login \  -H 'Content-Type: application/json' \  -d '{"email": "you@example.com", "password": "...", "totpCode": "123456"}'

Lost TOTP device (admin recovery): A tenant_admin user or the system API key can clear another user's TOTP secret by updating the user record. This requires a working login — if both the admin's password and TOTP device are unavailable, use the break-glass procedure below.

Break-glass: resetting a user password or clearing TOTP via the DB

Use only when the API is inaccessible or all admin accounts are locked out.

# Install Python bcrypt if neededpip3 install bcryptpython3 <<'EOF'import sqlite3, bcryptdb = '/var/lib/gslbd/gslbd.db'email = 'admin@example.com'new_password = 'temporary-reset-password'pw_hash = bcrypt.hashpw(new_password.encode(), bcrypt.gensalt(rounds=12)).decode()conn = sqlite3.connect(db)conn.execute('UPDATE users SET password_hash=?, totp_secret=NULL, updated_at=strftime("%s","now") WHERE email=?',             (pw_hash, email))conn.commit()rows = conn.execute('SELECT email, role, tenant_id FROM users WHERE email=?', (email,)).fetchall()print('Updated:', rows)conn.close()EOF

Restart gslbd after direct DB writes to ensure the running process reflects current state.

Daemon refuses to start: "GSLB_SECRET_KEY must be set"

Symptom: The daemon exits at startup with an error about GSLB_SECRET_KEY.

At least one user in the database has a TOTP secret stored, and GSLB_SECRET_KEY is not set (or is incorrect). Generate a new key and add it to the environment file:

openssl rand -hex 32# Add to /etc/gslb/env:# GSLB_SECRET_KEY=<output>

If the key was lost: The existing TOTP secrets are unrecoverable without the original key. Set a new key, then clear all TOTP secrets from the database:

python3 -c "import sqlite3c = sqlite3.connect('/var/lib/gslbd/gslbd.db')c.execute('UPDATE users SET totp_secret=NULL')c.commit()print('Cleared TOTP secrets for', c.execute('SELECT COUNT(*) FROM users WHERE totp_secret IS NOT NULL').fetchone()[0], 'users')"

Users will need to re-enroll their TOTP devices.

Setup endpoint returns 404 after first user created

Symptom: POST /api/v1/auth/setup returns 404.

This is expected — the setup endpoint is permanently disabled once any user exists. Use the API with admin credentials or --create-admin on a fresh database.

"No users exist" but setup endpoint still returns 404

Check auth.setupEndpoint in config.yaml — it may have been set to false. Set it to true and restart.

Rate limiting / license errors

Symptoms: Logs show no valid license; enforcing unlicensed RPS limit; DNS clients get REFUSED or timeouts under load.

Diagnosis:

journalctl -u gslbd | grep -i "license\|rps\|rate"curl -s http://localhost:8080/api/v1/health | jq .license

Fixes:

Cause	Fix
No license configured	Set `GSLB_LICENSE_KEY` and `GSLB_LICENSE_SECRET` environment variables (or `license.key`/`license.secret` in config).
License expired	Renew the license. Until renewed, the daemon enforces the unlicensed RPS cap.
Wrong credentials	Key/secret mismatch. The daemon logs the license tier and expiry on startup if credentials are valid.

Database errors / corruption

Symptoms: Logs show database disabled; API returns 503; DNS stops resolving DB-backed services.

Diagnosis:

# Check DB file integritysqlite3 /var/lib/gslbd/gslbd.db "PRAGMA integrity_check;"# Is the DB owned by gslbd?ls -la /var/lib/gslbd/# Check available disk spacedf -h /var/lib/gslbd

Fixes:

Cause	Fix
Disk full	Free space. SQLite WAL writes fail silently if the disk is full. `du -sh /var/lib/gslbd/*` to find large WAL/SHM files; `PRAGMA wal_checkpoint(TRUNCATE);` to compact.
Permission denied	`chown -R gslbd:gslbd /var/lib/gslbd`.
Corruption	Restore from backup: `cp /var/backups/gslbd/gslbd-<date>.db /var/lib/gslbd/gslbd.db && chown gslbd: /var/lib/gslbd/gslbd.db`. Restart the daemon.
WAL file left behind after crash	If the process was killed hard, WAL may be unapplied. SQLite recovers this automatically on next open; if `integrity_check` still reports errors, restore from backup.

Manual backup (safe while daemon is running):

sqlite3 /var/lib/gslbd/gslbd.db "VACUUM INTO '/var/backups/gslbd/gslbd-$(date +%Y%m%d-%H%M).db';"

Config validation errors at startup

Symptoms: Daemon exits immediately with a validation error message.

Run the validator before restarting:

gslbd -config /etc/gslb/config.yaml -validate

Common messages and fixes:

Message	Fix
`dns.port: must be between 1 and 65535`	Set `dns.port` to a valid port, e.g. 5353.
`state.nats.servers: must not be empty when state sync is enabled`	Add at least one NATS server URL, e.g. `nats://localhost:4222`.
`loadBalancer.algorithm: invalid value "roundrobin"`	Use one of: `round-robin`, `weighted-round-robin`, `geo-ip`, `map-file`.
`dnssec.zone: must be a valid FQDN ending in '.'`	Set `dnssec.zone: "example.com."` (trailing dot required).
`backup.intervalSeconds: must be > 0`	Set a positive interval, e.g. `backup.intervalSeconds: 3600`.

gslbctl cannot connect to API

Symptoms: gslbctl commands return connection errors.

Diagnosis:

# Is the API port listening?curl -v http://localhost:8080/api/v1/health# gslbctl uses GSLB_API_URL and GSLB_API_KEY env varsexport GSLB_API_URL=http://localhost:8080export GSLB_API_KEY=your-keygslbctl pools list

Fix: The API server only starts when database.path is configured. If api.enabled is true but no database path is set, the daemon logs API server disabled: database path not configured or failed to open and does not bind the port.

Collecting a diagnostics bundle

When filing a bug report:

#!/usr/bin/env bashOUT="gslbd-diag-$(date +%Y%m%d-%H%M)"mkdir "$OUT"# Logs (last 500 lines)journalctl -u gslbd -n 500 --output=json > "$OUT/journal.json" 2>/dev/null# Versiongslbd -version > "$OUT/version.txt" 2>&1# Config (redact secrets first)cp /etc/gslb/config.yaml "$OUT/config.yaml"sed -i 's/key:.*/key: REDACTED/g; s/secret:.*/secret: REDACTED/g; s/password:.*/password: REDACTED/g' "$OUT/config.yaml"# Metrics snapshotcurl -s http://localhost:9090/metrics > "$OUT/metrics.txt" 2>/dev/null# Health APIcurl -s http://localhost:8080/api/v1/health > "$OUT/health.json" 2>/dev/null# DB integrity (no data, just check)sqlite3 /var/lib/gslbd/gslbd.db "PRAGMA integrity_check;" > "$OUT/db-integrity.txt" 2>/dev/nulltar czf "${OUT}.tar.gz" "$OUT" && rm -rf "$OUT"echo "Bundle: ${OUT}.tar.gz"