Operations & Troubleshooting

Operations & Troubleshooting

This guide collects day-2 operations, common issues, and quick runbooks for Nexus GSLB.

Day-0/1 checklists

  • Binary/service is installed and starts on boot (systemd or container orchestrator)

  • Config file present at /etc/gslb/config.yaml and readable

  • Metrics endpoint exposed (optional) and scraped by Prometheus

  • NATS connectivity (if enabled) confirmed; health policy chosen

  • GitOps repo reachable, signatures verified (if enabled)

Operational commands

  • Check service status (systemd):

systemctl status gslbd
journalctl -u gslbd -f

  • Check metrics endpoint:

curl -s http://127.0.0.1:9090/metrics | head

  • DNS sanity check:

dig @127.0.0.1 -p 5353 A gslb.local

Common issues and fixes

  1. No DNS answers returned

  • Verify endpoints are valid IPs and reachable.

  • Check health metrics: gslbd_health_endpoints_total{family} and ..._healthy{family}.

  • Confirm policy: with prefer-local, global health won’t override local failures.

  1. GitOps changes not applied

  • Check metrics gslbd_gitops_* for result="error".

  • Inspect logs for signature verification failure.

  • Ensure gitops.pathPrefix points to a directory containing gslbd.yaml (or index.yaml).

  • Validate YAML locally using yamllint and confirm against the Configuration Guide.

  1. NATS state sync inactive or flapping

  • Check gslbd_state_nats_connected and reconnect logs.

  • Verify TLS cert paths and that NATS JetStream is enabled.

  • Ensure clocks are synchronized (NTP) to avoid excessive merge lag.

  1. HTTPS health checks failing with certificate errors

  • Ensure health.http.tls: true and the certificate matches health.http.host (SNI/Host header).

  • Avoid insecureSkipVerify: true in production; if required in lab, set it explicitly.

  1. Running on port 53

  • Either run as root (not recommended) or grant capability:

sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/gslbd

  • Then run as a non-root user.

Upgrades

  • Stop the service, replace the binary, and start again.

  • When using containers, roll out a new image gradually (blue/green or rolling update).

  • Config changes via GitOps: use signed commits; test in a staging branch/cluster first.

Backup and restore

  • GitOps: the Git repository is the source of truth—back it up using your standard VCS backups.

  • NATS JetStream: snapshot metadata and streams per NATS documentation (primarily for diagnostics; the system functions without historical state).

  • Config files: back up /etc/gslb/config.yaml if not using GitOps.

Runbooks

  • Incident: WAN partition / NATS down

    • Expect fallback to local-only behavior (prefer-local policy). No manual action required to continue serving local health.

  • Incident: all endpoints unhealthy

    • Validate backend service; verify health path/port and firewall rules; use curl from the node to test HTTP checks.

  • Rollback a bad config

    • Revert with a signed Git commit; wait for the next poll (or restart gslbd to force reconcile).

Diagnostics

  • Increase verbosity by running in foreground and inspecting logs.

  • Export metrics to a local Prometheus and view trends for health and membership.

Was this article helpful?