Health: Remote Probe Agents

Remote probe agents (gslb-probe) extend Nexus GSLB health and latency vantage beyond the cluster nodes. Drop a small probe-only binary anywhere — an edge VM, a Kubernetes pod, a branch office, a cloud region where no full node runs — and it contributes health verdicts and RTT samples from that vantage into the same aggregation the full nodes use.

Unlike SaaS external-monitoring products (Constellix Sonar, Catchpoint, ThousandEyes), probes are self-hosted and feed directly into GSLB routing decisions.

Licence: pro tier (remote_probes). See Licensing for the per-tier probe cap.

Vantage classes

Every health-reporting endpoint in the cluster is a vantage, of one of two kinds:

Kind Node ID Role
node e.g. lon-01 Authoritative full node — serves DNS, owns config, votes in quorum.
probe probe-<name> External observer — health + RTT only. Informational for serving decisions; never serves DNS.

Health aggregation is healthy-from-ANY-vantage, so a probe reporting an endpoint healthy keeps it in rotation; a probe reporting it unhealthy does not by itself remove it (a full node still decides). Probe RTTs feed latency routing (GlobalMinRTT) for free — adding probes quietly widens latency-routing coverage.

gslbctl cluster status and the WebUI cluster page list probes separately from nodes, with a last-seen/staleness indicator.

How it works

  1. When remote probes are licensed, each full node projects its health-check config + target IP set into a TTL'd KV bucket (gslb_<cluster>_probe_assignments). This is an ephemeral projection of the git/rqlite-owned config — not durable config — consistent with the NATS "ephemeral only" contract.
  2. gslb-probe connects to the cluster NATS, reads its assignment, builds a health checker over the assigned targets, and publishes health, heartbeat, and RTT into the same subjects/buckets full nodes use, as node probe-<name>.
  3. The (node_id, …) health schema and the healthy-from-ANY aggregation accept the probe vantage with no changes.

The probe hot-swaps its checker whenever the assignment changes, and derives its KV bucket TTLs from the assignment interval.

Running gslb-probe

<span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">gslb-probe</span> <span class="dt">\</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>  <span class="at">-nats</span> nats://10.100.0.1:4222 <span class="dt">\</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a>  <span class="at">-cluster</span> default <span class="dt">\</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a>  <span class="at">-name</span> edge-fra <span class="dt">\</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a>  <span class="at">-region</span> eu-central <span class="dt">\</span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a>  <span class="at">-nats-creds</span> /etc/gslb-probe/nats.creds</span>

The node ID becomes probe-edge-fra. Every flag has an environment-variable equivalent:

Flag Env Purpose
-nats NEXUS_NATS_SERVERS comma-separated NATS URLs (required)
-cluster NEXUS_CLUSTER_ID cluster ID; must match gslbd cluster.id
-name NEXUS_PROBE_NAME probe name → node ID probe-<name> (required)
-region NEXUS_PROBE_REGION informational vantage label
-nats-creds NEXUS_NATS_CREDS NATS credentials file
-nats-nkey NEXUS_NATS_NKEY NATS NKey seed file
-nats-user / -nats-pass NEXUS_NATS_USER / NEXUS_NATS_PASS NATS user/password
-nats-ca / -nats-cert / -nats-key NEXUS_NATS_CA / _CERT / _KEY NATS TLS
-webhook-allow NEXUS_WEBHOOK_ALLOW CIDRs allowed as webhook-check targets (local SSRF allowlist)
-allow-script-checks NEXUS_ALLOW_SCRIPT_CHECKS=true permit type:script checks (runs arbitrary commands; default off)

A Kubernetes DaemonSet example (one probe per node) ships at deploy/kubernetes/gslb-probe-daemonset.yaml.

ICMP checks

TCP, HTTP, and webhook checks work with no special privileges. ICMP (ping) checks need CAP_NET_RAW; grant it explicitly (the example DaemonSet drops all capabilities by default).

Security

A probe stamps its own webhook SSRF allowlist (-webhook-allow) and refuses script checks unless -allow-script-checks is set — a compromised assignment writer can neither widen a probe's egress nor make it run arbitrary commands.

Scope probe NATS credentials to health only

Probes must not be able to write anything but health/heartbeat/RTT. Issue probe credentials whose permissions cover only the health subjects and KV buckets, and read-only on the assignment bucket. A representative permission set (tune the exact $JS.API.* subjects to your NATS/JetStream deployment):

publish:
  allow:
    - &quot;gslb.<cluster>.health.probe-*.>&quot;      # own health events
    - &quot;gslb.<cluster>.membership.heartbeat&quot;  # liveness
    - &quot;$KV.gslb_<cluster>_health.>&quot;          # health KV
    - &quot;$KV.gslb_<cluster>_membership.>&quot;      # heartbeat KV
    - &quot;$KV.gslb_<cluster>_rtt.>&quot;             # RTT KV
subscribe:
  allow:
    - &quot;$KV.gslb_<cluster>_probe_assignments.>&quot;  # read assignment (watch)
    - &quot;_INBOX.>&quot;

Grant no access to any other subject or bucket. In this architecture durable config lives in git/rqlite (never in NATS), so a health-scoped probe credential cannot reach configuration at all — but keep the allowlist tight regardless. See the Security Guide for the broader hardening checklist.

Was this article helpful?