Routing: Service Discovery

Service Discovery

Nexus GSLB can auto-populate pool members from external service registries. When a service registers or deregisters with the registry, Nexus adds or removes the corresponding pool member automatically — no manual API calls required.

Discovered members coexist with manually-managed members in the same pool.

Supported registries

Registry Status License requirement
HashiCorp Consul Supported Paid tier or above
etcd Supported Pro tier or above

Discovery is disabled at startup for registries whose tier requirement is not met. Contact licensing@gslb.nexus to upgrade.

Consul

Prerequisites

  • A running Consul agent or cluster reachable from the gslbd node(s).
  • One or more services registered in Consul with an address and port.
  • Optional: Consul ACL token with service:read permission.

Configuration

<span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="fu">discovery</span><span class="kw">:</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">consul</span><span class="kw">:</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">addr</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;http://127.0.0.1:8500&quot;</span><span class="co"> # Consul HTTP API address</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">token</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # ACL token (empty = anonymous)</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">datacenter</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # empty = agent default datacenter</span></span>
<span id="cb1-6"><a href="#cb1-6" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">tls</span><span class="kw">:</span></span>
<span id="cb1-7"><a href="#cb1-7" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">caFile</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # CA cert for HTTPS verification</span></span>
<span id="cb1-8"><a href="#cb1-8" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">certFile</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # client cert (mTLS)</span></span>
<span id="cb1-9"><a href="#cb1-9" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">keyFile</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # client key (mTLS)</span></span>
<span id="cb1-10"><a href="#cb1-10" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">watches</span><span class="kw">:</span></span>
<span id="cb1-11"><a href="#cb1-11" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">service</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;web&quot;</span><span class="co"> # Consul service name to watch</span></span>
<span id="cb1-12"><a href="#cb1-12" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">poolID</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;web-prod&quot;</span><span class="co"> # Nexus pool to populate</span></span>
<span id="cb1-13"><a href="#cb1-13" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">tag</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # filter by Consul tag (empty = any)</span></span>
<span id="cb1-14"><a href="#cb1-14" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">onlyPassing</span><span class="kw">:</span><span class="at"> </span><span class="ch">true</span><span class="co"> # only include services with all checks passing</span></span>
<span id="cb1-15"><a href="#cb1-15" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">defaultPort</span><span class="kw">:</span><span class="at"> </span><span class="dv">0</span><span class="co"> # use if service port is 0 in Consul</span></span>
<span id="cb1-16"><a href="#cb1-16" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">defaultWeight</span><span class="kw">:</span><span class="at"> </span><span class="dv">1</span><span class="co"> # weight for members without Consul weights</span></span>
<span id="cb1-17"><a href="#cb1-17" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">region</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # region override (empty = Consul node metadata &quot;region&quot;, then datacenter)</span></span>

How it works

  1. On startup, gslbd connects to the Consul HTTP API and performs an initial fetch of all registered instances for each watched service.
  2. It then holds a blocking query (?index=X&wait=60s) — Consul blocks the request until the service catalog changes, then returns immediately with the updated list.
  3. On each update, gslbd reconciles the pool:
    • New instances → CreateMember (tagged with source: consul)
    • Changed instances (weight, region, name) → UpdateMember
    • Gone instances → DeleteMember
  4. If Consul is unreachable, the watcher retries with exponential backoff (1 s → 30 s maximum). Existing members remain untouched during the outage.

Member properties

Member field Source
ipAddress Service.Address, fallback to Node.Address
port Service.Port, fallback to defaultPort
weight Service.Weights.Passing, fallback to defaultWeight
region watch.region override → Node.Meta["region"] → Node.Datacenter
name {Node.Node}/{Service.ID}
source "consul"
priority Always 0 (equal priority)
enabled Always true

Health filtering

When onlyPassing: true (the default), Nexus passes ?passing=true to Consul's health API. This means only service instances where all registered health checks are passing are returned. Instances in warning or critical state are excluded from the pool automatically.

When onlyPassing: false, all registered instances are returned regardless of check state. Nexus's own health checker (if configured) will still probe them independently.

Example: two pools from one Consul cluster

<span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="fu">discovery</span><span class="kw">:</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">consul</span><span class="kw">:</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">addr</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;http://consul.internal:8500&quot;</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">token</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;a1b2c3d4-...&quot;</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">watches</span><span class="kw">:</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">service</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;web-frontend&quot;</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">poolID</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;frontend-pool&quot;</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">tag</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;production&quot;</span></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">onlyPassing</span><span class="kw">:</span><span class="at"> </span><span class="ch">true</span></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">defaultWeight</span><span class="kw">:</span><span class="at"> </span><span class="dv">10</span></span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">service</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;api-backend&quot;</span></span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">poolID</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;api-pool&quot;</span></span>
<span id="cb2-13"><a href="#cb2-13" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">onlyPassing</span><span class="kw">:</span><span class="at"> </span><span class="ch">true</span></span>
<span id="cb2-14"><a href="#cb2-14" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">defaultWeight</span><span class="kw">:</span><span class="at"> </span><span class="dv">5</span></span>
<span id="cb2-15"><a href="#cb2-15" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">region</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;us-east-1&quot;</span><span class="co"> # force region for all api-backend instances</span></span>

Combining with manual members

Discovered members are tagged with source: "consul". You can add manually-managed members to the same pool — they will have source: "" and are never touched by the discovery reconciler.

Visibility

  • API: GET /api/v1/pools/{poolId}/members — returned members include "source": "consul".
  • WebUI: Pool detail page shows a consul badge next to discovered member entries.
  • TUI: Member list shows [consul] suffix after discovered entries.
  • gslbctl: gslbctl members list --pool <id> — no change needed; source field is visible in the raw output.

Permissions

Discovery runs with the daemon's own DB access — no API key or user permission is required. The Consul ACL token is configured in gslbd.yaml only.


etcd

Prerequisites

  • A running etcd v3 cluster reachable from the gslbd node(s) via its HTTP gateway (default port 2379).
  • One or more keys registered under a common prefix, each containing a JSON value with at minimum ip and port.
  • Optional: etcd username/password for authentication, or client TLS certificates.

Configuration

<span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="fu">discovery</span><span class="kw">:</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">etcd</span><span class="kw">:</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">endpoints</span><span class="kw">:</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="st">&quot;http://127.0.0.1:2379&quot;</span><span class="co"> # etcd HTTP gateway address (only first endpoint is used)</span></span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">username</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # authentication username (empty = no auth)</span></span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">password</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # authentication password</span></span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">tls</span><span class="kw">:</span></span>
<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">caFile</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # CA cert for HTTPS verification</span></span>
<span id="cb3-9"><a href="#cb3-9" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">certFile</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # client cert (mTLS)</span></span>
<span id="cb3-10"><a href="#cb3-10" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">keyFile</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;&quot;</span><span class="co"> # client key (mTLS)</span></span>
<span id="cb3-11"><a href="#cb3-11" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">watches</span><span class="kw">:</span></span>
<span id="cb3-12"><a href="#cb3-12" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">prefix</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;/nexus/members/web-prod/&quot;</span><span class="co"> # KV key prefix to watch</span></span>
<span id="cb3-13"><a href="#cb3-13" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">poolID</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;web-prod&quot;</span><span class="co"> # Nexus pool to populate</span></span>
<span id="cb3-14"><a href="#cb3-14" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">defaultWeight</span><span class="kw">:</span><span class="at"> </span><span class="dv">1</span><span class="co"> # weight for members without a weight in their value</span></span>
Field Default Description
endpoints — List of etcd HTTP gateway URLs. Only the first entry is used.
username "" Username for etcd authentication. Leave empty to skip authentication.
password "" Password for etcd authentication.
tls.caFile "" Path to a CA certificate file for verifying the etcd server's TLS certificate.
tls.certFile "" Path to a client certificate for mTLS authentication.
tls.keyFile "" Path to the client certificate's private key.
watches[].prefix — Required. The key prefix to scan and watch. All keys with this prefix are treated as pool members.
watches[].poolID — Required. The Nexus pool to populate.
watches[].defaultWeight 1 Weight applied to members whose JSON value does not include a weight field, or where weight is 0.

Value format

Each key under the watched prefix must have a JSON value:

<span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="fu">{</span><span class="dt">&quot;ip&quot;</span><span class="fu">:</span> <span class="st">&quot;10.0.0.1&quot;</span><span class="fu">,</span> <span class="dt">&quot;port&quot;</span><span class="fu">:</span> <span class="dv">443</span><span class="fu">,</span> <span class="dt">&quot;weight&quot;</span><span class="fu">:</span> <span class="dv">10</span><span class="fu">,</span> <span class="dt">&quot;region&quot;</span><span class="fu">:</span> <span class="st">&quot;us-east&quot;</span><span class="fu">}</span></span>
Field Required Description
ip ✓ IP address of the member.
port ✓ Port number. Must be non-zero.
weight — Routing weight. Falls back to defaultWeight when absent or 0.
region — Region label for geo-aware routing. Defaults to "unknown" when absent.

Keys whose value is missing ip or has port equal to 0 are silently skipped.

How it works

  1. On startup, gslbd authenticates with etcd (if credentials are configured) via the etcd v3 /v3/auth/authenticate endpoint.
  2. It performs an initial key range fetch using /v3/kv/range for each watched prefix and reconciles the pool.
  3. gslbd then opens a streaming watch via the /v3/watch HTTP endpoint. The stream delivers change events for the key range as they occur.
  4. On each batch of events received, gslbd performs a full re-fetch of the key range and reconciles the pool:
    • New keys → CreateMember (tagged with source: etcd)
    • Changed keys (weight, region, port) → UpdateMember
    • Deleted keys → DeleteMember
  5. If the connection fails, the watcher retries with exponential backoff starting at 1 s, capping at 30 s. Existing members remain untouched during the outage.

Member properties

Member field Source
ipAddress ip field in the JSON value
port port field in the JSON value
weight weight field in the JSON value; fallback to defaultWeight (default 1)
region region field in the JSON value; fallback to "unknown"
name Full etcd key string (e.g. /nexus/members/web-prod/node-01)
source "etcd"
priority Always 0 (equal priority)
enabled Always true

Example: auto-register application nodes

<span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="fu">discovery</span><span class="kw">:</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">etcd</span><span class="kw">:</span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">endpoints</span><span class="kw">:</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="st">&quot;https://etcd.internal:2379&quot;</span></span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">tls</span><span class="kw">:</span></span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">caFile</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;/etc/ssl/etcd-ca.pem&quot;</span></span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">watches</span><span class="kw">:</span></span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">prefix</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;/services/web/nodes/&quot;</span></span>
<span id="cb5-9"><a href="#cb5-9" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">poolID</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;web-pool&quot;</span></span>
<span id="cb5-10"><a href="#cb5-10" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">defaultWeight</span><span class="kw">:</span><span class="at"> </span><span class="dv">10</span></span>
<span id="cb5-11"><a href="#cb5-11" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="kw">-</span><span class="at"> </span><span class="fu">prefix</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;/services/api/nodes/&quot;</span></span>
<span id="cb5-12"><a href="#cb5-12" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">poolID</span><span class="kw">:</span><span class="at"> </span><span class="st">&quot;api-pool&quot;</span></span>
<span id="cb5-13"><a href="#cb5-13" aria-hidden="true" tabindex="-1"></a><span class="at"> </span><span class="fu">defaultWeight</span><span class="kw">:</span><span class="at"> </span><span class="dv">5</span></span>

Application nodes register themselves on startup:

<span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="ex">etcdctl</span> put /services/web/nodes/web-01 <span class="st">&#39;{&quot;ip&quot;:&quot;10.0.1.10&quot;,&quot;port&quot;:80,&quot;region&quot;:&quot;eu-west&quot;}&#39;</span></span>

Deregistration on shutdown:

<span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="ex">etcdctl</span> del /services/web/nodes/web-01</span>

Combining with manual members

Discovered members are tagged with source: "etcd". You can add manually-managed members to the same pool — they will have source: "" and are never touched by the discovery reconciler.

Visibility

  • API: GET /api/v1/pools/{poolId}/members — returned members include "source": "etcd".
  • WebUI: Pool detail page shows an etcd badge next to discovered member entries.
  • TUI: Member list shows [etcd] suffix after discovered entries.
  • gslbctl: gslbctl members list --pool <id> — source field is visible in the raw output.

Operational notes

Multiple cluster nodes: Each gslbd node runs its own discovery watchers independently. All nodes write to their own local SQLite DB. The result is consistent across nodes via the normal NATS state sync. This means each node maintains a correctly reconciled member list without a central coordinator.

Deregistration delay: When a Consul instance deregisters, the gslbd watcher receives the update within the Consul ?wait=60s window. Under normal conditions the member is removed within seconds of deregistration.

Manual override: If you delete a discovered member via the API or TUI, the discovery reconciler will re-add it on the next sync. To permanently exclude an instance, use Consul's native maintenance mode or deregister it from Consul rather than from Nexus.


Was this article helpful?