Traffic Shadowing

License: pro tier and above. See Licensing.

Standing up a new datacenter or pool? Traffic shadowing clones a percentage of a service's real queries and resolves them a second time against a shadow pool — same candidate building, same filter chain — but the result is discarded, never returned to the client. You get real production query patterns hitting the candidate infrastructure before it ever serves a single real answer, with zero risk to the live path.

How It Works

query arrives ──▶ resolve against the primary pool (as always) ──▶ answer sent to client
       │
       └─(shadowPercent% of the time)─▶ go resolve against the shadow pool
                                          (same filter chain, discarded result)
                                          │
                                          └─▶ recorded: healthy? how long did it take?

Shadow resolution always runs in its own goroutine, fired independently of whatever the primary route decided (even if the primary pool is down and served a fallback CNAME) — dark-testing wants to see the traffic a query would generate, regardless of whether the live pool happened to serve it. It never blocks or slows down the real answer.

Configuring It

WebUI: open a service (from the topology view or its detail page) → Traffic Shadowing card → pick a shadow pool and a percentage.

API:

{  "shadowPoolId": "pool-uuid",  "shadowPercent": 10}

gslbctl:

gslbctl services update <id> --name web --domain www.example.com \  --shadow-pool <pool-id> --shadow-percent 10gslbctl services shadow-stats <id>

Terraform:

resource "nexus_service" "web" {
  name           = "web"
  domain         = "www.example.com"
  pool_id        = nexus_pool.primary.id
  shadow_pool_id = nexus_pool.candidate_dc.id
  shadow_percent = 10
}

Set shadowPercent to 100 to shadow every query while you're confident in the candidate pool's capacity; dial it down for a cautious ramp-up.

Watching the Results

The Traffic Shadowing card (WebUI) and gslbctl services shadow-stats <id> both show, from the last completed 60-second window:

  • Primary — the live path's QPS and average latency (the same numbers the dashboard already tracks).

  • Shadow — how many shadow resolutions ran, what fraction found a healthy candidate in the shadow pool, and the shadow path's own resolution latency.

The topology view also draws a dashed amber edge from the service to its shadow pool, labeled with the configured percentage — display only, it can't be dragged or deleted from the canvas (use the edit panel or API to change shadow config).

Notes

  • Shadow stats are in-memory, rolling-window telemetry — the same approach this codebase already uses for primary QPS/latency, not a per-query database write. At shadowPercent=100, writing one DB row per shadowed query would double the DNS engine's write load on the datastore for a feature that's pure observability; a 60-second in-memory window is the honest trade.

  • A shadow resolution reuses the service's own filter chain (or algorithm) against the shadow pool's members — so it exercises the same routing logic, health checks, and any scriptable steps, just against different backends.

  • Not gated by DNSSEC, RRL, or RPZ — shadow resolution runs after those checks, mirroring only the pool-selection part of the primary path.

Was this article helpful?