Encrypted DNS (DoT/DoH)

Encrypted DNS: DoT and DoH

Nexus GSLB supports two encrypted DNS transports:

Protocol	RFC	Default port	Use case
DNS over TLS (DoT)	RFC 7858	853	Stub resolvers, mobile OS, enterprise DNS policies
DNS over HTTPS (DoH)	RFC 8484	443	Browsers, applications, HTTP proxies

Both transports share the same resolution logic as plain UDP/53. DNSSEC signing works identically across all transports. Unlike UDP, TCP-based transports never need to truncate large responses — Nexus detects the transport automatically and skips the 1232-byte UDP payload limit for DoT and DoH connections.

Prerequisites

Both DoT and DoH require a valid TLS certificate and private key. The certificate must cover the hostname your clients will connect to.

Option A — Let's Encrypt (recommended for public deployments)

certbot certonly --standalone -d ns1.example.com# Certificate: /etc/letsencrypt/live/ns1.example.com/fullchain.pem# Key:         /etc/letsencrypt/live/ns1.example.com/privkey.pem

Option B — Self-signed (internal/lab use only)

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 \  -keyout /etc/nexus/dot.key -out /etc/nexus/dot.crt \  -days 365 -nodes -subj "/CN=ns1.example.com"

Clients querying a self-signed certificate will need to trust it explicitly (e.g. via a CA bundle or system trust store).

Configuration

Both listeners are disabled by default. Add the relevant block to your config.yaml under the dns: section.

DNS over TLS (DoT)

dns:  listenAddr: "0.0.0.0"  port: 53  domain: gslb.example.com  dot:    enabled: true    port: 853               # optional; 853 is the default    certFile: /etc/nexus/tls/fullchain.pem    keyFile:  /etc/nexus/tls/privkey.pem

DNS over HTTPS (DoH)

dns:  listenAddr: "0.0.0.0"  port: 53  domain: gslb.example.com  doh:    enabled: true    port: 443               # optional; 443 is the default    certFile: /etc/nexus/tls/fullchain.pem    keyFile:  /etc/nexus/tls/privkey.pem    path: /dns-query        # optional; /dns-query is the default

Both enabled simultaneously

DoT and DoH can run at the same time alongside plain UDP/53. All three listeners share the same certificate files if you want to reuse a single cert:

dns:  listenAddr: "0.0.0.0"  port: 53  domain: gslb.example.com  dot:    enabled: true    certFile: /etc/nexus/tls/fullchain.pem    keyFile:  /etc/nexus/tls/privkey.pem  doh:    enabled: true    certFile: /etc/nexus/tls/fullchain.pem    keyFile:  /etc/nexus/tls/privkey.pem

Configuration reference

Field	Type	Default	Description
`dns.dot.enabled`	bool	`false`	Enable DoT listener
`dns.dot.port`	int	`853`	TCP port for DoT
`dns.dot.certFile`	string	—	Path to PEM certificate (required when enabled)
`dns.dot.keyFile`	string	—	Path to PEM private key (required when enabled)
`dns.doh.enabled`	bool	`false`	Enable DoH listener
`dns.doh.port`	int	`443`	TCP port for DoH
`dns.doh.certFile`	string	—	Path to PEM certificate (required when enabled)
`dns.doh.keyFile`	string	—	Path to PEM private key (required when enabled)
`dns.doh.path`	string	`/dns-query`	HTTP path for the DoH endpoint

Firewall rules

Open the new ports in addition to the existing UDP/53 rule:

# DoTufw allow 853/tcp comment "DNS over TLS"# DoH (if not sharing port 443 with the web UI)ufw allow 443/tcp comment "DNS over HTTPS / HTTPS"

If the Nexus web UI and the DoH listener both need port 443 on the same address, put a reverse proxy (e.g. Caddy or nginx) in front and forward /dns-query to a non-standard DoH port (e.g. 8443):

dns:  doh:    enabled: true    port: 8443    path: /dns-query    certFile: /etc/nexus/tls/fullchain.pem    keyFile:  /etc/nexus/tls/privkey.pem

# Caddyfile excerpt
ns1.example.com {
    reverse_proxy /dns-query localhost:8443 {
        transport http { tls_insecure_skip_verify }
    }
}

TestingDoT with kdig (from knot-dnsutils)

kdig @ns1.example.com +tls app.gslb.example.com A# Self-signed cert:kdig @ns1.example.com +tls +tls-ca=/etc/nexus/tls/ca.crt app.gslb.example.com A

DoH with curl

# POST (wire format)DNS_MSG=$(printf '\x00\x01\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03app\x04gslb\x07example\x03com\x00\x00\x01\x00\x01')curl -s -X POST "https://ns1.example.com/dns-query" \  -H "Content-Type: application/dns-message" \  --data-binary "$DNS_MSG" \  --output - | hexdump -C# GET (base64url-encoded query — use dog or doh-client for convenience)dog app.gslb.example.com A --tls-host ns1.example.com

DoH with dog

# Install: https://github.com/ogham/dogdog @https://ns1.example.com/dns-query app.gslb.example.com A

Verify DNSSEC over DoH

kdig @ns1.example.com +https=/dns-query +dnssec app.gslb.example.com A# Expect: RRSIG record in the answer section, AD flag set

Certificate reload

Nexus loads the TLS certificate once at startup. To roll a renewed certificate without downtime:

Replace the cert/key files on disk (e.g. via a certbot deploy hook or cp).
Send SIGHUP or use the API restart endpoint:

curl -X POST https://nexus-api.example.com/api/v1/restart \  -H "Authorization: Bearer $GSLB_API_KEY"

The daemon performs a graceful restart — in-flight DNS queries complete before the listener is torn down. Certbot deploy hook example:

# /etc/letsencrypt/renewal-hooks/deploy/nexus-reload.sh#!/bin/bashcp /etc/letsencrypt/live/ns1.example.com/fullchain.pem /etc/nexus/tls/fullchain.pemcp /etc/letsencrypt/live/ns1.example.com/privkey.pem  /etc/nexus/tls/privkey.pemsystemctl reload-or-restart gslbd

Performance notes

DoT and DoH have higher per-query overhead than UDP/53 due to the TLS handshake. Modern clients reuse persistent connections, so the handshake cost is amortised over many queries.
GSLB responses are typically small (a single A or AAAA record) and will be served from the in-memory service cache without the TCP payload size limit ever being relevant.
If you are serving many short-lived clients (e.g. IoT devices that do not reuse connections), consider placing a caching DoH resolver (e.g. Unbound, Pi-hole) in front of Nexus and pointing it at UDP/53 for upstream queries.

Client configuration examplesAndroid (Private DNS)

Settings → Network & Internet → Private DNS → enter ns1.example.com (uses DoT on port 853 automatically).

iOS / macOS (DNS over HTTPS via profile)

Create a .mobileconfig profile with:

<key>DNSSettings</key><dict>  <key>DNSProtocol</key>  <string>HTTPS</string>  <key>ServerURL</key>  <string>https://ns1.example.com/dns-query</string></dict>

Deploy via Apple Configurator or MDM.

Firefox

about:config → set network.trr.mode to 2 and network.trr.uri to https://ns1.example.com/dns-query.

systemd-resolved (Linux)

# /etc/systemd/resolved.conf[Resolve]DNS=ns1.example.comDNSOverTLS=yes

systemctl restart systemd-resolvedresolvectl status

Unbound (forwarding resolver)

forward-zone:    name: "."    forward-tls-upstream: yes    forward-addr: <nexus-ip>@853#ns1.example.com