# M6.3 pdns-recursor Runbook

**Status**: Operator runbook for local self-resolution via pdns-recursor.
**Design**: ADR-0047 (`docs/adr/0047-pdns-recursor-local-self-resolution.md`).
**Plan**: `plans/m6.3-pdns-recursor.md` (7-step blueprint).

## Architecture at a glance

```
┌─────────────────────────┐   127.0.0.53:53    ┌──────────────────────┐
│  every glibc app        │ ───────────────►   │  systemd-resolved    │
│  (reads /etc/resolv.conf)│                   │  stub                │
└─────────────────────────┘                   └──────┬───────────────┘
                                                     │ DNS=127.0.0.1
                                                     ▼
                                              ┌──────────────────────┐
                    authoritative zones       │  pdns-recursor       │
                    via recursor.forwards ────►  127.0.0.1:53 (v4+v6)│
                                              │  allow-from=lo only  │
                                              └──────┬───────────────┘
                                          forward    │    recurse everything else
                                          127.0.0.1:5300  to 1.1.1.1 / 9.9.9.9
                                                     │
                                                     ▼
         ┌──────────────────────┐             ┌──────────────────────┐
         │  public internet     │ ◄──:53─────┤  pdns-server         │
         │  (NS delegation)     │   public IP │  :5300 loopback      │
         └──────────────────────┘             │  (authoritative)     │
                                              └──────────────────────┘
```

Key points:

- **pdns-server** keeps the public IP `:53` for authoritative external queries. Loopback is `:5300` only — recursor forwards into it. No change to how NS delegation works.
- **pdns-recursor** owns loopback `:53`. `allow-from=127.0.0.0/8, ::1/128` (narrower than Debian default; LXC bridge ranges are explicitly out).
- **systemd-resolved** stub still listens on `127.0.0.53:53` — glibc apps reading `/etc/resolv.conf` still hit the stub — but the stub's DNS= upstream is now `127.0.0.1` (recursor) thanks to the `zz-jabali-recursor.conf` drop-in.
- `/etc/powerdns/recursor.forwards` is **reconciler-owned**. Every edit flows through the agent (`pdns.recursor_add_zone` / `_remove_zone`) with atomic write + validator + `rec_control reload-zones` + SOA post-probe + rollback on fail.

## Verify health

On any healthy M6.3 host:

```bash
# 1. Both pdns services are active.
systemctl is-active pdns pdns-recursor systemd-resolved
# expect: active / active / active

# 2. Stub → recursor → public chain.
dig +short @127.0.0.53 deb.debian.org
# expect: one or more A records

# 3. Recursor answers directly.
dig +short @127.0.0.1 . NS | head -3
# expect: root NS names (a.root-servers.net etc.)

# 4. Recursor answers the panel's own hostname authoritatively.
dig @127.0.0.1 SOA $(hostname -f) +noall +answer
# expect: an SOA record with the panel hostname as the MNAME

# 5. systemd-resolved is pointed at 127.0.0.1 only (the M6.3 drop-in is working).
resolvectl status | grep "DNS Servers"
# expect: "DNS Servers: 127.0.0.1" (maybe repeated per-interface; NOT "1.1.1.1")

# 6. Bind surface — pdns-server public-IP :53, pdns-recursor loopback :53.
ss -lnup sport = :53
# expect lines like:
#   UNCONN  ...  ${PANEL_IPV4}:53      ... users:(("pdns_server",...))
#   UNCONN  ...  127.0.0.1:53          ... users:(("pdns-recursor",...))
#   UNCONN  ...  [::1]:53              ... users:(("pdns-recursor",...))
#   UNCONN  ...  127.0.0.53:53         ... users:(("systemd-resolve",...))
ss -lnup sport = :5300
# expect: pdns-server on 127.0.0.1:5300 + [::1]:5300
```

## Amplification-defense verification

This is operator-driven and MUST be done from **off-host** (a second VM, a laptop, any network not on the same loopback/LXC-internal L2). In-host probes can't verify amplification defense because the kernel rewrites the source address to loopback when the destination is loopback — the check always passes falsely.

From an off-host machine:

```bash
# Against the panel's public IP — authoritative path. Should answer
# panel zones (this is the public pdns-server :53).
dig @<panel-public-ip> <a-panel-domain> SOA
# expect: authoritative answer

# Against the panel's public IP — recursion attempt. Should REFUSE
# because pdns-server is authoritative-only.
dig @<panel-public-ip> google.com A
# expect: REFUSED or no answer

# Attempt to reach the recursor from outside. The recursor binds
# 127.0.0.1 + ::1 so this query CAN'T physically reach it — you'll
# see a connection refused / timeout at the TCP/UDP layer. If you
# DO get a REFUSED DNS response, something is badly wrong and the
# recursor might be bound to a routable address.
dig +time=3 +tries=1 @<panel-public-ip> -p 53 . NS
# expect: timeout (no response), NOT a DNS REFUSED packet
```

If any of these expectations doesn't hold, treat as a security incident:

1. `cat /etc/powerdns/recursor.conf | grep -E 'local-address|allow-from'`
2. `ss -lnup sport = :53` to confirm no recursor process is bound to a routable address.
3. If `allow-from=` contains anything outside `127.0.0.0/8, ::1/128`, revert to the managed config: delete `/etc/powerdns/recursor.conf` and re-run `./install.sh`. install.sh hard-fails if it ever renders a non-loopback config.

## Common operations

### Add / remove a zone forwarder manually

Use `jabali pdns backfill` — don't hand-edit `/etc/powerdns/recursor.forwards`. The reconciler owns that file and will overwrite hand edits on its next tick.

```bash
# Dry-run: print the plan, change nothing.
jabali pdns backfill

# Apply changes (prompts y/N by default).
jabali pdns backfill --yes

# Apply without prompt (scripted runs).
JABALI_PDNS_BACKFILL_NONINTERACTIVE=1 jabali pdns backfill --yes

# More detail (per-zone).
jabali pdns backfill --verbose
```

In steady state, `jabali pdns backfill` (dry-run) should report every zone as `noop`. Any `add` / `remove` / `update` lines mean the reconciler is paused, crashed, or behind.

### Override the upstream recursor target

`forward-zones-recurse=.=1.1.1.1;9.9.9.9` in `/etc/powerdns/recursor.conf` is the public-recursion fallback. If your deployment needs a specific upstream (corporate DNS, different quad), edit the line and restart:

```bash
# Edit recursor.conf; change the forward-zones-recurse line.
# Don't touch anything else — install.sh re-validates local-address +
# allow-from on re-run and will _die if you widened them.
sudo editor /etc/powerdns/recursor.conf

sudo systemctl restart pdns-recursor
sudo systemctl is-active pdns-recursor || journalctl -u pdns-recursor -n 50

# Your change survives `./install.sh` re-runs UNLESS install.sh's
# render-to-.new cmp says the file drifted from managed. Add a
# second drop-in under /etc/powerdns/recursor.conf.d/ if Debian's
# recursor supports drop-ins; otherwise document the override in
# your ops notes.
```

### Fall back when the recursor dies

If pdns-recursor is down and you need the host to resolve public names (debian-security.debian.org for updates, etc.) while you debug:

```bash
# 1. Point resolved at the fallback list directly (bypass recursor).
sudo tee /etc/systemd/resolved.conf.d/zz-jabali-recursor.conf > /dev/null <<'EOF'
[Resolve]
DNS=1.1.1.1 9.9.9.9
FallbackDNS=
DNSSEC=no
EOF
sudo systemctl restart systemd-resolved

# 2. Host resolves public names again. Panel zones WON'T resolve until
#    you bring the recursor back.
dig +short deb.debian.org

# 3. Fix + restore when ready.
sudo ./install.sh   # idempotent; re-writes zz-jabali-recursor.conf to managed state
```

### Clean up pre-M6.3 `/etc/hosts` overrides

Operators who added `/etc/hosts` entries as a pre-M6.3 workaround should remove them after M6.3 is live — glibc NSS resolves `/etc/hosts` BEFORE DNS, so stale hosts entries mask the recursor's authoritative answers.

```bash
# Find panel domains in /etc/hosts.
grep -E '^\s*(127\.0\.0\.1|\S+\s+(mail\.|autoconfig\.))' /etc/hosts

# Back up + edit. Leave the two standard lines (127.0.0.1 localhost,
# ::1 ip6-localhost); remove anything pointing panel-managed domains
# at 127.0.0.1.
sudo cp /etc/hosts /etc/hosts.pre-m6.3
sudo editor /etc/hosts

# Verify the recursor serves what /etc/hosts used to.
dig +short @127.0.0.53 mail.<a-panel-domain>
# expect: panel's public IP
```

### Inspect recursor cache / runtime state

`rec_control` is available to root:

```bash
sudo rec_control top-queries          # Most-queried names.
sudo rec_control current-queries      # What the recursor is chasing right now.
sudo rec_control get-qtypelist        # Histogram of query types.
sudo rec_control dump-cache /tmp/cache.dump    # Snapshot for forensics.
sudo rec_control reload-zones         # Re-read recursor.forwards.
```

Never run `rec_control` commands that modify running state (`reload-lua-config`, `quit`, etc.) without coordinating — a recursor restart briefly blackholes every DNS query on the host.

## Troubleshooting

### Symptom: `resolvectl status` shows `DNS Servers: 1.1.1.1 9.9.9.9` after install

install.sh's post-install probe should have hard-failed before you got here — but if you've arrived here with a half-installed host, the `zz-jabali-recursor.conf` `DNS=` reset didn't take globally. Two recovery paths:

1. **Preferred**: merge the files. Delete `jabali.conf` (if present) and roll its `DNS=` values into `zz-jabali-recursor.conf`'s `FallbackDNS=`:
   ```
   [Resolve]
   DNS=127.0.0.1
   FallbackDNS=1.1.1.1 9.9.9.9
   DNSSEC=no
   ```
   The panel UI's "DNS Resolvers" page will then edit `FallbackDNS=` instead of `DNS=` — note that the UI may need a follow-up patch to know this.
2. **Minimal**: edit `zz-jabali-recursor.conf` to add a trailing `DNS=` line repeating the reset. Usually unnecessary; file a bug if you need this.

### Symptom: `dig @127.0.0.53 <panel-zone>` returns SERVFAIL

Chain break. Isolate:

```bash
# Step 1 — recursor up?
systemctl is-active pdns-recursor || sudo systemctl restart pdns-recursor

# Step 2 — recursor answering at all?
dig +short @127.0.0.1 . NS | head -1
# if empty, the recursor is up but not recursing — check journal:
journalctl -u pdns-recursor -n 100

# Step 3 — forwarder for the zone present?
grep "<panel-zone>=" /etc/powerdns/recursor.forwards
# if missing, converge via CLI:
jabali pdns backfill --yes

# Step 4 — pdns-server answering on :5300?
dig +short @127.0.0.1 -p 5300 SOA <panel-zone>
# if SERVFAIL, pdns-server is alive but doesn't know the zone — check
# the domains table + bootstrap_pdns_self_zone for the hostname zone.

# Step 5 — rec_control can see the forwarder.
sudo rec_control get-qtypelist
sudo rec_control reload-zones
```

### Symptom: `jabali pdns backfill` shows `add` / `update` / `remove` in steady state

The reconciler isn't converging. Check:

```bash
# 1. Is jabali-panel running?
systemctl is-active jabali-panel || journalctl -u jabali-panel -n 200

# 2. Is the agent UDS reachable?
ls -l /run/jabali-agent/agent.sock
# expect: srw-rw---- root jabali (or similar)

# 3. Is the reconciler paused? (SSO key rotation uses a pause flag.)
# Check for "reconciler: paused" in recent panel logs.
journalctl -u jabali-panel --since "5 min ago" | grep -i reconciler
```

If the reconciler is healthy but still not converging, file a bug with:
- `jabali pdns backfill --verbose` output.
- Last 500 lines of jabali-panel journal.
- `rec_control get-remote-logins` output.

### Symptom: `install.sh` hard-fails with "resolvectl shows DNS Servers='…'"

This is the M6.3 drop-in merge gate working as intended. Your host's `jabali.conf` is bleeding `DNS=` values through the drop-in reset. Use the fallback from ADR-0047 §Decision 4:

1. Delete `/etc/systemd/resolved.conf.d/jabali.conf` (back it up first).
2. Move its values into `zz-jabali-recursor.conf`'s `FallbackDNS=` line.
3. Re-run `./install.sh` — the probe should now pass.
4. File a bug describing your systemd version + drop-in layout so we can harden the default.

### Symptom: `install.sh` hard-fails with "allow-from contains non-loopback CIDR"

Someone (local operator, merged patch, damaged config) edited `/etc/powerdns/recursor.conf` to widen the allow-from list. install.sh refuses to proceed rather than ship an open resolver. Recovery:

```bash
# Restore the managed config by deleting recursor.conf; install.sh
# re-writes it on re-run.
sudo mv /etc/powerdns/recursor.conf /etc/powerdns/recursor.conf.backup-drift
sudo ./install.sh
```

Then review the backup to understand what was added and whether the widening was intentional. If so, open a design discussion — the amplification defense is load-bearing.

## Package upgrade notes

### `apt-get upgrade pdns-recursor`

Debian's pdns-recursor package may drop our managed `/etc/powerdns/recursor.conf` via `dpkg --force-confnew` or equivalent. After any upgrade:

```bash
# Check the managed header survived.
head -1 /etc/powerdns/recursor.conf
# expect: "# Managed by jabali-panel install.sh (M6.3). ..."

# If not, re-run install.sh (idempotent on converged hosts).
sudo ./install.sh
```

### `apt-get upgrade pdns-server`

Same shape — Debian may restore the default `/etc/powerdns/pdns.conf`. The managed config lives under `/etc/powerdns/pdns.d/01-jabali-mysql.conf`, which Debian's upgrade won't touch. But verify:

```bash
grep local-address /etc/powerdns/pdns.d/01-jabali-mysql.conf
# expect: local-address=${JABALI_SRV_IPV4}:53, 127.0.0.1:5300, [::1]:5300
```

If the file is missing, `./install.sh` re-creates it.

### `apt-get upgrade systemd`

Our `resolved.conf.d/zz-jabali-recursor.conf` is under `/etc/`, not `/lib/` — systemd upgrade leaves it alone. After a systemd major version bump, re-verify `resolvectl status` shows `DNS Servers: 127.0.0.1` only.

## Security posture reference

- Recursor binds loopback only (`127.0.0.1`, `::1`); `allow-from` is `127.0.0.0/8, ::1/128`. `install.sh` hard-fails on any drift.
- No DNSSEC on the recursor (`DNSSEC=off`); systemd-resolved validates upstream.
- pdns-server still answers on the public IP `:53`, authoritative-only, no recursion.
- Every `recursor.forwards` edit goes through the agent with atomic write + validator + reload + SOA post-probe + rollback. Hand edits don't survive the next reconciler tick.
- Self-loop guard in the validator rejects forwarders pointing at `127.0.0.1:53` / `[::1]:53` — only port `5300` (pdns-server loopback bind) is legal.

## References

- ADR-0047 — decision record, alternatives considered, failure modes.
- `plans/m6.3-pdns-recursor.md` — full 7-step construction blueprint.
- `docs/BLUEPRINT.md` §M6.3 — milestone status.
- `internal/pdnsrecursor` package — atomic write + validator + probe implementation.
- `panel-api/cmd/server/pdns_cmd.go` — backfill CLI.
