# M6.3 — pdns-recursor for local self-resolution

**Objective**: The panel host resolves its own authoritative zones (`<domain>`, `mail.<domain>`, `autoconfig.<domain>`) via a local `pdns-recursor`, eliminating the `/etc/hosts` workaround and unblocking M6 SSO ergonomics (Bulwark → Stalwart `/.well-known/jmap`, ACME HTTP-01 pre-flight, etc.) on every install.

**Scope**: install-time bootstrap of `pdns-recursor`, split-port `pdns-server` reconfiguration (public `:53` kept, loopback moves to `:5300`), an agent command pair for zone forwarders with atomic reload + rollback, reconciler wiring on domain lifecycle, a backfill CLI. **Not in scope**: DNSSEC signing on the recursor, multi-host recursor failover, migrating operator `/etc/hosts` overrides, recursor cache tuning, changing how the panel UI "DNS Resolvers" page manages admin upstream (it stays in `jabali.conf`; our override layers on top).

**Branch**: `m6.3/pdns-recursor`

**Anchors** (verified 2026-04-22):
- `install.sh:1278` currently binds pdns-server to `local-address=127.0.0.1, ${JABALI_SRV_IPV4}, ::1` — this eats port 53 on loopback so nothing else can take it. Recursor needs `127.0.0.1:53` + `[::1]:53`; pdns-server moves off loopback to `:5300`.
- `install.sh:591` already writes `/etc/systemd/resolved.conf.d/jabali.conf` with admin-configured upstream DNS (managed via the panel UI "DNS Resolvers" page). Our recursor drop-in must coexist without stomping that file. Per `man systemd-resolved.conf`: "Setting this variable to an empty list (as in `DNS=`) resets the list of servers to the empty list, all prior assignments will be cleared." We rely on that global-across-drop-ins semantics — but **it is NOT load-bearing on a reasoned belief**: Step 2 gates its exit criterion on a real `resolvectl status` probe asserting `DNS Servers:` contains only `127.0.0.1`. Fallback path documented there if the man-page reading turns out wrong.
- Agent commands register via `Default.Register("dotted.name", handler)` pattern (example: `panel-agent/internal/commands/dns_zone_upsert.go:L_last`). Wire pattern is `panel-agent/internal/pdns/client.go` (MySQL gmysql backend, NOT DNS query — separate from recursor).
- Reconciler domain lifecycle entry points: `panel-api/internal/reconciler/reconciler.go:239 ReconcileAll` — the enabledDomains loop on every tick (comments at L299 confirm "every enabled DB domain gets a domain.create every pass"). Reconciler already converges domain state idempotently; recursor zone-add slots in beside domain.create.
- jabali CLI subcommand tree lives in `panel-api/cmd/server/*_cmd.go`. New file: `pdns_cmd.go`.
- `panel-agent/internal/pdns/` already exists (pdns-server client, gmysql backend). M6.3 adds a **new sibling** `panel-agent/internal/pdnsrecursor/` — keeping packages per-daemon because recursor and authoritative speak different control surfaces (`rec_control` vs `pdns_control` + SQL).
- `pdns-recursor` ships in Debian 13 main. Package name: `pdns-recursor`. Default config path: `/etc/powerdns/recursor.conf`. Runtime user: `pdns`. Binds via `local-address=`. Control tool: `rec_control`.

---

## Workflow conventions

1. **Every step is a PR into `m6.3/pdns-recursor`** off sub-branches `m6.3/pdns-recursor/<step-slug>`. The umbrella branch is integration-only; do not push code directly. Only after Step 7 green does the umbrella branch merge into `main`.
2. Before editing any symbol, run `mcp__gitnexus__impact({target, direction:"upstream"})` and record the blast radius in the PR description. Stop and flag if HIGH/CRITICAL.
3. Before committing, run `mcp__gitnexus__detect_changes()` and confirm the change set matches the step's declared scope.
4. After every step that builds Go code: `go build ./...` + `go test ./...` (with `-race` where the touched package has any goroutines).
5. After every step that touches `install.sh`: shellcheck clean + a dry-run on the test VM (192.168.100.150 over IPv6 — IPv4 recovery documented in `feedback_vm_local_mods_byte_identical.md`). Do NOT touch `/etc/systemd/network/*.network` files. Do NOT add a `.network` file with `[Match] Name=<iface>` unless it also carries `Address=` — prior incident (2026-04-22) took eth0 offline because an orphan Match= file sorted before the real config.
6. Rebase each sub-branch onto latest `origin/main` before opening PR; re-run tests post-rebase. This is dispatcher policy.

---

## Dependency graph

```
Step 1 (ADR-0047)     ────┐
Step 2 (install.sh)   ────┼──► Step 7 (runbook + merge to main)
Step 3 (pdnsrecursor pkg) ─► Step 4 (agent cmds) ──┬─► Step 5 (reconciler) ─┤
                                                   └─► Step 6 (backfill CLI) ┘
```

- **Parallel batch A** (immediately dispatchable against `main`): Step 1, Step 2, Step 3.
  - 1 is doc-only; 2 is `install.sh` only; 3 is a new Go package with no consumers yet.
- **Serial gate**: Step 4 requires Step 3 (imports the package). Step 5 requires Step 4 (calls the command). Step 6 requires Step 4 (also calls the command).
- **Integration gate**: Step 7 requires 2, 5, 6 all green on the umbrella branch AND a manual smoke run on the test VM (dig probes + Bulwark→Stalwart round-trip).

---

## Step 1 — ADR-0047 + BLUEPRINT entry

**Branch**: `m6.3/pdns-recursor/01-adr`
**Model**: strongest (architecture decision)
**Rollback**: `git reset --hard HEAD~1` — doc-only

### Context brief

The panel has no written decision for how it resolves its own authoritative zones on the host. Install.sh flips `/etc/resolv.conf` to systemd-resolved's stub and seeds upstream from `/etc/resolv.conf` at install time (or falls back to `1.1.1.1 9.9.9.9`). Nothing queries `pdns-server`. Capture the decision and the split-port binding BEFORE code lands, so a future operator reading `docs/adr/` knows *why* `pdns-server` is on `:5300` on loopback.

### Tasks

1. Write `docs/adr/0047-pdns-recursor-local-self-resolution.md`. Cover:
   - **Context**: problem statement (panel cannot resolve its own zones; `/etc/hosts` workarounds; blocks M6 SSO ergonomics and ACME HTTP-01 pre-flight).
   - **Decision**: run `pdns-recursor` on loopback `:53` (both v4+v6); move `pdns-server` loopback binding to `:5300`; keep `pdns-server` on the public IP `:53` unchanged. Forward authoritative zones from the recursor into `127.0.0.1:5300`. Point systemd-resolved at `127.0.0.1` via `zz-jabali-recursor.conf`.
   - **Alternatives considered & rejected**:
     - *dnsmasq*: good at DHCP/TFTP but weaker recursor; we already have pdns ecosystem tooling (`rec_control`); no extra dependency footprint.
     - *unbound*: solid recursor, but doesn't integrate with pdns's zone-aware forwarders config the way `recursor.forwards` does, adds an upstream to monitor. If we're going to operate a DNS stack, one upstream (pdns) is simpler.
     - *resolved-only, `DNSStubListener=no`*: impossible to resolve authoritative zones; systemd-resolved isn't a recursor that can forward per-zone.
     - *permanent `/etc/hosts`*: doesn't scale past a handful of domains; every domain.create would need hosts patching; hosts file is not reconciled.
     - *Have the panel talk straight to `pdns-server:53` via a loopback bypass*: requires app code changes in Bulwark/Stalwart/Kratos; not a general fix.
   - **Decision — DNS amplification defense**: `allow-from=127.0.0.0/8, ::1/128` explicitly on the recursor (default in Debian package is `127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16` — LXC bridges are in those ranges and would expose the resolver to every container on the host). Install.sh hard-fails if `local-address` resolves to anything non-loopback.
   - **Decision — startup ordering**: `After=pdns-server.service` on `pdns-recursor.service`; `After=pdns-recursor.service` on `systemd-resolved.service`. Systemd ordering drop-ins under `/etc/systemd/system/<unit>.service.d/10-jabali-after.conf`.
   - **Decision — atomic reload**: every `recursor.forwards` edit goes through the agent, which writes `.forwards.tmp`, validates each line against a strict regex, renames over the live file keeping `.forwards.bak`, calls `rec_control reload-zones`, post-probes `dig @127.0.0.1 SOA <zone>`, and rolls back on probe failure.
   - **Decision — no DNSSEC validation on the recursor**: `DNSSEC=no`. Rationale: systemd-resolved already validates DNSSEC upstream; doubling up burns CPU on every query without security benefit for a single-host lab/panel deployment.
   - **Consequences**: new package dependency (`pdns-recursor`); `/etc/powerdns/recursor.forwards` becomes reconciler-owned; loopback `:5300` must be reserved on the host (anything else that tries to bind it fails); `rec_control` is the interface for live reload.
   - **Link**: this plan at `plans/m6.3-pdns-recursor.md`; runbook at `plans/m6.3-pdns-recursor-runbook.md` (Step 7).
2. Update `docs/BLUEPRINT.md` — append M6.3 row to the milestone table: `| M6.3 | pdns-recursor local self-resolution | IN PROGRESS | plans/m6.3-pdns-recursor.md |`.

### Verification

```bash
test -f docs/adr/0047-pdns-recursor-local-self-resolution.md
grep -q "M6.3" docs/BLUEPRINT.md
```

### Exit criteria

- ADR committed, linked from `docs/BLUEPRINT.md`.
- No code changes.

---

## Step 2 — `install.sh`: split-port pdns-server + pdns-recursor + resolved drop-in + probe matrix

**Branch**: `m6.3/pdns-recursor/02-install-sh`
**Model**: strongest (risk is blast-radius: breaks self-resolution on every fresh install if wrong)
**Rollback**: revert branch. On a live host: `rm /etc/powerdns/recursor.conf /etc/powerdns/recursor.forwards /etc/systemd/resolved.conf.d/zz-jabali-recursor.conf`, `systemctl disable --now pdns-recursor`, restore `pdns-server` binding to `127.0.0.1, ${JABALI_SRV_IPV4}, ::1` in `/etc/powerdns/pdns.d/01-jabali-mysql.conf`, restart `pdns` + `systemd-resolved`.

### Context brief

This is the foundation. Until this ships, none of Steps 3–6 are useful (no recursor to forward into). It MUST ship as one PR — splitting `pdns-server` off loopback :53 without adding the recursor in the same PR would break install.sh's own self-tests (e.g. `getent hosts deb.debian.org` via the stub, which now routes to a `127.0.0.1:53` that nobody's listening on).

Three concrete changes in `install.sh`:

**A. pdns-server local-address flip** (`install_powerdns`, L1278 area).
Replace:
```
local-address=127.0.0.1, ${JABALI_SRV_IPV4}, ::1
```
with:
```
local-address=${JABALI_SRV_IPV4}:53, 127.0.0.1:5300, [::1]:5300
```
Each entry carries an explicit port — pdns-server defaults `local-port` to 53 when unspecified, but a future port flip would silently break only part of the binds. Pinned to pdns-server 4.9+ (Debian 13 default).

**B. New `install_pdns_recursor()` function** (add before `install_powerdns` so it can be ordered correctly in `main()`, placed AFTER `install_powerdns` + `bootstrap_pdns_self_zone` but BEFORE `setup_certbot`).

Tasks inside the function:
1. `apt-get install -y pdns-recursor` (policy-rc.d already gates auto-start from `install_base_packages`'s batch — but pdns-recursor ships in base too; verify by moving it into the batch at L527 to avoid a second apt call).
2. Ensure no process owns `127.0.0.1:53` (systemd-resolved listens on `127.0.0.53`, NOT `127.0.0.1` — confirm with `ss -lnup sport = :53` before proceeding; warn + abort install if another process holds it).
3. Write `/etc/powerdns/recursor.conf` with:
   ```
   local-address=127.0.0.1, ::1
   local-port=53
   allow-from=127.0.0.0/8, ::1/128
   forward-zones-file=/etc/powerdns/recursor.forwards
   forward-zones-recurse=.=1.1.1.1;9.9.9.9
   dnssec=off
   threads=2
   max-cache-entries=50000
   quiet=yes
   loglevel=4
   setuid=pdns
   setgid=pdns
   ```
   Permissions: `0644 root:pdns`.
4. **Pre-start validator (hard-fail)**: parse the written `recursor.conf` for `local-address=`, split on commas, verify every entry resolves to `127.0.0.0/8` or `::1`. `_die` on any non-loopback. This defends against an operator re-running `install.sh` with a modified template that accidentally exposes the resolver publicly.
5. Create `/etc/powerdns/recursor.forwards` empty, `0640 root:pdns-recursor`. Not `pdns` — Debian's pdns-recursor package ships its own `pdns-recursor` group, distinct from pdns-server's `pdns`. Verify `getent group pdns-recursor` exists after install before `chown`.
6. Write `/etc/systemd/system/pdns-recursor.service.d/10-jabali-after.conf`:
   ```
   [Unit]
   After=pdns.service
   Wants=pdns.service
   ```
7. `systemctl daemon-reload && systemctl enable --now pdns-recursor.service`.
8. Post-start probe:
   ```
   dig +short +timeout=2 @127.0.0.1 . NS | grep -q '\.'
   ```
   hard-fail on empty (recursor isn't recursing) with `_die` + `journalctl -u pdns-recursor --no-pager -n 50`.

**C. systemd-resolved drop-in `zz-jabali-recursor.conf`**
- Filename `zz-` prefix sorts AFTER the existing `jabali.conf` (panel-UI-managed). systemd-resolved merges `DNS=` across drop-ins, so to make `DNS=127.0.0.1` authoritative we have to reset first.
- Write `/etc/systemd/resolved.conf.d/zz-jabali-recursor.conf`:
  ```
  # Managed by install.sh (M6.3). Do not hand-edit — panel DNS Resolvers UI
  # continues to manage /etc/systemd/resolved.conf.d/jabali.conf (admin
  # upstream); this file forces resolution through the local recursor at
  # 127.0.0.1 so panel-owned zones resolve to the authoritative pdns-server.
  [Resolve]
  DNS=
  DNS=127.0.0.1
  FallbackDNS=1.1.1.1 9.9.9.9
  DNSSEC=no
  ```
  Permissions: `0644 root:root`.
- Add systemd ordering drop-in `/etc/systemd/system/systemd-resolved.service.d/10-jabali-after.conf`:
  ```
  [Unit]
  After=pdns-recursor.service
  Wants=pdns-recursor.service
  ```
- `systemctl daemon-reload && systemctl restart systemd-resolved`.

**D. Post-install probe matrix** — at the end of `install_pdns_recursor()`:
```bash
# 1. Stub (resolved) must resolve a public name via recursor.
dig +short +timeout=3 @127.0.0.53 deb.debian.org | grep -qE '^[0-9.]+$' || _die "resolved→recursor→public chain broken"
# 2. Recursor must resolve a public name directly.
dig +short +timeout=3 @127.0.0.1 deb.debian.org | grep -qE '^[0-9.]+$' || _die "recursor→public chain broken"
# 3. DNS drop-in merge sanity — resolvectl must show DNS Servers: 127.0.0.1 ONLY.
#    If jabali.conf's DNS=1.1.1.1 9.9.9.9 bleeds through, our `DNS=` reset in
#    zz-jabali-recursor.conf isn't working and recursor isn't authoritative;
#    abort with a clear error so the operator can switch to the fallback
#    plan (merge jabali.conf into zz-jabali-recursor.conf — see "fallback" below).
dns_servers="$(resolvectl status 2>/dev/null | awk '/^ *DNS Servers:/{sub(/^ *DNS Servers: */,""); print; exit}')"
case "$dns_servers" in
  "127.0.0.1"|"127.0.0.1 "*"127.0.0.1") : ;;  # 127.0.0.1 only, or repeated only (both v4+v6 link views)
  *) _die "resolvectl shows DNS Servers='$dns_servers' — expected '127.0.0.1' only. zz-jabali-recursor.conf merge is not producing the expected state; see plans/m6.3-pdns-recursor.md §Step 2 fallback." ;;
esac
# 4. Config-level amplification defense — verify allow-from in recursor.conf
#    is loopback-only. We do NOT attempt `dig -b <public-ip>` source-spoofing
#    here: the kernel rewrites the source address to loopback when the
#    destination is 127.0.0.1, so the probe tests nothing useful. Real
#    external-source testing lives in the runbook (operator-driven from
#    off-host). Config parsing is deterministic and sufficient on install.
allow_from="$(awk -F= '/^allow-from=/{print $2; exit}' /etc/powerdns/recursor.conf)"
case "$allow_from" in
  *" "*|*",") _die "recursor.conf allow-from='$allow_from' has unexpected whitespace/trailing comma" ;;
esac
for cidr in ${allow_from//,/ }; do
  case "$cidr" in
    127.0.0.0/8|"::1/128") : ;;
    *) _die "recursor.conf allow-from contains non-loopback CIDR '$cidr' — amplification defense breach" ;;
  esac
done
```
Probe #3 defends the critical assumption about systemd-resolved drop-in merge semantics (see review feedback on the `DNS=` reset pattern). Probe #4 replaces a prior source-spoof probe that didn't actually test what it claimed; real external-source verification lives in the Step 7 runbook.

**Fallback if probe #3 fails** (systemd merge semantics aren't what we assumed): delete `zz-jabali-recursor.conf` and instead REPLACE `jabali.conf` with a single file carrying `DNS=127.0.0.1` + `FallbackDNS=<preserved-upstream>`. The panel UI "DNS Resolvers" page then has to be taught that it edits `FallbackDNS=` instead of `DNS=` — follow-up UI ticket, out of scope for Step 2 but documented here so the fallback is implementable without re-blueprinting.

**E. `main()` call order** — add `install_pdns_recursor` between `bootstrap_pdns_self_zone` and `setup_certbot`. Rationale: the self-zone must exist in `jabali_pdns` before recursor starts (otherwise `dig @127.0.0.1 SOA <hostname>` probes during later steps fail); certbot must run AFTER the recursor is live so ACME HTTP-01 pre-flight resolves correctly.

### Tasks

1. Edit the one hunk in `install_powerdns` (split-port `local-address`).
2. Add `install_pdns_recursor()` per A–D above.
3. Move `pdns-recursor` into the `install_base_packages` apt batch (L527). After the batch, add a hard-fail guard: `getent group pdns-recursor >/dev/null || _die "pdns-recursor group missing — package postinst failed; check 'apt-get install -f' output"`. This catches the subtle race where postinst order within a batch doesn't create the group before a later package's chown tries to use it.
4. Wire `install_pdns_recursor` into `main()` between `bootstrap_pdns_self_zone` and `setup_certbot`.
5. **Idempotency** — guard each write + service touch so a second run of `install.sh` on the same host is a no-op (no service restarts, no file rewrites):
   - If `/etc/powerdns/pdns.d/01-jabali-mysql.conf` already contains `:5300` in its `local-address` line, skip the split-port rewrite.
   - If `/etc/powerdns/recursor.conf` already exists and its first line matches our managed header (`# Managed by jabali-panel install.sh (M6.3)`), skip the rewrite.
   - Never touch an existing `/etc/powerdns/recursor.forwards` — only create if absent.
   - For `zz-jabali-recursor.conf`: compute SHA-256 of our templated-output against the on-disk file; skip write on match; warn + overwrite on drift.
   - Restart services only when their backing config actually changed in this invocation (track via local bash flag variables `pdns_changed=1`, `recursor_changed=1`, `resolved_changed=1`; `systemctl restart` only when the corresponding flag is set).
6. Test on the VM with a fresh `jabali update && ./install.sh`. Verify:
   - `dig @127.0.0.53 <panel-hostname>` returns the panel's own IP (not REFUSED).
   - `resolvectl status` shows `DNS Servers: 127.0.0.1` (only — no 1.1.1.1 leakage from `jabali.conf`).
   - `systemctl status pdns-recursor` is active.
7. **Idempotency test** — run `./install.sh` a second time on the same VM. Capture:
   - `journalctl -u pdns -u pdns-recursor -u systemd-resolved --since "30 seconds ago"` — zero `Started` or `Stopped` entries for those services.
   - File mtimes of `/etc/powerdns/pdns.d/01-jabali-mysql.conf`, `/etc/powerdns/recursor.conf`, `/etc/systemd/resolved.conf.d/zz-jabali-recursor.conf` — unchanged from before the second run.
   - `diff <snapshot-before> <snapshot-after>` of the three file contents — empty.

### Verification

```bash
shellcheck install.sh
bash -n install.sh
```
Then on the VM (192.168.100.150, IPv6 fallback `fc42:5009:ba4b:5ab0:7cec:37ff:fe4b:b0c4`):
```bash
systemctl is-active pdns pdns-recursor systemd-resolved
dig +short @127.0.0.1 . NS | head -3
dig +short @127.0.0.53 deb.debian.org
ss -lnup sport = :53 | grep -E 'pdns-recursor|systemd-resolve'
ss -lnup sport = :5300 | grep pdns_server
```

### Exit criteria

- Fresh install on a clean VM completes install.sh without errors.
- All four probes above return expected results.
- `getent hosts deb.debian.org` works (install.sh's own DNS-check at L650 passes).
- `pdns-server` still answers on public IP `:53` (external authoritative still works).

### Known pitfalls

- `pdns-recursor` and `pdns-server` installed together on fresh Debian 13 images race on user/group creation — if you see `chown: invalid group: 'pdns-recursor'`, the package's `postinst` hasn't run yet. Install in the same apt batch to force Debian's postinst ordering.
- `systemd-resolved` Debian package has historically shipped `DNSStubListenerExtra=` support — do NOT use it. The stub on `127.0.0.53` is all we need; extra listeners conflict with our `:53` binding on `127.0.0.1`.
- `systemctl restart systemd-resolved` briefly drops DNS on the host. Sequence matters: restart resolved LAST, after recursor is confirmed active.
- Do NOT touch `/etc/systemd/network/*.network`. Do NOT add any `.network` file with a `[Match] Name=<iface>` block that lacks `Address=`. Prior incident (2026-04-22) took eth0's IPv4 offline because a `DNSDefaultRoute=no` drop-in sorted alphabetically before the real `eth0.network`.
- Debian's `pdns-recursor` postinst auto-enables + starts the service. Our apt batch has `policy-rc.d` set to deny auto-start, so the service stays down until our explicit `systemctl enable --now` call.
- `policy-rc.d` gates service **start**, not user/group creation. Postinst scripts can still `adduser --system pdns-recursor` and create the group — but ordering within the apt batch is not guaranteed. Task 3's explicit `getent group pdns-recursor` hard-fail catches batches where the group didn't land. Do NOT move the chown before this check.

---

## Step 3 — `internal/pdnsrecursor` shared package

**Branch**: `m6.3/pdns-recursor/03-pdnsrecursor-pkg`
**Model**: strongest (correctness-critical; tested in isolation)
**Rollback**: delete the new package directory (no consumers yet).

### Context brief

Encapsulate the atomic-reload dance in a package that the agent command (Step 4) imports. This package never runs on the panel side — agent-only — but it is written against an injectable dependency set (clock, exec-runner, filesystem, DNS probe) so unit tests cover rollback paths without shelling out.

**Package path**: `git.jabali-panel.com/shukivaknin/jabali2/panel-agent/internal/pdnsrecursor` (sibling of existing `internal/pdns/`).

**Public API**:
```go
package pdnsrecursor

// Entry is one line in recursor.forwards: "<zone>=<ip>:<port>".
type Entry struct {
    Zone string // apex, no trailing dot, lowercase
    Addr string // IPv4 or IPv6 literal (no brackets in the struct)
    Port int    // REQUIRED (>0) — manager returns error if zero; callers default explicitly
}

// Manager wraps recursor.forwards writes + rec_control reload + post-probe.
// Construct with New; methods are safe to call concurrently (internal mutex).
type Manager struct { /* ... */ }

func New(opts Options) (*Manager, error)

// AddZone is idempotent: no-op if the entry already matches.
func (m *Manager) AddZone(ctx context.Context, e Entry) error

// RemoveZone is idempotent: no-op if the zone is absent.
func (m *Manager) RemoveZone(ctx context.Context, zone string) error

// List returns the current forwards snapshot (reads from disk).
func (m *Manager) List(ctx context.Context) ([]Entry, error)
```

**Options**:
```go
type Options struct {
    ForwardsPath string           // default "/etc/powerdns/recursor.forwards"
    BackupPath   string           // default ForwardsPath + ".bak"
    TempPath     string           // default ForwardsPath + ".tmp"
    Exec         ExecRunner       // default osExecRunner{}; stubbed in tests
    Prober       Prober           // default miekgDNSProbe; stubbed in tests
    Clock        func() time.Time // default time.Now
    FileMode     os.FileMode      // default 0640
    Owner        string           // default "root:pdns-recursor"
}

type ExecRunner interface {
    Run(ctx context.Context, name string, args ...string) ([]byte, error)
}
type Prober interface {
    // Query "zone SOA" against 127.0.0.1:53; return nil if authoritative.
    ProbeSOA(ctx context.Context, zone string) error
}
```

**Validator regex** (package-local):
```go
var entryRe = regexp.MustCompile(`^([a-z0-9][a-z0-9.-]*[a-z0-9]|[a-z0-9])=(\d{1,3}(?:\.\d{1,3}){3}|\[?[0-9a-fA-F:]+\]?)(?::(\d{1,5}))?$`)
```
Plus a post-regex pass:
- Reject `zone == ""` or any zone ending in `.`.
- Reject forward address `127.0.0.1:53` / `[::1]:53` (self-loop guard — recursor forwarding to itself on its own bind port would loop and exhaust threads).
- Reject port 0 or >65535.

**Atomic write sequence** (`AddZone`):
1. Acquire package mutex.
2. Read current `ForwardsPath` (empty if missing). Parse into `map[zone]Entry`.
3. Compute desired map (apply add).
4. If desired == current → no-op; return nil.
5. Marshal desired map to `ForwardsPath.tmp`, sorted by zone for deterministic diff.
6. Run validator against every line in `.tmp`; bail on any reject.
7. `os.Link(ForwardsPath, BackupPath)` (or `os.Rename` if bak exists — keep a single rolling bak).
8. `os.Rename(ForwardsPath.tmp, ForwardsPath)`.
9. `chown` + `chmod` to Owner/FileMode.
10. `exec rec_control reload-zones` — 5s timeout.
11. Post-probe: `Prober.ProbeSOA(ctx, zone)` for the added zone. If fail, restore `.bak` → rename over, `rec_control reload-zones` again, return the probe error (wrapped).
12. Release mutex.

**RemoveZone**: same sequence with a removal instead of add; no post-probe (removal should always succeed, nothing to probe).

### Tasks

1. Create `panel-agent/internal/pdnsrecursor/manager.go` with the API above.
2. Create `panel-agent/internal/pdnsrecursor/validator.go` (the regex + semantic checks). Exported `ValidateLine(s string) error` for reuse.
3. Create `panel-agent/internal/pdnsrecursor/exec.go` with the default `osExecRunner` wrapping `exec.CommandContext`, 5s timeout baked in.
4. Create `panel-agent/internal/pdnsrecursor/probe.go`:
   - **First, audit deps**: `grep -r "miekg/dns" panel-agent/go.mod panel-api/go.mod`. If `github.com/miekg/dns` is already present, use it (`dns.Msg` with `dns.TypeSOA`, dial `127.0.0.1:53`). If NOT present, do NOT add it as a new top-level dep for a single call. Instead, implement a hand-rolled SOA probe using `net.Dial("udp", "127.0.0.1:53")` + a 17-byte DNS query construction. Document which path you took in a comment at the top of `probe.go`.
   - Either path, timeout is 2s. Success = RCODE 0 and at least one answer with `Type == SOA` (or RR type 6 for the hand-rolled path).
5. Unit tests covering:
   - Validator accepts `example.com=127.0.0.1:5300`, `[::1]` bracketed, bare IPv4 with default-port inference.
   - Validator rejects: trailing-dot zone, `127.0.0.1:53` (self-loop), empty zone, `UPPERCASE.COM`, zone starting with `-`, port 0, port 65536.
   - AddZone idempotent (second call with same entry: no file rewrite, no rec_control call).
   - AddZone rollback on `rec_control` fail: `.bak` restored, original file intact.
   - AddZone rollback on probe fail: same.
   - RemoveZone no-op when zone absent.
   - Concurrency: two `AddZone` calls for different zones in goroutines serialize correctly (mutex), final file contains both.
6. No integration tests in this step (no live recursor) — those live in Step 4.

### Verification

```bash
cd panel-agent && go test -race ./internal/pdnsrecursor/...
go vet ./internal/pdnsrecursor/...
```

### Exit criteria

- `go test -race` passes.
- `go vet` clean.
- Package has ≥85% line coverage (it's small and self-contained; aim high).
- No consumers yet — package exists but isn't imported.

### Known pitfalls

- Using `os.Link` requires the source+dest on the same filesystem; `/etc/powerdns/` is always one FS. But on tests we use `t.TempDir()` which is also one FS — fine.
- `rec_control` returns 0 on "reload succeeded even if some zones failed to load" in some versions. Don't rely on exit code alone — always post-probe.
- `miekg/dns` SOA probe vs `net.Resolver`: `net.Resolver.LookupNS` doesn't return authoritative-flag info. Prefer `miekg/dns` for a true SOA query (already used by pdns internals, cheap dep).

---

## Step 4 — panel-agent commands `pdns.recursor_add_zone` + `pdns.recursor_remove_zone`

**Branch**: `m6.3/pdns-recursor/04-agent-commands`
**Model**: default
**Rollback**: revert branch — removes the two new commands; no DB state touched. Reconciler (Step 5) is not live yet, so no production calls.

### Context brief

Thin handlers that unmarshal params, construct a singleton `*pdnsrecursor.Manager`, and call `AddZone`/`RemoveZone`. Follow the `dns_zone_upsert.go` pattern: one file per command, type block at the top, handler function, `init()` registering on `Default`.

**Wire contract**:
```json
// Request (panel → agent)
{
  "command": "pdns.recursor_add_zone",
  "params": {
    "zone": "example.com",
    "addr": "127.0.0.1",
    "port": 5300
  }
}
// Success
{ "ok": true, "result": { "zone": "example.com", "changed": true } }
// Removal
{
  "command": "pdns.recursor_remove_zone",
  "params": { "zone": "example.com" }
}
```
`changed` reports whether the forwards file was rewritten; `false` on idempotent no-op so the reconciler can skip subsequent unnecessary probes.

### Tasks

1. Create `panel-agent/internal/commands/pdns_recursor_add_zone.go`:
   ```go
   type pdnsRecursorAddZoneParams struct {
       Zone string `json:"zone"`
       Addr string `json:"addr"`
       Port int    `json:"port,omitempty"` // default 5300 — MUST be filled by handler, not by AddZone
   }
   type pdnsRecursorAddZoneResponse struct {
       Zone    string `json:"zone"`
       Changed bool   `json:"changed"`
   }
   func pdnsRecursorAddZoneHandler(ctx context.Context, params json.RawMessage) (any, error) {
       var p pdnsRecursorAddZoneParams
       if err := json.Unmarshal(params, &p); err != nil { /* return CodeInvalidArgument */ }
       if p.Port == 0 { p.Port = 5300 } // default resolved here, NOT inside AddZone
       // ... call manager.AddZone(ctx, pdnsrecursor.Entry{Zone: p.Zone, Addr: p.Addr, Port: p.Port})
   }
   func init() { Default.Register("pdns.recursor_add_zone", pdnsRecursorAddZoneHandler) }
   ```
   **Invariant**: `pdnsrecursor.Manager.AddZone` receives `Entry.Port > 0` always. The handler is the ONLY defaulter; the manager is strict (errors on port==0). This keeps the package usable from the CLI backfill path too (Step 6) where the CLI might bypass the handler's default — the strict manager forces the CLI to default explicitly as well, avoiding two subtly-different defaults.
2. Create `panel-agent/internal/commands/pdns_recursor_remove_zone.go` similarly.
3. Package-level singleton `var recursorMgr *pdnsrecursor.Manager` with `sync.Once`-initialized `getRecursorMgr()` that returns it. This isolates test-time wiring (tests can replace the var).
4. Unit tests with a `fakeManager` type-switching on `*pdnsrecursor.Manager` methods — skip if upstream interface isn't already ergonomic; in that case test by temporarily swapping the singleton.
5. Golden request/response JSON fixtures at `panel-api/internal/agent/testdata/` (follow `mailbox_create_request.json` pattern — pattern established in M6).
6. Security hardening: handler rejects `zone` containing anything `ValidateLine` would reject at the line level, with `CodeInvalidArgument`, BEFORE invoking the manager (defense in depth).

### Verification

```bash
cd panel-agent && go test -race ./internal/commands/pdns_recursor_*
go vet ./internal/commands/...
```
Integration: start the agent locally, send a POST against its UDS with the golden request fixture. Assert 200 + expected response + `/etc/powerdns/recursor.forwards` now contains the added line.

### Exit criteria

- `go test -race` green for the two new command files.
- Unknown-command smoke test still passes (registry didn't break on the new Register calls).
- Manual UDS integration test succeeds on the VM against a real `pdns-recursor`.

### Known pitfalls

- Do not call `pdnsrecursor.New` inside the handler — the manager caches state (backup-file tracking) and constructing per-call breaks idempotency invariants.
- `agentwire.CodeInvalidArgument` is the right error code for malformed zone; `CodeInternal` for rec_control failures.

---

## Step 5 — panel-api reconciler wiring on domain lifecycle

**Branch**: `m6.3/pdns-recursor/05-reconciler`
**Model**: default
**Rollback**: revert branch. Leaves `recursor.forwards` with whatever state was last written; the backfill CLI (Step 6) can converge.

### Context brief

`reconciler.ReconcileAll` already walks enabled domains every tick (`reconciler.go:299`). For every enabled domain it calls `domain.create` on the agent. We add a second call: `pdns.recursor_add_zone` with `{zone: domain.Name, addr: "127.0.0.1", port: 5300}`. Idempotent on the agent side (Step 3 invariant), so calling it every pass is safe and self-healing.

For deletes, the reconciler's "agent has it but DB doesn't" branch (around `reconciler.go:336`) calls `domain.create` with `is_enabled=false`. Extend: when DB domain is **absent** (not just disabled), also call `pdns.recursor_remove_zone`. Disabled-but-present domains keep their forwarder — they still need local resolution for reactivation and for nginx `disabled_vhost` static page serving.

### Tasks

1. In `ReconcileAll` enabled-domain loop: after the `domain.create` call returns success, issue `pdns.recursor_add_zone` for the same zone. No state-tracking flag needed — the agent command is idempotent (Step 3 invariant, asserted by unit test), and the returned `Changed: false` distinguishes real writes from no-ops for metrics.
2. BEFORE the enabled-domain loop (once per pass), issue `pdns.recursor_add_zone` for the self-zone `JABALI_SRV_HOSTNAME` — boot-strapped in `install.sh:bootstrap_pdns_self_zone`, not a row in the `domains` table. Same idempotent call; no tracking flag. This is cheap (sub-millisecond agent RPC on no-op) and self-healing across restarts.
3. Add a new helper `reconcileRecursorForwards(ctx, agentAddr, domainName)` that wraps the agent call with structured logging.
4. For the orphan-on-agent branch: after the domain.delete-equivalent (currently disabled_vhost), call `pdns.recursor_remove_zone`.
5. Add a reconciler metric: `jabali_reconciler_recursor_zone_add_total{result="changed|noop|error"}` and corresponding `_remove_total`. Reuse the existing prometheus registry (grep `promauto`).
6. Integration test: `reconciler_test.go` with a fake agent that records received commands; seed DB with two enabled domains + one deleted; assert exact command sequence: two add_zones for enabled + one remove_zone for the deleted + one add_zone for the self-zone.
7. No DB schema change. No migration.

### Verification

```bash
cd panel-api && go test -race ./internal/reconciler/...
go vet ./internal/reconciler/...
```

### Exit criteria

- `go test -race` green.
- Integration test asserts correct command sequence.
- On the VM: create a domain via the panel UI, confirm within one reconciler tick (60s) that `/etc/powerdns/recursor.forwards` contains the new zone. Delete the domain, confirm within 60s that the line is gone.
- `dig @127.0.0.53 <new-zone> SOA` returns the panel's SOA record (proves the full chain: resolved → recursor → forwarder → pdns-server:5300).

### Known pitfalls

- `domain.create` and `pdns.recursor_add_zone` are two separate agent RPCs — either can fail independently. If `recursor_add_zone` fails, don't mark the domain state inconsistent; the reconciler retries next tick. Log warn, increment `error` metric, continue.
- The self-zone (`JABALI_SRV_HOSTNAME`) isn't a row in the `domains` table — `install.sh:bootstrap_pdns_self_zone` writes it directly via SQL. So the reconciler has to add the recursor forwarder for it explicitly, outside the enabled-domains loop.
- **500-domains scale concern**: every reconciler tick issues one `pdns.recursor_add_zone` agent RPC per domain. Agent UDS RPC latency is sub-millisecond on no-op (the manager's `AddZone` returns after comparing the in-memory desired map to the on-disk map — no file rewrite, no `rec_control`). On first-pass or on drift, the actual rewrite runs once: `map recompute + ~N lines written + rec_control reload-zones + post-probe SOA`. At N=500 that's ~N*30 bytes + one reload + 500 probes; the post-probes batch matters — consider deferring probes to the LAST zone change per reconcile pass rather than per-zone. Open a follow-up if scale hurts; not a blocker for first ship.

---

## Step 6 — Backfill CLI `jabali pdns backfill`

**Branch**: `m6.3/pdns-recursor/06-backfill-cli`
**Model**: default
**Rollback**: revert branch — CLI subcommand removed; no state touched (--dry-run default).

### Context brief

For hosts that were installed BEFORE M6.3 (pre-recursor), after `install.sh` upgrades them the `/etc/powerdns/recursor.forwards` file is empty. The reconciler will eventually converge it over 60s — but operators who SSH in wanting to verify state, or who need to converge before the next tick, need an explicit CLI. Also useful post-incident: "did we lose a zone forwarder?" → `jabali pdns backfill --dry-run` shows the diff.

**Command tree**:
```
jabali pdns backfill [--yes] [--verbose]
```
- No flag (default) = dry run. Prints planned adds/removes/no-ops. Exits 0.
- `--yes` actually performs the adds/removes via the agent commands.
- `--verbose` prints per-zone details including the exact recursor.forwards line.

### Tasks

1. Create `panel-api/cmd/server/pdns_cmd.go`. Register under `jabali` root as a new `pdns` subcommand with `backfill` child.
2. Implementation:
   ```go
   func (o *pdnsBackfillOpts) run(ctx context.Context) error {
       // 1. Walk enabled domains from the domains repo.
       // 2. For each, build Entry{Zone: name, Addr: "127.0.0.1", Port: 5300}.
       // 3. Add the self-zone (JABALI_SRV_HOSTNAME from env).
       // 4. Call agent "pdns.recursor_list" — NEW command? see below.
       // 5. Diff planned vs actual: adds, removes, no-ops.
       // 6. Print table.
       // 7. If --yes, call agent pdns.recursor_add_zone / pdns.recursor_remove_zone for each delta.
   }
   ```
3. Add a new agent command `pdns.recursor_list` (returns the parsed `recursor.forwards`). Thin handler over `Manager.List`. This keeps the CLI honest about what's currently live, rather than re-parsing the file panel-side.
4. Safety rail: `--yes` without `--dry-run` run first is fine, but print a one-line summary + `--confirm` final prompt (read from stdin, case-insensitive `yes`) UNLESS `JABALI_PDNS_BACKFILL_NONINTERACTIVE=1` is set (for CI / scripted runs).
5. Output format — always print to stdout, color-code on TTY, plain on pipe:
   ```
   ZONE                   FORWARDER              ACTION
   example.com            127.0.0.1:5300         add
   panel.local            127.0.0.1:5300         noop
   old-domain.com         —                      remove
   ```
6. Exit codes: 0 on success (including dry-run reporting changes); 1 on partial failure (any add/remove returned error); 2 on fatal (agent unreachable, DB unreachable).
7. Unit test the diff logic with a fake agent + fake repo; integration test by running against a live panel-agent on the VM.

### Verification

```bash
cd panel-api && go test ./cmd/server/ -run "TestPdnsBackfill.*"
jabali pdns backfill --verbose   # dry-run, prints plan
jabali pdns backfill --yes       # confirms + applies
```

### Exit criteria

- `go test` green for the new test cases.
- CLI help (`jabali pdns --help`) reads clean.
- On the VM: running `jabali pdns backfill` after a fresh `install.sh` upgrade from a pre-M6.3 snapshot produces a plan with one add per existing enabled domain; `--yes` applies them; running again produces all `noop`.

### Known pitfalls

- `panel-api/cmd/server/*_cmd.go` subcommands share root flag parsing (see `root.go`). Follow the existing pattern; don't invent a new flag parser.
- The backfill reads from the panel DB directly, NOT via the HTTP API. Be sure to open a read-only connection; don't accidentally trigger a migration.
- If `JABALI_SRV_HOSTNAME` is empty (rare — only on broken installs), skip self-zone backfill with a warning rather than erroring.

---

## Step 7 — Runbook + memory + BLUEPRINT + merge to main

**Branch**: `m6.3/pdns-recursor/07-runbook-merge`
**Model**: default
**Rollback**: revert branch — doc + memory only. The feature branch itself is not rolled back at this step; if Step 7 needs to be redone, it's appended.

### Context brief

Final docs, memory updates, and the gated merge. This step does NOT run until all of 2, 5, 6 are green on the umbrella branch AND a smoke run on the test VM passes.

### Tasks

1. Write `plans/m6.3-pdns-recursor-runbook.md`:
   - How to verify the recursor is healthy: `systemctl status pdns-recursor`, `dig @127.0.0.1 . NS`, `dig @127.0.0.1 <panel-zone> SOA`.
   - How to run the amplification-defense probe (`dig @<public-ip> -b <public-ip> . NS` expect REFUSED; if it answers, alert).
   - How to override the upstream recursor forwarders (`/etc/powerdns/recursor.conf` `forward-zones-recurse=` line, rec_control reload-zones).
   - How to fall back if the recursor dies: edit `/etc/systemd/resolved.conf.d/zz-jabali-recursor.conf` → set `DNS=1.1.1.1 9.9.9.9` → `systemctl restart systemd-resolved` (removes the recursor from the path; panel won't resolve its own zones until the recursor is back, but the rest of the host stays up).
   - How to manually add/remove a zone forwarder: `jabali pdns backfill --yes` (preferred) or edit `/etc/powerdns/recursor.forwards` + `rec_control reload-zones` (not recommended; next reconciler tick will re-sync).
   - How to clean up pre-M6.3 `/etc/hosts` overrides: grep for entries pointing at `127.0.0.1` for panel domains, remove, run `jabali pdns backfill --yes` to confirm equivalent forwarders are in place.
   - Troubleshooting: `rec_control` commands (`top-queries`, `current-queries`, `dump-cache`).
2. Update `docs/BLUEPRINT.md`: flip M6.3 row to SHIPPED with final SHA on main (fill in at merge time).
3. Append to `MEMORY.md`: `- [M6.3 pdns-recursor SHIPPED](project_m6_3_pdns_recursor.md) — local self-resolution via pdns-recursor on loopback :53; pdns-server split to public :53 + loopback :5300; zz-jabali-recursor.conf; recursor.forwards reconciler-owned; amplification defense; ADR-0047`.
4. Write `project_m6_3_pdns_recursor.md` memory file with: merged SHA, key decisions, out-of-scope items preserved from this plan, known follow-ups (DNSSEC, multi-host, recursor cache tuning).
5. Run `mcp__gitnexus__detect_changes` one last time on the umbrella branch to verify the full change set matches the plan.
6. Rebase `m6.3/pdns-recursor` onto latest `origin/main`, re-run `go test ./...` + `shellcheck install.sh` + VM smoke.
7. Open PR from `m6.3/pdns-recursor` → `main`. Body includes: dependency graph, Step-by-step SHA list, smoke probe output, amplification-defense probe output, blast-radius summary.

### Verification

```bash
test -f plans/m6.3-pdns-recursor-runbook.md
grep -q "M6.3" docs/BLUEPRINT.md
grep -q "M6.3 pdns-recursor SHIPPED" ~/.claude/projects/-home-shuki-projects-jabali2/memory/MEMORY.md
```
On the VM, smoke test the full chain end-to-end:
```bash
# 1. Panel self-zone via resolved stub — proves resolved→recursor→pdns-server chain.
dig +short @127.0.0.53 "$(hostname -f)" | grep -qE '^[0-9.]+$'
# 2. A created domain's mail.<domain> resolves to the panel IP via the stub.
dig +short @127.0.0.53 mail.example-test.com | grep -qE '^[0-9.]+$'
# 3. Bulwark → Stalwart round-trip WITHOUT --resolve override — this is the
#    M6 anchor use-case and it proves the recursor is actually on the path.
#    (Previous draft used `curl --resolve mail.example-test.com:443:<ip>`
#    which bypasses DNS and thus tests NOTHING about the recursor.)
#    Uses the system resolver (→ stub @127.0.0.53 → recursor @127.0.0.1 →
#    forward to pdns-server :5300 → authoritative answer).
curl -sSI https://mail.example-test.com/.well-known/jmap | head -1 | grep -q '200'
# 4. External amplification defense — run from OFF-HOST (operator laptop
#    or a second VM on a different L2 segment). In-host probes don't work
#    because the kernel rewrites the source address when the destination
#    is loopback. Details + the exact dig invocation live in the runbook.
#    Step 7 requires operator attestation that this was run.
```

### Exit criteria

- All four smoke probes pass.
- Runbook merged.
- Memory index updated.
- M6.3 marked SHIPPED in BLUEPRINT.md with the merge-to-main SHA.
- PR merged to `main`.

---

## Anti-patterns to avoid

1. **Do not** add a systemd `.network` drop-in with only `[Match] Name=<iface>` and no `Address=`. The resolved drop-ins live under `resolved.conf.d/` — if you accidentally create a `*.network` file, networkd's alphabetical precedence can silently disable the real interface config. (Source: 2026-04-22 VM network break.)
2. **Do not** set `DNSStubListener=no` or `DNSStubListenerExtra=` on systemd-resolved to "free up" port 53. The stub is needed by every app that reads `/etc/resolv.conf` (which points at the stub). We own `127.0.0.1:53` (recursor), stub stays on `127.0.0.53`.
3. **Do not** point pdns-recursor's `forward-zones-recurse=.=` at `127.0.0.1:5300`. Only per-zone forwarders go to the authoritative; the recursor handles root queries directly against public upstream (`1.1.1.1;9.9.9.9`).
4. **Do not** store recursor forwarder state in the panel DB. `recursor.forwards` is the source of truth on disk; the reconciler + backfill CLI are idempotent converge-only. Adding a DB table would create a second source-of-truth to sync.
5. **Do not** let the panel UI edit `recursor.forwards` directly. Every write goes through the agent command (Step 4) so the atomic-write + validator + post-probe dance can't be bypassed. UI operators use `jabali pdns backfill`.
6. **Do not** use `rec_control` from panel-api. Agent owns `rec_control` — panel-api only calls the agent command. This preserves the rest of the panel's "never execute privileged local commands" invariant.
7. **Do not** rely on `rec_control`'s exit code alone. It returns 0 on partial success. Always post-probe with `dig SOA` to confirm the new forwarder actually answers.
8. **Do not** fold Step 2's three install.sh changes (pdns split-port, recursor install, resolved drop-in) into separate PRs. They depend on each other to leave the host resolvable; splitting them guarantees a broken intermediate state. One PR.
9. **Do not** touch `/etc/systemd/resolved.conf.d/jabali.conf`. That file is panel-UI-managed (admin DNS Resolvers page). Our `zz-jabali-recursor.conf` layers on top with alphabetical precedence + explicit `DNS=` reset.
10. **Do not** set `allow-from=` to include any RFC1918 range, even for "convenience." LXC bridge interfaces live in those ranges and would expose the resolver to every co-tenant container. `127.0.0.0/8, ::1/128` only.

---

## Risks & mitigations

| Risk | Mitigation |
|---|---|
| VM network break (as on 2026-04-22) from bad systemd drop-in | Resolved drop-in lives under `resolved.conf.d/`, not `network/`. No `.network` file is created by M6.3. Step 2 known-pitfalls block flags this explicitly. |
| Amplification attack via public recursor | `allow-from=127.0.0.0/8, ::1/128` + install.sh pre-start hard-fail if `local-address` resolves to non-loopback + runbook probe (`dig @<public-ip> -b <public-ip> . NS` expect REFUSED). |
| Self-loop if recursor forwards its own zone to itself | Validator rejects `<zone>=127.0.0.1:53` / `[::1]:53`. Port 5300 is the only legal forwarder target. |
| Stale `recursor.forwards` on upgrade from pre-M6.3 | Backfill CLI (`jabali pdns backfill --yes`). Reconciler also converges within 60s on next tick. |
| `pdns-server` apt upgrade drops our `local-address` override | Config lives in `/etc/powerdns/pdns.d/01-jabali-mysql.conf`, a drop-in; Debian package ships `/etc/powerdns/pdns.conf` which is untouched. `dpkg --verify` clean. |
| `pdns-recursor` apt upgrade drops our `allow-from` | Same — `/etc/powerdns/recursor.conf` is package-owned. Use `recursor.conf.d/` drop-ins if Debian supports that; otherwise, document `apt-get upgrade` re-provisioning and add to runbook. Install.sh re-runs idempotent. |
| Partial-push incident (per `feedback_no_partial_blueprint_to_main.md`) | Umbrella branch `m6.3/pdns-recursor`; sub-branches merge into the umbrella, not main. Only Step 7 merges umbrella → main, after all steps green. |
| M6.3 runs on a host where the admin has manually added `/etc/hosts` entries for panel domains | Runbook includes cleanup step. `/etc/hosts` takes precedence over DNS in glibc NSS, so lingering entries mask the recursor — not a correctness issue (same IP resolved), but a source of confusion for the admin. |

---

## Verification checklist (every PR into the umbrella branch)

- [ ] `go build ./...` and `go test -race ./...` green (for steps 3–6).
- [ ] `shellcheck install.sh` + `bash -n install.sh` clean (for step 2).
- [ ] No new `*.network` file created under `/etc/systemd/network/`.
- [ ] Impact analysis run on every edited symbol (`mcp__gitnexus__impact`).
- [ ] Rebased onto latest `origin/main` post-implementation and re-built.
- [ ] PR description lists blast radius + affected files.
- [ ] For steps touching `install.sh`: smoke run on the test VM (fresh install OR upgrade-from-pre-M6.3 OR re-run-existing) passes.

---

## Out of scope

- DNSSEC validation on the recursor — `DNSSEC=off`. Systemd-resolved handles upstream validation.
- Multi-host recursor failover / anycast — single-node panel.
- Recursor cache tuning beyond package defaults (+ `max-cache-entries=50000` baseline).
- Migrating admin `/etc/hosts` entries — runbook covers hand cleanup.
- Changing the panel UI "DNS Resolvers" page to edit something other than `jabali.conf`. That file continues to be the admin-upstream config; `zz-jabali-recursor.conf` is layered on top install-side.
- Exposing the recursor externally — explicitly blocked by `allow-from` and the install.sh config validator.
- Metrics / dashboard for recursor query load — Prometheus scrape of `pdns-recursor` is a separate concern; revisit after M6.3 ships.
- Panel UI "DNS Status" card surfacing recursor health. V1 operator tooling is `jabali pdns backfill` + runbook. Add a UI surface in a follow-up if operators ask for it; not a blocker.
- Install-time source-spoofed amplification probe. The probe that was drafted (`dig -b <public-ip>`) doesn't test what it claims — kernel rewrites the source to loopback when the destination is loopback. Real external-source verification is operator-driven from off-host (covered in the runbook).
