# M6.4 — Panel hostname as primary mail domain — Operator runbook

**Audience:** operators running jabali-panel on a real server.
**Context:** M6.4 auto-registers the panel hostname (`$JABALI_SRV_HOSTNAME`) as a first-class email domain at install time. This runbook covers verification, hostname changes, cert rotation, and troubleshooting.
**See also:** ADR-0048 (design), `plans/m6.4-panel-hostname-mail-domain.md` (blueprint).

---

## Architecture snapshot

```
┌────────────────────────────────────────────────────────────────────┐
│  Browser hits https://<panel-hostname>/webmail                     │
└────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌────────────────────────────────────────────────────────────────────┐
│  nginx :443 default vhost                                          │
│    include phpmyadmin.conf                                         │
│    location = /webmail  → 301 https://mail.<panel-hostname>/      │
│    location /           → 444                                      │
└────────────────────────────────────────────────────────────────────┘
                                 │
                                 ▼
┌────────────────────────────────────────────────────────────────────┐
│  nginx mail.<panel-hostname>:443 vhost (jabali-mail-vhost.conf)    │
│    location /             → Bulwark (127.0.0.1:3000)               │
│    location /jmap         → Stalwart (127.0.0.1:8446)              │
│    location = /sso/webmail → panel-api (127.0.0.1:8443)            │
└────────────────────────────────────────────────────────────────────┘

Auto-registration chain:

install.sh main()
  ├─ bootstrap_pdns_self_zone       (pdns row for <hostname>)
  ├─ provision_tls_cert             (SAN: DNS:<host>, DNS:mail.<host>)
  ├─ start_and_verify               (panel-api → BootstrapAdmin creates admin)
  ├─ install_stalwart_apply         (Stalwart ready for domain-add)
  ├─ install_panel_primary_domain   (jabali-panel panel-primary ensure)
  │       └─ CLI inserts domains row: is_panel_primary=1, email_enabled=1
  └─ install_bulwark                (webmail ready)

Then reconciler tick:
  ├─ sees is_panel_primary=1, email_enabled=1, DkimSelector IS NULL
  ├─ domain.email_enable → agent:
  │     ├─ DKIM keypair at /etc/jabali-panel/dkim/<host>.key
  │     ├─ Stalwart domain add
  │     └─ dns_record.upsert → MX, SPF, DKIM TXT, DMARC TXT, mail A
  └─ webmail.vhost_apply → nginx mail.<host> vhost + reload
```

---

## Health probes

Run on the panel host after install.sh completes:

```bash
# 1. panel-primary domain row exists.
mysql -uroot -Ns jabali_panel -e \
  "SELECT id, name, is_panel_primary, email_enabled, \
          dkim_selector IS NOT NULL AS dkim_set \
   FROM domains WHERE is_panel_primary=1"
# Expected: one row, name=$JABALI_SRV_HOSTNAME, flags all 1.

# 2. Cert has the mail.<hostname> SAN.
openssl x509 -in /etc/jabali/tls/panel.crt -noout -ext subjectAltName \
  | grep -qE "DNS:mail\\.$(hostname -f)" \
  && echo "OK" || echo "MISSING mail.<hostname> SAN"

# 3. DNS self-zone has the mail records.
dig @127.0.0.1 mail.$(hostname -f) A +short
dig @127.0.0.1 $(hostname -f) MX +short
dig @127.0.0.1 $(hostname -f) TXT +short | grep -q '^"v=spf1' && echo "SPF OK"
dig @127.0.0.1 _dmarc.$(hostname -f) TXT +short | grep -q '^"v=DMARC1' && echo "DMARC OK"

# 4. DKIM TXT record.
selector=$(mysql -Ns jabali_panel -e "SELECT dkim_selector FROM domains WHERE is_panel_primary=1")
dig @127.0.0.1 "${selector}._domainkey.$(hostname -f)" TXT +short | grep -q 'p=' \
  && echo "DKIM OK" || echo "DKIM NOT PUBLISHED"

# 5. nginx mail vhost exists + reloads clean.
test -f "/etc/nginx/sites-available/$(hostname -f)-mail.conf" && echo "vhost OK"
nginx -t

# 6. /webmail 301 is live.
curl -skI "https://$(hostname -f)/webmail" | head -3
# Expected: HTTP/2 301 ... location: https://mail.<hostname>/

# 7. mail.<hostname> TLS handshake succeeds.
echo | openssl s_client -servername "mail.$(hostname -f)" \
  -connect "mail.$(hostname -f):443" 2>/dev/null \
  | openssl x509 -noout -text \
  | grep -A1 "Subject Alternative Name"
# Expected: DNS:mail.<hostname> present in SANs.

# 8. Settings → Email endpoint.
curl -sk -X GET "https://127.0.0.1:8443/api/v1/admin/settings/email" \
  -H "Cookie: <admin kratos cookie>" | jq .
# Expected: {primary_domain_name, webmail_url, dkim_published: true, email_enabled_at}
```

All eight green → M6.4 is healthy on this host.

---

## Common operations

### Create the admin's mailbox

The panel-primary domain is up but has no mailboxes. Operator creates their own:

1. Log in to the admin panel as the admin user.
2. Navigate to Domains → `<panel-hostname>` → Email / Mailboxes tab.
3. Create `admin@<panel-hostname>` (or any local-part). Reveal-once password pattern — copy out before leaving the page.
4. Test login at `https://<panel-hostname>/webmail` (which 301s to `https://mail.<panel-hostname>/`) with the credentials.

### Rotate the panel cert's DKIM private key

DKIM rotation is out of scope for M6.4; follow the M6/ADR-0043 runbook for tenant domains — it works the same way for panel-primary.

### Rotate the self-signed panel cert (long-lived install)

The install.sh-generated cert is valid for 3650 days. If you need to rotate earlier (key compromise, cipher upgrade, etc.):

```bash
# Force regen on the next install.sh run by deleting both files.
rm /etc/jabali/tls/panel.crt /etc/jabali/tls/panel.key
bash /path/to/install.sh   # re-run; provision_tls_cert sees missing files, regenerates
# install.sh reloads nginx and RESTARTS jabali-panel automatically.
```

### Disable email on the panel-primary domain (rare)

The row is delete-protected, but email_enabled can still be toggled off via the usual API:

```bash
# Via panel API (requires admin session cookie):
curl -sk -X DELETE "https://<panel-hostname>:8443/api/v1/domains/<id>/email" \
  -H "Cookie: <admin cookie>"
# The reconciler tears down Stalwart domain + nginx mail vhost + MX/SPF/DKIM rows
# on next tick.
```

Deleting the row itself is blocked — a 403 `panel_primary_protected` is returned.

---

## Hostname change

The supported path is "update in place" per ADR-0048 Decision 3:

1. Edit `/etc/jabali/panel.env`, set `JABALI_SRV_HOSTNAME=<new-hostname>`.
2. (Optional but recommended) `hostnamectl set-hostname <new-hostname>`.
3. Re-run `bash /path/to/install.sh`. The following happen in order:
   - `provision_tls_cert` sees CN drift (old CN ≠ new hostname) → regenerates cert + restarts panel-api.
   - `install_panel_primary_domain` sees the existing `is_panel_primary=1` row with old name → UPDATEs it to new name in place.
   - Reconciler on next tick: sees `email_enabled=1` but the Stalwart domain / nginx vhost / self-zone are all still under the OLD hostname → provisions NEW set for the new hostname.
4. Manual cleanup of the OLD hostname's residue:
   ```bash
   # 4a. Old DKIM private key.
   rm /etc/jabali-panel/dkim/<old-hostname>.key
   # 4b. Old nginx mail vhost.
   rm /etc/nginx/sites-available/<old-hostname>-mail.conf
   rm /etc/nginx/sites-enabled/<old-hostname>-mail.conf
   systemctl reload nginx
   # 4c. Old self-zone in pdns (if the old zone name is no longer needed).
   mysql jabali_pdns -e "DELETE FROM records WHERE domain_id=(SELECT id FROM domains WHERE name='<old-hostname>'); DELETE FROM domains WHERE name='<old-hostname>';"
   # 4d. Old Stalwart domain entry (requires stalwart-cli — follow M6 runbook).
   ```

M6.4 does NOT automate step 4 because any operator who hostname-changed a production panel probably has mailboxes attached to the old domain and needs to make a migration decision.

---

## Troubleshooting

### `provision_tls_cert`: "nginx reload failed"

`journalctl -u nginx` will show the underlying issue. Usually a syntax error from a sibling vhost (not M6.4). Fix that vhost, re-run install.sh.

### `provision_tls_cert`: "jabali-panel restart failed"

`systemctl status jabali-panel` + `journalctl -u jabali-panel -n 100` for the root cause. The cert itself is already in place; panel-api just needs to start. Usually an env var or migration issue unrelated to M6.4.

### `install_panel_primary_domain`: "JABALI_SRV_HOSTNAME not set"

Edit `/etc/jabali/panel.env` to set `JABALI_SRV_HOSTNAME=<your-hostname>`. Re-run install.sh.

### `install_panel_primary_domain`: "pdns self-zone '\<host\>' not found"

Means `bootstrap_pdns_self_zone` hasn't run (or ran for a different hostname and then the env var changed). Check `mysql jabali_pdns -e "SELECT id,name FROM domains"`. If your current `$JABALI_SRV_HOSTNAME` isn't listed, the self-zone never bootstrapped — this is an install.sh ordering or MariaDB error. Check `install.sh` logs for the `bootstrap_pdns_self_zone` step; the log says `self-zone '<host>' created in jabali_pdns` on success.

### `install_panel_primary_domain`: "no admin user exists yet; deferring"

Expected on a fresh install where BootstrapAdmin hasn't yet created the first admin (rare — start_and_verify waits for the panel to be healthy, which implies admin creation completed). If you see it repeatedly on re-run: check `journalctl -u jabali-panel` for BootstrapAdmin errors, or verify `/etc/jabali/panel.env` has `JABALI_BOOTSTRAP_ADMIN_EMAIL=` set.

### Settings → Email card stuck on "Initializing" for more than 2 minutes

Reconciler isn't converging. Sequence to diagnose:

```bash
# 1. Panel service healthy?
systemctl status jabali-panel
# 2. Reconciler log — look for the panel-primary domain.
journalctl -u jabali-panel --since '5 minutes ago' | grep -iE 'reconcil|email_enable|dkim'
# 3. Agent reachable?
systemctl status jabali-agent
# 4. Stalwart accepting domains?
systemctl status jabali-stalwart
journalctl -u jabali-stalwart --since '5 minutes ago' | tail -30
# 5. Force a panel restart to re-tick the reconciler immediately.
systemctl restart jabali-panel
```

The most common cause is an agent or Stalwart crash loop — those have their own runbooks (M6 runbook, M6.2 runbook).

### `/webmail` returns TLS handshake error

Expected during the first ~30 seconds of a fresh install while the reconciler converges. If it persists:

```bash
# 1. Does mail.<hostname> resolve locally?
dig @127.0.0.1 mail.$(hostname -f) A +short
# 2. Is the nginx mail vhost in sites-enabled?
ls -la /etc/nginx/sites-enabled/ | grep mail
# 3. Is bulwark running?
systemctl status jabali-webmail
```

If DNS resolution fails but M6.3 is supposed to be live, see the M6.3 runbook for recursor group/ownership issues.

### Admin accidentally deleted the panel-primary row via raw SQL

This is an "operator broke the guard" case. Repair:

```bash
# Re-run install_panel_primary_domain to recreate. This is what install.sh
# would do on next run anyway.
jabali panel-primary ensure --hostname $(hostname -f)
```

The CLI also MarkPanelPrimary-enforces at-most-one via the Go repo transaction, so re-running is idempotent.

---

## Package upgrade notes

No new apt packages in M6.4 — all plumbing reuses M6 components:
- Stalwart v0.16.x (M6)
- Bulwark 1.4.14 (M6)
- pdns-recursor 5.2.x (M6.3)

No version-specific hazards for this increment. The migration 000057 is additive (one new TINYINT column, one index). Rollback is a single `.down.sql`.

---

## Integration smoke checklist (dispatch-gated for Step 6)

Run against a clean VM (e.g. 192.168.100.150) before declaring M6.4 ready to merge:

- [ ] `bash install.sh --uninstall && bash install.sh` completes without error.
- [ ] All 8 health probes above pass.
- [ ] Navigate to `https://<panel-hostname>:8443/jabali-admin/settings` → Email tab → card shows "Published" with the hostname.
- [ ] Navigate to `https://<panel-hostname>:8443/jabali-admin/domains` → panel-primary row has purple "System" tag, no Delete option in the dropdown.
- [ ] Navigate to `https://<panel-hostname>/webmail` → 301 → `https://mail.<panel-hostname>/` → Bulwark login page renders (TLS trusted by the same self-signed cert, accept or import into browser).
- [ ] Create a mailbox via the UI, send a test mail to itself in Bulwark, verify receipt + DKIM-Signature header present in the delivered message.
- [ ] `curl -sk -X DELETE https://<panel-hostname>:8443/api/v1/domains/<panel-primary-id> -H "Cookie: <admin>"` returns HTTP 403 with body `{"error":"panel_primary_protected"}`.
- [ ] Re-run `bash install.sh` → idempotent (cert skipped, panel-primary row no-op, all services stay up).

Pasting the above command outputs into the Step 6 PR description is the dispatch gate for merge-to-main.

---

## Reserved-TLD hostnames (.local, .test, .invalid, .localhost, .example)

**Context**: Stalwart v0.16 rejects RFC 6761 reserved TLDs at `Domain/set` with
`{"type":"invalidPatch","description":"Invalid domain name"}`. Without the
guard (added 2026-04-23), `jabali-panel panel-primary ensure` would create
the row with `email_enabled=1` and the reconciler's `ensurePanelPrimaryDKIM`
loop would crash-spin every tick.

**Current behavior on reserved TLDs** (enforced in `panel-primary ensure`):

- Row created with `is_panel_primary=1, email_enabled=0, dkim_selector=NULL`.
- Reconciler's panel-primary branch no-ops (it checks `EmailEnabled` before
  calling `domain.email_enable`).
- `mail.<hostname>` nginx vhost is NOT created.
- `/webmail` still 301s to `https://mail.<hostname>/`; the browser hits the
  default nginx vhost (no per-domain vhost for `mail.<hostname>`).

**Reserved TLDs recognized**: `local`, `localhost`, `invalid`, `test`,
`example`. Matches the final label case-insensitively after trimming a
trailing dot. Source: `panel-api/cmd/server/panel_primary_cmd.go :: hostnameIsMailRoutable`.

**Remediation** (operator sets a routable hostname):

```bash
hostnamectl set-hostname mail.example.com
vi /etc/jabali/config.toml   # update JABALI_SRV_HOSTNAME
jabali update
```

`panel-primary ensure` detects drift, updates the row name, flips
`email_enabled=1`, and the reconciler provisions DKIM on next tick.

**Back-rollover** (routable → reserved, rare): `ensure` flips `email_enabled=0`
and clears `dkim_selector` / `dkim_public_key` / `email_enabled_at` so a
future flip-back-to-routable triggers a fresh DKIM provisioning pass.

**Migration note**: Existing installs with a `.local` panel hostname that
landed before the guard will have `email_enabled=1` + infinite reconciler
errors. Fixed by running `jabali update` once — `ensure`'s same-hostname
branch now reconciles the email_enabled bit to match routability and
clears any stale DKIM state.
