# M6.6 — Per-Domain Mail TLS Runbook

Operational guide for the per-domain mail TLS feature shipped under
ADR-0117. Covers opt-in, troubleshooting, force-reissue, opt-out, and
LE rate-limit cap management.

## Concept refresher

- One row per opted-in domain in `mail_certificate`.
- Reconciler dispatches `ssl.mail.issue` once per row each tick until
  the row reaches `issued` (then it's renewal-only).
- Cert covers four SANs: `mail.<d>`, `autoconfig.<d>`,
  `autodiscover.<d>`, `mta-sts.<d>`.
- Stalwart picks the right cert via SNI; panel-hostname cert remains
  the default for connections without a matching cert.

## Opt-in (tenant or admin)

1. UI: domain detail page → Email tab → **Mail TLS** section. Flip
   the toggle.
2. Confirm the four DNS A records the panel asks for exist + point
   at the panel public IP. The DNS preview table makes it obvious.
3. Wait one reconciler tick (~60s). Status should move
   `pending → issuing → issued`. Renewals happen automatically when
   <30d to expiry.

## Status meanings

| Chip | Means | Operator action |
|------|-------|-----------------|
| `Not opted in` | Row doesn't exist | Toggle on if you want it |
| `Pending` | Row exists; reconciler hasn't dispatched yet | Wait 60s |
| `Issuing` | `ssl.mail.issue` is in flight | Wait up to 4 min |
| `DNS missing` | One+ SAN A records don't resolve to panel IP | Fix DNS, then **Re-issue** to clear the 1h backoff |
| `Issued` | Cert installed in Stalwart | None; renewal is automatic |
| `Failed` | certbot error or rate-limit hit | Read `last_error`; fix the cause; **Re-issue** to clear 3h backoff |
| `Disabled` | Operator opted out | Toggle on to re-opt-in |

## Troubleshooting

### Status stuck at `dns_missing`

```bash
# Per SAN, from the panel host (matches what the agent saw):
for h in mail autoconfig autodiscover mta-sts; do
  dig +short A "$h.<domain>"
done
# Compare to:
curl -s https://ipinfo.io/ip   # what the panel believes its public IP is
# Or:
mysql -u jabali -p jabali_panel -e \
  "SELECT public_ipv4 FROM server_settings;"
```

If the A records exist but point at a different IP, fix DNS first.
If `server_settings.public_ipv4` is wrong, the panel-cert flow
should heal it; in the meantime patch via the Admin → IP Addresses
page.

### Status stuck at `failed` with rate-limit error

`last_error` will mention LE rate-limit. Inspect:

```sql
SELECT COUNT(*) AS first_issue_attempts_last_7d
FROM mail_certificate
WHERE attempt_count >= 1
  AND updated_at > NOW() - INTERVAL 7 DAY
  AND issued_at IS NULL;
```

If the count is >=40, the panel hit the soft cap. Options:
1. Wait for the rolling 7d window to clear.
2. Request a rate-limit increase from Let's Encrypt
   (https://letsencrypt.org/docs/rate-limits/). Once granted,
   raise the constant `mailCertNewOrderSoftCap` in
   `panel-api/internal/reconciler/mail_certificate_reconciler.go`.

### Stalwart still serves the panel-hostname cert for `mail.<d>`

Check what Stalwart sees:

```bash
sudo -u jabali-mail /usr/local/bin/stalwart-cli query x:Certificate --json
# Should show an entry with name="domain-<ulid>" + your domain's SANs.
```

If missing, force the deploy-hook to run:

```bash
JABALI_PANEL_CERT_KIND=mail-domain \
JABALI_MAIL_DOMAIN_ID=<ulid> \
RENEWED_LINEAGE=/etc/letsencrypt/live/mail.<domain> \
/etc/letsencrypt/renewal-hooks/deploy/jabali-panel-cert.sh
```

Verify the cert is now served:

```bash
openssl s_client -connect mail.<domain>:465 \
  -servername mail.<domain> -showcerts </dev/null \
  | openssl x509 -noout -subject -issuer -dates
```

### Force re-issue without clicking the UI

```sql
UPDATE mail_certificate
SET status = 'pending', last_error = NULL, next_retry_at = NULL
WHERE domain_id = '<ulid>';
```

Next reconciler tick will dispatch.

## Revoking a per-domain mail cert

```bash
certbot revoke --cert-name mail.<domain>
certbot delete --cert-name mail.<domain>

# Remove from Stalwart so SNI falls back to default:
sudo -u jabali-mail /usr/local/bin/stalwart-cli \
  query x:Certificate --json \
  | jq -r '.[] | select(.name=="domain-<ulid>") | .id' \
  | xargs -I{} sudo -u jabali-mail /usr/local/bin/stalwart-cli \
      delete x:Certificate --ids {}
```

Then flip the domain to `disabled`:

```bash
curl -X POST https://<panel>/api/v1/domains/<ulid>/mail-certificate/disable \
  -H "Authorization: Bearer <token>"
```

## Raising the LE new-order soft cap

After LE grants a rate-limit increase:

1. Edit `panel-api/internal/reconciler/mail_certificate_reconciler.go`.
2. Bump `mailCertNewOrderSoftCap`.
3. Rebuild + redeploy panel-api.

This is intentionally a code change, not a runtime config: operators
who haven't requested a rate-limit increase should not be able to
raise the cap accidentally and burn their LE quota.

## Audit / observability

```sql
-- Per-status counts:
SELECT status, COUNT(*) FROM mail_certificate GROUP BY status;

-- Domains nearing expiry:
SELECT d.name, mc.status, mc.expires_at,
       TIMESTAMPDIFF(DAY, NOW(), mc.expires_at) AS days_left
FROM mail_certificate mc
JOIN domains d ON d.id = mc.domain_id
WHERE mc.status = 'issued'
ORDER BY mc.expires_at ASC
LIMIT 50;

-- Recent failures + their reasons:
SELECT d.name, mc.status, mc.last_error, mc.updated_at
FROM mail_certificate mc
JOIN domains d ON d.id = mc.domain_id
WHERE mc.status IN ('failed','dns_missing')
ORDER BY mc.updated_at DESC
LIMIT 50;
```
