# M30.1 Backup Destinations + Schedules — Operator Runbook

ADR: 0078. Plan: `plans/m30.1-backup-schedules-destinations.md`.

## Add a remote destination

1. Admin UI: `/jabali-admin/backups` → Destinations tab → New destination.
2. Pick the backend.

### SFTP

The SFTP form has dedicated host / user / port / path fields plus an
auth picker:

- **SSH key** (default): pick from a dropdown of `/root/.ssh/id_*`
  private keys. Click **Generate new key** for a fresh ed25519 (or
  rsa-4096) pair. The modal returns the public-key body — copy it
  into the remote's `~/.ssh/authorized_keys` *before* saving.
  Passphrase-protected keys are listed but disabled (BatchMode auth
  fails on them).
- **Password**: stored encrypted-at-rest as `SSHPASS=<value>` in
  `/etc/jabali-panel/restic-remotes/<dest-id>.env` (root:root 0600).
  Requires the `sshpass` package, which `install.sh` provisions
  alongside `restic`.

The wrapper composes `-o sftp.command=...` automatically: `ssh -i
<key>` for key auth, `sshpass -e ssh` for password auth, and adds
`-p <port>` when the port is non-22.

### S3 / B2 / Azure / GCS / REST

3. Enter the restic URL — see the inline hint per backend, e.g.
   - S3: `s3:s3.amazonaws.com/<bucket>/<path>`
   - B2: `b2:<bucket-name>:<path>`
4. Paste credentials as KEY=VALUE lines into the credentials field. Common sets:
   - S3: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
   - B2: `B2_ACCOUNT_ID`, `B2_ACCOUNT_KEY`
   - Azure: `AZURE_ACCOUNT_NAME`, `AZURE_ACCOUNT_KEY`
   - GCS: `GOOGLE_APPLICATION_CREDENTIALS=/etc/jabali-panel/restic-remotes/<dest-id>.gcs.json` (you must place the JSON keyfile at that path manually; M30.2 adds a binary upload form)

### Verify + save

5. Click **Test** before saving. The button runs `restic snapshots` against the URL with the supplied creds; success means restic could authenticate AND read the repo. Empty repo = success ("0 snapshots").
6. Save. The DB row is created and the env file is written to
   `/etc/jabali-panel/restic-remotes/<dest-id>.env` (root:root 0600).

### Initialize a brand-new remote repo

The first backup that copies to a fresh remote will fail with
"unable to open config file: …" if the remote isn't a restic repo
yet. Initialize manually:

```
RESTIC_REPOSITORY="<your URL>" \
RESTIC_PASSWORD_FILE=/etc/jabali-panel/restic-repo.password \
AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… \
restic init
```

(Replace the AWS_ env vars with whatever the backend needs.) Then the
M30.1 copy worker takes over.

## Add a scheduled backup

1. `/jabali-admin/backups` → Schedules tab → New schedule.
2. Pick **Account** (per-user) or **System** (full server snapshot).
3. For Account: pick the user from the dropdown.
4. Pick a cadence preset (Daily / Weekly / Monthly) or **Custom** for raw cron.
5. Multi-select the destinations to mirror to. Local repo always receives a copy regardless; the picker is for remotes only.
6. (Advanced) Override `keep_daily` / `keep_weekly` / `keep_monthly` if you want different retention than the server defaults. Leave blank to inherit.
7. Save.

The next 60s tick fires the schedule if `next_run_at <= NOW()`. Use
**Run now** to advance `next_run_at` to NOW manually.

## Pipeline

```
backup_schedules (cron)
        │  60s tick
        ▼
panel-agent backup.create
        │
        ▼
backup_jobs.status='running'
        │
        │  agent writes manifest snapshot
        ▼
finalizer (30s tick) sees manifest -> mark succeeded
        │
        ▼
enqueue backup_copy_jobs per destination
        │  60s tick
        ▼
copy worker spawns systemd-run transient unit:
   jabali-backup-copy-<copy-job-id>.service
        │
        ▼
restic copy --from-repo=local --to=<dest URL>
        │
        ▼
backup_copy_jobs.status='succeeded'
```

Retention runs once daily at 04:30 against the local repo (M30 timer).
Per-destination retention is M30.2.

## Failure modes + diagnosis

### Test connection fails
- Bad creds: error returned in the toast verbatim. Fix env, save.
- Network unreachable: `journalctl -u jabali-panel -n 200` shows the
  outbound restic stderr.
- Wrong URL prefix: validation rejects (must start with `s3:`, `sftp:`,
  `b2:`, etc.).

### Schedule never fires
- Check `next_run_at` in DB:
  ```sql
  SELECT id, kind, cron_expr, enabled, last_run_at, next_run_at
  FROM backup_schedules ORDER BY next_run_at ASC;
  ```
- `enabled=0` means UI toggle is off.
- `next_run_at` in the future (>1h) means the cron is correct but the
  next tick is later. Click **Run now** to force.
- Panel logs: `journalctl -u jabali-panel | grep "backup scheduler"`.

### Copy job stuck in queued
- Worker tick 60s; longest expected wait = 60s.
- `backup_copy_jobs.next_attempt_at IN THE FUTURE` = backoff after a
  prior fail. Wait or manually:
  ```sql
  UPDATE backup_copy_jobs SET next_attempt_at = NOW(), retry_count = 0
  WHERE id = '<copy-job-id>';
  ```
- Persistent failures past 3 retries → `status='failed'`, `error_text`
  shows the last restic stderr.

### Copy job runs but writes nothing
- restic copy reuses the same password for source + dest. v1 enforces
  this; if a remote was init'd with a different password, recover by:
  1. delete the remote repo (or rotate panel password — destroys local
     too).
  2. recreate dest in admin UI; first run inits the remote with the
     panel password.
- `bytes_copied` is always 0 in v1 (restic copy lacks JSON summary;
  observability debt tracked in ADR-0078 Notes).

### restic-remotes env file lost / corrupted
- Files at `/etc/jabali-panel/restic-remotes/<dest-id>.env` are part
  of the M30 system_backup `panel_config` stage — restore from the
  most recent system backup or recreate via PATCH /api/v1/admin/backup-destinations/<id>
  with a new `credentials_env` body.

## CLI reference

- `jabali backup retention apply` — manual retention sweep (M30; uses
  server_settings.backup_keep_*).
- `jabali backup copy run --copy-job-id=<id>` — invoked by the systemd-run
  transient unit; runs `restic copy` for one queue row. Operators can
  call it manually for debugging.

## Manual rollback

To uninstall M30.1 without losing M30:

```sql
SET FOREIGN_KEY_CHECKS=0;
DROP TABLE backup_copy_jobs;
DROP TABLE backup_schedule_destinations;
DROP TABLE backup_schedules;
DROP TABLE backup_destinations;
ALTER TABLE backup_jobs DROP KEY idx_backup_jobs_schedule, DROP COLUMN schedule_id;
DELETE FROM schema_migrations WHERE version IN (86, 87, 88, 89, 90);
SET FOREIGN_KEY_CHECKS=1;
```

Restart panel-api; the in-process tickers fail-fast (deps missing) and
log the disabled state without crashing.
