# M30.1 — Backup Schedules + Remote Destinations

> **M30.2 amendment (2026-04-28; supersedes sections below):** the
> async-copy + copy-worker design described in this plan was retired
> in M30.2 (ADR-0080). Per-destination model now writes DIRECTLY
> from the local backup stage to each destination — no source repo,
> no copy worker, no `backup_copy_jobs` queue. References below to
> `backup_copy_jobs`, `jabali backup copy run`,
> `jabali-backup-copy-worker.timer`, and the 'Async copy' constraint
> are all stale; current shape is a single backup phase that fans
> out to every linked destination in parallel. See ADR-0080 for
> the rationale; `jabali backup copy` cobra cmd prints the
> retirement notice + points operators at the per-destination
> workflow.

**Status:** SHIPPED (M30.1 + M30.2)
**Builds on:** M30 (ADR-0075). Local-only restic repo + manual backup + retention timer already shipped.
**ADR target:** **0077** — backup destinations + scheduling model.

## Goal

Add two admin-managed capabilities on top of M30:

1. **Multiple restic destinations** (local always; admin-defined remotes: SFTP / S3-compat / B2 / Azure / GCS / REST server). No rclone — restic-native only.
2. **Scheduled backups** — admin-defined cron schedules per kind (account_backup / system_backup) and per user, each linked to one or more destinations. After local backup completes, `restic copy` mirrors the snapshot to every linked remote **asynchronously** (does not block the backup job).

Out of scope:
- Per-user destination management (admin only).
- User-side scheduling UI (admin only).
- Auto-created defaults on user-add.
- Per-destination retention policies (single policy across all repos in v1; revisit if real demand surfaces).
- rclone / Google Drive / Dropbox / OneDrive (operators wanting these run their own restic-rest-server or S3 gateway).

## Constraints + invariants

- **Restic-native only.** Backend menu = `local | sftp | s3 | b2 | azure | gcs | rest`. S3 covers AWS, Backblaze B2 (S3 API), Wasabi, MinIO, Cloudflare R2.
- **Credentials never in DB.** `backup_destinations.credentials_ref` is a path under `/etc/jabali-panel/restic-remotes/<dest-id>.env` (root:root 0600). Each env file holds the restic env vars for that backend (`AWS_ACCESS_KEY_ID`, `B2_ACCOUNT_ID`, etc.) plus the dest's repo password if rotated independently. DB only stores the file pointer.
- **Local repo is implicit.** Schedule with zero linked destinations = local-only. `restic copy` to local target = no-op (skip).
- **Async copy.** Local backup marks job `succeeded` first. A separate `backup_copy_jobs` queue row per (backup_job, destination) is enqueued and processed by the copy worker. Backup UI shows local status + per-destination copy status.
- **Single repo password across all remotes.** `restic copy --from-repo … --from-password-file=…` requires source + dest passwords. v1 reuses `/etc/jabali-panel/restic-repo.password` for all destinations (same password, different storage). Per-destination password rotation = M30.2.
- **Cron parser:** `github.com/robfig/cron/v3` with `cron.WithSeconds()` disabled. Standard 5-field cron. Presets render to canonical strings: Daily=`0 3 * * *`, Weekly=`0 3 * * 0`, Monthly=`0 3 1 * *`.
- **Tick cadence:** every 60s. Single systemd timer `jabali-backup-scheduler.timer` → `jabali backup scheduler tick`. Idempotent; missed ticks (host suspend) are tolerated — the next tick picks up overdue schedules.
- **Concurrency:** scheduler dispatches via panel-agent the same way manual backups do. Existing per-(kind,user-id) job lock in panel-agent rejects duplicates (409); scheduler logs and skips on overlap.
- **Copy worker = systemd-run transient unit per copy job** (M29 carry-over). Survives `jabali update`. Unit name `jabali-backup-copy-<copy-job-id>.service`. ProtectHome=read-only; ReadWritePaths covers the local repo + cache.
- **Snapshot tagging carries through copy.** `restic copy` preserves tags. Retention timer scans by `--tag jabali` on every dest separately.
- **Retention runs per-destination.** v1 timer runs `restic forget --keep-* --prune` against the local repo. v2 add: enumerate destinations + run retention against each. Same `--keep-*` knobs from `server_settings`.

## ADR-0078 decision points

1. Restic-native backends only (no rclone).
2. Async copy via `restic copy` after local success.
3. Admin-only destination + schedule management.
4. Single shared repo password across destinations in v1.
5. Cron-string field (not separate hour/min/dow cols) — preset UI maps to canonical strings.
6. Scheduler = single 60s tick worker reading DB, not per-schedule systemd timers.

## Schema (migrations 086 + 087 + 088)

### 086_create_backup_destinations.up.sql
```sql
CREATE TABLE backup_destinations (
  id                   CHAR(26)     NOT NULL PRIMARY KEY,
  name                 VARCHAR(64)  NOT NULL,
  kind                 ENUM('local','sftp','s3','b2','azure','gcs','rest') NOT NULL,
  url                  VARCHAR(512) NOT NULL,
  credentials_ref      VARCHAR(255) NULL,    -- /etc/jabali-panel/restic-remotes/<id>.env
  enabled              TINYINT(1)   NOT NULL DEFAULT 1,
  created_at           DATETIME     NOT NULL,
  updated_at           DATETIME     NOT NULL,
  UNIQUE KEY uniq_backup_dest_name (name)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```

### 087_create_backup_schedules.up.sql
```sql
CREATE TABLE backup_schedules (
  id                   CHAR(26)     NOT NULL PRIMARY KEY,
  kind                 ENUM('account_backup','system_backup') NOT NULL,
  user_id              CHAR(26)     NULL,    -- NULL only for system_backup
  cron_expr            VARCHAR(64)  NOT NULL,
  enabled              TINYINT(1)   NOT NULL DEFAULT 1,
  keep_daily           INT          NULL,    -- NULL = inherit server_settings
  keep_weekly          INT          NULL,
  keep_monthly         INT          NULL,
  last_run_at          DATETIME     NULL,
  next_run_at          DATETIME     NULL,
  created_at           DATETIME     NOT NULL,
  updated_at           DATETIME     NOT NULL,
  KEY idx_backup_sched_due (enabled, next_run_at),
  KEY idx_backup_sched_user (user_id, kind),
  CONSTRAINT fk_backup_sched_user FOREIGN KEY (user_id)
    REFERENCES users(id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```

### 088_create_backup_schedule_destinations.up.sql
```sql
CREATE TABLE backup_schedule_destinations (
  schedule_id          CHAR(26) NOT NULL,
  destination_id       CHAR(26) NOT NULL,
  created_at           DATETIME NOT NULL,
  PRIMARY KEY (schedule_id, destination_id),
  CONSTRAINT fk_bsd_schedule    FOREIGN KEY (schedule_id)
    REFERENCES backup_schedules(id) ON DELETE CASCADE,
  CONSTRAINT fk_bsd_destination FOREIGN KEY (destination_id)
    REFERENCES backup_destinations(id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```

### Extension to `backup_jobs` (migration 089)
```sql
ALTER TABLE backup_jobs
  ADD COLUMN schedule_id CHAR(26) NULL AFTER user_id,
  ADD KEY idx_backup_jobs_schedule (schedule_id);
```
(`schedule_id` populated when scheduler-triggered; NULL for manual.)

### New `backup_copy_jobs` table (migration 090)
```sql
CREATE TABLE backup_copy_jobs (
  id              CHAR(26) NOT NULL PRIMARY KEY,
  backup_job_id   CHAR(26) NOT NULL,
  destination_id  CHAR(26) NOT NULL,
  status          ENUM('queued','running','succeeded','failed','cancelled') NOT NULL DEFAULT 'queued',
  started_at      DATETIME NULL,
  finished_at     DATETIME NULL,
  bytes_copied    BIGINT   NULL,
  error_message   TEXT     NULL,
  created_at      DATETIME NOT NULL,
  updated_at      DATETIME NOT NULL,
  KEY idx_bcj_queue (status, created_at),
  KEY idx_bcj_backup (backup_job_id),
  CONSTRAINT fk_bcj_backup FOREIGN KEY (backup_job_id)
    REFERENCES backup_jobs(id) ON DELETE CASCADE,
  CONSTRAINT fk_bcj_dest   FOREIGN KEY (destination_id)
    REFERENCES backup_destinations(id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```

Migration count: 086, 087, 088, 089, 090 (5 new).

## Architecture

### Components

```
admin UI ──CRUD──▶ panel-api ──▶ DB
                                  │
                                  ▼
              ┌──────────────────────────────────┐
              │ backup_schedules (cron, enabled) │
              │ backup_destinations              │
              │ backup_schedule_destinations     │
              └──────────────────────────────────┘
                                  ▲
                                  │ poll every 60s
                                  │
       jabali-backup-scheduler.timer ──▶ jabali backup scheduler tick
                                          │
                                          │ overdue rows → dispatch
                                          ▼
                                  panel-agent backup.create
                                          │
                                  local backup completes
                                          │
                                          ▼
                              enqueue backup_copy_jobs
                              (one per linked destination)
                                          │
                                          ▼
                  jabali-backup-copy-worker.timer (60s) OR
                  reconciler tick → spawn systemd-run transient
                                          │
                                          ▼
                              restic copy --from-repo=local --to=<dest>
```

### `jabali backup scheduler tick`

```go
func (s *Scheduler) Tick(ctx context.Context) error {
  due, err := s.repo.ListDue(ctx, time.Now())
  for _, sched := range due {
    if !sched.Enabled { continue }
    if running, _ := s.agent.IsBackupRunning(ctx, sched.Kind, sched.UserID); running {
      log.Infof("schedule %s: skip — backup already running", sched.ID)
      s.repo.UpdateNextRun(ctx, sched.ID, nextFireFromCron(sched.CronExpr, time.Now()))
      continue
    }
    if err := s.agent.DispatchBackup(ctx, BackupRequest{
      Kind: sched.Kind, UserID: sched.UserID, ScheduleID: sched.ID,
    }); err != nil {
      log.Errorf("schedule %s: dispatch failed: %v", sched.ID, err)
      continue
    }
    s.repo.MarkRan(ctx, sched.ID, time.Now(), nextFireFromCron(sched.CronExpr, time.Now()))
  }
  return nil
}
```

### Copy worker

After backup_jobs marks `succeeded`, panel-api hook (or post-backup callback) fans out:

```go
for _, dest := range schedule.Destinations {
  if dest.Kind == "local" { continue }   // no-op
  copyJob := backup.CopyJob{ID: ulid.Make(), BackupJobID: bj.ID, DestinationID: dest.ID, Status: "queued"}
  copyJobs.Create(copyJob)
}
```

Worker tick (60s timer, separate from scheduler):

```go
func (w *CopyWorker) Tick(ctx context.Context) error {
  queued, _ := w.repo.ListQueued(ctx, 5)   // 5 concurrent max
  for _, cj := range queued {
    unitName := fmt.Sprintf("jabali-backup-copy-%s.service", cj.ID)
    cmd := exec.Command("systemd-run",
      "--unit="+unitName, "--collect", "--quiet",
      "/usr/local/bin/jabali", "backup", "copy", "run", "--copy-job-id", cj.ID,
    )
    if err := cmd.Run(); err != nil { ... }
    w.repo.MarkRunning(ctx, cj.ID)
  }
  return nil
}
```

`jabali backup copy run --copy-job-id=<id>`:
```
restic --repo /var/lib/jabali-backups/repo \
       --password-file /etc/jabali-panel/restic-repo.password \
       copy \
       --to-repo <dest.URL> \
       --to-password-file /etc/jabali-panel/restic-repo.password \
       --tag job-id=<bj.JobID>           # only this job's snapshots
```
Env loaded from `dest.CredentialsRef` (`/etc/jabali-panel/restic-remotes/<dest-id>.env`).

### UI

**Admin Server Settings → Backups tab** (new sub-page or extend existing):
1. **Destinations** sub-tab
   - SearchableTable: `name | kind | url | enabled | actions`
   - Drawer (Create/Edit): name + kind dropdown + URL field + creds form (varies by kind)
   - Test-connection button → calls `restic snapshots` (or `restic init` for new) on the dest with form-supplied creds; result in toast.
2. **Schedules** sub-tab
   - SearchableTable: `kind | user | cron | next_run | destinations | enabled | actions`
   - Drawer (Create/Edit): kind dropdown + user picker (account_backup only) + cron field with preset buttons (Daily/Weekly/Monthly/Custom) + multi-select destinations + retention overrides (collapsed advanced section)
   - Cron preview shows next 5 firings.

**Admin Backups page (existing /jabali-admin/backups)** extends:
- New column "destinations" on Account/System tabs showing copy-status pills (`local ✓ | s3 ✓ | sftp ⌛ | b2 ✗`)
- Job detail drawer shows per-copy-job log + retry button on failed copies

## Step plan (5 waves, 11 steps)

### Wave A — schema + models (parallel-safe)
1. Migrations 086-090; reverse migrations.
2. GORM models: `BackupDestination`, `BackupSchedule`, `BackupScheduleDestination`, `BackupCopyJob`. Validation: cron parse, user_id mandatory for account_backup, NULL for system_backup, URL prefix matches kind.
3. Repositories with sqlmock tests: `BackupDestinationRepository` (CRUD + ListEnabled), `BackupScheduleRepository` (CRUD + ListDue + MarkRan + UpdateNextRun), `BackupCopyJobRepository` (CRUD + ListQueued + ListByBackupJob).

### Wave B — agent + CLI
4. `panel-agent/internal/commands/backup_copy.go`: `backup.copy` command. Reads dest from DB via panel-api or via shared config; runs `restic copy` with both passwords + dest env file.
5. `panel-api/cmd/server/backup_scheduler_cmd.go`: `jabali backup scheduler tick` (cron parser + dispatch loop).
6. `panel-api/cmd/server/backup_copy_cmd.go`: `jabali backup copy run --copy-job-id=<id>` + `jabali backup copy worker tick`.

### Wave C — REST
7. Admin REST endpoints (`panel-api/internal/api/backup_destinations.go`):
   - `GET /admin/backup-destinations` (list)
   - `POST /admin/backup-destinations` (create)
   - `PATCH /admin/backup-destinations/:id`
   - `DELETE /admin/backup-destinations/:id`
   - `POST /admin/backup-destinations/:id/test` → runs `restic snapshots` against the dest, returns success or error

8. Admin REST endpoints (`panel-api/internal/api/backup_schedules.go`):
   - `GET /admin/backup-schedules` (list, filter by kind / user_id)
   - `POST /admin/backup-schedules`
   - `PATCH /admin/backup-schedules/:id`
   - `DELETE /admin/backup-schedules/:id`
   - `POST /admin/backup-schedules/:id/run-now` → bypass cron, dispatch immediately

   Plus `GET /admin/backups/:job_id/copy-jobs` for status drilldown on the existing Backups page.

### Wave D — install + systemd
9. `install/systemd/jabali-backup-scheduler.{service,timer}` (every 60s) + `jabali-backup-copy-worker.{service,timer}` (every 60s). Same hardening pattern as retention timer (PrivateTmp / ProtectSystem=strict / ProtectHome=read-only / ReadWritePaths=/var/lib/jabali-backups + RESTIC_CACHE_DIR redirect / runs as root).
10. `install.sh`: `install -d -m 0700 -o root -g root /etc/jabali-panel/restic-remotes/`. Enable + start both new timers in install_backup_foundation.

### Wave E — UI + docs
11. Admin UI pages (`panel-ui/src/shells/admin/backups/destinations/*.tsx` + `panel-ui/src/shells/admin/backups/schedules/*.tsx`). Table + Drawer + per-kind credential forms (sftp_host/path/key, s3_endpoint/bucket/access_key/secret, b2_account_id/key/bucket, etc.). Cron preview via `cronstrue` (existing dep) or quick custom helper. Mount as Tabs on existing Backups admin page.

12. Docs: `docs/runbooks/backup-format.md` extension covering destinations + schedules; `docs/adr/0077-backup-destinations-and-schedules.md`; `plans/m30.1-backup-destinations-runbook.md` (operator how-to: add S3, add SFTP, common pitfalls).

## Files to modify / add

```
panel-api/internal/db/migrations/
  000086_create_backup_destinations.up.sql + .down.sql
  000087_create_backup_schedules.up.sql + .down.sql
  000088_create_backup_schedule_destinations.up.sql + .down.sql
  000089_alter_backup_jobs_schedule_id.up.sql + .down.sql
  000090_create_backup_copy_jobs.up.sql + .down.sql

panel-api/internal/models/
  backup_destination.go (new)
  backup_schedule.go (new)
  backup_copy_job.go (new)
  backup_job.go (add ScheduleID *string)

panel-api/internal/repository/
  backup_destination_repository.go (new) + _test.go
  backup_schedule_repository.go (new) + _test.go
  backup_copy_job_repository.go (new) + _test.go

panel-api/internal/api/
  backup_destinations.go (new)
  backup_schedules.go (new)
  backups.go (extend with copy-jobs endpoint)

panel-api/cmd/server/
  backup_scheduler_cmd.go (new)
  backup_copy_cmd.go (new)
  serve.go (wire schedulers + workers into deps + start tick goroutines OR rely on systemd timers)
  root.go (add subcommands)

panel-agent/internal/commands/
  backup_copy.go (new) — reads creds env file + runs restic copy

internal/backup/
  copy.go (Client.Copy wrapper around restic copy + env loader)
  cron.go (cron parser shim)
  *_test.go

install.sh
  apt + dirs already done; add /etc/jabali-panel/restic-remotes/
  install_backup_foundation: enable scheduler + copy-worker timers

install/systemd/
  jabali-backup-scheduler.service + .timer
  jabali-backup-copy-worker.service + .timer

panel-ui/src/shells/admin/backups/
  destinations/{ListPage,Drawer,kindForms/*}.tsx
  schedules/{ListPage,Drawer,CronPicker}.tsx
  AdminBackupsPage.tsx (add new tabs)
  copyJobsCell.tsx (per-row pills on existing table)

docs/adr/0077-backup-destinations-and-schedules.md
docs/runbooks/backup-format.md (extend)
plans/m30.1-backup-destinations-runbook.md
docs/BLUEPRINT.md (M30.1 row)
```

## Verification (per wave)

**Wave A:**
- `migrate up` to 90 then `migrate down` to 85 cleanly. DESCRIBE shows all FKs intact.
- Unit tests: `go test ./panel-api/internal/repository/...` covers CRUD + ListDue + ListQueued + cron-derived next_run.

**Wave B:**
- Manual: insert one local-only schedule, run `jabali backup scheduler tick` → schedules ran, backup_jobs row exists.
- Manual: insert SFTP destination + creds env + a schedule, tick → backup_jobs succeeded, backup_copy_jobs queued, worker tick fires systemd-run, copy succeeded, snapshot visible on remote with `restic --repo sftp:… snapshots`.

**Wave C:**
- E2E: POST destination → 201; POST schedule referencing it → 201; PATCH enabled=false → list filters; DELETE schedule cascades schedule_destinations rows; cross-user/admin enforcement on every endpoint (401 for unauth, 403 for non-admin).

**Wave D:**
- Fresh VM install lays down both timers active. `systemctl list-timers | grep jabali-backup` shows scheduler + copy-worker + retention.
- Hardening: `systemd-analyze security jabali-backup-scheduler.service` ≤ 1.5.

**Wave E:**
- Playwright: admin creates SFTP dest, creates daily schedule, "Run now" → backups page shows new row with `local ✓ | sftp ⌛` then `sftp ✓` after copy worker tick.

## Risk + rollout

- **Credential leak surface.** Env files at 0600 root:root. UI never returns secrets in GET responses; PATCH with empty secret = no change. Documented in runbook.
- **Tick storm if 1000 schedules due simultaneously after host suspend.** ListDue caps at 50 per tick; remainder picked up next tick.
- **Copy failures cascade.** Failed copy jobs retry up to 3× via worker (exponential backoff: 1m, 5m, 30m); after 3 fails, status=failed and M14 notification fires.
- **`restic copy` requires both passwords.** v1 uses same password file for source + dest. If a remote rotates its password, current pattern breaks. M30.2 = per-destination password file at `/etc/jabali-panel/restic-remotes/<id>.password` mirroring the env file pattern.
- **Backwards compat.** All M30 manual flows preserved; schedules + destinations are pure additions. Existing manual backups create no copy jobs (no schedule = no destinations linked).

## Out of scope (future M30.2 / M30.3)

- Per-destination retention policies.
- Per-destination passwords / key rotation.
- Encryption-at-rest different from repo password.
- Bandwidth limits on copy (`restic` doesn't expose; would need traffic-shaping wrapper).
- Cross-region replication topology UI.
- rclone backends (Google Drive et al.) — operators run own restic-rest-server.
