# M33 Malware Detection — Operator Runbook

**Status:** SHIPPED 2026-04-27 (all 10 steps on `m33/malware-detection`).
**ADR:** [ADR-0072](../docs/adr/0072-malware-detection-stack.md)
**Plan:** [m33-malware-detection.md](./m33-malware-detection.md)

## Architecture

```
[ClamAV freshclam]──daily──>[/var/lib/clamav/*.cvd]
[rfxn maldet -u]──daily──>[/usr/local/maldetect/sigs]
                                │
                                ▼
[maldet --monitor USERS] ─inotify─> [/usr/local/maldetect/quarantine/]
   uses ClamAV as scanner                │
   inotify_minuid=1000                   │
                                         │
[/usr/local/maldetect/sess/session.<id>] (one per scan)
                                         │
                  ┌──────────────────────┘
                  ▼
[panel-agent sessionwatcher (5s poll)] ──parse──> [POST /admin/security/malware/event]
                                                      │ (RequireLocalhost)
                                                      ▼
                                         [malware_events + malware_quarantine]
                                                      │
                                                      ▼
                                         [eventsources/malware.go ticker (30s)]
                                                      │
                                                      ▼
                                         [M14 dispatcher → 6 channels]

[Tetragon] ──TracingPolicy match──> [/var/log/tetragon/tetragon.log]
                                         │
                                         ▼ (manual journalctl tail until tetragon-relay shim lands)
                                         (defer to next minor: malware_events.source=tetragon ingest)
```

## Quick verification

```bash
# Stack health (run on VM):
ssh root@192.168.100.150 'systemctl is-active clamav-daemon clamav-freshclam jabali-maldet-monitor tetragon'
ssh root@192.168.100.150 'systemctl list-timers | grep jabali-mal'
ssh root@192.168.100.150 'maldet --version | head -1'
ssh root@192.168.100.150 'clamscan --version | head -1'

# Tab smoke (from browser):
# https://<panel>/jabali-admin/security?tab=malware
# Overview should show: ClamAV daemon active + Freshclam active + Realtime monitor active
# Tetragon: active OR "Disabled (kernel does not expose BTF)" on older kernels

# Plant an EICAR test file under tenant docroot (creates hit + notification):
ssh root@192.168.100.150 'sudo -u testuser sh -c "echo \"X5O!P%@AP[4\\PZX54(P^)7CC)7}\\\$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!\\\$H+H*\" > /home/testuser/public_html/eicar.txt"'
sleep 60   # inotify_minutes can be tight; monitor catches within a minute
ssh root@192.168.100.150 'tail /var/log/maldet/event_log'
# Verify in /jabali-admin/security?tab=malware → Quarantine card row appears
# Verify in-app bell (or whichever channel the admin enabled) shows alert
```

## Common operations

### Restore quarantined file (false positive)

1. Open `/jabali-admin/security?tab=malware` → Quarantine card.
2. Click **Restore** on the row.
3. Drawer requires reason (>= 5 chars). Enter why this file is safe.
4. Click **Restore** in drawer. File moves back to original path.
5. Audit row: `restored_at`, `restored_by`, `restored_reason` populated in `malware_quarantine`.

### Manual scan a path or user

1. Open Manual scan card.
2. Choose **Path** or **User** radio.
3. Type the absolute path (`/home/alice/public_html`) or username (`alice`).
4. Click **Scan**. UI polls `/scan/:id/status` every 1.5s and tails the log.
5. Hits land in Quarantine card (and Events card) when scan completes.

### Force a fresh signature pull

1. Overview card → **Update signatures** button.
2. Runs `freshclam --quiet && maldet -u` (5-min budget).
3. Status card shows updated `clamav_sig_date` + `maldet_sig_version`.

### Add a custom YARA rule

1. YARA rules tab → **Upload rule**.
2. Drawer: filename matches `^[a-z0-9][a-z0-9._-]{0,63}\.yar$`, content <= 64KB.
3. Pre-flight `yara -w <tmp> /dev/null` runs server-side; UI surfaces parser error on rejection.
4. Lands at `/etc/jabali/yara/<filename>` (mode 0644).
5. Audit row in `yara_custom_rules` (uploaded_by, sha256).
6. maldet picks up new rule on next scan (yara_rules glob expanded each scan).

### Toggle a Tetragon policy

1. Tetragon tab → table row → **Enabled** Switch.
2. Agent renames `.yaml` ↔ `.yaml.disabled` + `systemctl reload tetragon`.
3. `tetragon_policy_state` row reflects new state with `updated_by`.

## Troubleshooting

### clamd consuming too much RAM

ClamAV daemon idles ~80 MB. If RSS climbs above 500 MB, signature DB
may be corrupt:

```bash
systemctl stop clamav-freshclam clamav-daemon
rm -f /var/lib/clamav/*.cvd /var/lib/clamav/*.cld
systemctl start clamav-freshclam   # pulls fresh DB
sleep 30
systemctl start clamav-daemon
```

### freshclam DNS / network failures

`/var/log/clamav/freshclam.log` shows `Can't get download URL`. Verify
DNS first (`resolvectl query database.clamav.net`). The Jabali pdns
recursor only forwards configured zones; outbound for arbitrary names
is the system resolver. If the host has only `127.0.0.1` (pdns) in
`/etc/resolv.conf`, freshclam fails.

Fix: ensure `systemd-resolved` upstream is configured in
`/etc/systemd/resolved.conf.d/jabali.conf` (install.sh seeds this).

### maldet inotify watches exhausted

Symptom: Realtime monitor card shows watch count growing then stuck;
`/var/log/maldet/event_log` shows `inotify_add_watch failed: No space left on device`.

```bash
sysctl fs.inotify.max_user_watches   # should be 524288 (install.sh sets it)
# If lower, reapply:
sysctl --system
systemctl restart jabali-maldet-monitor
```

### Tetragon BPF load fails

`journalctl -u tetragon` shows `loading BPF program failed`. Common
causes: kernel < 5.10, BTF missing, BPF JIT off.

```bash
[ -f /sys/kernel/btf/vmlinux ] && echo "BTF ok" || echo "BTF MISSING"
sysctl kernel.unprivileged_bpf_disabled    # 0 or 1 = ok; "2" blocks all unpriv BPF — but tetragon runs as root so this should be fine
uname -r                                    # need >= 5.10
```

If unsupported: `touch /etc/jabali/tetragon-disabled` and the UI
shows the disabled-with-reason Alert. Other malware detection
(maldet/ClamAV/YARA) keeps working.

### Quarantine dir growing unbounded

Default retention is 30 days, applied via
`jabali-malware-quarantine-purge.timer` (daily 04:00, calls
`jabali malware-purge`). If timer is disabled or retention is too
long, run manually:

```bash
jabali malware-purge
df -h /usr/local/maldetect
```

Operator can lower retention via `/admin/security/malware` →
Settings → Quarantine retention (days).

### Event ingestion stuck

If quarantine fills up but Events card stays empty: agent
sessionwatcher may have stalled. Cache file at
`/var/lib/jabali-agent/sessionwatcher.json` tracks last-seen mtime
per session.

```bash
journalctl -u jabali-agent | grep sessionwatcher
# To force backfill of all historical sessions:
systemctl stop jabali-agent
rm /var/lib/jabali-agent/sessionwatcher.json
systemctl start jabali-agent
```

## Reference: notification routing

| Source/severity            | EventKind                     | M14 severity | Default channels                    |
| -------------------------- | ----------------------------- | ------------ | ----------------------------------- |
| maldet/clamav/yara warn    | `malware.quarantine.added`    | warning      | gated by `notify_threshold` setting |
| maldet/clamav/yara critical| `malware.realtime.critical`   | critical     | all admin-enabled channels          |
| Tetragon (any severity)    | `malware.realtime.critical`   | critical     | all admin-enabled channels (always) |

Threshold setting suppresses `info`/`warn` if set to `critical`;
Tetragon source ignores threshold entirely (runtime exec from `/tmp`
should never be silenced).

## Reference: file paths

| Path                                              | Purpose                              |
| ------------------------------------------------- | ------------------------------------ |
| `/usr/local/maldetect/conf.maldet`                | maldet config (Jabali drop-in merged via JABALI-DROPIN-BEGIN/END markers) |
| `/etc/jabali/maldet/conf.maldet.d/00-jabali.conf` | Jabali maldet drop-in source        |
| `/usr/local/maldetect/sess/session.<id>`          | Per-scan session file (sessionwatcher reads) |
| `/usr/local/maldetect/quarantine/`                | Quarantined files (NOT under tenant docroot) |
| `/etc/jabali/yara/*.yar`                          | Admin-uploaded YARA rules           |
| `/etc/tetragon/tetragon.tp.d/jabali-*.yaml`       | Default Tetragon TracingPolicies    |
| `/var/log/tetragon/tetragon.log`                  | Tetragon JSON event export          |
| `/etc/jabali/tetragon-disabled`                   | Sentinel file: skip Tetragon (set by install.sh on no-BTF) |
| `/var/lib/jabali-agent/sessionwatcher.json`       | sessionwatcher last-seen-mtime cache |

## Rollback

Full uninstall (ADR-0072 rollback):

```bash
systemctl disable --now jabali-maldet-monitor jabali-maldet-update-signatures.timer \
  jabali-maldet-scan-daily.timer jabali-malware-quarantine-purge.timer tetragon
apt-get -y remove --purge clamav clamav-daemon clamav-freshclam yara
rm -rf /opt/tetragon /etc/tetragon /var/log/tetragon /usr/local/bin/tetragon /usr/local/bin/tetra
/usr/local/maldetect/uninstall.sh   # ships with LMD
rm -rf /etc/jabali/maldet /etc/jabali/yara /etc/jabali/tetragon-disabled
rm -f /etc/systemd/system/jabali-maldet-*.service /etc/systemd/system/jabali-maldet-*.timer
rm -f /etc/systemd/system/jabali-malware-quarantine-purge.* /etc/systemd/system/tetragon.service
systemctl daemon-reload
```

Schema rollback:

```bash
jabali migrate down 1   # reverts migration 000080
```

UI tab disappears automatically when malware repos are nil
(graceful degradation).
