# ADR-0072 — Malware detection stack: ClamAV + maldet + YARA + Tetragon

**Status:** accepted
**Date:** 2026-04-27 (shipped same day on `m33/malware-detection`)
**Supersedes:** none
**Superseded by:** none

## Context

Jabali is shared hosting. Tenants upload PHP/JS/binaries to their
docroots; some are compromised, some are intentional malware staging,
some are crypto miners running out of `/tmp`. Until M33 we had:

- **CrowdSec AppSec** (M27) — HTTP-layer WAF; catches signatures in
  request bodies. Doesn't see what already lives on disk.
- **CrowdSec scenarios** — detect attack patterns at the request log
  level. Won't catch a web shell that's already installed and called
  by the tenant themselves.
- **No file-system scanner.** No quarantine. No runtime detection.
- **No notification path** for "your shared host has malware on it."

The gap: a tenant uploads a PHP web shell (typical pattern: dynamic
code execution sourced from POST data) into
`/home/alice/public_html/uploads/x.php`. CrowdSec doesn't see it
(no abnormal request shape). The admin doesn't see it. It runs until
either the IP gets blocked for outbound abuse (CrowdSec post-fact) or
the tenant's bill spikes from miner CPU (also post-fact).

## Decision

Deploy four orthogonal detectors as a single defence-in-depth pipeline,
unified into one MariaDB schema, surfaced in one admin tab, alerting
through the existing M14 notification dispatcher.

| Layer                     | Tool                          | Catches                                |
| ------------------------- | ----------------------------- | -------------------------------------- |
| File signature engine     | **ClamAV** + freshclam        | Known-bad files via daily sig updates  |
| Orchestrator + quarantine | **Linux Malware Detect**      | Per-user inotify watches + quarantine  |
| Behavior signatures       | **YARA** (rfxn + custom)      | Pattern-based catches LMD doesn't ship |
| Runtime exec watcher      | **Tetragon** (eBPF)           | Exec from `/tmp`, curl-piped-bash, ptrace |

Wired together: maldet uses ClamAV as its scanner backend
(`scan_clamscan=1`) + monitors per-user docroots via inotify
(`--monitor USERS`, `inotify_minuid=1000`) + quarantines hits to
`/usr/local/maldetect/quarantine/`. The agent fsnotify-watches
`/usr/local/maldetect/sess/` and parses session files; Tetragon
events flow through `tetragon-relay` (Go shim tailing the JSON log).
Both sources POST to a loopback-only `/admin/security/malware/event`
on panel-api which writes `malware_quarantine` + `malware_events`.
A ticker in `eventsources/malware.go` drains unnotified events into
the M14 dispatcher.

## Alternatives considered

### ClamAV-only

- **Pros:** simplest, ships in Debian, signatures via freshclam.
- **Cons:** no real-time inotify (only on-demand `clamscan`); no YARA;
  no eBPF runtime catch. Misses miners running from /tmp.
- **Verdict:** rejected. Coverage gap on the most common shared-host
  attack patterns (web-shell drop + crypto miner exec).

### Wazuh / OSSEC

- **Pros:** mature HIDS, comprehensive coverage, dashboards.
- **Cons:** heavyweight; runs its own server + agent + dashboard.
  Overlaps M14 (notifications) and admin UI; we'd duplicate dispatch
  and storage. Memory footprint exceeds the entire current Jabali
  panel by 2x.
- **Verdict:** rejected. Wrong scale for shared-host control plane;
  we already own the surface they'd duplicate.

### Tetragon-only

- **Pros:** strong runtime detection; modern eBPF; first-class K8s
  support.
- **Cons:** doesn't catch a web-shell that's uploaded but never exec'd
  (tenant has it ready for exfil/sale, never runs it locally — caught
  only when called via HTTP, by which point CrowdSec is the layer).
  No file-signature catch, no YARA.
- **Verdict:** rejected as sole scanner. Adopted as one of four.

### maldet without ClamAV backend

- **Pros:** one fewer service.
- **Cons:** LMD's own scanner (`scan_clamscan=0`) is slower and has
  weaker hit rate on common WordPress shells per upstream notes.
  Practical impact: a full nightly scan on a 10k-domain box rises
  from ~30 min to ~2 hr without ClamAV.
- **Verdict:** rejected. ClamAV daemon is the marginal cost; LMD
  becomes the orchestrator that uses it.

## Consequences

### Positive

- **Single admin surface:** one Malware tab, one notification path,
  one quarantine table. Operators don't context-switch.
- **Defence in depth:** orthogonal failure modes covered. A miss in
  one layer is caught by another (web-shell upload → maldet inotify;
  miner exec → Tetragon; new signature → freshclam next pull).
- **Reuse M14 infrastructure:** zero new senders, zero new dispatchers.
  Email/ntfy/slack/discord/webpush/in-app bell already exist.
- **Reuse M29 transient-systemd pattern:** manual scans run as
  `jabali-maldet-scan-<id>.service` — same lifecycle as M29 admin
  updates. Operator already trained.
- **Restorable quarantine:** every quarantined file can be inspected,
  restored (with reason + audit), or downloaded for forensics. Not a
  one-way street.

### Negative

- **~150 MB extra RAM** per host: clamd ~80 MB + maldet+inotify ~30 MB
  + tetragon ~40 MB. Documented; settings.realtime_enabled toggle lets
  operators stop the inotify monitor (clamd stays).
- **Disk overhead** for signature DBs (~250 MB ClamAV + ~50 MB rfxn) +
  quarantine retention (configurable, default 30 days).
- **Tetragon kernel requirement:** ≥ 5.10 + BTF. install.sh probes
  `bpftool btf list` and gracefully skips on unsupported kernels;
  UI shows "Disabled (no BTF)". Older Debian 11 / RHEL 7 hosts lose
  the runtime-detection layer.
- **Notification storm risk during attack:** 1000 quarantined files
  in seconds → eventsources/malware.go batches per
  (user, signature) within a 60s window before dispatching.

### Risks accepted

- **Restore is a foot-gun.** Operator restoring a real web-shell
  re-introduces it. Mitigation: required reason text + 3-second
  confirm + audit log entry showing signature.
- **YARA rule upload DoS.** A pathological regex could stall maldet.
  Mitigation: pre-flight `yara -w` on a 1KB sample + maldet's
  `scan_max_filesize` + max 100 custom rule files quota.
- **GitHub release tarball SHA drift.** Tetragon has no Debian apt
  repo; install.sh pins `TETRAGON_VERSION` + `TETRAGON_SHA256`.
  Bumps go through normal PR review.

## Implementation

10-step plan at [`plans/m33-malware-detection.md`](../../plans/m33-malware-detection.md).

- Schema: migration `000081_malware_detection`
- Routes: `RegisterSecurityMalwareRoutes` mounted alongside
  `RegisterSecurityCrowdSecRoutes` and `RegisterSecurityUFWRoutes`
  in `panel-api/internal/app/app.go`
- Tab: `panel-ui/src/shells/admin/security/AdminSecurityPage.tsx`
  TAB_KEYS extended `["crowdsec", "malware", "ufw"]`
- Event source: `panel-api/internal/eventsources/malware.go` +
  two new sources (`malware_quarantine_added`, `malware_realtime_critical`)
  registered alongside existing 6

## Amendment 2026-04-27 — drop ClamAV, add php-malware-finder

### Decision change

ClamAV + clamav-daemon + clamav-freshclam REMOVED from the M33 stack.
php-malware-finder (PMF) YARA rule pack added in their place, fed
into LMD's existing `yara_rules` glob.

### Why

- `clamd` held 1.7 GB RAM resident for the full sig DB load — measured
  on `mx.jabali-panel.com` (Server Status → Processes). On a small VPS
  that's a third of total RAM for a single defensive engine.
- Most threats on Jabali (PHP-only shared hosting) are PHP webshells,
  obfuscated PHP, eval/base64 patterns. ClamAV's general-purpose sig
  set catches Windows-malware mail attachments and similar — relevant
  on a mail gateway, not on a hosting docroot.
- PMF (`github.com/jvoisin/php-malware-finder`) ships a single
  `php.yar` covering exactly the PHP threat classes Jabali sees:
  obfuscation, e-modifier preg_replace, dangerous functions,
  webshell signatures. Drops straight into
  `/etc/jabali/yara/php-malware-finder.yar` with no extra runtime
  cost beyond LMD's existing YARA scan path.
- LMD's built-in pattern engine + rfxn YARA pack + PMF YARA pack +
  Tetragon eBPF still give defence-in-depth. ClamAV was the layer
  with the worst RAM-per-coverage ratio for our workload.

### What changed

- `install.sh`:
  - `clamav clamav-daemon clamav-freshclam` removed from apt list.
  - First-run purge step (`clamav-purged` marker) stops + masks +
    `apt-get purge` + wipes `/var/lib/clamav` on existing installs.
    Operators who want ClamAV back touch `/etc/jabali/clamav-keep`.
  - `scan_clamscan="1"` → `"0"` in maldet drop-in.
  - `freshclam --quiet` removed from sig-update timer.
  - New `jabali-pmf-update.{service,timer}` — daily fetch from upstream,
    atomic write, `maldet --monitor RELOAD`.
- `panel-agent`:
  - `mwStatusResponse` keeps ClamAV fields for wire-compat (always
    false/empty) and adds `pmf_rules_present`, `pmf_rules_count`,
    `pmf_rules_updated`.
  - `mwUpdateSignaturesHandler` no longer calls `freshclam`; triggers
    `jabali-pmf-update.service` instead.
- `panel-ui`:
  - Drops the ClamAV daemon, Freshclam, ClamAV version, ClamAV sig age
    tiles from the Malware Overview.
  - Adds PMF rules + PMF rules updated tiles.
- DB enum: `enum('maldet','clamav','yara','tetragon')` LEFT IN PLACE.
  Existing rows still resolve; new rows just won't use `'clamav'`.

### Operator action

`jabali update` is sufficient on existing installs:
- `install_malware_stack` runs the purge step once (idempotent via
  `/etc/jabali/clamav-purged` marker)
- Frees ~1.7 GB RAM + ~180 MB disk
- maldet inotify continues running with PMF + rfxn YARA + LMD's own
  signatures

## Amendment 2 — 2026-04-28 — clamscan on-demand + admin-opt-in monitor

### Decision change

Amendment 1 (2026-04-27) was **wrong on the load-bearing technical
claim** that PMF would "drop straight into LMD's existing YARA scan
path with no extra runtime cost". Verified live on 192.168.100.150:
`scan_clamscan=0` removes ALL YARA scanning from maldet. There is no
"native YARA path" — maldet's YARA invocation lives entirely inside
clamscan via `$clamav_db -d <yara file>` (see
`/usr/local/maldetect/internals/functions:1820`). Without ClamAV,
maldet runs MD5 + HEX only; rfxn.yara + PMF rules sit on disk unread.

Amendment 1's EICAR smoke pass (project_m33_shipped.md) worked via
MD5 sig hit, not YARA. ClamAV's removal hadn't changed observed
behavior because the test never exercised the YARA path.

This amendment reinstates ClamAV in **on-demand** form (binary only,
no daemons) and gates the inotify monitor behind admin opt-in.

### Why

**On-demand instead of resident:**
- maldet's monitor batches changed files via inotifywait then runs ONE
  `clamscan -f <list>` per cycle (default 120s when no clamd)
- Sig DB peaks at ~1.7 GB during scan, frees on exit
- Avg resident RAM ≈ 0 (vs amendment-1 prior daemon: 1.7 GB constant)
- 30-120 s detection latency (vs daemon's 5-15 s) — acceptable for
  shared hosting

**Admin opt-in for the monitor:**
- Even on-demand, the per-cycle 1.7 GB RAM peak is too costly to enable
  by default on small VPS hosts
- Daily scheduled scans + on-demand panel scans + Tetragon eBPF still
  cover threats without the monitor running
- Operators with headroom flip Settings → "Real-time scanning" on; the
  agent reconciles `jabali-maldet-monitor.service` start/stop

### What changed

- `install.sh`:
  - `clamav` + `clamav-freshclam` reinstalled `--no-install-recommends`;
    `clamav-daemon.service` + `clamav-freshclam.service` `systemctl mask`'d
  - New `jabali-freshclam.{service,timer}` — daily 02:15 one-shot
    `freshclam`. First-install runs synchronously so clamscan has sigs
    immediately.
  - `scan_clamscan="0"` → `"1"` in maldet drop-in (reverted from
    amendment 1)
  - `jabali-maldet-monitor.service` is `systemctl disable --now`'d
    unconditionally on every install run — opt-out is enforced at the
    host level, not only in the DB. Admin re-enables via UI; agent
    `security.malware.monitor.set_enabled` runs `enable --now`.
  - `clamav-purged` marker removed (amendment 1 artifact)
  - `/etc/jabali/maldet/build-rfxn-yara.sh` appends PMF php.yar +
    `private rule IsWhitelisted { condition: false }` stub to
    `/usr/local/maldetect/sigs/rfxn.yara`. Re-applied via
    `ExecStartPost` on `jabali-maldet-update-signatures.service` so PMF
    survives `maldet -u` rfxn.yara overwrites.

- DB: migration `000082_malware_realtime_optin.up.sql` flips
  `malware_settings.realtime_enabled` `DEFAULT 0` and updates the
  existing row to 0.

- `panel-agent`:
  - New `security.malware.monitor.set_enabled` command (start/stop
    monitor unit)
  - `mwStatusResponse` now returns clamscan binary version + freshclam
    timer liveness + ClamAV sig DB date (via `/var/lib/clamav/main.cvd`
    mtime). `ClamAVDaemonActive` always false (daemons masked).

- `panel-api`:
  - `putSettings` calls
    `security.malware.monitor.set_enabled` BEFORE `Upsert` (apply-then-
    persist). If the agent call fails the DB stays old, so the next
    PUT retries instead of silently desyncing host vs DB.

- `panel-ui`:
  - Settings tab "Real-time scanning" Switch with description of the
    RAM cost
  - Overview shows info Alert + "off (opt-in)" tile state when
    `realtime_enabled=false`

- `jabali update`: now invokes `install_malware_stack` after the
  rebuild + migration + restart cycle (see
  `panel-api/cmd/server/update.go`). Existing hosts converge on
  ClamAV-on-demand + opt-in monitor without manual SSH.

### libclamav YARA gap (load-bearing constraint)

**clamscan's libclamav YARA is a strict subset of upstream YARA.** It
does NOT implement the `hash` module — both `import "hash"` and bare
`hash.sha1()` calls fail with `LibClamAV Error: yyerror(): undefined
identifier "hash"`. PMF's whitelist machinery (whitelist.yar +
whitelists/*.yar) is 100 % SHA1-lookup tables of legitimate framework
files; it cannot run under clamscan.

`build-rfxn-yara.sh` therefore ships ONLY php.yar's webshell heuristics
(NonPrintableChars / PasswordProtection / ObfuscatedPhp / DodgyPhp /
DangerousPhp), each of which references `not IsWhitelisted`, and
provides a stub `private rule IsWhitelisted { condition: false }` so
the references resolve.

Trade-off: lose the framework-file false-positive filter. Vanilla
WordPress / Drupal / Magento files that match a webshell heuristic
will flag. Acceptable for shared-hosting webshell hunting; admin
investigates via the Quarantine tab.

`yara` CLI accepts everything libclamav rejects, so
`yara <rules> <file>` parse checks DO NOT prove clamscan will accept
the same rules. Verify via
`clamscan -d <rules> /etc/passwd 2>&1 | grep -i error`.

### Operator action

`jabali update` is sufficient. The update path now sources
`install.sh` and runs `install_malware_stack`, which:
- Installs clamav + clamav-freshclam packages
- Masks both daemon services
- Enables `jabali-freshclam.timer` + first-install freshclam
- Force-disables `jabali-maldet-monitor.service` (admin re-enables via
  Settings tab)
- Rebuilds the PMF YARA block onto rfxn.yara

### Live verification

End-to-end on 192.168.100.150 (2026-04-28):
- Webshell planted in user docroot
- maldet inotify cycle → clamscan with rfxn + PMF rules
- Hit: `YARA.ObfuscatedPhp` quarantined to `/usr/local/maldetect/quarantine/`
- `malware_quarantine` row + `malware_events.notified=1` recorded
- Pipeline depth: docroot write → inotify → clamscan + YARA →
  maldet quarantine → sessionwatcher poll → panel-agent ingest →
  panel-api event → M14 dispatch (bell channel)

## Amendment 2026-04-30 — Remove Tetragon (M39)

### Decision change

Tetragon and the four bundled TracingPolicies (`exec-from-tmp`,
`chmod-x-docroot`, `curl-bash`, `suspicious-syscalls`) are removed.
The L3 forensic audit role is taken over by narrow-scoped auditd:
11 `execve` rules tagged `jabali_susp_exec`, filtered to `auid>=1000`
real users. See ADR-0085.

### Why

- **Built for k8s/cloud-native, not bare-metal LAMP.** Cilium's audience
  is per-pod workload identity; we have per-user-slice on a single
  tenant-per-host model.
- **Default 4 TracingPolicies are noise cannons** on shared hosting.
  WordPress ImageMagick, Composer tempfiles, wp-cli auto-update,
  plugin uploads, Let's Encrypt webroot probes — every legit PHP path
  trips at least one default daily.
- **BTF gate fragmentation.** install_tetragon shipped with
  skip+sentinel for no-BTF kernels; M33 effectively shipped in two
  operating modes neither symmetrically tested.
- **tetragon-relay log shim deferred indefinitely.** Events sat in
  `/var/log/tetragon/tetragon.log`; nobody ingested them.
- **CRD/YAML policy model is wrong shape for hosting.** N users × M
  policies = YAML explosion or matchBinaries explosion.

### What changed

- `install.sh`:
  - `install_tetragon()` deleted (~270 lines including BPF mount,
    4 TracingPolicies, systemd unit, GitHub-release tarball install,
    BTF probe, jabali-tetragon-relay install).
  - `cleanup_tetragon_legacy()` added — idempotent sweep for hosts
    upgrading from M33 (mask + stop + disable units; remove
    `/opt/tetragon`, `/etc/tetragon`, `/var/log/tetragon`,
    `/usr/local/lib/tetragon`, sentinel file, `/sys/fs/bpf/tetragon`
    map dir, binaries under `/usr/local/bin/tetragon*`).
  - `bpftool` apt block dropped (only Tetragon used it).
  - `install_audit_exec()` added (see ADR-0085).

- `panel-agent`:
  - `cmd/jabali-tetragon-relay/` deleted.
  - `mwTetragonSetEnabledHandler`, `mwListTetragonPoliciesHandler`,
    `mwToggleTetragonPolicyHandler` deleted.
  - `mwStatusResponse.{TetragonAvailable,TetragonReason}` removed.
  - BTF/sentinel probe removed from `mwStatusHandler`.
  - `systemctlIsActiveOrActivating` helper removed (only Tetragon
    used it).
  - `ScannerTetragon` const dropped from `MalwareEventIngest.Source`
    enum docs.

- `panel-api`:
  - `internal/repository/tetragon_policy_state_repository.go`
    deleted.
  - `models.TetragonPolicyState` struct + `ScannerTetragon` const
    removed.
  - `MalwareSettings.TetragonEnabled` column removed.
  - Enum `'tetragon'` removed from `malware_quarantine.scanner` and
    `malware_events.source`.
  - `TetragonPolicies` dep removed from
    `SecurityMalwareHandlerConfig` + `app.Deps` + `serve.go`.
  - `/tetragon/policies` GET + POST routes removed.
  - `eventsources/malware.go` Tetragon-always-critical override
    replaced with severity=`critical` short-circuit.

- `panel-ui`:
  - `useTetragonPolicies` + `useToggleTetragonPolicy` hooks deleted.
  - `tetragon_available` + `tetragon_reason` + `tetragon_enabled`
    fields removed.
  - `TetragonCard` component deleted.
  - `tetragon` source filter option + `tetragon` scanner colour
    + Settings switch removed.
  - Sub-tab list rewritten to drop `tetragon`; `exec-audit` added
    (ExecAuditCard backed by `/admin/security/malware/audit/recent`).

- Migration `000106_drop_tetragon_state` — `DROP TABLE tetragon_policy_state` + `ALTER TABLE malware_settings DROP COLUMN tetragon_enabled`, both `IF EXISTS`.

### Operator action

`jabali update` is sufficient — the update path runs
`install_malware_stack`, which calls `cleanup_tetragon_legacy()`
before provisioning the rest. No manual SSH required.

### Status

Accepted (this ADR retains its top-level **Accepted** status; the M33
Tetragon decision documented above is superseded by this amendment +
ADR-0085).
