amadan
plan: the rollout gets clamd's resource limits, and a corrected assertion
Two additions and one correction. **Resource control on the clamd unit**, which matters more than sizing the box. This runs on the console box: if clamd drives that into the OOM killer we do not lose scanning, we lose deploys, routing and config for every app on the deployment. MemoryMax on clamd's own cgroup makes the kernel kill clamd and leave the console standing. **The reload spike.** ClamAV defaults to holding the old and new signature databases in memory at once during a freshclam update, roughly doubling it. A box sized for steady state and not for the reload falls over hours after the last change anyone made, which is the worst kind of incident to diagnose. **The correction: step 5 asserted `unscanned: 0` and `objects == scanned + skipped`.** Both are wrong, and wrong in the direction that matters — scanned and unscanned are not complements, and a skipped object counts as unscanned because it was never judged. An operator following the old text would have read the feature's central honesty property as a bug on the first store with a large object in it. The plan now states the arithmetic and says a non-zero unscanned is the second sentence of the audit answer, not a failure. Also: the rewrite check now names the version binding rather than the ETag alone, and says why both halves are needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YJkJkxZZLgW8X8A5tk69Sq
1 file changed,
+78
−11
docs/superpowers/plans/2026-09-01-carloku-scanning-rollout.md+78 −11
diff --git a/docs/superpowers/plans/2026-09-01-carloku-scanning-rollout.md b/docs/superpowers/plans/2026-09-01-carloku-scanning-rollout.md| index e26f04610..8a2cdd57c 100644 |
| --- a/docs/superpowers/plans/2026-09-01-carloku-scanning-rollout.md |
| +++ b/docs/superpowers/plans/2026-09-01-carloku-scanning-rollout.md |
| @@ -51,8 +51,30 @@ This is the `tito/carlos-infrastructure` half. Per spec §10: |
| signatures reads exactly like a fresh one |
| - `StreamMaxLength` matched to whatever `CARLOS_CLAMD_MAX_BYTES` will be |
| - the console unit's user added to the `clamav` group |
| -- roughly 1–1.5 GB of RAM for resident signatures — check the box has it |
| - before assuming it does |
| +- roughly 1–1.5 GB of RAM for resident signatures, held whether or not |
| + anything is being scanned — check the box has it rather than assuming |
| +- **resource control on the unit**, which matters more than the headroom: |
| + |
| + ```ini |
| + [Service] |
| + MemoryMax=2G # the kernel kills clamd, never the console |
| + MemoryHigh=1600M |
| + CPUQuota=50% |
| + Nice=10 |
| + ``` |
| + |
| + This runs on the CONSOLE box. If clamd drives that into the OOM killer we |
| + do not lose scanning, we lose deploys, routing and config for every app on |
| + the deployment. With `MemoryMax` on clamd's own cgroup the kernel kills |
| + clamd and the console carries on; without it the two compete for the same |
| + memory and the wrong one can lose. |
| +- **the reload spike, which is the number that catches people.** Current |
| + ClamAV defaults to `ConcurrentDatabaseReload yes`, holding the old and new |
| + databases in memory at once so scanning never pauses during a `freshclam` |
| + update — briefly around doubling it. Either size for ~3 GB of headroom, or |
| + set `ConcurrentDatabaseReload no` and size for steady state. The second |
| + trade is nearly free: it blocks scanning during a reload rather than |
| + doubling memory, and a pass that fails a tick resumes on the next one |
| **How to know it worked:** |
| @@ -81,6 +103,17 @@ supposed to require consent for. |
| Set `CARLOS_STORE_SCANNER=clamav`, `CARLOS_CLAMD_SOCKET=/run/clamav/clamd.ctl` |
| and `CARLOS_CLAMD_MAX_BYTES` on the Tito console. Restart. |
| +The other three have defaults and only need setting deliberately: |
| +`CARLOS_CLAMD_MAX_SIG_AGE` (168h — when `scan status` starts warning), |
| +`CARLOS_STORE_SCAN_BUDGET` (60s of scanning per app per hourly tick) and |
| +`CARLOS_STORE_SCAN_INTERVAL` (24h floor before a *completed* sweep re-runs; a |
| +sweep still in progress ignores it, which is how a bucket bigger than one |
| +budget ever finishes). |
| + |
| +Setting `CARLOS_STORE_SCANNER` without `CARLOS_STORE_BACKEND` is refused at |
| +boot, so a typo here fails loudly rather than leaving a scanner with nothing |
| +to scan. |
| + |
| **How to know it worked:** the console log shows the scan pass starting and |
| then finding nothing to do. `carlos store scan status` on any app now answers |
| `engine: clamav`, `requested: no`. **No bucket has been read** — the pass |
| @@ -102,19 +135,46 @@ carlos store scan sweep |
| carlos store scan status |
| ``` |
| -**How to know it worked:** `status` reports a completed sweep with |
| -`objects == scanned + skipped`, `unscanned: 0`, a signature version matching |
| -step 2, and a `last complete` timestamp. Then verify the two properties that |
| -are easy to assume and worth proving: |
| +**How to know it worked:** `status` reports a completed sweep with a |
| +`last complete` timestamp and a signature version matching step 2. |
| + |
| +Read the arithmetic carefully, because it is deliberately not the obvious one. |
| +`scanned` and `unscanned` are **not complements of each other against |
| +`objects`** — they are two independent facts: |
| + |
| +``` |
| +objects = scanned + skipped + denied + failed |
| +unscanned = skipped + denied + failed (NOT objects - scanned) |
| +``` |
| + |
| +An object too large to stream was never judged, so it counts as both `skipped` |
| +and `unscanned`. `unscanned: 0` is therefore only the right expectation on a |
| +store where nothing was skipped, denied or failed — likely on a small |
| +plaintext app, but do not treat a non-zero number as a bug. It is the feature: |
| +the second sentence of the audit answer. |
| + |
| +`status` also splits coverage into verdicts reached during this sweep and |
| +verdicts carried forward from an earlier one. On a first sweep everything is |
| +fresh; on a re-sweep of an unchanged bucket everything is carried forward, |
| +which is correct and is why the engine database version is printed as what it |
| +is *now* rather than as a claim about every object beneath it. |
| + |
| +Then verify the two properties that are easy to assume and worth proving: |
| - **Verdicts are on the objects.** `aws s3api get-object-tagging` on any |
| scanned key shows `carlos-scan`, `carlos-scan-version` and `carlos-scan-at`, |
| and the etag matches the object's current one. |
| -- **A rewrite invalidates the verdict.** Overwrite one object, re-run |
| - `scan status` — that object must count as unscanned until the next sweep |
| - rescans it. If it still reads clean, the ETag binding is not working and |
| +- **A rewrite invalidates the verdict.** Overwrite one object, then run |
| + `carlos store scan sweep` — that object must be re-scanned rather than |
| + reading back its old verdict, which shows up as the fresh count going up. |
| + If it is carried forward instead, the version binding is not working and |
| **stop the rollout**: verdicts that outlive the bytes they judged are worse |
| - than no verdicts. |
| + than no verdicts, because they are trusted. |
| + |
| + The tag records the object's ETag *and* its last-modified time. Both halves |
| + matter: a single-part upload's ETag is the content MD5, and chosen-prefix |
| + MD5 collisions are practical, so an attacker who controls two uploads could |
| + otherwise hand a payload the ETag of the clean object it replaced. |
| ### 6. Prove the destructive paths before any member sees this |
| @@ -165,7 +225,14 @@ ask for it. |
| - **`skipped` climbing.** Over-ceiling objects are not scanned and are not |
| clean. A store that is mostly skips has not been meaningfully scanned and |
| the compliance answer must say so. |
| -- **Console box memory**, now carrying clamd's resident signatures. |
| +- **`denied` appearing at all.** It means the console could not read those |
| + objects — an SSE-KMS key policy that does not cover it, or an object owned |
| + by another account in an adopted bucket. It is re-issued every sweep rather |
| + than cached, so a permissions fix repairs it on the next tick with nothing |
| + to clean up. A `denied` count that stays flat after a fix means the fix did |
| + not work. |
| +- **Console box memory**, now carrying clamd's resident signatures — and the |
| + reload spike, not just the steady state (step 2). |
| ## Rollback |