amadan
plan: the Carloku rollout for object-storage scanning
An ordered operational sequence, Tito first. The shape follows from the feature being inert until three things are separately true — binary shipped, engine configured, app granted — so the binary can land and sit harmlessly while the infra half catches up. Two steps exist because of specific bugs earlier drafts would have shipped, and both are proved on a real deployment rather than trusted to a unit test: that a rewritten object loses its verdict rather than inheriting one for bytes nobody read, and that enforce off still works with no engine built. Disclosure comes before the first member grant, not after. The engine reads tenant object bytes, and a member whose objects are encrypted should know a clean result means nothing before someone hands them one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 file changed,
+182
−0
docs/superpowers/plans/2026-09-01-carloku-scanning-rollout.md+182 −0
diff --git a/docs/superpowers/plans/2026-09-01-carloku-scanning-rollout.md b/docs/superpowers/plans/2026-09-01-carloku-scanning-rollout.md| new file mode 100644 |
| index 000000000..49673fdc2 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-09-01-carloku-scanning-rollout.md |
| @@ -0,0 +1,182 @@ |
| +# Rolling object-storage scanning out to Carloku |
| + |
| +**Not TDD tasks — an ordered operational sequence.** Do not start it until the |
| +implementation plan (`2026-09-01-object-storage-virus-scanning.md`) is merged |
| +and on `main`. Each step states how to know it worked. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-09-01-object-storage-virus-scanning-design.md` |
| + |
| +**Tito first throughout**, for the reason the console-as-an-app rollout gives: |
| +smaller blast radius, and Tito's console going down does not take Carloku with |
| +it. |
| + |
| +## What is actually being rolled out |
| + |
| +An opt-in scanner that is **inert until three separate things are true**: |
| +`CARLOS_STORE_SCANNER` is set on the deployment, a member has requested |
| +scanning for their app, and an operator has granted it. Shipping the binary |
| +changes nothing on its own, which is what makes this rollout low-risk and why |
| +step 1 is safe to do first and sit on. |
| + |
| +The failure mode worth naming up front: the ClamAV engine **reads tenant |
| +object bytes**. Nothing reads anything until an operator grants a specific |
| +app, but the capability arrives with the binary, and the sequence below puts |
| +the disclosure before the first grant rather than after it. |
| + |
| +--- |
| + |
| +### 1. Ship the binary with the scanner unset |
| + |
| +Deploy `main` to Tito, then Carloku, with `CARLOS_STORE_SCANNER` **unset**. |
| + |
| +**How to know it worked:** `carlos store scan status` on any app answers |
| +`engine: none` and the operator doors 503 naming `CARLOS_STORE_SCANNER`. No |
| +scan pass appears in the console log. `carlos store status` is unchanged for |
| +every existing app, and no `scan.json` exists anywhere: |
| + |
| +``` |
| +aws s3 ls s3://<deployment-bucket>/apps/ --recursive | grep scan.json |
| +``` |
| + |
| +should print nothing. If it prints anything, stop — a sweep ran when nothing |
| +should have. |
| + |
| +### 2. Stand up clamd on the Tito console box |
| + |
| +This is the `tito/carlos-infrastructure` half. Per spec §10: |
| + |
| +- a `clamav-daemon` systemd unit, socket at `/run/clamav/clamd.ctl` |
| +- `freshclam` on a timer — **a stale signature database is the silent failure |
| + mode of this whole feature**; a clean verdict from three-month-old |
| + signatures reads exactly like a fresh one |
| +- `StreamMaxLength` matched to whatever `CARLOS_CLAMD_MAX_BYTES` will be |
| +- the console unit's user added to the `clamav` group |
| +- roughly 1–1.5 GB of RAM for resident signatures — check the box has it |
| + before assuming it does |
| + |
| +**How to know it worked:** |
| + |
| +``` |
| +systemctl is-active clamav-daemon |
| +sudo -u <console-user> test -w /run/clamav/clamd.ctl && echo reachable |
| +echo -n 'zVERSION' | sudo -u <console-user> socat - UNIX-CONNECT:/run/clamav/clamd.ctl |
| +``` |
| + |
| +The last prints `ClamAV <engine>/<sig-version>/<sig-date>`. Check the date is |
| +today or yesterday. If `freshclam` has never run, it will not be. |
| + |
| +### 3. Grant the console its S3 read and tag permissions |
| + |
| +Add `s3:GetObject`, `s3:GetObjectTagging` and `s3:PutObjectTagging` on |
| +`arn:aws:s3:::<BucketPrefix>*` to the Tito console's role. It already lists |
| +those buckets for the economics storage pass; this adds read and tag. |
| + |
| +**How to know it worked:** step 5's first sweep completes. There is no useful |
| +intermediate check that does not involve reading a tenant object, and doing |
| +that by hand before a member has opted in is exactly what this feature is |
| +supposed to require consent for. |
| + |
| +### 4. Turn the engine on, still with nothing opted in |
| + |
| +Set `CARLOS_STORE_SCANNER=clamav`, `CARLOS_CLAMD_SOCKET=/run/clamav/clamd.ctl` |
| +and `CARLOS_CLAMD_MAX_BYTES` on the Tito console. Restart. |
| + |
| +**How to know it worked:** the console log shows the scan pass starting and |
| +then finding nothing to do. `carlos store scan status` on any app now answers |
| +`engine: clamav`, `requested: no`. **No bucket has been read** — the pass |
| +skips every app without an `enabled` block, and there are none. |
| + |
| +This is the state to sit in for a day. A scan pass that wakes hourly and |
| +correctly does nothing is the cheapest possible proof that the enumeration and |
| +the gating work. |
| + |
| +### 5. First real store: one Tito app you own |
| + |
| +Pick an app whose objects are **plaintext** and whose data you own — not a |
| +member's. Then: |
| + |
| +``` |
| +carlos store scan request --plaintext |
| +carlos store scan grant |
| +carlos store scan sweep |
| +carlos store scan status |
| +``` |
| + |
| +**How to know it worked:** `status` reports a completed sweep with |
| +`objects == scanned + skipped`, `unscanned: 0`, a signature version matching |
| +step 2, and a `last complete` timestamp. Then verify the two properties that |
| +are easy to assume and worth proving: |
| + |
| +- **Verdicts are on the objects.** `aws s3api get-object-tagging` on any |
| + scanned key shows `carlos-scan`, `carlos-scan-etag` and `carlos-scan-at`, |
| + and the etag matches the object's current one. |
| +- **A rewrite invalidates the verdict.** Overwrite one object, re-run |
| + `scan status` — that object must count as unscanned until the next sweep |
| + rescans it. If it still reads clean, the ETag binding is not working and |
| + **stop the rollout**: verdicts that outlive the bytes they judged are worse |
| + than no verdicts. |
| + |
| +### 6. Prove the destructive paths before any member sees this |
| + |
| +On a throwaway app in Tito: |
| + |
| +- `carlos store scan grant`, then destroy the app. Confirm no `scan.json` |
| + remains under its prefix and **the app name is immediately claimable |
| + again** — an orphan sidecar makes a name permanently unusable and is the |
| + failure this feature's teardown path exists to prevent. |
| +- `carlos store scan enforce on`, then unset `CARLOS_STORE_SCANNER` and |
| + restart. Confirm `carlos store scan enforce off` still works with no engine |
| + built. This is the deadlock draft 3 of the spec would have shipped; prove it |
| + is gone on a real deployment, not only in a unit test. |
| + |
| +### 7. Disclose before the first member grant |
| + |
| +Before granting scanning on any store CARLOS does not own, the member has to |
| +know what they are consenting to: |
| + |
| +- the deployment will **read their object bytes** to scan them |
| +- a clean verdict on a **client-side-encrypted** store means nothing, and they |
| + should request `--client-encrypted` if that is their shape |
| +- verdicts are **eventually consistent** — an object is readable before it is |
| + scanned, and there is no upload hook |
| + |
| +The `scan request` verb prints this, which is the point of making the |
| +attestation an explicit flag. But a member who has not run the verb has not |
| +read it, so an operator granting on someone's behalf must have had the |
| +conversation first. |
| + |
| +### 8. Carloku, same sequence |
| + |
| +Repeat steps 2 through 6 on Carloku. Then stop and wait: on Carloku the first |
| +grant should be **driven by an actual member request**, not by an operator |
| +enabling it speculatively. This feature exists to answer a compliance |
| +question, and the person with the compliance question is the one who should |
| +ask for it. |
| + |
| +## Watch after each grant |
| + |
| +- **Signature age.** `scan status` warns past `CARLOS_CLAMD_MAX_SIG_AGE`. |
| + Treat the warning as an incident, not a notice — every clean verdict written |
| + while it was stale is worth less than it looks. |
| +- **Sweep duration against the budget.** If a sweep spans many ticks, the |
| + audit answer's T is as old as the sweep is long. Raise |
| + `CARLOS_STORE_SCAN_BUDGET` or shorten the interval, and say the real number |
| + when anyone quotes coverage. |
| +- **`skipped` climbing.** Over-ceiling objects are not scanned and are not |
| + clean. A store that is mostly skips has not been meaningfully scanned and |
| + the compliance answer must say so. |
| +- **Console box memory**, now carrying clamd's resident signatures. |
| + |
| +## Rollback |
| + |
| +Unset `CARLOS_STORE_SCANNER` and restart. The pass stops, the doors 503, and |
| +every existing verdict tag stays on its object — harmless, and still readable |
| +if the engine comes back. Nothing needs undoing in a bucket. |
| + |
| +The one thing that does **not** roll back by unsetting the variable is an |
| +active `enforce` deny, which lives in the app's IAM policy. Clear it first |
| +with `carlos store scan enforce off` (which works with no engine built, by |
| +design) before unsetting, or an app keeps a Deny statement referring to a tag |
| +nothing will ever write again. Harmless in practice — the tag will not appear, |
| +so nothing is denied — but it is a stale statement in a live policy and it |
| +should not be left behind. |