| 1 | > **2026-10-05:** steps 2 and 4 are superseded by docs/superpowers/specs/2026-10-04-scanner-host-rollout-design.md. clamd runs on its own box, the switch is carlos system antivirus enable, and CARLOS_STORE_SCANNER is deprecated. Steps 5 to 8 and the watch list still stand. |
| 2 | |
| 3 | # Rolling object-storage scanning out to Carloku |
| 4 | |
| 5 | **Not TDD tasks — an ordered operational sequence.** Do not start it until the |
| 6 | implementation plan (`2026-09-01-object-storage-virus-scanning.md`) is merged |
| 7 | and on `main`. Each step states how to know it worked. |
| 8 | |
| 9 | **Spec:** `docs/superpowers/specs/2026-09-01-object-storage-virus-scanning-design.md` |
| 10 | |
| 11 | **Tito first throughout**, for the reason the console-as-an-app rollout gives: |
| 12 | smaller blast radius, and Tito's console going down does not take Carloku with |
| 13 | it. |
| 14 | |
| 15 | ## What is actually being rolled out |
| 16 | |
| 17 | An opt-in scanner that is **inert until three separate things are true**: |
| 18 | `CARLOS_STORE_SCANNER` is set on the deployment, a member has requested |
| 19 | scanning for their app, and an operator has granted it. Shipping the binary |
| 20 | changes nothing on its own, which is what makes this rollout low-risk and why |
| 21 | step 1 is safe to do first and sit on. |
| 22 | |
| 23 | The failure mode worth naming up front: the ClamAV engine **reads tenant |
| 24 | object bytes**. Nothing reads anything until an operator grants a specific |
| 25 | app, but the capability arrives with the binary, and the sequence below puts |
| 26 | the disclosure before the first grant rather than after it. |
| 27 | |
| 28 | --- |
| 29 | |
| 30 | ### 1. Ship the binary with the scanner unset |
| 31 | |
| 32 | Deploy `main` to Tito, then Carloku, with `CARLOS_STORE_SCANNER` **unset**. |
| 33 | |
| 34 | **How to know it worked:** `carlos store scan status` on any app answers |
| 35 | `engine: none` and the operator doors 503 naming `CARLOS_STORE_SCANNER`. No |
| 36 | scan pass appears in the console log. `carlos store status` is unchanged for |
| 37 | every existing app, and no `scan.json` exists anywhere: |
| 38 | |
| 39 | ``` |
| 40 | aws s3 ls s3://<deployment-bucket>/apps/ --recursive | grep scan.json |
| 41 | ``` |
| 42 | |
| 43 | should print nothing. If it prints anything, stop — a sweep ran when nothing |
| 44 | should have. |
| 45 | |
| 46 | ### 2. Stand up clamd on the Tito console box |
| 47 | |
| 48 | This is the `tito/carlos-infrastructure` half. Per spec §10: |
| 49 | |
| 50 | - a `clamav-daemon` systemd unit, socket at `/run/clamav/clamd.ctl` |
| 51 | - `freshclam` on a timer — **a stale signature database is the silent failure |
| 52 | mode of this whole feature**; a clean verdict from three-month-old |
| 53 | signatures reads exactly like a fresh one |
| 54 | - `StreamMaxLength` matched to whatever `CARLOS_CLAMD_MAX_BYTES` will be |
| 55 | - the console unit's user added to the `clamav` group |
| 56 | - roughly 1–1.5 GB of RAM for resident signatures, held whether or not |
| 57 | anything is being scanned — check the box has it rather than assuming |
| 58 | - **resource control on the unit**, which matters more than the headroom: |
| 59 | |
| 60 | ```ini |
| 61 | [Service] |
| 62 | MemoryMax=2G # the kernel kills clamd, never the console |
| 63 | MemoryHigh=1600M |
| 64 | CPUQuota=50% |
| 65 | Nice=10 |
| 66 | ``` |
| 67 | |
| 68 | This runs on the CONSOLE box. If clamd drives that into the OOM killer we |
| 69 | do not lose scanning, we lose deploys, routing and config for every app on |
| 70 | the deployment. With `MemoryMax` on clamd's own cgroup the kernel kills |
| 71 | clamd and the console carries on; without it the two compete for the same |
| 72 | memory and the wrong one can lose. |
| 73 | - **the reload spike, which is the number that catches people.** Current |
| 74 | ClamAV defaults to `ConcurrentDatabaseReload yes`, holding the old and new |
| 75 | databases in memory at once so scanning never pauses during a `freshclam` |
| 76 | update — briefly around doubling it. Either size for ~3 GB of headroom, or |
| 77 | set `ConcurrentDatabaseReload no` and size for steady state. The second |
| 78 | trade is nearly free: it blocks scanning during a reload rather than |
| 79 | doubling memory, and a pass that fails a tick resumes on the next one |
| 80 | |
| 81 | **How to know it worked:** |
| 82 | |
| 83 | ``` |
| 84 | systemctl is-active clamav-daemon |
| 85 | sudo -u <console-user> test -w /run/clamav/clamd.ctl && echo reachable |
| 86 | echo -n 'zVERSION' | sudo -u <console-user> socat - UNIX-CONNECT:/run/clamav/clamd.ctl |
| 87 | ``` |
| 88 | |
| 89 | The last prints `ClamAV <engine>/<sig-version>/<sig-date>`. Check the date is |
| 90 | today or yesterday. If `freshclam` has never run, it will not be. |
| 91 | |
| 92 | ### 3. Grant the console its S3 read and tag permissions |
| 93 | |
| 94 | Add `s3:GetObject`, `s3:GetObjectTagging` and `s3:PutObjectTagging` on |
| 95 | `arn:aws:s3:::<BucketPrefix>*` to the Tito console's role. It already lists |
| 96 | those buckets for the economics storage pass; this adds read and tag. |
| 97 | |
| 98 | **How to know it worked:** step 5's first sweep completes. There is no useful |
| 99 | intermediate check that does not involve reading a tenant object, and doing |
| 100 | that by hand before a member has opted in is exactly what this feature is |
| 101 | supposed to require consent for. |
| 102 | |
| 103 | ### 4. Turn the engine on, still with nothing opted in |
| 104 | |
| 105 | Set `CARLOS_STORE_SCANNER=clamav`, `CARLOS_CLAMD_SOCKET=/run/clamav/clamd.ctl` |
| 106 | and `CARLOS_CLAMD_MAX_BYTES` on the Tito console. Restart. |
| 107 | |
| 108 | The other three have defaults and only need setting deliberately: |
| 109 | `CARLOS_CLAMD_MAX_SIG_AGE` (168h — when `scan status` starts warning), |
| 110 | `CARLOS_STORE_SCAN_BUDGET` (60s of scanning per app per hourly tick) and |
| 111 | `CARLOS_STORE_SCAN_INTERVAL` (24h floor before a *completed* sweep re-runs; a |
| 112 | sweep still in progress ignores it, which is how a bucket bigger than one |
| 113 | budget ever finishes). |
| 114 | |
| 115 | Setting `CARLOS_STORE_SCANNER` without `CARLOS_STORE_BACKEND` is refused at |
| 116 | boot, so a typo here fails loudly rather than leaving a scanner with nothing |
| 117 | to scan. |
| 118 | |
| 119 | **How to know it worked:** the console log shows the scan pass starting and |
| 120 | then finding nothing to do. `carlos store scan status` on any app now answers |
| 121 | `engine: clamav`, `requested: no`. **No bucket has been read** — the pass |
| 122 | skips every app without an `enabled` block, and there are none. |
| 123 | |
| 124 | This is the state to sit in for a day. A scan pass that wakes hourly and |
| 125 | correctly does nothing is the cheapest possible proof that the enumeration and |
| 126 | the gating work. |
| 127 | |
| 128 | ### 5. First real store: one Tito app you own |
| 129 | |
| 130 | Pick an app whose objects are **plaintext** and whose data you own — not a |
| 131 | member's. Then: |
| 132 | |
| 133 | ``` |
| 134 | carlos store scan request --plaintext |
| 135 | carlos store scan grant |
| 136 | carlos store scan sweep |
| 137 | carlos store scan status |
| 138 | ``` |
| 139 | |
| 140 | **How to know it worked:** `status` reports a completed sweep with a |
| 141 | `last complete` timestamp and a signature version matching step 2. |
| 142 | |
| 143 | Read the arithmetic carefully, because it is deliberately not the obvious one. |
| 144 | `scanned` and `unscanned` are **not complements of each other against |
| 145 | `objects`** — they are two independent facts: |
| 146 | |
| 147 | ``` |
| 148 | objects = scanned + skipped + denied + failed |
| 149 | unscanned = skipped + denied + failed (NOT objects - scanned) |
| 150 | ``` |
| 151 | |
| 152 | An object too large to stream was never judged, so it counts as both `skipped` |
| 153 | and `unscanned`. `unscanned: 0` is therefore only the right expectation on a |
| 154 | store where nothing was skipped, denied or failed — likely on a small |
| 155 | plaintext app, but do not treat a non-zero number as a bug. It is the feature: |
| 156 | the second sentence of the audit answer. |
| 157 | |
| 158 | `status` also splits coverage into verdicts reached during this sweep and |
| 159 | verdicts carried forward from an earlier one. On a first sweep everything is |
| 160 | fresh; on a re-sweep of an unchanged bucket everything is carried forward, |
| 161 | which is correct and is why the engine database version is printed as what it |
| 162 | is *now* rather than as a claim about every object beneath it. |
| 163 | |
| 164 | Then verify the two properties that are easy to assume and worth proving: |
| 165 | |
| 166 | - **Verdicts are on the objects.** `aws s3api get-object-tagging` on any |
| 167 | scanned key shows `carlos-scan`, `carlos-scan-version` and `carlos-scan-at`, |
| 168 | and the etag matches the object's current one. |
| 169 | - **A rewrite invalidates the verdict.** Overwrite one object, then run |
| 170 | `carlos store scan sweep` — that object must be re-scanned rather than |
| 171 | reading back its old verdict, which shows up as the fresh count going up. |
| 172 | If it is carried forward instead, the version binding is not working and |
| 173 | **stop the rollout**: verdicts that outlive the bytes they judged are worse |
| 174 | than no verdicts, because they are trusted. |
| 175 | |
| 176 | The tag records the object's ETag *and* its last-modified time. Both halves |
| 177 | matter: a single-part upload's ETag is the content MD5, and chosen-prefix |
| 178 | MD5 collisions are practical, so an attacker who controls two uploads could |
| 179 | otherwise hand a payload the ETag of the clean object it replaced. |
| 180 | |
| 181 | ### 6. Prove the destructive paths before any member sees this |
| 182 | |
| 183 | On a throwaway app in Tito: |
| 184 | |
| 185 | - `carlos store scan grant`, then destroy the app. Confirm no `scan.json` |
| 186 | remains under its prefix and **the app name is immediately claimable |
| 187 | again** — an orphan sidecar makes a name permanently unusable and is the |
| 188 | failure this feature's teardown path exists to prevent. |
| 189 | - `carlos store scan enforce on`, then unset `CARLOS_STORE_SCANNER` and |
| 190 | restart. Confirm `carlos store scan enforce off` still works with no engine |
| 191 | built. This is the deadlock draft 3 of the spec would have shipped; prove it |
| 192 | is gone on a real deployment, not only in a unit test. |
| 193 | |
| 194 | ### 7. Disclose before the first member grant |
| 195 | |
| 196 | Before granting scanning on any store CARLOS does not own, the member has to |
| 197 | know what they are consenting to: |
| 198 | |
| 199 | - the deployment will **read their object bytes** to scan them |
| 200 | - a clean verdict on a **client-side-encrypted** store means nothing, and they |
| 201 | should request `--client-encrypted` if that is their shape |
| 202 | - verdicts are **eventually consistent** — an object is readable before it is |
| 203 | scanned, and there is no upload hook |
| 204 | |
| 205 | The `scan request` verb prints this, which is the point of making the |
| 206 | attestation an explicit flag. But a member who has not run the verb has not |
| 207 | read it, so an operator granting on someone's behalf must have had the |
| 208 | conversation first. |
| 209 | |
| 210 | ### 8. Carloku, same sequence |
| 211 | |
| 212 | Repeat steps 2 through 6 on Carloku. Then stop and wait: on Carloku the first |
| 213 | grant should be **driven by an actual member request**, not by an operator |
| 214 | enabling it speculatively. This feature exists to answer a compliance |
| 215 | question, and the person with the compliance question is the one who should |
| 216 | ask for it. |
| 217 | |
| 218 | ## Watch after each grant |
| 219 | |
| 220 | - **Signature age.** `scan status` warns past `CARLOS_CLAMD_MAX_SIG_AGE`. |
| 221 | Treat the warning as an incident, not a notice — every clean verdict written |
| 222 | while it was stale is worth less than it looks. |
| 223 | - **Sweep duration against the budget.** If a sweep spans many ticks, the |
| 224 | audit answer's T is as old as the sweep is long. Raise |
| 225 | `CARLOS_STORE_SCAN_BUDGET` or shorten the interval, and say the real number |
| 226 | when anyone quotes coverage. |
| 227 | - **`skipped` climbing.** Over-ceiling objects are not scanned and are not |
| 228 | clean. A store that is mostly skips has not been meaningfully scanned and |
| 229 | the compliance answer must say so. |
| 230 | - **`denied` appearing at all.** It means the console could not read those |
| 231 | objects — an SSE-KMS key policy that does not cover it, or an object owned |
| 232 | by another account in an adopted bucket. It is re-issued every sweep rather |
| 233 | than cached, so a permissions fix repairs it on the next tick with nothing |
| 234 | to clean up. A `denied` count that stays flat after a fix means the fix did |
| 235 | not work. |
| 236 | - **Console box memory**, now carrying clamd's resident signatures — and the |
| 237 | reload spike, not just the steady state (step 2). |
| 238 | |
| 239 | ## Rollback |
| 240 | |
| 241 | Unset `CARLOS_STORE_SCANNER` and restart. The pass stops, the doors 503, and |
| 242 | every existing verdict tag stays on its object — harmless, and still readable |
| 243 | if the engine comes back. Nothing needs undoing in a bucket. |
| 244 | |
| 245 | The one thing that does **not** roll back by unsetting the variable is an |
| 246 | active `enforce` deny, which lives in the app's IAM policy. Clear it first |
| 247 | with `carlos store scan enforce off` (which works with no engine built, by |
| 248 | design) before unsetting, or an app keeps a Deny statement referring to a tag |
| 249 | nothing will ever write again. Harmless in practice — the tag will not appear, |
| 250 | so nothing is denied — but it is a stale statement in a live policy and it |
| 251 | should not be left behind. |
| 252 | |
amadan