amadan
spec: scanner host, getting object-storage scanning running on Carloku and Tito
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+250
−0
docs/superpowers/specs/2026-10-04-scanner-host-rollout-design.md+250 −0
diff --git a/docs/superpowers/specs/2026-10-04-scanner-host-rollout-design.md b/docs/superpowers/specs/2026-10-04-scanner-host-rollout-design.md| new file mode 100644 |
| index 000000000..35c38b601 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-04-scanner-host-rollout-design.md |
| @@ -0,0 +1,250 @@ |
| +# Scanner host: getting object-storage scanning running on Carloku and Tito |
| + |
| +**Status:** draft 1, design agreed in conversation 2026-10-04. |
| +**Builds on:** `2026-09-01-object-storage-virus-scanning-design.md` (the sweep, |
| +verdict tags, enforcement) and `2026-09-02-scan-on-demand-design.md` §9 (clamd |
| +off the console box, reachable over TCP). |
| +**Supersedes:** steps 2 and 4 of `../plans/2026-09-01-carloku-scanning-rollout.md` |
| +(clamd on the console box, `CARLOS_STORE_SCANNER` as the switch). Its steps 5–8 |
| +and its watch list still stand. |
| + |
| +--- |
| + |
| +## 1. Where things stand |
| + |
| +The code is shipped and live. Both consoles run builds that carry the sweep, |
| +`carlos store scan request|status|grant|revoke|sweep|enforce` and |
| +`carlos system antivirus status|enable|disable` (Carloku `7edd4b33`, Tito |
| +`fe2088e3`). `CARLOS_CLAMD_ADDR` already takes `host:port` as well as a socket |
| +path (`2eaf2aae`, in every release since v0.18.0). |
| + |
| +What neither deployment has is a clamd to talk to, permission for the console |
| +to read and tag store objects, or a scan tag key. This spec supplies those three |
| +things and nothing else. **No platform Go code changes.** |
| + |
| +## 2. Decisions |
| + |
| +| # | Decision | Why | |
| +|---|---|---| |
| +| D1 | clamd runs on its own box, not the console box | Its ~1.5 GB of resident signatures (about 3 GB during a reload) leave the control plane. Carloku's console box is a `t4g.large` that also serves IE apps. | |
| +| D2 | One box per deployment, in the home region only | Only the console talks to clamd, and every store bucket is in the one region `CARLOS_STORE_REGION` names (`cmd/console/main.go:546-585`). A box elsewhere would add cross-region transfer, not remove it. | |
| +| D3 | Console to clamd is plain TCP inside the VPC, closed by security group | clamd has no TLS and no authentication. Same-VPC, single-source security group is the same trust model as the console's own S3 traffic. TLS (stunnel or a dialler change) or Tailscale are later options, not v1. **(Paul, 2026-10-04: "A is fine for now")** | |
| +| D4 | The console keeps fetching from S3 and streams to clamd | Scan-on-demand §9.4: tenant-data credentials stay on one box. The scanner box gets no S3 or parameter grants. | |
| +| D5 | Both pieces are platform modules | Carloku consumes the platform the way a self-hoster does. Hand-written root resources would be a workaround and would duplicate clamd's setup per root. | |
| +| D6 | Carloku first, then Tito | Tito is a third party, and its console was only unpinned on 2026-10-03 with #201 phase 2 pending. Prove it on our own flagship first. | |
| +| D7 | Scope ends at "engine wired, switched on, no member granted" | Rollout plan steps 7–8: a member grant follows a member's request and the disclosure, never an operator's speculation. | |
| + |
| +**Out of scope:** the unmerged scan-on-demand API (`worktree-store-scan-on-demand`), |
| +TLS to clamd, alerting on signature age, GuardDuty, per-region scanning. If |
| +store buckets ever become per-region, scanning would carry tenant bytes out of |
| +their region; the answer then is a regional scanner that fetches for itself, |
| +which reopens D4. That is a new decision, not more boxes. |
| + |
| +## 3. The `scanner-host` module (`infra/modules/scanner-host`) |
| + |
| +### 3.1 Resources |
| + |
| +- **`aws_instance`**: `ami_id` from the caller (the caller passes |
| + `module.<host>.ami_id`, as Tito's `svc-pdf.tf` does), `instance_type` default |
| + `t4g.medium`, 20 GiB encrypted gp3 root, IMDSv2 required |
| + (`http_tokens = "required"`), `user_data_replace_on_change = false`, |
| + `lifecycle { ignore_changes = [ami, user_data] }`, so a newer image or an |
| + edited script never silently replaces the box; changing either (including |
| + `stream_max_bytes`) is an explicit `-replace`. Optional `subnet_id`; null |
| + means the default subnet, as `region-host` does, and a precondition then |
| + requires the client security group's VPC to be the default VPC. |
| +- **`aws_security_group`**: ingress TCP 3310 from `client_security_group_id` |
| + **only**. No other ingress, no SSH. Egress open, for `freshclam` and apt. It |
| + lands in the client security group's VPC (`vpc_id` from a data source on that |
| + group), so the two can never end up in different VPCs. |
| +- **IAM role and instance profile**: `AmazonSSMManagedInstanceCore` attached, |
| + nothing else. No bucket, no parameter, no KMS. The box cannot fetch tenant |
| + data even if clamd were compromised. |
| + |
| +### 3.2 `user_data` |
| + |
| +A templated script (`scanner-init.sh.tftpl`): |
| + |
| +1. Install `clamav-daemon` and `clamav-freshclam` from Debian. |
| +2. Run `freshclam` once in the foreground and **refuse to continue** until |
| + `main.cvd` (or `.cld`), `daily` and `bytecode` exist. clamd never starts on an |
| + empty database. |
| +3. Set these keys in `/etc/clamav/clamd.conf` (clamd has no include |
| + directory, so the script rewrites each key idempotently, replacing any |
| + existing line): |
| + - `TCPSocket 3310` |
| + - `TCPAddr <private IPv4>` from IMDSv2. **Never `0.0.0.0`.** The security |
| + group is the fence; binding the private address is the second rail. |
| + - `StreamMaxLength <stream_max_bytes>`, plus `MaxScanSize` and `MaxFileSize` |
| + no lower than it, so clamd never silently truncates a stream the console |
| + believes it sent whole. |
| + - `ConcurrentDatabaseReload` from a variable, default `yes`. |
| + - The unix socket `LocalSocket /run/clamav/clamd.ctl` stays, for local checks |
| + over SSM. |
| +4. A systemd drop-in for `clamav-daemon.service`: `MemoryMax` from a variable |
| + (default `3G`), `MemoryHigh` at about 85% of it, `Restart=on-failure`. |
| +5. `freshclam` keeps its Debian service (checks hourly by default). |
| +6. Debian ships `clamav-daemon.socket` for systemd socket activation, which |
| + passes clamd only the unix socket. Mask the socket unit so clamd binds both |
| + listeners from its own config; the check below fails if TCP is not up. |
| +7. Enable and start `clamav-daemon`; finish by checking `zVERSION` answers on |
| + both the socket and `<private ip>:3310`, and print the result to the console |
| + log so `aws ec2 get-console-output` shows it. |
| + |
| +### 3.3 Variables and outputs |
| + |
| +| Variable | Default | Notes | |
| +|---|---|---| |
| +| `name` | — | Prefixes every resource | |
| +| `ami_id` | — | Debian arm64, from `region-host` | |
| +| `client_security_group_id` | — | The console box's group | |
| +| `instance_type` | `t4g.medium` | `t4g.small` only with `concurrent_database_reload = false` | |
| +| `stream_max_bytes` | `26214400` | Must equal the console's `CARLOS_CLAMD_MAX_BYTES` | |
| +| `memory_max` | `"3G"` | clamd's cgroup ceiling | |
| +| `concurrent_database_reload` | `true` | | |
| +| `subnet_id` | `null` | | |
| + |
| +Validation: `instance_type` beginning `t4g.small`/`t4g.micro`/`t4g.nano` with |
| +`concurrent_database_reload = true` is refused with a message naming the reload |
| +spike. |
| + |
| +Outputs: `clamd_addr` (`"<private_ip>:3310"`), `instance_id`, |
| +`security_group_id`, `stream_max_bytes`. |
| + |
| +### 3.4 Replacement |
| + |
| +A replaced box gets a new private IP, so `CARLOS_CLAMD_ADDR` must be updated by |
| +hand. Accepted: replacements are explicit (`-replace`) and the failure is loud |
| +(sweeps fail per tick, `scan status` and `antivirus status` say so, the console |
| +stays up). The README says this in its first section. |
| + |
| +## 4. The console's read-and-tag grant |
| + |
| +### 4.1 `region-host` (Carloku) |
| + |
| +- New variable `store_scanning` (bool, default `false`). A precondition refuses |
| + `true` with an empty `store_bucket_prefix`. |
| +- When `true`, a **separate** `aws_iam_role_policy`, `${iam_name}-store-scan`: |
| + `s3:GetObject`, `s3:GetObjectTagging`, `s3:PutObjectTagging` on |
| + `arn:aws:s3:::${store_bucket_prefix}*/*`. These are exactly the calls the |
| + scanner makes (`HeadObject` is authorised by `GetObject`; `ListObjectsV2` is |
| + already granted by `host_store`). Store buckets use SSE-S3, so no KMS grant. |
| +- Separate, so an opted-out deployment plans zero changes, and so the existing |
| + `host_store` comment ("No GetObject: the console provisions stores, it does not |
| + read tenant data") stays true. That comment gains one line pointing at |
| + `store_scanning`. |
| +- The store-user boundary is untouched. Apps still cannot tag their own objects. |
| + |
| +### 4.2 Tito (`object-store.tf`) |
| + |
| +A new statement `StoreScan` in `host_ie_object_store` with the same three |
| +actions on `arn:aws:s3:::carlos-store-tito-*/*` and |
| +`arn:aws:s3:::titogo-uploads-665106592142/*`. Tito keeps its own store policy |
| +(#201 settled this), so `store_scanning` is not used there. |
| + |
| +## 5. Release and consumption |
| + |
| +- The module change and `scanner-host` land on amadan main through a branch and |
| + `amadan branch merge -squash`, then the GitHub mirror is pushed forward |
| + (fast-forward checked first), then **v0.23.0** is cut per the README's |
| + "Releasing the platform" recipe and pushed to the mirror. |
| +- **Carloku** (`carloku-infrastructure`): move **only `host_ie`'s** `?ref=` to |
| + v0.23.0, set `store_scanning = true`, add `module "scanner"` with |
| + `client_security_group_id = module.host_ie.security_group_id`. The other three |
| + regions stay on v0.21.0. Every apply is `-target`ed |
| + (`module.host_ie.aws_iam_role_policy.host_store_scan`, `module.scanner`), and |
| + the plan is read for any `user_data` line before it runs, because a |
| + `user_data` change stops and starts a box. Any other `host_ie` diff the |
| + v0.21.0 to v0.23.0 bump shows is listed in the PR and not applied. |
| +- **Tito** (`teamtito/carlos-infrastructure`): a branch from the |
| + `store-boundary-201` line (main cannot plan while the Canada box is only on |
| + `region-ca`), `module "scanner"` at `?ref=v0.23.0`, the `StoreScan` |
| + statement. Targeted applies. Tito pins `region-host` to a SHA; that pin does |
| + not move. |
| + |
| +## 6. Wiring the console |
| + |
| +### 6.1 The scan tag key |
| + |
| +- 32 random bytes, hex. Generated once per deployment, never in git, tofu state |
| + or a transcript: generated and written in one pipeline. |
| +- Durable copy: SSM SecureString `/carlos/scan-tag-key` in the deployment's |
| + account (inside the `/carlos/*` path the box can already read). Losing it makes |
| + every stored verdict unverifiable. |
| +- `CARLOS_SCAN_TAG_KEY_PREVIOUS` stays unset; rotation exists and is not needed. |
| + |
| +### 6.2 The console's environment |
| + |
| +First confirm over SSM, read-only, which file `console.service` loads. Then, |
| +after a dated backup of that file: |
| + |
| +``` |
| +CARLOS_CLAMD_ADDR=<module.scanner.clamd_addr> |
| +CARLOS_CLAMD_MAX_BYTES=<module.scanner.stream_max_bytes> |
| +CARLOS_SCAN_TAG_KEY=<from /carlos/scan-tag-key> |
| +``` |
| + |
| +`systemctl restart console`, then `curl -sI` the console for `x-carlos-version`. |
| +If it fails to start: restore the backup, `systemctl reset-failed console && |
| +systemctl restart console` (a start-limited unit ignores try-restart; seen |
| +2026-09-11). |
| + |
| +`CARLOS_STORE_SCANNER` stays unset. It is deprecated, and the switch is the |
| +record (`system/antivirus.json`), so the restart wires the engine without |
| +turning it on. |
| + |
| +### 6.3 Switching on |
| + |
| +`carlos system antivirus status`: engine wired, clamd answering, signature date |
| +today or yesterday. Then `carlos system antivirus enable`. With no app granted |
| +the hourly pass runs and correctly does nothing. Sit in that state until §7 |
| +passes. |
| + |
| +## 7. Proving it on Carloku |
| + |
| +On a throwaway app we own, with a plaintext store: |
| + |
| +1. Upload one clean object and the EICAR test file. `scan request --plaintext`, |
| + `grant`, `sweep`. **Pass:** `scan status` shows a completed sweep, one clean, |
| + one infected; both objects' tags verify under this deployment's key and carry |
| + the object's current ETag. |
| +2. Overwrite the clean object, `sweep`. **Pass:** it is re-scanned, not carried |
| + forward. **If it is carried forward, stop the rollout:** a verdict that |
| + outlives its bytes is worse than none. |
| +3. `enforce on`. Remove `CARLOS_CLAMD_ADDR`, restart. **Pass:** `enforce off` |
| + still works with no engine built. Restore the address, restart. |
| +4. Destroy the app. **Pass:** no `scan.json` remains under its prefix and the |
| + name is claimable at once. |
| + |
| +Then stop. Scanning stays enabled; no member store is granted. |
| + |
| +## 8. Tito |
| + |
| +After §7 passes and after #201 phase 2 has landed (so two changes do not edit |
| +Tito's console environment on the same day): §5's Tito apply, §6 in Tito's |
| +account, `antivirus status`, `enable`. **Tito's proof stops at status and |
| +enable.** §7 exercises platform code identical to Carloku's, and a throwaway app |
| +in a customer's deployment needs the customer's say-so. |
| + |
| +## 9. Docs |
| + |
| +- `infra/modules/scanner-host/README.md`: what it is, the three outputs the |
| + console needs, replacement (§3.4), and the sizing rule. |
| +- `infra/modules/region-host/README.md`: `store_scanning`. |
| +- A dated note at the top of `../plans/2026-09-01-carloku-scanning-rollout.md`: |
| + steps 2 and 4 are superseded by this spec, `CARLOS_STORE_SCANNER` is |
| + deprecated. |
| +- An amadan discussion on `carlos/platform` tracking the rollout. |
| + |
| +## 10. Cost |
| + |
| +One `t4g.medium` plus 20 GiB gp3 per deployment: about $25 a month each, $50 |
| +for the pair. |
| + |
| +## 11. What would change this design |
| + |
| +- `t4g.medium` turning out too small for the reload spike in practice: raise |
| + the instance type, or set `concurrent_database_reload = false`. |
| +- A member asking for scanning on a client-encrypted store: no change here, the |
| + existing `--client-encrypted` path handles it. |
| +- Store buckets becoming per-region: see §2, out of scope. |