| 1 | # Scanner host: getting object-storage scanning running on Carloku and Tito |
| 2 | |
| 3 | **Status:** draft 6. Closes Astra review 1 (`CODEX-FINDINGS-SCANNER-HOST.md`, |
| 4 | findings 1–5, all accepted), review 2 (`CODEX-FINDINGS-SCANNER-HOST-R2.md`, |
| 5 | findings 6–8: 7 and 8 accepted, 6 declined with the reason in §4.3) and |
| 6 | review 3 (`CODEX-FINDINGS-SCANNER-HOST-R3.md`, finding 9, accepted and tested |
| 7 | against a real engine, §3.4) and review 4 (`CODEX-FINDINGS-SCANNER-HOST-R4.md`, |
| 8 | findings 10 and 11, both accepted; 10 reshaped how the box listens, §3.2) and |
| 9 | review 5 (`CODEX-FINDINGS-SCANNER-HOST-R5.md`, finding 12, accepted). Review 5 |
| 10 | found the gating design sound at spec level once 12 is fixed, and left restart |
| 11 | timing, connection cut-off and EICAR transport to the real-box acceptance |
| 12 | checks in §3.3. Accepted findings are marked **(codex N)**. |
| 13 | **Builds on:** `2026-09-01-object-storage-virus-scanning-design.md` (the sweep, |
| 14 | verdict tags, enforcement) and `2026-09-02-scan-on-demand-design.md` §9 (clamd |
| 15 | off the console box, reachable over TCP). |
| 16 | **Supersedes:** steps 2 and 4 of `../plans/2026-09-01-carloku-scanning-rollout.md` |
| 17 | (clamd on the console box, `CARLOS_STORE_SCANNER` as the switch). Its steps 5–8 |
| 18 | and its watch list still stand. |
| 19 | |
| 20 | --- |
| 21 | |
| 22 | ## 1. Where things stand |
| 23 | |
| 24 | The code is shipped and live. Both consoles run builds that carry the sweep, |
| 25 | `carlos store scan request|status|grant|revoke|sweep|enforce` and |
| 26 | `carlos system antivirus status|enable|disable` (Carloku `7edd4b33`, Tito |
| 27 | `fe2088e3`). `CARLOS_CLAMD_ADDR` already takes `host:port` as well as a socket |
| 28 | path (`2eaf2aae`, in every release since v0.18.0). |
| 29 | |
| 30 | What neither deployment has is a clamd to talk to, permission for the console |
| 31 | to read and tag store objects, or a scan tag key. This spec supplies those three |
| 32 | things, plus one small client change (§4.3) that stops a partly scanned archive |
| 33 | being signed clean. **(codex 4)** |
| 34 | |
| 35 | ## 2. Decisions |
| 36 | |
| 37 | | # | Decision | Why | |
| 38 | |---|---|---| |
| 39 | | D1 | clamd runs on its own box, not the console box | Its ~1.5 GB of resident signatures (about 3 GB during a reload) leave the control plane. Carloku's console box is a `t4g.large` that also serves IE apps. | |
| 40 | | D2 | One box per deployment, in the home region only | Only the console talks to clamd, and every store bucket is in the one region `CARLOS_STORE_REGION` names (`cmd/console/main.go:546-585`). A box elsewhere would add cross-region transfer, not remove it. | |
| 41 | | D3 | Console to clamd is plain TCP inside the VPC, closed by security group | clamd has no TLS and no authentication. Same-VPC, single-source security group is the same trust model as the console's own S3 traffic. TLS (stunnel or a dialler change) or Tailscale are later options, not v1. **(Paul, 2026-10-04: "A is fine for now")** | |
| 42 | | D4 | The console keeps fetching from S3 and streams to clamd | Scan-on-demand §9.4: tenant-data credentials stay on one box. The scanner box gets no S3 or parameter grants. | |
| 43 | | D5 | Both pieces are platform modules | Carloku consumes the platform the way a self-hoster does. Hand-written root resources would be a workaround and would duplicate clamd's setup per root. | |
| 44 | | D6 | Carloku first, then Tito | Tito is a third party, and its console was only unpinned on 2026-10-03 with #201 phase 2 pending. Prove it on our own flagship first. | |
| 45 | | D7 | Scope ends at "engine wired, switched on, no member granted" | Rollout plan steps 7–8: a member grant follows a member's request and the disclosure, never an operator's speculation. | |
| 46 | | D8 | An object clamd only partly scanned, with no malware signature matched, is recorded **unscanned**, never clean, and never infected on the strength of the limit alone | A signed clean verdict over bytes nobody read is the failure the verdict design exists to prevent; flagging large legitimate archives infected would make enforce block them. Unscanned fits the existing audit arithmetic (`unscanned = skipped + denied + failed`). A real signature match is positive evidence and stays **infected** even when the scan was partial (§4.3). **(Paul, 2026-10-04, on codex 4)** | |
| 47 | |
| 48 | **Out of scope:** the unmerged scan-on-demand API (`worktree-store-scan-on-demand`), |
| 49 | TLS to clamd, alerting on signature age, GuardDuty, per-region scanning. If |
| 50 | store buckets ever become per-region, scanning would carry tenant bytes out of |
| 51 | their region; the answer then is a regional scanner that fetches for itself, |
| 52 | which reopens D4. That is a new decision, not more boxes. |
| 53 | |
| 54 | ## 3. The `scanner-host` module (`infra/modules/scanner-host`) |
| 55 | |
| 56 | ### 3.1 Resources |
| 57 | |
| 58 | - **`aws_instance`**: `ami_id` from the caller (the caller passes |
| 59 | `module.<host>.ami_id`, as Tito's `svc-pdf.tf` does), `instance_type` default |
| 60 | `t4g.medium`, 20 GiB encrypted gp3 root, IMDSv2 required |
| 61 | (`http_tokens = "required"`), `user_data_replace_on_change = false`, |
| 62 | `lifecycle { ignore_changes = [ami, user_data] }`, so a newer image or an |
| 63 | edited script never silently replaces the box; changing either (including |
| 64 | `stream_max_bytes`) is an explicit `-replace`. Optional `subnet_id`; null |
| 65 | means the default subnet, as `region-host` does, and a precondition then |
| 66 | requires the client security group's VPC to be the default VPC. |
| 67 | - **`aws_security_group`**: ingress TCP 3310 from `client_security_group_id` |
| 68 | **only**. No other ingress, no SSH. Egress open, for `freshclam` and apt. It |
| 69 | lands in the client security group's VPC (`vpc_id` from a data source on that |
| 70 | group), so the two can never end up in different VPCs. |
| 71 | - **IAM role and instance profile**: `AmazonSSMManagedInstanceCore` attached, |
| 72 | for Session Manager. That managed policy grants `ssm:GetParameter*` on `*`, |
| 73 | which on its own would let the box read `/carlos/scan-tag-key` and |
| 74 | `/carlos/release-key`. So the role also carries an explicit Deny on |
| 75 | `ssm:GetParameter`, `GetParameters`, `GetParameterHistory` and |
| 76 | `GetParametersByPath` on `*`, the same shape as `region-host`'s |
| 77 | `host_ssm_parameter_scope` (`main.tf:418-450`) with no allowed parameter at |
| 78 | all. Session Manager's channel actions are untouched. No bucket, no KMS. The |
| 79 | box cannot fetch tenant data or the key even if clamd were compromised. |
| 80 | **(codex 1)** |
| 81 | |
| 82 | ### 3.2 How the box listens: the invariant |
| 83 | |
| 84 | **TCP 3310 is open only while a self-test has passed against the clamd process |
| 85 | that is running now.** Not "the boot script checked once": a failed test, a |
| 86 | clamd restart, or a ClamAV package upgrade closes the port until the test |
| 87 | passes again. The console's own checks cannot stand in for this, because |
| 88 | `antivirus status` and `enable` only ask clamd for `VERSION` |
| 89 | (`internal/objstore/clamav.go:108`, `internal/console/apisystem.go:85,133`), |
| 90 | and an engine that signs truncated scans clean answers `VERSION` perfectly |
| 91 | well. **(codex 10)** |
| 92 | |
| 93 | So clamd itself never listens on TCP. Three units: |
| 94 | |
| 95 | - **`clamav-daemon`**, Debian's own unit and socket activation, **untouched** |
| 96 | except a drop-in for `MemoryMax` (variable, default `3G`), `MemoryHigh` at |
| 97 | about 85% of it, and `Restart=on-failure`. It listens on the unix socket |
| 98 | `/run/clamav/clamd.ctl` only; `clamd.conf` has no `TCPSocket` and no |
| 99 | `TCPAddr`. That is why the earlier replacement unit is gone: socket |
| 100 | activation handing clamd only the unix socket is now exactly what is wanted. |
| 101 | (Finding 2's problem no longer arises.) A drop-in `Wants=scanner-tcp.socket` |
| 102 | makes every start of clamd, including an automatic restart, ask for the |
| 103 | gate below to be re-run. |
| 104 | - **`scanner-selftest.service`**, `Type=oneshot`, `RemainAfterExit=yes`, |
| 105 | `BindsTo=clamav-daemon.service`, `After=clamav-daemon.service`. Runs |
| 106 | `/usr/local/sbin/scanner-selftest` against the unix socket. Being bound to |
| 107 | clamd, it goes inactive the moment clamd stops or restarts. |
| 108 | - **`scanner-tcp.socket`** and **`scanner-tcp.service`**: the socket listens on |
| 109 | `<private IPv4>:3310` (from IMDSv2, written into the unit at first boot; |
| 110 | never `0.0.0.0`) and hands connections to `systemd-socket-proxyd |
| 111 | /run/clamav/clamd.ctl`, running as a dedicated user in group `clamav`. Both |
| 112 | carry `Requires=scanner-selftest.service`, `After=` it and `BindsTo=` it. So |
| 113 | the port cannot open before the test passes, and it closes, cutting |
| 114 | connections in flight, when the test's unit goes inactive. |
| 115 | |
| 116 | The socket unit sets **`DefaultDependencies=no`**. A socket unit is ordered |
| 117 | `Before=sockets.target` by default, `sockets.target` comes before |
| 118 | `basic.target`, and every service (the self-test included) comes after |
| 119 | `basic.target`; a socket that must also come *after* the self-test closes |
| 120 | that loop, and systemd breaks an ordering cycle by dropping a job, which |
| 121 | could leave the port shut after a reboot. So the unit opts out, restores what |
| 122 | it still needs by hand (`After=sysinit.target`, `Conflicts=shutdown.target`, |
| 123 | `Before=shutdown.target`), and is started by `WantedBy=multi-user.target` and |
| 124 | clamd's `Wants=`, never by `sockets.target`. **(codex 12)** |
| 125 | |
| 126 | `clamd.conf` keys, each rewritten idempotently (clamd has no include |
| 127 | directory): |
| 128 | |
| 129 | - `StreamMaxLength <stream_max_bytes / 1 MiB>M`. Written in whole MiB, not |
| 130 | bytes: Debian's debconf round trip strips an `M` and appends one, so a bare |
| 131 | byte count would become a 25 TiB limit on the first configure (final |
| 132 | review I1). `clamav-daemon/debconf` is also set false after install, so |
| 133 | later upgrades leave `clamd.conf` alone. |
| 134 | - `AlertExceedsMax yes`. With the default `no`, an archive that expands past |
| 135 | `MaxScanSize`, `MaxFileSize`, `MaxFiles` or `MaxRecursion` is partly |
| 136 | scanned and answered `OK`. With `yes` clamd answers |
| 137 | `Heuristics.Limits.Exceeded.<limit> FOUND`, which §4.3 records as |
| 138 | unscanned (D8). **(codex 4)** |
| 139 | - `MaxFileSize 100M` and `MaxScanSize 100M`: **the same value, and fixed in |
| 140 | this release.** `AlertExceedsMax` alone is not enough. ClamAV's zip |
| 141 | extractor trims a member that expands past `MaxFileSize` and scans the |
| 142 | trimmed copy without raising any limit (`libclamav/unzip.c:222-226`, |
| 143 | 1.4.3). With the two limits equal, the trimmed copy is as large as the |
| 144 | whole scan budget, so scanning it always trips `MaxScanSize`, which does |
| 145 | alert. Tested, §3.4. **(codex 9)** It is a constant, not a variable, |
| 146 | because the self-test's fixtures and expected answers are sized for |
| 147 | 100 MiB; a configurable limit means deriving both from it, which is a |
| 148 | follow-up (§11). **(codex 11)** |
| 149 | - `ConcurrentDatabaseReload` from a variable, default `yes`. |
| 150 | |
| 151 | ### 3.3 `user_data` |
| 152 | |
| 153 | A templated script (`scanner-init.sh.tftpl`), `set -eu`, each step logged to |
| 154 | the serial console so `aws ec2 get-console-output` shows how far it got. |
| 155 | |
| 156 | 1. **SSM agent first.** The AMI is the base Debian image `region-host` boots |
| 157 | from (`outputs.tf:26`), not a configured host, and Debian ships no SSM agent. |
| 158 | Install the arm64 `amazon-ssm-agent` `.deb` from the regional bucket and |
| 159 | enable it, exactly as `host-init.sh.tftpl` does. With SSH closed this is the |
| 160 | only way onto the box, so it goes before anything that can fail. A |
| 161 | `region` variable feeds the URL. Also `unattended-upgrades`, as region hosts |
| 162 | have. **(codex 5)** |
| 163 | 2. **Install without starting.** Write a `policy-rc.d` that exits 101, then |
| 164 | `apt-get install clamav-daemon clamav-freshclam clamdscan python3`, then |
| 165 | remove the `policy-rc.d`. Debian's postinst would otherwise start |
| 166 | `clamav-freshclam` (`clamav-freshclam.postinst.in:333`) and the daemon |
| 167 | before they are configured. **(codex 3)** |
| 168 | 3. **Write `clamd.conf`'s keys, the drop-in, the self-test script and the |
| 169 | three units** of §3.2. `systemctl daemon-reload`. |
| 170 | 4. **First signatures.** `freshclam` once in the foreground, with no |
| 171 | `clamav-freshclam` service running (step 2 kept it stopped), so the two never |
| 172 | contend for the lock or the update log. Refuse to continue unless `main`, |
| 173 | `daily` and `bytecode` (`.cvd` or `.cld`) exist. Then enable and start |
| 174 | `clamav-freshclam` (hourly checks by default). **(codex 3)** |
| 175 | 5. **Enable and start** `clamav-daemon`, `scanner-selftest.service` and |
| 176 | `scanner-tcp.socket`. The ordering in §3.2 does the rest: clamd loads, the |
| 177 | self-test runs, and only on a pass does the port open. |
| 178 | 6. **Verify, and say so.** `systemctl is-active scanner-selftest.service |
| 179 | scanner-tcp.socket` both `active`; `zVERSION` answers on |
| 180 | `<private ip>:3310`; `ss -ltn` shows 3310 bound to the private IP only. |
| 181 | Write `/var/lib/scanner-host/ready` and print `SCANNER-HOST READY` plus the |
| 182 | `zVERSION` line. Any failure prints `SCANNER-HOST FAILED: <which check>` and |
| 183 | exits non-zero, with the port closed by construction. |
| 184 | |
| 185 | **The self-test** (`/usr/local/sbin/scanner-selftest`) builds the §3.4 |
| 186 | fixtures with `python3` in a temp directory (members written from a temp file, |
| 187 | so their zip headers are ordinary rather than zip64, which clamd would skip), |
| 188 | streams each to the unix socket with `clamdscan --stream`, and requires the |
| 189 | class of reply in the table's last column. **An over-limit fixture answering |
| 190 | `OK` is a failure.** It re-proves the behaviour on whatever ClamAV version the |
| 191 | image or a later upgrade installs, rather than trusting what 1.4.3 did. |
| 192 | |
| 193 | **Acceptance checks for the module**, run on the first real box before any |
| 194 | console is wired (§6.2), and recorded in the README: |
| 195 | |
| 196 | - kill clamd: 3310 closes at once, and reopens only after the self-test has |
| 197 | re-run and passed; |
| 198 | - force the self-test to fail (an environment override in a drop-in, used only |
| 199 | for this check): clamd comes up, 3310 **stays closed**, the unit shows |
| 200 | failed; remove the override and restart: it opens; |
| 201 | - an INSTREAM scan of EICAR through `<private ip>:3310` from the console box |
| 202 | answers `Eicar-Test-Signature FOUND`, so the proxy carries the protocol; |
| 203 | - **reboot the box**: it comes back with the self-test passed and 3310 open, |
| 204 | `systemd-analyze verify` on the three units is clean, and `journalctl -b` |
| 205 | has no `ordering cycle` line. First boot cannot show this, because |
| 206 | `basic.target` and `sockets.target` are already reached by the time |
| 207 | `user_data` runs. **(codex 12)** |
| 208 | |
| 209 | ### 3.4 What the limits do, measured |
| 210 | |
| 211 | Run 2026-10-05 against Debian trixie's clamd (ClamAV 1.4.3), `StreamMaxLength |
| 212 | 25M`, `AlertExceedsMax yes`, every fixture streamed with `clamdscan --stream`. |
| 213 | Members are a repeating 16 KiB random block, distinct per member, so the clean |
| 214 | cache cannot short-circuit them. "Limit" means a |
| 215 | `Heuristics.Limits.Exceeded.*` reply. |
| 216 | |
| 217 | | Fixture | `MaxFileSize 25M`, `MaxScanSize 100M` | Both 100M (this spec) | Self-test requires | |
| 218 | |---|---|---|---| |
| 219 | | small clean file | OK | OK | OK | |
| 220 | | EICAR | infected | infected | infected | |
| 221 | | zip, one member expanding to 500 MiB | **OK** (trimmed, silent) | Limit | Limit | |
| 222 | | zip, a zip inside holding that member | **OK** (silent) | Limit | Limit | |
| 223 | | zip, 30 distinct 20 MiB members | Limit | Limit | Limit | |
| 224 | | gzip of 500 MiB | Limit | Limit | Limit | |
| 225 | | tar.gz, 5 × 40 MiB | Limit | Limit | Limit | |
| 226 | | bzip2, xz, tar.xz of the same | Limit | Limit | not in self-test | |
| 227 | | zip, 500 MiB member then EICAR | infected | Limit | Limit | |
| 228 | | zip, 60 MiB member then EICAR | infected | infected | infected | |
| 229 | | zip, one 60 MiB member | **OK** (trimmed, silent) | OK (fully scanned) | OK | |
| 230 | |
| 231 | The first column is finding 9, observed: three fixtures signed clean over |
| 232 | content nobody read. The second is what this spec configures. |
| 233 | |
| 234 | **Known and not fixed: zip64 local headers.** A zip member written with zip64 |
| 235 | local headers is not extracted at all by 1.4.3 (`local header - stream out of |
| 236 | file`), and EICAR inside one reads `OK` under every config. That is parser |
| 237 | coverage, not a limit, and no clamd setting changes it. Encrypted archives are |
| 238 | the same class. So a clean verdict means "clamd found nothing in what it could |
| 239 | open", never "this object is safe", and the disclosure a member reads before |
| 240 | opting in (rollout plan step 7) should say so in those words. Recorded in §11 |
| 241 | as a follow-up; it does not block this rollout, because it is true of the |
| 242 | engine wherever it runs. |
| 243 | |
| 244 | ### 3.5 Variables and outputs |
| 245 | |
| 246 | | Variable | Default | Notes | |
| 247 | |---|---|---| |
| 248 | | `name` | — | Prefixes every resource | |
| 249 | | `ami_id` | — | Debian arm64, from `region-host` | |
| 250 | | `client_security_group_id` | — | The console box's group | |
| 251 | | `instance_type` | `t4g.medium` | `t4g.small` only with `concurrent_database_reload = false` | |
| 252 | | `stream_max_bytes` | `26214400` | Must equal the console's `CARLOS_CLAMD_MAX_BYTES`. Validation: a whole number of MiB, between 8 MiB and 100 MiB, so the self-test's fixtures (about 5 MB compressed) always fit the stream and a whole stream always fits the scan budget **(codex 11)** | |
| 253 | | `memory_max` | `"3G"` | clamd's cgroup ceiling | |
| 254 | | `concurrent_database_reload` | `true` | | |
| 255 | | `subnet_id` | `null` | | |
| 256 | | `region` | — | For the SSM agent download URL | |
| 257 | |
| 258 | Validation: `instance_type` beginning `t4g.small`/`t4g.micro`/`t4g.nano` with |
| 259 | `concurrent_database_reload = true` is refused with a message naming the reload |
| 260 | spike. |
| 261 | |
| 262 | Outputs: `clamd_addr` (`"<private_ip>:3310"`), `instance_id`, |
| 263 | `security_group_id`, `stream_max_bytes`. |
| 264 | |
| 265 | ### 3.6 Replacement |
| 266 | |
| 267 | A replaced box gets a new private IP, so `CARLOS_CLAMD_ADDR` must be updated by |
| 268 | hand. Accepted: replacements are explicit (`-replace`) and the failure is loud |
| 269 | (sweeps fail per tick, `scan status` and `antivirus status` say so, the console |
| 270 | stays up). The README says this in its first section. |
| 271 | |
| 272 | ## 4. The console's read-and-tag grant |
| 273 | |
| 274 | ### 4.1 `region-host` (Carloku) |
| 275 | |
| 276 | - New variable `store_scanning` (bool, default `false`). A precondition refuses |
| 277 | `true` with an empty `store_bucket_prefix`. |
| 278 | - When `true`, a **separate** `aws_iam_role_policy`, `${iam_name}-store-scan`: |
| 279 | `s3:GetObject`, `s3:GetObjectTagging`, `s3:PutObjectTagging` on |
| 280 | `arn:aws:s3:::${store_bucket_prefix}*/*`. These are exactly the calls the |
| 281 | scanner makes (`HeadObject` is authorised by `GetObject`; `ListObjectsV2` is |
| 282 | already granted by `host_store`). Store buckets use SSE-S3, so no KMS grant. |
| 283 | - Separate, so an opted-out deployment plans zero changes, and so the existing |
| 284 | `host_store` comment ("No GetObject: the console provisions stores, it does not |
| 285 | read tenant data") stays true. That comment gains one line pointing at |
| 286 | `store_scanning`. |
| 287 | - The store-user boundary is untouched. Apps still cannot tag their own objects. |
| 288 | |
| 289 | ### 4.2 Tito (`object-store.tf`) |
| 290 | |
| 291 | A new statement `StoreScan` in `host_ie_object_store` with the same three |
| 292 | actions on `arn:aws:s3:::carlos-store-tito-*/*` and |
| 293 | `arn:aws:s3:::titogo-uploads-665106592142/*`. Tito keeps its own store policy |
| 294 | (#201 settled this), so `store_scanning` is not used there. |
| 295 | |
| 296 | ### 4.3 The client change: limits exceeded is unscanned (D8) |
| 297 | |
| 298 | `internal/clamd/clamd.go` `parseReply` (`:273`) maps any `FOUND` to |
| 299 | `StatusInfected`. With `AlertExceedsMax yes`, a limit hit arrives as |
| 300 | `stream: Heuristics.Limits.Exceeded.MaxScanSize FOUND` (likewise `.MaxFileSize`, |
| 301 | `.MaxFiles`, `.MaxRecursion`, `.MaxScanTime`). Add two cases **before** the |
| 302 | `FOUND` case, both keeping the raw reply: |
| 303 | |
| 304 | - signature exactly `Heuristics.Limits.Exceeded.MaxScanTime` becomes |
| 305 | `StatusError`. A timeout depends on load, not on the bytes, so it must be |
| 306 | retried. `verdictFor` maps `StatusError` to `VerdictFailed`, which |
| 307 | `clamav.go:249-251` re-issues every sweep rather than carrying forward. |
| 308 | **(codex 7)** |
| 309 | - any other signature beginning `Heuristics.Limits.Exceeded` becomes |
| 310 | `StatusSizeExceeded`. These limits are deterministic for given bytes and |
| 311 | config. `clamav.go:374-383` maps it to `VerdictSkipped`, so it lands in |
| 312 | `skipped` and `unscanned`, never `clean` or `infected`, and like an oversize |
| 313 | object it is carried forward until the bytes change. |
| 314 | |
| 315 | **Declined: codex 6** (a malware match hiding an earlier limit hit). When |
| 316 | clamd reports a real match, the client records infected, whatever limit was |
| 317 | also hit. That is correct under D8: the object is known to contain malware, |
| 318 | whatever else went unread. Setting `HeuristicScanPrecedence yes` to surface the |
| 319 | limit instead would turn a known infection into `unscanned`, which is worse. |
| 320 | When nothing matches, clamd answers with the limit heuristic, so the case D8 |
| 321 | exists for is covered. Which of the two an archive gets depends on what clamd |
| 322 | reaches before the budget runs out (§3.4: a 500 MiB member ahead of EICAR |
| 323 | answers the limit, a 60 MiB member ahead of it answers EICAR). Both are honest |
| 324 | and neither is clean. §7 step 2 tests both. |
| 325 | |
| 326 | Tests in `clamd_test.go`: `MaxScanTime` maps to error; each of the other four |
| 327 | limit names maps to size-exceeded; |
| 328 | `Eicar-Signature FOUND` still maps to infected; a signature that merely |
| 329 | contains the phrase elsewhere (`Win.Heuristics.Limits.Exceeded.Fake`) still |
| 330 | maps to infected, so the match is a prefix on the signature, not a substring of |
| 331 | the reply. The docs for `carlos store scan status` that explain `skipped` gain |
| 332 | one sentence naming over-limit archives. |
| 333 | |
| 334 | Because this is console code, **both consoles roll to a build containing it |
| 335 | before they are wired** (§6.2). A console without it, wired to a clamd with |
| 336 | `AlertExceedsMax yes`, would record over-limit archives as infected. |
| 337 | |
| 338 | ## 5. Release and consumption |
| 339 | |
| 340 | - The module change, `scanner-host` and the §4.3 client change land on amadan main through a branch and |
| 341 | `amadan branch merge -squash`, then the GitHub mirror is pushed forward |
| 342 | (fast-forward checked first), then **v0.24.1** is cut per the README's |
| 343 | "Releasing the platform" recipe and pushed to the mirror. |
| 344 | - **Carloku** (`carloku-infrastructure`): move **only `host_ie`'s** `?ref=` to |
| 345 | v0.24.1, set `store_scanning = true`, add `module "scanner"` with |
| 346 | `client_security_group_id = module.host_ie.security_group_id`. The other three |
| 347 | regions stay on v0.21.0. Every apply is `-target`ed |
| 348 | (`module.host_ie.aws_iam_role_policy.host_store_scan`, `module.scanner`), and |
| 349 | the plan is read for any `user_data` line before it runs, because a |
| 350 | `user_data` change stops and starts a box. Any other `host_ie` diff the |
| 351 | v0.21.0 to v0.24.1 bump shows is listed in the PR and not applied. |
| 352 | - **Tito** (`teamtito/carlos-infrastructure`): a branch from the |
| 353 | `store-boundary-201` line (main cannot plan while the Canada box is only on |
| 354 | `region-ca`), `module "scanner"` at `?ref=v0.24.1`, the `StoreScan` |
| 355 | statement. Targeted applies. Tito pins `region-host` to a SHA; that pin does |
| 356 | not move. |
| 357 | |
| 358 | ## 6. Wiring the console |
| 359 | |
| 360 | ### 6.1 The scan tag key |
| 361 | |
| 362 | - 32 random bytes, hex. Generated once per deployment, never in git, tofu state |
| 363 | or a transcript: generated and written in one pipeline. |
| 364 | - Durable copy: SSM SecureString `/carlos/scan-tag-key` in the deployment's |
| 365 | account (inside the `/carlos/*` path the box can already read). Losing it makes |
| 366 | every stored verdict unverifiable. |
| 367 | - `CARLOS_SCAN_TAG_KEY_PREVIOUS` stays unset; rotation exists and is not needed. |
| 368 | |
| 369 | ### 6.2 The console's environment |
| 370 | |
| 371 | **Two preconditions, both checked before any variable is written.** |
| 372 | |
| 373 | *The scanner box is ready.* Over SSM on the scanner instance: `cloud-init |
| 374 | status --wait` reports `done` with no error; `/var/lib/scanner-host/ready` |
| 375 | exists; `systemctl is-active scanner-selftest.service scanner-tcp.socket` says |
| 376 | `active` twice; and the module's acceptance checks (§3.3) have been run on |
| 377 | this box. Nothing the console can ask tells a tested engine from an untested |
| 378 | one, so this is checked on the box, on **each** deployment, Tito included. |
| 379 | **(codex 10)** |
| 380 | |
| 381 | *The console **process** runs a build containing §4.3.* Promoting only moves the channel |
| 382 | pointer (`internal/release/release.go:718`); adoption is separate and can defer |
| 383 | (`adopt.go:956`); and `x-carlos-version` comes from the edge's route state |
| 384 | (`internal/edge/edge.go:2192`), not from the running process. So neither a |
| 385 | promote nor the header proves it. The check is the one `roll-console.sh` |
| 386 | already makes (`scripts/roll-console.sh:93-97`): over SSM, `readlink |
| 387 | /proc/$(systemctl show -p MainPID --value console)/exe` must resolve to |
| 388 | `versions/<sha>/console` for the exact SHA built. Roll with `roll-console.sh` |
| 389 | (it ships, promotes, adopts, restarts and runs that check); if the check fails, |
| 390 | **stop**: nothing is wired. Boxes are untouched. On Tito, the same from Tito's |
| 391 | bucket. **(codex 8)** |
| 392 | |
| 393 | First confirm over SSM, read-only, which file `console.service` loads. Then, |
| 394 | after a dated backup of that file: |
| 395 | |
| 396 | ``` |
| 397 | CARLOS_CLAMD_ADDR=<module.scanner.clamd_addr> |
| 398 | CARLOS_CLAMD_MAX_BYTES=<module.scanner.stream_max_bytes> |
| 399 | CARLOS_SCAN_TAG_KEY=<from /carlos/scan-tag-key> |
| 400 | ``` |
| 401 | |
| 402 | `systemctl restart console`, then repeat the `/proc/<pid>/exe` check (same SHA, |
| 403 | new PID), and `carlos system antivirus status` answers with the engine wired. |
| 404 | If it fails to start: restore the backup, `systemctl reset-failed console && |
| 405 | systemctl restart console` (a start-limited unit ignores try-restart; seen |
| 406 | 2026-09-11). |
| 407 | |
| 408 | `CARLOS_STORE_SCANNER` stays unset. It is deprecated, and the switch is the |
| 409 | record (`system/antivirus.json`), so the restart wires the engine without |
| 410 | turning it on. |
| 411 | |
| 412 | ### 6.3 Switching on |
| 413 | |
| 414 | `carlos system antivirus status`: engine wired, clamd answering, signature date |
| 415 | today or yesterday. Then `carlos system antivirus enable`. With no app granted |
| 416 | the hourly pass runs and correctly does nothing. Sit in that state until §7 |
| 417 | passes. |
| 418 | |
| 419 | ## 7. Proving it on Carloku |
| 420 | |
| 421 | On a throwaway app we own, with a plaintext store: |
| 422 | |
| 423 | 1. Upload one clean object and the EICAR test file. `scan request --plaintext`, |
| 424 | `grant`, `sweep`. **Pass:** `scan status` shows a completed sweep, one clean, |
| 425 | one infected; both objects' tags verify under this deployment's key and carry |
| 426 | the object's current ETag. |
| 427 | 2. Upload three archives from the §3.4 fixture set and `sweep`: |
| 428 | - the zip with one member expanding to 500 MiB. **Pass:** `skipped` and |
| 429 | `unscanned`, never `clean` or `infected`; its tag carries no clean |
| 430 | verdict. **(codex 4, codex 9)** |
| 431 | - the zip with that member followed by EICAR. **Pass:** `skipped` and |
| 432 | `unscanned` (clamd stops at the limit before reaching EICAR). |
| 433 | - the zip with a 60 MiB member followed by EICAR. **Pass:** `infected` |
| 434 | (D8: a real match stands). |
| 435 | |
| 436 | If any of the three reads `clean`, stop. |
| 437 | 3. Overwrite the clean object, `sweep`. **Pass:** it is re-scanned, not carried |
| 438 | forward. **If it is carried forward, stop the rollout:** a verdict that |
| 439 | outlives its bytes is worse than none. |
| 440 | 4. `enforce on`. Remove `CARLOS_CLAMD_ADDR`, restart. **Pass:** `enforce off` |
| 441 | still works with no engine built. Restore the address, restart. |
| 442 | 5. Destroy the app. **Pass:** no `scan.json` remains under its prefix and the |
| 443 | name is claimable at once. |
| 444 | |
| 445 | Then stop. Scanning stays enabled; no member store is granted. |
| 446 | |
| 447 | ## 8. Tito |
| 448 | |
| 449 | After §7 passes and after #201 phase 2 has landed (so two changes do not edit |
| 450 | Tito's console environment on the same day): §5's Tito apply, the Tito console |
| 451 | roll and the scanner-box readiness check (§6.2's two preconditions), §6 in |
| 452 | Tito's account, `antivirus status`, `enable`. **Tito's proof stops at status and |
| 453 | enable.** §7 exercises platform code identical to Carloku's, and a throwaway app |
| 454 | in a customer's deployment needs the customer's say-so. |
| 455 | |
| 456 | ## 9. Docs |
| 457 | |
| 458 | - `infra/modules/scanner-host/README.md`: what it is, the three outputs the |
| 459 | console needs, replacement (§3.6), the invariant (§3.2), the limits and why they are equal (§3.4), the acceptance checks (§3.3), and the sizing rule. |
| 460 | - `infra/modules/region-host/README.md`: `store_scanning`. |
| 461 | - A dated note at the top of `../plans/2026-09-01-carloku-scanning-rollout.md`: |
| 462 | steps 2 and 4 are superseded by this spec, `CARLOS_STORE_SCANNER` is |
| 463 | deprecated. |
| 464 | - An amadan discussion on `carlos/platform` tracking the rollout. |
| 465 | |
| 466 | ## 10. Cost |
| 467 | |
| 468 | One `t4g.medium` plus 20 GiB gp3 per deployment: about $25 a month each, $50 |
| 469 | for the pair. |
| 470 | |
| 471 | ## 11. What would change this design |
| 472 | |
| 473 | - `t4g.medium` turning out too small for the reload spike in practice: raise |
| 474 | the instance type, or set `concurrent_database_reload = false`. |
| 475 | - A member asking for scanning on a client-encrypted store: no change here, the |
| 476 | existing `--client-encrypted` path handles it. |
| 477 | - Store buckets becoming per-region: see §2, out of scope. |
| 478 | - **zip64 and encrypted archive members are not opened by clamd** (§3.4), so a |
| 479 | clean verdict never covers them. Follow-ups, each needing its own decision: |
| 480 | say so in the `scan request` disclosure (user-facing copy, so copy review); |
| 481 | consider `AlertEncrypted yes` with a mapping to unscanned, the way §4.3 maps |
| 482 | limits. |
| 483 | - Making the scan limit configurable: the self-test's fixture sizes and |
| 484 | expected answers must then be derived from it (codex 11). Fixed at 100 MiB |
| 485 | until someone needs otherwise. |
| 486 | - Raising clamd's limits later: objects already recorded `skipped` under the |
| 487 | old limits are carried forward until their bytes change, as oversize objects |
| 488 | are today. A forced re-sweep of skipped objects is a follow-up if anyone |
| 489 | raises a limit, not part of this rollout. |
| 490 | |
amadan