| new file mode 100644 |
| index 000000000..708b6b094 |
| --- /dev/null |
| +++ b/infra/modules/scanner-host/README.md |
| @@ -0,0 +1,181 @@ |
| +# scanner-host: a clamd box for a deployment's console |
| + |
| +> 🤖 Written by an LLM (Claude), per factor X. |
| + |
| +This module makes one small box that runs clamd, the ClamAV scanning daemon, so the console can virus-scan the objects in app stores. The console reads each object from S3 and streams it to this box. The box reads nothing itself. Its role has Session Manager and nothing else, with an explicit Deny on every SSM parameter read, so a compromised clamd cannot fetch a tenant object, the scan tag key or the release key. There is no SSH. You reach the box over SSM Session Manager. |
| + |
| +```hcl |
| +module "scanner" { |
| + source = "git::https://github.com/carlosframework/platform.git//infra/modules/scanner-host?ref=main" |
| + name = "carlos-flagship-scanner" |
| + ami_id = module.host_ie.ami_id |
| + client_security_group_id = module.host_ie.security_group_id |
| + region = "eu-west-1" |
| +} |
| +``` |
| + |
| +Pass the console box's own AMI, so the two boxes are built from the same base. Pass the console box's security group as `client_security_group_id`. It is the only thing allowed to reach clamd, which authenticates nobody, so that group is the whole fence. The module creates the scanner's security group in the same VPC. It admits TCP 3310 from that one group and never from a CIDR. |
| + |
| +## If you replace the box, the address changes |
| + |
| +A replaced box gets a new private IP, and the console does not follow it. After any replacement, update `CARLOS_CLAMD_ADDR` on the console and restart it. Until you do, every sweep fails on its tick, and `carlos store scan status` and `carlos system antivirus status` say so. The console itself stays up. |
| + |
| +Replacement is always explicit. A newer AMI, an edited init script or a new `stream_max_bytes` does nothing to a running box. To apply any of them: |
| + |
| +```sh |
| +tofu apply -replace=module.<name>.aws_instance.scanner |
| +``` |
| + |
| +## Wiring the console |
| + |
| +The module has two outputs the console needs. Each becomes a console environment variable. |
| + |
| +| Output | Console variable | |
| +|---|---| |
| +| `clamd_addr` (`<private_ip>:3310`) | `CARLOS_CLAMD_ADDR` | |
| +| `stream_max_bytes` | `CARLOS_CLAMD_MAX_BYTES` | |
| + |
| +The console also needs `CARLOS_SCAN_TAG_KEY`. It signs every verdict the console writes, and the console refuses to boot with an address and no key. Generate 32 random bytes as hex, once per deployment. Keep the only copy in your secret store, never in git, tofu state or a transcript. If you lose it, every stored verdict becomes unverifiable. |
| + |
| +The console's own role also needs `store_scanning = true` on its `region-host`, which lets it read and tag store objects. See that module's README. |
| + |
| +Wire the console only after the box has passed the readiness check and the four acceptance checks below. Nothing the console can ask tells a tested engine from an untested one. The console must also run a build that records an archive over the engine's limits as not scanned. A console without that change would record such an archive as infected. |
| + |
| +Wiring an address does not switch scanning on. Switch it on with `carlos system antivirus enable`. |
| + |
| +## How the port is kept shut |
| + |
| +Port 3310 is open only while a self-test has passed against the clamd process that is running now. A failed test, a clamd restart or a ClamAV package upgrade closes the port until the test passes again. The console cannot check this for you, because `antivirus status` and `enable` only ask clamd for its version, and an engine that signs a truncated scan as clean answers that perfectly well. |
| + |
| +To get there, clamd never listens on TCP. It listens on the unix socket `/run/clamav/clamd.ctl` only. `scanner-tcp.socket` listens on the box's private IPv4 address on port 3310 and hands each connection to `scanner-tcp.service`, which carries it to the unix socket. Both units are bound to `scanner-selftest.service`, which is bound to `clamav-daemon.service`. When the self-test unit goes inactive, the port closes and connections in flight are cut. |
| + |
| +The self-test builds ten archive fixtures, streams each to clamd, and requires the right class of answer. An archive that expands past the limits must answer with a limit hit. An `OK` there is a failure. It runs every time clamd starts, on whatever ClamAV version is installed at that moment. |
| + |
| +## Why the two limits are equal and fixed |
| + |
| +clamd runs with `MaxFileSize 100M`, `MaxScanSize 100M` and `AlertExceedsMax yes`. These are constants in this release, not variables. |
| + |
| +`AlertExceedsMax yes` makes clamd answer `Heuristics.Limits.Exceeded.<limit> FOUND` when an archive expands past a limit. The console records that as not scanned. With the default `no`, clamd scans part of the archive and answers `OK`. |
| + |
| +That is not enough on its own. ClamAV 1.4.3 trims a zip member that expands past `MaxFileSize`, scans the trimmed copy, and raises no limit. With the two limits equal, the trimmed copy is as large as the whole scan budget, so scanning it always trips `MaxScanSize`, which does alert. The fixtures and expected answers in the self-test are sized for 100 MiB, so changing the limit means deriving both from it. |
| + |
| +Measured on 2026-10-05 against Debian trixie's clamd (ClamAV 1.4.3), with `StreamMaxLength` at 25M and every fixture streamed with `clamdscan --stream`. Each member is a repeating 16 KiB random block, distinct per member, so clamd's clean cache cannot short-circuit it. "Limit" means a `Heuristics.Limits.Exceeded.*` reply. |
| + |
| +| Fixture | `MaxFileSize 25M`, `MaxScanSize 100M` | Both 100M (this module) | Self-test requires | |
| +|---|---|---|---| |
| +| small clean file | OK | OK | OK | |
| +| EICAR | infected | infected | infected | |
| +| zip, one member expanding to 500 MiB | **OK** (trimmed, silent) | Limit | Limit | |
| +| zip, a zip inside holding that member | **OK** (silent) | Limit | Limit | |
| +| zip, 30 distinct 20 MiB members | Limit | Limit | Limit | |
| +| gzip of 500 MiB | Limit | Limit | Limit | |
| +| tar.gz, 5 x 40 MiB | Limit | Limit | Limit | |
| +| bzip2, xz, tar.xz of the same | Limit | Limit | not in self-test | |
| +| zip, 500 MiB member then EICAR | infected | Limit | Limit | |
| +| zip, 60 MiB member then EICAR | infected | infected | infected | |
| +| zip, one 60 MiB member | **OK** (trimmed, silent) | OK (fully scanned) | OK | |
| + |
| +The first column is what an unequal pair does: three fixtures were signed clean over content nobody read. |
| + |
| +`stream_max_bytes` is the one limit you can change. It is clamd's `StreamMaxLength`, the largest object it accepts on a stream. It defaults to 26214400 (25 MiB) and must be between 8 MiB and 100 MiB. The floor keeps the self-test's fixtures, about 5 MB each, inside the stream. The ceiling keeps a whole stream inside the scan budget. The console's `CARLOS_CLAMD_MAX_BYTES` must equal it. Use the `stream_max_bytes` output rather than typing the number twice. |
| + |
| +## What a clean verdict does not cover |
| + |
| +A zip member written with zip64 local headers is not extracted at all by ClamAV 1.4.3 (`local header - stream out of file`), and EICAR inside one reads `OK` under every configuration. Encrypted archives are the same. This is parser coverage, not a limit, and no clamd setting changes it. A clean verdict means clamd found nothing in what it could open. It does not mean the object is safe. |
| + |
| +## Checking that a box is ready |
| + |
| +First boot takes several minutes. It installs packages, downloads about 110 MB of signatures and runs the self-test. In an SSM session on the box: |
| + |
| +```sh |
| +cloud-init status --wait |
| +cat /var/lib/scanner-host/ready |
| +systemctl is-active scanner-selftest.service scanner-tcp.socket |
| +journalctl -u scanner-selftest.service --no-pager | tail -14 |
| +``` |
| + |
| +You should see `status: done`, a ClamAV version line, `active` twice, and then ten `pass` lines and `scanner-selftest: PASS`. The init script also writes each step to the serial console, ending in `SCANNER-HOST READY <version>` or `SCANNER-HOST FAILED: <reason>`. To read it without a session: |
| + |
| +```sh |
| +aws ec2 get-console-output --instance-id <scanner-instance-id> --latest --output text | grep 'SCANNER-HOST' |
| +``` |
| + |
| +If the box failed, fix the module and replace the box. Do not patch the box. |
| + |
| +## Acceptance checks |
| + |
| +Run all four on the first real box in a deployment, before the console is wired, and record the results. Each one proves a part of the invariant that first boot cannot. Run the commands as root in an SSM session on the box, except the third. |
| + |
| +**1. Killing clamd closes the port, and it reopens only after the self-test passes.** |
| + |
| +```sh |
| +systemctl kill -s KILL clamav-daemon.service |
| +sleep 3 |
| +echo "after kill: $(ss -ltnH 'sport = :3310' | wc -l) listeners, selftest $(systemctl is-active scanner-selftest.service)" |
| +for i in $(seq 1 60); do [ "$(systemctl is-active scanner-tcp.socket)" = active ] && break; sleep 10; done |
| +echo "recovered: socket $(systemctl is-active scanner-tcp.socket), selftest $(systemctl is-active scanner-selftest.service), $(ss -ltnH 'sport = :3310' | wc -l) listeners" |
| +``` |
| + |
| +Expected: `after kill: 0 listeners`, with the self-test anything but `active`. Then `recovered: socket active, selftest active, 1 listeners`. If the port is still open right after the kill, or never reopens, the unit dependencies do not work on this systemd. Fix the module and replace the box. |
| + |
| +**2. A failed self-test keeps the port shut.** The script honours `SCANNER_SELFTEST_FORCE_FAIL=1`. Set it in a drop-in for this check only. |
| + |
| +```sh |
| +mkdir -p /etc/systemd/system/scanner-selftest.service.d |
| +printf '[Service]\nEnvironment=SCANNER_SELFTEST_FORCE_FAIL=1\n' > /etc/systemd/system/scanner-selftest.service.d/99-force-fail.conf |
| +systemctl daemon-reload |
| +systemctl restart clamav-daemon.service |
| +sleep 90 |
| +systemctl is-active clamav-daemon.service scanner-selftest.service scanner-tcp.socket |
| +ss -ltnH 'sport = :3310' | wc -l |
| + |
| +rm /etc/systemd/system/scanner-selftest.service.d/99-force-fail.conf |
| +systemctl daemon-reload |
| +systemctl restart clamav-daemon.service |
| +for i in $(seq 1 60); do [ "$(systemctl is-active scanner-tcp.socket)" = active ] && break; sleep 10; done |
| +systemctl is-active clamav-daemon.service scanner-selftest.service scanner-tcp.socket |
| +ss -ltnH 'sport = :3310' | wc -l |
| +``` |
| + |
| +Expected from the first print: `active`, `failed`, then `inactive` or `failed`, then `0`. Expected from the second: `active` three times, then `1`. |
| + |
| +**3. The scan protocol works through the proxy, from the console's box.** Run this in bash in an SSM session on the console box. `/dev/tcp` is a bash feature. It streams the EICAR test string, a harmless standard test file that every engine flags, to the scanner. |
| + |
| +```bash |
| +exec 3<>/dev/tcp/<scanner-private-ip>/3310 |
| +printf 'zINSTREAM\0\0\0\0\104X5O!P%%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*\0\0\0\0' >&3 |
| +timeout 20 head -c 200 <&3 | tr -d '\0'; echo |
| +``` |
| + |
| +`\0\0\0\104` is the 68-byte chunk length (octal 104 is 68). The four zero bytes after the payload end the stream. Expected: `stream: Eicar-Test-Signature FOUND`. Also confirm the fence from a box that is not the console's: the same connect attempt should time out. |
| + |
| +**4. The box comes back from a reboot with the port open.** First boot cannot show this, because `basic.target` and `sockets.target` are already reached by the time the init script runs. A systemd ordering cycle would only appear on a later boot. |
| + |
| +```sh |
| +aws ec2 reboot-instances --instance-ids <scanner-instance-id> |
| +``` |
| + |
| +Wait five minutes, then in an SSM session on the box: |
| + |
| +```sh |
| +systemctl is-active clamav-daemon.service scanner-selftest.service scanner-tcp.socket |
| +ss -ltnH 'sport = :3310' | wc -l |
| +journalctl -b --no-pager | grep -ci "ordering cycle" |
| +systemd-analyze verify /etc/systemd/system/scanner-selftest.service /etc/systemd/system/scanner-tcp.socket /etc/systemd/system/scanner-tcp.service 2>&1 | grep -v '^$' | head |
| +``` |
| + |
| +Expected: `active` three times, `1`, `0`, and nothing from `systemd-analyze verify` about these three units. |
| + |
| +## Sizing |
| + |
| +The default is `t4g.medium`. clamd holds about 1.5 GB of signatures in memory, and about twice that for a short time while it loads a fresh set. A `t4g.small` or smaller cannot hold two databases at once. `tofu plan` refuses that pairing unless you set `concurrent_database_reload = false`, which pauses scanning during each reload. |
| + |
| +clamd runs under a cgroup ceiling (`memory_max`, default `3G`, with a throttle at `memory_high`, default `2560M`). Under memory pressure the kernel kills clamd, never the SSM agent, which is the only way onto the box. systemd restarts clamd, and the self-test re-runs before the port reopens. The root volume is 20 GiB, encrypted gp3. |
| + |
| +## Network |
| + |
| +The box needs a route out. It downloads packages, the SSM agent (from the regional bucket for `region`) and signature updates. With `subnet_id` null, the module uses the default subnet of the default VPC, and refuses at plan time unless the client security group is in the default VPC. Otherwise pass a `subnet_id` in the client group's VPC. Egress is open. There is no ingress except TCP 3310 from the client group. |
| + |
| +## Variables and outputs |
| + |
| +Variables are described in `variables.tf`. Outputs: `clamd_addr`, `stream_max_bytes`, `instance_id`, `security_group_id`. |
amadan