carlos / platform Public

Download this file

1	# scanner-host: a clamd box for a deployment's console
2
3	> 🤖 Written by an LLM (Claude), per factor X.
4
5	This module makes one small box that runs clamd, the ClamAV scanning daemon, so the console can virus-scan the objects in app stores. The console reads each object from S3 and streams it to this box. The box reads nothing itself. Its role has Session Manager and nothing else, with an explicit Deny on every SSM parameter read, so a compromised clamd cannot fetch a tenant object, the scan tag key or the release key. There is no SSH. You reach the box over SSM Session Manager.
6
7	```hcl
8	module "scanner" {
9	source = "git::https://github.com/carlosframework/platform.git//infra/modules/scanner-host?ref=main"
10	name = "carlos-flagship-scanner"
11	ami_id = module.host_ie.ami_id
12	client_security_group_id = module.host_ie.security_group_id
13	region = "eu-west-1"
14	}
15	```
16
17	Pass the console box's own AMI, so the two boxes are built from the same base. Pass the console box's security group as `client_security_group_id`. It is the only thing allowed to reach clamd, which authenticates nobody, so that group is the whole fence. The module creates the scanner's security group in the same VPC. It admits TCP 3310 from that one group and never from a CIDR.
18
19	## If you replace the box, the address changes
20
21	A replaced box gets a new private IP, and the console does not follow it. After any replacement, update `CARLOS_CLAMD_ADDR` on the console and restart it. Until you do, every sweep fails on its tick, and `carlos store scan status` and `carlos system antivirus status` say so. The console itself stays up.
22
23	Replacement is always explicit. A newer AMI, an edited init script or a new `stream_max_bytes` does nothing to a running box. To apply any of them:
24
25	```sh
26	tofu apply -replace=module.<name>.aws_instance.scanner
27	```
28
29	## Wiring the console
30
31	The module has two outputs the console needs. Each becomes a console environment variable.
32
33	| Output | Console variable |
34	|---|---|
35	| `clamd_addr` (`<private_ip>:3310`) | `CARLOS_CLAMD_ADDR` |
36	| `stream_max_bytes` | `CARLOS_CLAMD_MAX_BYTES` |
37
38	The console also needs `CARLOS_SCAN_TAG_KEY`. It signs every verdict the console writes, and the console refuses to boot with an address and no key. Generate 32 random bytes as hex, once per deployment. Keep the only copy in your secret store, never in git, tofu state or a transcript. If you lose it, every stored verdict becomes unverifiable.
39
40	The console's own role also needs `store_scanning = true` on its `region-host`, which lets it read and tag store objects. See that module's README.
41
42	Wire the console only after the box has passed the readiness check and the four acceptance checks below. Nothing the console can ask tells a tested engine from an untested one. The console must also run a build that records an archive over the engine's limits as not scanned. A console without that change would record such an archive as infected.
43
44	Wiring an address does not switch scanning on. Switch it on with `carlos system antivirus enable`.
45
46	## How the port is kept shut
47
48	Port 3310 is open only while a self-test has passed against the clamd process that is running now. A failed test, a clamd restart or a ClamAV package upgrade closes the port until the test passes again. The console cannot check this for you, because `antivirus status` and `enable` only ask clamd for its version, and an engine that signs a truncated scan as clean answers that perfectly well.
49
50	To get there, clamd never listens on TCP. It listens on the unix socket `/run/clamav/clamd.ctl` only. `scanner-tcp.socket` listens on the box's private IPv4 address on port 3310 and hands each connection to `scanner-tcp.service`, which carries it to the unix socket. Both units are bound to `scanner-selftest.service`, which is bound to `clamav-daemon.service`. When the self-test unit goes inactive, the port closes and connections in flight are cut.
51
52	The self-test builds ten archive fixtures, streams each to clamd, and requires the right class of answer. An archive that expands past the limits must answer with a limit hit. An `OK` there is a failure. It runs every time clamd starts, on whatever ClamAV version is installed at that moment.
53
54	## Why the two limits are equal and fixed
55
56	clamd runs with `MaxFileSize 100M`, `MaxScanSize 100M` and `AlertExceedsMax yes`. These are constants in this release, not variables.
57
58	`AlertExceedsMax yes` makes clamd answer `Heuristics.Limits.Exceeded.<limit> FOUND` when an archive expands past a limit. The console records that as not scanned. With the default `no`, clamd scans part of the archive and answers `OK`.
59
60	That is not enough on its own. ClamAV 1.4.3 trims a zip member that expands past `MaxFileSize`, scans the trimmed copy, and raises no limit. With the two limits equal, the trimmed copy is as large as the whole scan budget, so scanning it always trips `MaxScanSize`, which does alert. The fixtures and expected answers in the self-test are sized for 100 MiB, so changing the limit means deriving both from it.
61
62	Measured on 2026-10-05 against Debian trixie's clamd (ClamAV 1.4.3), with `StreamMaxLength` at 25M and every fixture streamed with `clamdscan --stream`. Each member is a repeating 16 KiB random block, distinct per member, so clamd's clean cache cannot short-circuit it. "Limit" means a `Heuristics.Limits.Exceeded.*` reply. The self-test does not count `Heuristics.Limits.Exceeded.MaxScanTime` as one. That reply is a timeout, not a limit, and it fails the test.
63
64	| Fixture | `MaxFileSize 25M`, `MaxScanSize 100M` | Both 100M (this module) | Self-test requires |
65	|---|---|---|---|
66	| small clean file | OK | OK | OK |
67	| EICAR | infected | infected | infected |
68	| zip, one member expanding to 500 MiB | **OK** (trimmed, silent) | Limit | Limit |
69	| zip, a zip inside holding that member | **OK** (silent) | Limit | Limit |
70	| zip, 30 distinct 20 MiB members | Limit | Limit | Limit |
71	| gzip of 500 MiB | Limit | Limit | Limit |
72	| tar.gz, 5 x 40 MiB | Limit | Limit | Limit |
73	| bzip2, xz, tar.xz of the same | Limit | Limit | not in self-test |
74	| zip, 500 MiB member then EICAR | infected | Limit | Limit |
75	| zip, 60 MiB member then EICAR | infected | infected | infected |
76	| zip, one 60 MiB member | **OK** (trimmed, silent) | OK (fully scanned) | OK |
77
78	The first column is what an unequal pair does: three fixtures were signed clean over content nobody read.
79
80	`stream_max_bytes` is the one limit you can change. It is clamd's `StreamMaxLength`, the largest object it accepts on a stream. It defaults to 26214400 (25 MiB). It must be a whole number of MiB, between 8 MiB and 100 MiB. The floor keeps the self-test's fixtures, about 5 MB each, inside the stream. The ceiling keeps a whole stream inside the scan budget. The console's `CARLOS_CLAMD_MAX_BYTES` must equal it. Use the `stream_max_bytes` output rather than typing the number twice.
81
82	## What a clean verdict does not cover
83
84	A zip member written with zip64 local headers is not extracted at all by ClamAV 1.4.3 (`local header - stream out of file`), and EICAR inside one reads `OK` under every configuration. Encrypted archives are the same. This is parser coverage, not a limit, and no clamd setting changes it. A clean verdict means clamd found nothing in what it could open. It does not mean the object is safe.
85
86	## Checking that a box is ready
87
88	First boot takes several minutes. It installs packages, downloads about 110 MB of signatures and runs the self-test. In an SSM session on the box:
89
90	```sh
91	cloud-init status --wait
92	cat /var/lib/scanner-host/ready
93	systemctl is-active scanner-selftest.service scanner-tcp.socket
94	journalctl -u scanner-selftest.service --no-pager | tail -14
95	```
96
97	You should see `status: done`, a ClamAV version line, `active` twice, and then ten `pass` lines and `scanner-selftest: PASS`. The init script writes each step to `/var/log/cloud-init-output.log`, and copies it to the serial console when it can, ending in `SCANNER-HOST READY <version>` or `SCANNER-HOST FAILED: <reason>`. To read the console copy without a session:
98
99	```sh
100	aws ec2 get-console-output --instance-id <scanner-instance-id> --latest --output text | grep 'SCANNER-HOST'
101	```
102
103	If the box failed, fix the module and replace the box. Do not patch the box.
104
105	## Acceptance checks
106
107	Run all four on the first real box in a deployment, before the console is wired, and record the results. Each one proves a part of the invariant that first boot cannot. Run the commands as root in an SSM session on the box, except the third.
108
109	**1. Killing clamd closes the port, and it reopens only after the self-test passes.**
110
111	```sh
112	systemctl kill -s KILL clamav-daemon.service
113	sleep 3
114	echo "after kill: $(ss -ltnH 'sport = :3310' | wc -l) listeners, selftest $(systemctl is-active scanner-selftest.service)"
115	for i in $(seq 1 60); do [ "$(systemctl is-active scanner-tcp.socket)" = active ] && break; sleep 10; done
116	echo "recovered: socket $(systemctl is-active scanner-tcp.socket), selftest $(systemctl is-active scanner-selftest.service), $(ss -ltnH 'sport = :3310' | wc -l) listeners"
117	```
118
119	Expected: `after kill: 0 listeners`, with the self-test anything but `active`. Then `recovered: socket active, selftest active, 1 listeners`. If the port is still open right after the kill, or never reopens, the unit dependencies do not work on this systemd. Fix the module and replace the box.
120
121	**2. A failed self-test keeps the port shut.** The script honours `SCANNER_SELFTEST_FORCE_FAIL=1`. Set it in a drop-in for this check only.
122
123	```sh
124	mkdir -p /etc/systemd/system/scanner-selftest.service.d
125	printf '[Service]\nEnvironment=SCANNER_SELFTEST_FORCE_FAIL=1\n' > /etc/systemd/system/scanner-selftest.service.d/99-force-fail.conf
126	systemctl daemon-reload
127	systemctl restart clamav-daemon.service
128	sleep 90
129	systemctl is-active clamav-daemon.service scanner-selftest.service scanner-tcp.socket
130	ss -ltnH 'sport = :3310' | wc -l
131
132	rm /etc/systemd/system/scanner-selftest.service.d/99-force-fail.conf
133	systemctl daemon-reload
134	systemctl restart clamav-daemon.service
135	for i in $(seq 1 60); do [ "$(systemctl is-active scanner-tcp.socket)" = active ] && break; sleep 10; done
136	systemctl is-active clamav-daemon.service scanner-selftest.service scanner-tcp.socket
137	ss -ltnH 'sport = :3310' | wc -l
138	```
139
140	Expected from the first print: `active`, `failed`, then `inactive` or `failed`, then `0`. Expected from the second: `active` three times, then `1`.
141
142	**3. The scan protocol works through the proxy, from the console's box.** Run this in bash in an SSM session on the console box. `/dev/tcp` is a bash feature. It streams the EICAR test string, a harmless standard test file that every engine flags, to the scanner.
143
144	```bash
145	exec 3<>/dev/tcp/<scanner-private-ip>/3310
146	printf 'zINSTREAM\0\0\0\0\104X5O!P%%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*\0\0\0\0' >&3
147	timeout 20 head -c 200 <&3 | tr -d '\0'; echo
148	```
149
150	`\0\0\0\104` is the 68-byte chunk length (octal 104 is 68). The four zero bytes after the payload end the stream. Expected: `stream: Eicar-Test-Signature FOUND`. Also confirm the fence from a box that is not the console's: the same connect attempt should time out.
151
152	**4. The box comes back from a reboot with the port open.** First boot cannot show this, because `basic.target` and `sockets.target` are already reached by the time the init script runs. A systemd ordering cycle would only appear on a later boot.
153
154	```sh
155	aws ec2 reboot-instances --instance-ids <scanner-instance-id>
156	```
157
158	Wait five minutes, then in an SSM session on the box:
159
160	```sh
161	systemctl is-active clamav-daemon.service scanner-selftest.service scanner-tcp.socket
162	ss -ltnH 'sport = :3310' | wc -l
163	journalctl -b --no-pager | grep -ci "ordering cycle"
164	systemd-analyze verify /etc/systemd/system/scanner-selftest.service /etc/systemd/system/scanner-tcp.socket /etc/systemd/system/scanner-tcp.service 2>&1 | grep -v '^$' | head
165	```
166
167	Expected: `active` three times, `1`, `0`, and nothing from `systemd-analyze verify` about these three units.
168
169	## Sizing
170
171	The default is `t4g.medium`. clamd holds about 1.5 GB of signatures in memory, and about twice that for a short time while it loads a fresh set. A `t4g.small` or smaller cannot hold two databases at once. `tofu plan` refuses that pairing unless you set `concurrent_database_reload = false`, which pauses scanning during each reload.
172
173	clamd runs under a cgroup ceiling (`memory_max`, default `3G`, with a throttle at `memory_high`, default `2560M`). Under memory pressure the kernel kills clamd, never the SSM agent, which is the only way onto the box. systemd restarts clamd, and the self-test re-runs before the port reopens. The root volume is 20 GiB, encrypted gp3.
174
175	## Network
176
177	The box needs a route out. It downloads packages, the SSM agent (from the regional bucket for `region`) and signature updates. With `subnet_id` null, the module uses the default subnet of the default VPC, and refuses at plan time unless the client security group is in the default VPC. Otherwise pass a `subnet_id` in the client group's VPC. Egress is open. There is no ingress except TCP 3310 from the client group.
178
179	## Variables and outputs
180
181	Variables are described in `variables.tf`. Outputs: `clamd_addr`, `stream_max_bytes`, `instance_id`, `security_group_id`.
182