carlos / platform Public

Download this file

1	# region-host: bringing the box up
2
3	> 🤖 Written by an LLM (Claude), per factor X.
4
5	`tofu apply` gives you a machine with an IP, an IAM role and no CARLOS on
6	it. This is the other half: what to put on the box and what to tell it, so
7	that it becomes a region of your deployment.
8
9	It lives beside the module on purpose. The knowledge here used to sit only
10	in the private infrastructure repo, which meant a deployment could triple
11	its fleet with no committed way to bring a box up — and did (discussion #7).
12
13	## The four steps
14
15	Run them over SSM Session Manager. There is no SSH ingress and there is not
16	meant to be.
17
18	1. **Put the `linux/arm64` binary at `/opt/carlos/carlos`**, mode 755. How it
19	gets there is your deployment's business — the host role can read your
20	deployment bucket, so pulling it from wherever your build publishes it is
21	the usual answer. The binary is static and has no runtime dependencies.
22
23	2. **Write `/etc/carlos/host.env`**, mode 0600, root-owned. The minimum:
24
25	```sh
26	CARLOS_DEPLOYMENT_BUCKET=carlos-myco-platform-123456789012
27	CARLOS_RELEASE_PUBKEY=<the deployment's public release key>
28	CARLOS_REGION=ie
29	AWS_REGION=eu-west-1
30	```
31
32	That is enough to make the box an **edge**: it will terminate TLS, adopt
33	releases, converge routes and serve static sites. It is not enough to run
34	an app. If this region is to **host instances**, it also needs its own
35	data bucket — the one `tofu apply` already made and granted the host role:
36
37	```sh
38	CARLOS_HIBERNATE_BUCKET=carlos-myco-ie-123456789012
39	CARLOS_HIBERNATE_REGION=eu-west-1
40	CARLOS_HIBERNATE_PREFIX=instances
41	```
42
43	Without those, the hibernation activator never arms, and nothing on the
44	box ever starts an instance's process. The failure is quiet from the
45	box's side — the record is accepted, the route is minted and the binary
46	is staged, and only the request finds out — so the box now publishes
47	what it was told as `hibernation_configured` in its edges file, and the
48	console refuses instance creates into a region whose boxes report false
49	rather than accepting one that would strand. Add the variables and
50	restart `carlos-edge` and the region arms itself within a minute or two.
51
52	`CARLOS_HIBERNATE_REGION` is the AWS region of the **data bucket**, and
53	like `AWS_REGION` above it is a real AWS region name, not a placement
54	code.
55
56	`AWS_REGION` is the one that bites. It must name the region the
57	**deployment bucket** lives in, not the region this box is in. The AWS Go
58	SDK does not follow S3's cross-region redirects, so a box in Sydney
59	pointed at an Irish deployment bucket with `AWS_REGION=ap-southeast-2`
60	fails every read with `PermanentRedirect: 301` — a day was lost to this in
61	one deployment before anyone wrote it down.
62
63	`CARLOS_REGION` is the region **code** this box serves (`ie`, `au`, `pi`),
64	not an AWS region name. It is what makes the box claim its own region's
65	work.
66
67	3. **Run `carlos bootstrap`** (as root). It creates the service user and the
68	consumers group, writes the data and runtime dirs, installs the systemd
69	units from inside the binary, and arms the timers and path units. It is
70	idempotent: re-running it is how you refresh unit files after a binary
71	upgrade, and `carlos bootstrap -root <dir>` stages the same files into a
72	directory tree without touching the live system, which is what an
73	automated upgrade path should use.
74
75	4. **Start the edge**: `systemctl start carlos-edge`. From here the box
76	polls `control/serial`, adopts releases, and serves.
77
78	## There is no per-region secret
79
80	Worth stating plainly, because at least one deployment's region files
81	asserted the opposite and the wrong comment was copy-pasted into three more
82	before anyone caught it. Following it would have put the root signing key in
83	six regions.
84
85	A host **verifies** releases. It does that with `CARLOS_RELEASE_PUBKEY`,
86	which is public — the whole point of the signing scheme is that the private
87	half never leaves your secret store and never touches a box. There is
88	nothing region-specific to create, copy or rotate when you add a region.
89
90	What a region does need is in the four steps above and nothing more: the
91	binary, `host.env`, `carlos bootstrap`, start the edge. The only thing that
92	is easy to leave out is the hibernation block in step 2 — and leaving it out
93	is what makes a region an edge that cannot host.
94
95	## Arming the root passes
96
97	Two path units let the unprivileged agent ask a root pass to do something it
98	cannot do itself — stage a release into `/opt/carlos` (`carlos-stage.path`)
99	and cycle a unit-backed route (`carlos-restart.path`). `carlos bootstrap`
100	arms both. If you upgrade the binary by hand rather than through your own
101	platform-update script, re-run `carlos bootstrap` so any new unit is
102	installed and armed; a stage relay that is not armed fails silently, which
103	looks exactly like a deploy that did nothing.
104
105	## Object-storage scanning
106
107	`store_scanning = true` lets this host's role read and tag objects under `store_bucket_prefix` (`s3:GetObject`, `s3:GetObjectTagging` and `s3:PutObjectTagging`), which is what the console's virus scanner needs. It is off by default, so a deployment that has not switched scanning on plans no change and its console still never reads tenant data. Set it on the console's box only, and only on a deployment that runs a [scanner-host](../scanner-host/README.md). It requires `store_bucket_prefix`.
108
109	## Checking it worked
110
111	```sh
112	systemctl status carlos-edge
113	curl -sI --resolve <host>:443:<eip> https://<host>/ | grep -i x-carlos-version
114	```
115
116	Use `--resolve` against the box's own EIP rather than trusting DNS: a stale
117	answer from a resolver has produced more than one false "verified" report.
118
119	## Termination evidence
120
121	The module creates a queue, `<iam_name>-terminations`, that receives an
122	event whenever an EC2 instance in this account and region reaches
123	`terminated`. Region HA reads it to prove a dead box is gone before its
124	instances run elsewhere. Only the module's own EventBridge rule can write
125	to the queue, and the box role can only read and delete from it.
126
127	After the first apply, check that nobody but the rule can write to it.
128	From an administrator's session:
129
130	aws sqs send-message --queue-url <termination_queue_url> --message-body '{"forged":true}'
131
132	must fail with `AccessDenied`. Then terminate a throwaway instance in the
133	region and confirm its event arrives.
134
135	The root volume is deleted when the instance is terminated. Region HA
136	depends on that: a terminated box must leave no disk behind.
137
138	Open item for region HA plan 1b: the box role's `ec2:DescribeInstances`
139	is unconditioned, because the infrastructure inventory reads every
140	region. The HA spec asks for an `ec2:Region` condition on the call the
141	placer makes. Decide there whether that is a second, narrower statement.
142