region-host: bringing the box up

> 🤖 Written by an LLM (Claude), per factor X.

tofu apply gives you a machine with an IP, an IAM role and no CARLOS on it. This is the other half: what to put on the box and what to tell it, so that it becomes a region of your deployment.

It lives beside the module on purpose. The knowledge here used to sit only in the private infrastructure repo, which meant a deployment could triple its fleet with no committed way to bring a box up — and did (discussion #7).

The four steps

Run them over SSM Session Manager. There is no SSH ingress and there is not meant to be.

1. Put the linux/arm64 binary at /opt/carlos/carlos, mode 755. How it gets there is your deployment's business — the host role can read your deployment bucket, so pulling it from wherever your build publishes it is the usual answer. The binary is static and has no runtime dependencies.

1. Write /etc/carlos/host.env, mode 0600, root-owned. The minimum:

``sh CARLOS_DEPLOYMENT_BUCKET=carlos-myco-platform-123456789012 CARLOS_RELEASE_PUBKEY=<the deployment's public release key> CARLOS_REGION=ie AWS_REGION=eu-west-1 ``

That is enough to make the box an edge: it will terminate TLS, adopt releases, converge routes and serve static sites. It is not enough to run an app. If this region is to host instances, it also needs its own data bucket — the one tofu apply already made and granted the host role:

``sh CARLOS_HIBERNATE_BUCKET=carlos-myco-ie-123456789012 CARLOS_HIBERNATE_REGION=eu-west-1 CARLOS_HIBERNATE_PREFIX=instances ``

Without those, the hibernation activator never arms, and nothing on the box ever starts an instance's process. The failure is quiet from the box's side — the record is accepted, the route is minted and the binary is staged, and only the request finds out — so the box now publishes what it was told as hibernation_configured in its edges file, and the console refuses instance creates into a region whose boxes report false rather than accepting one that would strand. Add the variables and restart carlos-edge and the region arms itself within a minute or two.

CARLOS_HIBERNATE_REGION is the AWS region of the data bucket, and like AWS_REGION above it is a real AWS region name, not a placement code.

AWS_REGION is the one that bites. It must name the region the deployment bucket lives in, not the region this box is in. The AWS Go SDK does not follow S3's cross-region redirects, so a box in Sydney pointed at an Irish deployment bucket with AWS_REGION=ap-southeast-2 fails every read with PermanentRedirect: 301 — a day was lost to this in one deployment before anyone wrote it down.

CARLOS_REGION is the region code this box serves (ie, au, pi), not an AWS region name. It is what makes the box claim its own region's work.

1. Run carlos bootstrap (as root). It creates the service user and the consumers group, writes the data and runtime dirs, installs the systemd units from inside the binary, and arms the timers and path units. It is idempotent: re-running it is how you refresh unit files after a binary upgrade, and carlos bootstrap -root <dir> stages the same files into a directory tree without touching the live system, which is what an automated upgrade path should use.

1. Start the edge: systemctl start carlos-edge. From here the box polls control/serial, adopts releases, and serves.

There is no per-region secret

Worth stating plainly, because at least one deployment's region files asserted the opposite and the wrong comment was copy-pasted into three more before anyone caught it. Following it would have put the root signing key in six regions.

A host verifies releases. It does that with CARLOS_RELEASE_PUBKEY, which is public — the whole point of the signing scheme is that the private half never leaves your secret store and never touches a box. There is nothing region-specific to create, copy or rotate when you add a region.

What a region does need is in the four steps above and nothing more: the binary, host.env, carlos bootstrap, start the edge. The only thing that is easy to leave out is the hibernation block in step 2 — and leaving it out is what makes a region an edge that cannot host.

Arming the root passes

Two path units let the unprivileged agent ask a root pass to do something it cannot do itself — stage a release into /opt/carlos (carlos-stage.path) and cycle a unit-backed route (carlos-restart.path). carlos bootstrap arms both. If you upgrade the binary by hand rather than through your own platform-update script, re-run carlos bootstrap so any new unit is installed and armed; a stage relay that is not armed fails silently, which looks exactly like a deploy that did nothing.

Object-storage scanning

store_scanning = true lets this host's role read and tag objects under store_bucket_prefix (s3:GetObject, s3:GetObjectTagging and s3:PutObjectTagging), which is what the console's virus scanner needs. It is off by default, so a deployment that has not switched scanning on plans no change and its console still never reads tenant data. Set it on the console's box only, and only on a deployment that runs a scanner-host. It requires store_bucket_prefix.

Checking it worked

systemctl status carlos-edge
curl -sI --resolve <host>:443:<eip> https://<host>/ | grep -i x-carlos-version

Use --resolve against the box's own EIP rather than trusting DNS: a stale answer from a resolver has produced more than one false "verified" report.

Termination evidence

The module creates a queue, <iam_name>-terminations, that receives an event whenever an EC2 instance in this account and region reaches terminated. Region HA reads it to prove a dead box is gone before its instances run elsewhere. Only the module's own EventBridge rule can write to the queue, and the box role can only read and delete from it.

After the first apply, check that nobody but the rule can write to it. From an administrator's session:

aws sqs send-message --queue-url <terminationqueueurl> --message-body '{"forged":true}'

must fail with AccessDenied. Then terminate a throwaway instance in the region and confirm its event arrives.

The root volume is deleted when the instance is terminated. Region HA depends on that: a terminated box must leave no disk behind.

Open item for region HA plan 1b: the box role's ec2:DescribeInstances is unconditioned, because the infrastructure inventory reads every region. The HA spec asks for an ec2:Region condition on the call the placer makes. Decide there whether that is a second, narrower statement.