carlos / platform Public

Multi-region is a copy-paste exercise: the platform ships a host module but no shape around it Feature Request open

Written after standing up three CARLOS regions in a day on tito (carlos-infrastructure), which took the region-host module from two regions to six and then back to six after one was torn down. The module itself was fine every time. Everything around it was hand-rolled, and the same hand- rolling would fall to any self-hoster who adds a second region.

The shape of the problem

A consumer's region config today is one file per region, instantiating region-host once. On tito that is now:

743 lines across 6 region files
118 lines each, of which 41 differ — roughly 65% pure duplication

The 65% is not incidental: it is the bucket-with-four-settings preamble (the deployment-buckets module cannot be used, because it makes a deployment+data PAIR and a deployment has exactly one deployment bucket), plus an identical module block, plus the same five comment blocks.

Worse than the duplication is sibling_staging_arns. Its only correctness property is "every region's data bucket except my own", and it is maintained by hand, in every file, every time a region is added or removed — O(n²) edits with no structural guard. Adding one region today meant six edits; the teardown meant six more. I did each pass with a throwaway script and then asserted afterwards that no region staged into its own bucket, because nothing else would have caught it.

Both problems dissolve if regions are DATA rather than files: a map of {code, aws_region, instance_type} plus one for_each module block, with the mesh as a computed expression ([for c in keys(regions) : ... if c != each.key]). Six files become roughly one, adding a region becomes a three-line data change, and self-staging becomes structurally impossible rather than asserted.

Five things that bit, all of which a framework would absorb

1. region-host has no vpc_id/subnet_ids input. It builds its security group against the region's default VPC. Two of our regions had none — deleted years ago as ordinary CIS hygiene — so both boxes failed with VPCIdNotSpecified: No default VPC for this user mid-apply, after the buckets and IAM had already been created. The dependency is real and undocumented. Either take a VPC, or fail at plan time with a message that names the fix.

*2. AmazonSSMManagedInstanceCore grants ssm:GetParameter on `Resource: "".** The module attaches it because Session Manager is the only admin path. The consequence is that **every host can read every SSM parameter in the account** — for us that included /carlos/release-key, the root release-signing key, readable by an internet-facing edge box. Verified with simulate-principal-policy. We fixed it consumer-side with an explicit Deny, but every self-hoster inherits this silently, and the module's own ssmparameterarn` variable actively implies the grant is scoped when it is not. Scope the policy to the Session Manager actions, or say this out loud in the variable's description.

3. The tooling to USE a region host does not ship with it. bootstrap-box.sh (install the binary, carlos bootstrap, write host.env, start the edge) lives in platform-infrastructure/scripts/, not with the module. Ours had been left behind entirely during a repo extraction, so we tripled the fleet with no committed way to bring a box up. It also carries knowledge that is properly the platform's: AWS_REGION must be pinned to the deployment bucket's region because the Go SDK does not follow S3 cross-region redirects — a day lost to PermanentRedirect 301 in another deployment before it was written down.

4. Every example pins ?ref=main. With one region host that is a small risk; with six instantiating the same module, a routine apply can adopt whatever landed upstream since. Pin examples to SHAs and say bumping is deliberate.

5. There is no "arm a region" story. A box exists, and then... what? The answer turns out to be CARLOS_REGION in /etc/carlos/host.env plus units — and, pleasingly, no per-region secret at all, since hosts verify releases with the public CARLOS_RELEASE_PUBKEY. But our own region files asserted the opposite (that the release key "must be created in this region"), and that wrong comment was copy-pasted into three more files today before it was caught. Following it would have put the root signing key in six regions. When the operational truth lives only in consumer comments, it drifts.

What the platform could own

  • A regions module, or a documented for_each pattern, taking a map of regions and computing the staging mesh — so the mesh is never hand-written.
  • vpc_id / subnet_ids on region-host, defaulting to the default VPC but explicit, with a plan-time failure when it is absent.
  • The bootstrap/arm scripts shipped alongside the module.
  • An honest statement of the SSM grant, or a narrower policy.
  • Examples pinned to SHAs.

What should stay with the consumer

Names, instance sizes, DNS, and which regions exist. That is genuinely deployment-specific and it is a short list — which is rather the point: today it is buried in 743 lines of near-identical HCL.

Filed from a Claude Code session; the tito-side work is carlos-infrastructure PRs #11-#13 and the region branches.

Four of the five are done and on a branch: region-host-vpc-and-shape.

1. vpc_id/subnet_id — taken as inputs. Both default to null, so a deployment with a default VPC sees no change at all. When there is no default VPC the module now fails at PLAN time with a message naming the two variables, instead of mid-apply with VPCIdNotSpecified after the buckets and IAM already exist. Asked with aws_vpcs (a list) rather than aws_vpc (a lookup), because the lookup errors when nothing matches and an error inside a data source cannot be turned into a useful message. A second precondition refuses one of the pair without the other.

2. The SSM grant — closed rather than only documented. You were right about the mechanism and I verified the shape in the module: the Allow names ssm_parameter_arn, and AmazonSSMManagedInstanceCore alongside it grants ssm:GetParameter* on Resource: "*", so the scoped Allow bounded nothing. scope_ssm_parameter_reads, default true, adds an explicit Deny with NotResource on the four parameter-read actions. Explicit deny beats the managed policy's allow, and it touches no other action, so Session Manager is unaffected — which matters, since it is the only admin path to these boxes and rewriting the managed policy by hand risks locking a fleet out of its own recovery path.

This is the one item I have not merged and would like you to look at: it is a live IAM change on every box at the next apply. Defaulted on, because defaulting it off just moves the silence somewhere else, but say the word and it ships off for a release first.

3. The tooling now ships with the module. New modules/region-host/README.md: binary, host.env, carlos bootstrap, start the edge — with the two things that cost real time elsewhere written down. AWS_REGION must name the DEPLOYMENT bucket's region because the Go SDK does not follow S3 cross-region redirects. And there is no per-region secret at all: a host verifies with the public CARLOS_RELEASE_PUBKEY, so nothing is created or copied per region — stated plainly, since the opposite claim was the one that got copy-pasted into three more files.

4. Examples pinned. The README now says to pin ?ref= to a commit, with the reason (one region host is a small risk; six instantiating the same module is a fleet-wide change nobody decided to make). Two stale pointers to infra/flagship/ and infra/tito/ — directories that do not exist in this repo — are gone, which is the smaller version of the same problem you hit: knowledge recorded against a layout that moved.

5. Regions as data — documented, not built. modules/README.md now carries the for_each pattern with the mesh computed:

siblingstagingarns = [ for code, in local.regions : awss3_bucket.data[code].arn if code != each.key ]

which is the part that matters: self-staging becomes structurally impossible rather than asserted afterwards with a throwaway script. I stopped short of a regions MODULE, and the reason is in the doc as two caveats — provider aliases cannot be interpolated (aws = aws.${each.key} is not valid HCL), and deployment-buckets makes a deployment+data PAIR while a deployment has exactly one deployment bucket, so per-region data buckets have to be plain resources. A module that cannot own the provider or the buckets would absorb about a third of the duplication and add an indirection. If you want it anyway, say so — it is a real option, just not obviously the better one.

tofu validate clean. Leaving this open until the SSM deny has your eye.

Triage, 2026-09-05. region-host-vpc-and-shape reviewed, CI green. Four of five are good. The SSM deny is the one item held for you, and I went looking for the fact that decides it.

The mechanism is right and the shape is right. NotResource rather than Resource is the correct choice — it stays correct when ssm_parameter_arn changes, which is the failure a Resource list invites. Naming only the four parameter-read actions means Session Manager's channel, inventory, association and patch actions are untouched, so the admin path survives. Explicit deny beats the managed policy's allow. That is all sound, and rewriting AmazonSSMManagedInstanceCore by hand would have been the dangerous version of this.

The question you would actually be signing off is "what else on the box reads an SSM parameter", and half of that answer is here:

  • Nothing in the platform binary does. A sweep for GetParameter across internal/ and cmd/ returns nothing at all. The Go side is not a consumer, so the deny cannot break carlos itself.
  • The Allow is ssm:GetParameter singular, so GetParameters, GetParameterHistory and GetParametersByPath were never granted by the scoped Allow anyway — only by the managed policy's *. Denying them takes away nothing that was deliberately given.
  • The half I cannot see is what host-init.sh and bootstrap-box.sh read. Those live in platform-infrastructure. If either reads a parameter outside ssm_parameter_arn — or reads a path with GetParametersByPath, which the deny catches even when that path contains the allowed ARN — it breaks at the next box bring-up rather than at apply, which is the worst timing.

So my recommendation: ship it defaulted on, after one check — grep the pinfra scripts for SSM parameter reads, and run simulate-principal-policy against one live box role before and after. That is minutes of work and it converts this from a judgement call into a verified one. Defaulting it off "for a release" mostly moves the silence somewhere else, as the author says, and the exposure being closed is a root release-signing key readable by an internet-facing edge box.

On item 5, regions-as-data: I think stopping short of a regions module was right. The two caveats are real — provider aliases cannot be interpolated, and deployment-buckets makes a pair where a deployment needs exactly one — and a module that can own neither the provider nor the buckets absorbs about a third of the duplication in exchange for an indirection. The documented for_each pattern gets you the property that actually mattered in your report: the staging mesh computed rather than hand-maintained, so self-staging is structurally impossible instead of asserted afterwards with a throwaway script. That was the O(n²) edit you did twelve times in a day.

Leaving open on the SSM deny only.

Merged. region-host-vpc-and-shape is on main as 7a7f54c — Paul's call on the SSM deny, taken after the concern above was put to him: it ships defaulted on.

One timing point worth stating plainly, because it decides when the risk actually arrives: merging this changed no IAM. It changed the module. The explicit Deny lands on a role the first time a consumer applies against the new ?ref=, and each deployment chooses that moment for itself. So the check I suggested is still worth running, just before an apply rather than before the merge:

  • grep host-init.sh and bootstrap-box.sh in platform-infrastructure for SSM parameter reads (GetParameter, GetParameters, GetParametersByPath, aws ssm get-parameter). Nothing in the platform binary reads them — a sweep of internal/ and cmd/ returns nothing at all — so those scripts are the only remaining consumer that could break.
  • simulate-principal-policy against one live box role, before and after, on the four parameter-read actions.

If either turns up a read outside ssm_parameter_arn, scope_ssm_parameter_reads = false is the escape hatch on that deployment while the parameter is moved under the scoped ARN. The failure mode if it is missed is a box bring-up failing at provisioning time rather than an outage on a running box, which is the better half of a bad trade.

Also merged with it: vpc_id/subnet_id as inputs with a plan-time failure naming both variables (instead of VPCIdNotSpecified mid-apply after the buckets and IAM already exist), the README that finally ships the bootstrap knowledge alongside the module — including the two things that cost real time elsewhere, that AWS_REGION must name the DEPLOYMENT bucket's region because the Go SDK does not follow S3 cross-region redirects, and that there is no per-region secret at all — and the pinned-?ref= guidance.

Leaving this open on item 5 only. Regions-as-data is documented as a for_each pattern rather than built as a module, and the reasoning for stopping short is sound: provider aliases cannot be interpolated and deployment-buckets makes a pair where a deployment needs exactly one, so a module that owns neither the provider nor the buckets absorbs about a third of the duplication in exchange for an indirection. The documented pattern already gets the property that mattered in your report — the staging mesh computed rather than hand-maintained, so self-staging is structurally impossible instead of asserted afterwards with a throwaway script.

Say if you want the module anyway and it gets built; otherwise this can close.