amadan
Merge branch 'region-host-vpc-and-shape' into main
4 files changed,
+356
−11
infra/modules/README.md+133 −9infra/modules/region-host/README.md+87 −0infra/modules/region-host/main.tf+82 −2infra/modules/region-host/variables.tf+54 −0
CI passed — run details
diff --git a/infra/modules/README.md b/infra/modules/README.md| index 1765debb2..8526879fe 100644 |
| --- a/infra/modules/README.md |
| +++ b/infra/modules/README.md |
| @@ -2,10 +2,12 @@ |
| > š¤ Written by an LLM (Claude), per factor X. |
| -Reusable OpenTofu modules for a CARLOS deployment. Our own stacks |
| -(`infra/flagship/`, `infra/tito/`) consume them by relative path |
| -(`source = "../modules/region-host"`); a self-hoster ā or Tito, from its |
| -own repo ā consumes the exact same modules by git source URL: |
| +Reusable OpenTofu modules for a CARLOS deployment. Every consumer reaches |
| +them the same way ā by git source URL ā including the deployments Paul |
| +operates, whose roots live in the private |
| +[platform-infrastructure](https://github.com/carlosframework/platform-infrastructure) |
| +repo. There are no stacks in this repo to copy from, which is deliberate: |
| +nothing deployment-specific belongs here. |
| ```hcl |
| source = "git::https://github.com/carlosframework/platform.git//infra/modules/region-host?ref=main" |
| @@ -64,7 +66,10 @@ bucket, so a wrong-region wake fails loudly at the IAM layer. |
| | `user_data` | First-boot init script *content* (pass `file("${path.module}/host-init.sh")`). Changes never replace the instance. | |
| | `deployment_bucket_arn` | ARN of the deployment bucket. | |
| | `data_bucket_arn` | ARN of THIS region's data bucket. | |
| -| `ssm_parameter_arn` | SSM parameter ARN pattern the host may read, e.g. `arn:aws:ssm:<region>:<account>:parameter/carlos/*`. | |
| +| `vpc_id` | VPC to place the host in. Null (default) uses the region's **default VPC** ā see the warning below. Set it together with `subnet_id`. | |
| +| `subnet_id` | Subnet to launch in; required when `vpc_id` is set, and must belong to it. Null (default) lets EC2 pick the default VPC's default subnet. | |
| +| `ssm_parameter_arn` | SSM parameter ARN pattern the host may read, e.g. `arn:aws:ssm:<region>:<account>:parameter/carlos/*`. Bounds parameter reads only in combination with `scope_ssm_parameter_reads` ā see below. | |
| +| `scope_ssm_parameter_reads` | Deny every SSM parameter read outside `ssm_parameter_arn`. Default `true`. Set `false` only if this deployment's hosts genuinely read parameters outside that pattern ā see below. | |
| | `route53_zone_ids` | Hosted zones the box may publish ACME DNS-01 challenge TXTs into (wildcard certs). Bare `Z...` ids; empty = no Route53 access. | |
| | `steering_zone_ids` | Hosted zones the DNS-steering converger may write latency A records into, plus the health checks that withdraw a dead edge. Bare `Z...` ids; empty = no steering grant (the default ā steering also needs `CARLOS_STEERING_ZONE_ID` in the box env). | |
| | `steering_record_names` | Name bound on that grant, e.g. `["*.oncarlos.com"]`. Empty (default) grants A-record UPSERT/DELETE on the WHOLE zone, apex included ā set it. | |
| @@ -77,7 +82,34 @@ Outputs: `eip_public_ip` (point your DNS here), `instance_id`, |
| `security_group_id` (for sibling services that admit fleet traffic |
| only), `role_arn`, `role_name`, `ami_id`. |
| -Not a module: Tito's `svc/pdf` box stays in `infra/tito/svc-pdf.tf` ā it |
| +`README.md` in the module directory is the other half: what to put on the |
| +box once tofu has made it, and what to tell it. That runbook used to live |
| +only in the private infrastructure repo, which is how one deployment came to |
| +triple its fleet with no committed way to bring a box up. |
| + |
| +### Two things to know before you apply |
| + |
| +**Default VPCs.** With `vpc_id` unset the module builds the security group |
| +in the region's default VPC and lets EC2 pick that VPC's default subnet. |
| +Plenty of accounts no longer have one ā deleting them is ordinary CIS |
| +hygiene ā and without a default VPC the apply fails **half-way through**, |
| +after the buckets, the role and the instance profile already exist, with |
| +`VPCIdNotSpecified: No default VPC for this user`. The module now checks at |
| +plan time and tells you to pass `vpc_id`/`subnet_id`, which is the whole fix. |
| + |
| +**The SSM grant is wider than `ssm_parameter_arn` suggests.** Session Manager |
| +is the only admin path to these boxes, so the module attaches the AWS-managed |
| +`AmazonSSMManagedInstanceCore` policy ā and that policy grants |
| +`ssm:GetParameter*` on `Resource: "*"`. On its own, the module's own scoped |
| +Allow bounds nothing: every host could read every SSM parameter in the |
| +account, which on one deployment included the root release-signing key, |
| +readable by an internet-facing edge box. `scope_ssm_parameter_reads` (default |
| +`true`) adds an explicit Deny on the four parameter-read actions for |
| +everything outside `ssm_parameter_arn`. It touches no other action, so |
| +Session Manager is unaffected. If your boxes read a parameter outside that |
| +pattern, widen the pattern rather than turning the deny off. |
| + |
| +Not a module: Tito's `svc/pdf` box stays in Tito's own root (`svc-pdf.tf`) ā it |
| is deliberately deployment-local (a stateless Gotenberg whose SG admits |
| only the fleet, with *no* bucket or parameter grants; statelessness is |
| imposed, not promised). Generalise it only when a second deployment |
| @@ -162,6 +194,98 @@ resource "aws_route53_record" "ie" { |
| } |
| ``` |
| -Copy a `host-init.sh` from `infra/flagship/` or `infra/tito/` as your |
| -starting point, then `tofu init && tofu plan`. Pin `?ref=` to a tag or |
| -commit rather than `main` once you're live. |
| +Then `tofu init && tofu plan`. Write your own `host-init.sh` ā the module |
| +takes its *content*, and the box-side steps it needs to end up performing are |
| +in `modules/region-host/README.md`. |
| + |
| +**Pin `?ref=` to a commit, not to `main`.** The examples above say `main` so |
| +that they keep working as the modules move, and that is exactly the wrong |
| +setting for a live deployment: a routine `tofu apply` would adopt whatever |
| +landed upstream since your last one. With one region host that is a small |
| +risk; with six instantiating the same module it is a fleet-wide change |
| +nobody decided to make. Pin it, and treat bumping the pin as its own |
| +deliberate act: |
| + |
| +```hcl |
| +source = "git::https://github.com/carlosframework/platform.git//infra/modules/region-host?ref=f975b47cc88a7c033daab6453111298d3d86d401" |
| +``` |
| + |
| +## More than one region |
| + |
| +Six regions is the same file six times if you write it that way. One |
| +deployment measured its own: 743 lines across six region files, 118 each, of |
| +which 41 differed ā about 65% pure duplication, and the duplicated part |
| +included a value that has to be **different in every file**. |
| + |
| +That value is `sibling_staging_arns`, whose only correctness property is |
| +"every region's data bucket except my own". Maintained by hand it is O(nĀ²) |
| +edits with nothing checking them: adding one region means editing all six, |
| +and the failure it invites ā a region staging into its own bucket ā is |
| +silent. |
| + |
| +Both problems dissolve if regions are data rather than files. One map, one |
| +`for_each` on each module block, and the mesh as a computed expression: |
| + |
| +```hcl |
| +locals { |
| + account = "123456789012" |
| + |
| + # The only thing that changes when you add or remove a region. |
| + regions = { |
| + ie = { aws_region = "eu-west-1", instance_type = "t4g.medium" } |
| + uk = { aws_region = "eu-west-2", instance_type = "t4g.small" } |
| + au = { aws_region = "ap-southeast-2", instance_type = "t4g.small" } |
| + } |
| +} |
| + |
| +# One provider alias per region, because a bucket and a box are created in |
| +# the region they live in. This is the one part that stays literal: provider |
| +# blocks cannot be generated. |
| +provider "aws" { |
| + alias = "ie" |
| + region = local.regions.ie.aws_region |
| +} |
| +# ...one per region. |
| + |
| +module "region" { |
| + for_each = local.regions |
| + source = "git::https://github.com/carlosframework/platform.git//infra/modules/region-host?ref=<sha>" |
| + providers = { |
| + aws = aws.${each.key} # see the note below |
| + } |
| + |
| + name = "carlos-myco-${each.key}" |
| + sg_description = "carlos myco ${upper(each.key)} host" |
| + iam_name = "carlos-myco-host-${each.key}" |
| + instance_type = each.value.instance_type |
| + root_volume_size = 20 |
| + user_data = file("${path.module}/host-init.sh") |
| + |
| + deployment_bucket_arn = module.buckets.deployment_bucket_arn |
| + data_bucket_arn = aws_s3_bucket.data[each.key].arn |
| + ssm_parameter_arn = "arn:aws:ssm:${each.value.aws_region}:${local.account}:parameter/carlos/*" |
| + |
| + # The mesh: every region's data bucket except this one's. Computed, so it |
| + # cannot be wrong, and adding a region does not touch this line. |
| + sibling_staging_arns = [ |
| + for code, _ in local.regions : aws_s3_bucket.data[code].arn if code != each.key |
| + ] |
| +} |
| +``` |
| + |
| +Adding a region becomes a three-line data change instead of six file edits, |
| +and a region staging into its own bucket becomes structurally impossible |
| +rather than something you assert afterwards with a script. |
| + |
| +Two honest caveats. Provider aliases cannot be interpolated ā `providers = { |
| +aws = aws.${each.key} }` is not valid HCL ā so in practice you either keep |
| +one small module block per region that passes its own provider into a shared |
| +inner module, or generate the root from your own templating. And |
| +`deployment-buckets` makes a deployment+data *pair*, while a deployment has |
| +exactly one deployment bucket, so the per-region data buckets in the sketch |
| +above are plain `aws_s3_bucket` resources with the four settings |
| +`deployment-buckets` would have applied (versioning, encryption, public |
| +access block, ownership controls). |
| + |
| +Neither caveat changes the shape: regions are data, the mesh is computed, |
| +and the thing you edit when the fleet changes is a map. |
diff --git a/infra/modules/region-host/README.md b/infra/modules/region-host/README.md| new file mode 100644 |
| index 000000000..ace82e086 |
| --- /dev/null |
| +++ b/infra/modules/region-host/README.md |
| @@ -0,0 +1,87 @@ |
| +# region-host: bringing the box up |
| + |
| +> š¤ Written by an LLM (Claude), per factor X. |
| + |
| +`tofu apply` gives you a machine with an IP, an IAM role and no CARLOS on |
| +it. This is the other half: what to put on the box and what to tell it, so |
| +that it becomes a region of your deployment. |
| + |
| +It lives beside the module on purpose. The knowledge here used to sit only |
| +in the private infrastructure repo, which meant a deployment could triple |
| +its fleet with no committed way to bring a box up ā and did (discussion #7). |
| + |
| +## The four steps |
| + |
| +Run them over SSM Session Manager. There is no SSH ingress and there is not |
| +meant to be. |
| + |
| +1. **Put the `linux/arm64` binary at `/opt/carlos/carlos`**, mode 755. How it |
| + gets there is your deployment's business ā the host role can read your |
| + deployment bucket, so pulling it from wherever your build publishes it is |
| + the usual answer. The binary is static and has no runtime dependencies. |
| + |
| +2. **Write `/etc/carlos/host.env`**, mode 0600, root-owned. The minimum: |
| + |
| + ```sh |
| + CARLOS_DEPLOYMENT_BUCKET=carlos-myco-platform-123456789012 |
| + CARLOS_RELEASE_PUBKEY=<the deployment's public release key> |
| + CARLOS_REGION=ie |
| + AWS_REGION=eu-west-1 |
| + ``` |
| + |
| + `AWS_REGION` is the one that bites. It must name the region the |
| + **deployment bucket** lives in, not the region this box is in. The AWS Go |
| + SDK does not follow S3's cross-region redirects, so a box in Sydney |
| + pointed at an Irish deployment bucket with `AWS_REGION=ap-southeast-2` |
| + fails every read with `PermanentRedirect: 301` ā a day was lost to this in |
| + one deployment before anyone wrote it down. |
| + |
| + `CARLOS_REGION` is the region **code** this box serves (`ie`, `au`, `pi`), |
| + not an AWS region name. It is what makes the box claim its own region's |
| + work. |
| + |
| +3. **Run `carlos bootstrap`** (as root). It creates the service user and the |
| + consumers group, writes the data and runtime dirs, installs the systemd |
| + units from inside the binary, and arms the timers and path units. It is |
| + idempotent: re-running it is how you refresh unit files after a binary |
| + upgrade, and `carlos bootstrap -root <dir>` stages the same files into a |
| + directory tree without touching the live system, which is what an |
| + automated upgrade path should use. |
| + |
| +4. **Start the edge**: `systemctl start carlos-edge`. From here the box |
| + polls `control/serial`, adopts releases, and serves. |
| + |
| +## There is no per-region secret |
| + |
| +Worth stating plainly, because at least one deployment's region files |
| +asserted the opposite and the wrong comment was copy-pasted into three more |
| +before anyone caught it. Following it would have put the root signing key in |
| +six regions. |
| + |
| +A host **verifies** releases. It does that with `CARLOS_RELEASE_PUBKEY`, |
| +which is public ā the whole point of the signing scheme is that the private |
| +half never leaves your secret store and never touches a box. There is |
| +nothing region-specific to create, copy or rotate when you add a region. |
| + |
| +What a region does need is in the four steps above and nothing more: the |
| +binary, `host.env`, `carlos bootstrap`, start the edge. |
| + |
| +## Arming the root passes |
| + |
| +Two path units let the unprivileged agent ask a root pass to do something it |
| +cannot do itself ā stage a release into `/opt/carlos` (`carlos-stage.path`) |
| +and cycle a unit-backed route (`carlos-restart.path`). `carlos bootstrap` |
| +arms both. If you upgrade the binary by hand rather than through your own |
| +platform-update script, re-run `carlos bootstrap` so any new unit is |
| +installed and armed; a stage relay that is not armed fails silently, which |
| +looks exactly like a deploy that did nothing. |
| + |
| +## Checking it worked |
| + |
| +```sh |
| +systemctl status carlos-edge |
| +curl -sI --resolve <host>:443:<eip> https://<host>/ | grep -i x-carlos-version |
| +``` |
| + |
| +Use `--resolve` against the box's own EIP rather than trusting DNS: a stale |
| +answer from a resolver has produced more than one false "verified" report. |
diff --git a/infra/modules/region-host/main.tf b/infra/modules/region-host/main.tf| index deb71e96f..ecc59f522 100644 |
| --- a/infra/modules/region-host/main.tf |
| +++ b/infra/modules/region-host/main.tf |
| @@ -24,9 +24,41 @@ data "aws_ami" "debian_arm64" { |
| } |
| } |
| +# Does this region have a default VPC at all? Asked with aws_vpcs (a LIST) |
| +# rather than aws_vpc (a lookup), on purpose: the lookup ERRORS when nothing |
| +# matches, and an error inside a data source cannot be turned into a useful |
| +# message. The list comes back empty instead, so the precondition below can |
| +# say what to do about it. Not evaluated at all when vpc_id is set. |
| +data "aws_vpcs" "default" { |
| + count = var.vpc_id == null ? 1 : 0 |
| + filter { |
| + name = "is-default" |
| + values = ["true"] |
| + } |
| +} |
| + |
| resource "aws_security_group" "host" { |
| name = var.name |
| description = var.sg_description |
| + # Null = the region's default VPC, which is what this module did before it |
| + # took the input at all. |
| + vpc_id = var.vpc_id |
| + |
| + lifecycle { |
| + # Both preconditions exist to move a failure from the middle of an apply |
| + # to the start of a plan. The buckets and the IAM role are created before |
| + # the instance, so a mid-apply failure here leaves a half-built region |
| + # that the next run has to reconcile. |
| + precondition { |
| + condition = var.vpc_id != null || length(try(data.aws_vpcs.default[0].ids, [])) > 0 |
| + error_message = "This region has no default VPC, so the host has nowhere to go. Pass vpc_id and subnet_id naming the VPC and subnet this box should live in. (A default VPC is created per region per account and deleting them is ordinary hygiene; without this check the apply fails half-way through with `VPCIdNotSpecified: No default VPC for this user`, after the buckets and IAM already exist.)" |
| + } |
| + precondition { |
| + condition = (var.vpc_id == null) == (var.subnet_id == null) |
| + error_message = "vpc_id and subnet_id go together: name both, or neither. A security group in your VPC and an instance in the default one is an apply that fails at the EC2 layer with a much worse message than this one." |
| + } |
| + } |
| + |
| ingress { |
| from_port = 80 |
| to_port = 80 |
| @@ -327,19 +359,67 @@ resource "aws_iam_role_policy" "host_metrics" { |
| }) |
| } |
| +# Session Manager. This is the ONLY admin path to these boxes ā there is no |
| +# SSH ingress at all ā so the managed policy is attached rather than |
| +# hand-rolled: getting its action list wrong locks everyone out of a fleet |
| +# whose recovery path is the thing that just broke. |
| resource "aws_iam_role_policy_attachment" "host_ssm" { |
| role = aws_iam_role.host.name |
| policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" |
| } |
| +# ...and the hole that comes with it, closed. |
| +# |
| +# AmazonSSMManagedInstanceCore grants `ssm:GetParameter*` on `Resource: "*"`. |
| +# So the Allow above ā one parameter ARN pattern, the variable named |
| +# ssm_parameter_arn, whose whole description is about scoping ā bounds |
| +# nothing at all on its own: every host can read every SSM parameter in the |
| +# account. On one deployment that included /carlos/release-key, the root |
| +# release-signing key, readable by an internet-facing edge box. Verified with |
| +# simulate-principal-policy; found standing up a fleet, not in review |
| +# (discussion #7). |
| +# |
| +# An explicit Deny is the fix that does not risk the admin path: IAM's |
| +# explicit deny wins over any Allow, including a managed policy's, and this |
| +# one names ONLY the four parameter-read actions. Session Manager's channel |
| +# actions, inventory, association and patch actions are untouched, so an |
| +# operator's `aws ssm start-session` behaves exactly as it did. |
| +# |
| +# NotResource, not Resource: the deny covers everything the Allow does not |
| +# name, which is the shape that stays correct when ssm_parameter_arn changes. |
| +resource "aws_iam_role_policy" "host_ssm_parameter_scope" { |
| + count = var.scope_ssm_parameter_reads ? 1 : 0 |
| + name = "${var.iam_name}-ssm-parameter-scope" |
| + role = aws_iam_role.host.id |
| + policy = jsonencode({ |
| + Version = "2012-10-17" |
| + Statement = [ |
| + { |
| + Effect = "Deny" |
| + Action = [ |
| + "ssm:GetParameter", |
| + "ssm:GetParameters", |
| + "ssm:GetParameterHistory", |
| + "ssm:GetParametersByPath", |
| + ] |
| + NotResource = [var.ssm_parameter_arn] |
| + }, |
| + ] |
| + }) |
| +} |
| + |
| resource "aws_iam_instance_profile" "host" { |
| name = var.iam_name |
| role = aws_iam_role.host.name |
| } |
| resource "aws_instance" "host" { |
| - ami = data.aws_ami.debian_arm64.id |
| - instance_type = var.instance_type |
| + ami = data.aws_ami.debian_arm64.id |
| + instance_type = var.instance_type |
| + # Null = the default VPC's default subnet, EC2's own behaviour and this |
| + # module's original one. The security group's precondition guarantees this |
| + # is only null when a default VPC exists to hold it. |
| + subnet_id = var.subnet_id |
| vpc_security_group_ids = [aws_security_group.host.id] |
| iam_instance_profile = aws_iam_instance_profile.host.name |
| user_data = var.user_data |
diff --git a/infra/modules/region-host/variables.tf b/infra/modules/region-host/variables.tf| index b508af535..eefd8b9b5 100644 |
| --- a/infra/modules/region-host/variables.tf |
| +++ b/infra/modules/region-host/variables.tf |
| @@ -43,6 +43,60 @@ variable "ssm_parameter_arn" { |
| type = string |
| } |
| +variable "vpc_id" { |
| + description = <<-EOT |
| + VPC to place this host in. Null (the default) keeps the module's original |
| + behaviour: the security group is built in the region's DEFAULT VPC and the |
| + instance lands in that VPC's default subnet. |
| + |
| + That default is not safe to rely on. A default VPC is created per region |
| + per account, and deleting them is ordinary CIS hygiene ā an account that |
| + has done so has none, and the failure arrives MID-APPLY as |
| + `VPCIdNotSpecified: No default VPC for this user`, after the buckets, the |
| + IAM role and the instance profile have already been created (discussion |
| + #7: two of six regions, standing up a fleet in a day). The module now |
| + fails at PLAN time with a message naming this variable instead. |
| + |
| + Set it together with subnet_id; naming one without the other is refused, |
| + because a security group in your VPC and an instance in the default one |
| + is an apply that fails at the EC2 layer with a much worse message. |
| + EOT |
| + type = string |
| + default = null |
| +} |
| + |
| +variable "subnet_id" { |
| + description = "Subnet to launch the host in. Required when vpc_id is set, and must belong to it. Null (the default) lets EC2 pick the default subnet of the default VPC, which is the module's original behaviour." |
| + type = string |
| + default = null |
| +} |
| + |
| +variable "scope_ssm_parameter_reads" { |
| + description = <<-EOT |
| + Deny this role every SSM parameter read outside ssm_parameter_arn. |
| + |
| + The module attaches the AWS-managed AmazonSSMManagedInstanceCore policy, |
| + because Session Manager is the only admin path to these boxes. That policy |
| + grants `ssm:GetParameter*` on `Resource: "*"` ā so without this deny, |
| + every host can read EVERY SSM parameter in the account, whatever |
| + ssm_parameter_arn says. On one deployment that included the root |
| + release-signing key, readable by an internet-facing edge box (discussion |
| + #7, verified with simulate-principal-policy). |
| + |
| + On (the default) adds an explicit Deny on the four parameter-read actions |
| + for everything the Allow above does not name. It does not touch any other |
| + action, so Session Manager keeps working exactly as before ā the deny is |
| + scoped to parameter reads and nothing else. |
| + |
| + Set it to false only if this deployment's hosts genuinely read parameters |
| + outside ssm_parameter_arn's pattern. The better fix in that case is to |
| + widen the pattern, because a role that may read every parameter in the |
| + account is not the role this module's variables describe. |
| + EOT |
| + type = bool |
| + default = true |
| +} |
| + |
| variable "extra_ingress" { |
| description = "Additional ingress rules beyond the standard 80/443 (e.g. tcp/25 for a host running an inbound MX). The SG uses in-line rules, so extra rules MUST come through here ā a standalone aws_vpc_security_group_ingress_rule in the root would fight the in-line set and be stripped on every apply." |
| type = list(object({ |