| 1 | # 🤖 CARLOS infra modules |
| 2 | |
| 3 | > 🤖 Written by an LLM (Claude), per factor X. |
| 4 | |
| 5 | Reusable OpenTofu modules for a CARLOS deployment. Every consumer reaches |
| 6 | them the same way — by git source URL — including the deployments Paul |
| 7 | operates, whose roots live in the private |
| 8 | [platform-infrastructure](https://github.com/carlosframework/platform-infrastructure) |
| 9 | repo. There are no stacks in this repo to copy from, which is deliberate: |
| 10 | nothing deployment-specific belongs here. |
| 11 | |
| 12 | ```hcl |
| 13 | source = "git::https://github.com/carlosframework/platform.git//infra/modules/region-host?ref=main" |
| 14 | ``` |
| 15 | |
| 16 | The division of labour is unchanged by modularisation: **tofu owns boxes, |
| 17 | buckets, IAM, and static DNS; the platform owns apps, releases, |
| 18 | instances, and dynamic DNS.** Nothing account-shaped (tenant hostnames, |
| 19 | routes) may ever enter tofu state — `tofu apply` must always be safe. |
| 20 | Providers, backends, and DNS records stay in your root module: they are |
| 21 | deployment-specific, the modules are not. |
| 22 | |
| 23 | ## deployment-buckets |
| 24 | |
| 25 | The durable pair: the **deployment bucket** (apps, releases, channels, |
| 26 | instance records, host config — routing metadata only, nothing |
| 27 | customer-shaped) and one **regional data bucket** (parked databases, WAL |
| 28 | streams, leases, blobs — the residency promise). Both get versioning, |
| 29 | AES256 SSE, a full public-access block, and `prevent_destroy`. Boxes are |
| 30 | disposable; these buckets are not. |
| 31 | |
| 32 | | Variable | Description | |
| 33 | |---|---| |
| 34 | | `deployment_bucket_name` | Name of the deployment bucket. | |
| 35 | | `data_bucket_name` | Name of the regional data bucket. | |
| 36 | |
| 37 | Outputs: `deployment_bucket`, `deployment_bucket_arn`, `data_bucket`, |
| 38 | `data_bucket_arn`. |
| 39 | |
| 40 | Adding a second region later means a second `deployment-buckets`-shaped |
| 41 | data bucket; for now each deployment is one region, so the module is a |
| 42 | pair. |
| 43 | |
| 44 | ## region-host |
| 45 | |
| 46 | One region of a deployment: a Debian arm64 (Graviton) box running |
| 47 | `carlos agent`, behind an Elastic IP. Provides the AMI lookup, a security |
| 48 | group admitting 80/443 only (**no SSH** — admin access is SSM Session |
| 49 | Manager, and the carlos binary arrives from the deployment bucket, not |
| 50 | over scp), an IAM role scoped to exactly the deployment bucket + this |
| 51 | region's data bucket + the deployment's SSM parameter prefix, the |
| 52 | instance profile with `AmazonSSMManagedInstanceCore` attached, the |
| 53 | instance (encrypted gp3 root), and the EIP. |
| 54 | |
| 55 | Residency is enforced by construction: a host in another region gets a |
| 56 | different module instance whose role does not include this region's data |
| 57 | bucket, so a wrong-region wake fails loudly at the IAM layer. |
| 58 | |
| 59 | | Variable | Description | |
| 60 | |---|---| |
| 61 | | `name` | Security group name + `Name` tag (e.g. `carlos-flagship-ie`). | |
| 62 | | `sg_description` | SG description (changing it forces SG replacement — keep it stable). | |
| 63 | | `iam_name` | IAM role + instance profile name; inline policy is `<iam_name>-buckets`. | |
| 64 | | `instance_type` | arm64 instance type, e.g. `t4g.small`. | |
| 65 | | `root_volume_size` | Root gp3 volume size, GiB. | |
| 66 | | `user_data` | First-boot init script *content* (pass `file("${path.module}/host-init.sh")`). Changes never replace the instance. | |
| 67 | | `deployment_bucket_arn` | ARN of the deployment bucket. | |
| 68 | | `data_bucket_arn` | ARN of THIS region's data bucket. | |
| 69 | | `vpc_id` | VPC to place the host in. Null (default) uses the region's **default VPC** — see the warning below. Set it together with `subnet_id`. | |
| 70 | | `subnet_id` | Subnet to launch in; required when `vpc_id` is set, and must belong to it. Null (default) lets EC2 pick the default VPC's default subnet. | |
| 71 | | `ssm_parameter_arn` | SSM parameter ARN pattern the host may read, e.g. `arn:aws:ssm:<region>:<account>:parameter/carlos/*`. Bounds parameter reads only in combination with `scope_ssm_parameter_reads` — see below. | |
| 72 | | `scope_ssm_parameter_reads` | Deny every SSM parameter read outside `ssm_parameter_arn`. Default `true`. Set `false` only if this deployment's hosts genuinely read parameters outside that pattern — see below. | |
| 73 | | `route53_zone_ids` | Hosted zones the box may publish ACME DNS-01 challenge TXTs into (wildcard certs). Bare `Z...` ids; empty = no Route53 access. | |
| 74 | | `steering_zone_ids` | Hosted zones the DNS-steering converger may write latency A records into, plus the health checks that withdraw a dead edge. Bare `Z...` ids; empty = no steering grant (the default — steering also needs `CARLOS_STEERING_ZONE_ID` in the box env). | |
| 75 | | `steering_record_names` | Name bound on that grant, e.g. `["*.oncarlos.com"]`. Empty (default) grants A-record UPSERT/DELETE on the WHOLE zone, apex included — set it. | |
| 76 | | `ami_name_pattern` | AMI name filter (default `debian-13-arm64-*`). | |
| 77 | | `ami_owner` | AMI owner account (default `136693071363`, Debian). | |
| 78 | | `alerts_email` | Email for the host's `MemAvailablePercent < 15% for 3 min` alarm (memory-pressure spec §2.4). The module creates a same-region SNS topic + subscription per host (an alarm cannot notify a topic in another region). Empty (default): no topic, no alarm. Set it only once the box is publishing — missing data is treated as breaching. The `cloudwatch:PutMetricData` grant on namespace `Carlos/Host` is unconditional, so the agent's memwatch duty publishes either way. | |
| 79 | | `enable_auto_recovery` | Arm the EC2 auto-recovery alarm (HA ladder rung 0) — a system-status-check failure recovers the instance onto healthy hardware, same instance id, EIP and volumes. Default `true`; set `false` only where an external supervisor owns recovery. | |
| 80 | |
| 81 | Outputs: `eip_public_ip` (point your DNS here), `instance_id`, |
| 82 | `security_group_id` (for sibling services that admit fleet traffic |
| 83 | only), `role_arn`, `role_name`, `ami_id`. |
| 84 | |
| 85 | `README.md` in the module directory is the other half: what to put on the |
| 86 | box once tofu has made it, and what to tell it. That runbook used to live |
| 87 | only in the private infrastructure repo, which is how one deployment came to |
| 88 | triple its fleet with no committed way to bring a box up. |
| 89 | |
| 90 | ### Two things to know before you apply |
| 91 | |
| 92 | **Default VPCs.** With `vpc_id` unset the module builds the security group |
| 93 | in the region's default VPC and lets EC2 pick that VPC's default subnet. |
| 94 | Plenty of accounts no longer have one — deleting them is ordinary CIS |
| 95 | hygiene — and without a default VPC the apply fails **half-way through**, |
| 96 | after the buckets, the role and the instance profile already exist, with |
| 97 | `VPCIdNotSpecified: No default VPC for this user`. The module now checks at |
| 98 | plan time and tells you to pass `vpc_id`/`subnet_id`, which is the whole fix. |
| 99 | |
| 100 | **The SSM grant is wider than `ssm_parameter_arn` suggests.** Session Manager |
| 101 | is the only admin path to these boxes, so the module attaches the AWS-managed |
| 102 | `AmazonSSMManagedInstanceCore` policy — and that policy grants |
| 103 | `ssm:GetParameter*` on `Resource: "*"`. On its own, the module's own scoped |
| 104 | Allow bounds nothing: every host could read every SSM parameter in the |
| 105 | account, which on one deployment included the root release-signing key, |
| 106 | readable by an internet-facing edge box. `scope_ssm_parameter_reads` (default |
| 107 | `true`) adds an explicit Deny on the four parameter-read actions for |
| 108 | everything outside `ssm_parameter_arn`. It touches no other action, so |
| 109 | Session Manager is unaffected. If your boxes read a parameter outside that |
| 110 | pattern, widen the pattern rather than turning the deny off. |
| 111 | |
| 112 | Not a module: Tito's `svc/pdf` box stays in Tito's own root (`svc-pdf.tf`) — it |
| 113 | is deliberately deployment-local (a stateless Gotenberg whose SG admits |
| 114 | only the fleet, with *no* bucket or parameter grants; statelessness is |
| 115 | imposed, not promised). Generalise it only when a second deployment |
| 116 | wants one. |
| 117 | |
| 118 | ## scanner-host |
| 119 | |
| 120 | One box that runs clamd for the console's object-storage virus scanning, on its own host so signature updates and scan CPU stay off the control plane. The console streams each object to it over TCP 3310. The box reads no buckets and no parameters: its role has Session Manager and an explicit Deny on SSM parameter reads. The security group admits 3310 from the console box's group and nothing else, and there is no SSH. |
| 121 | |
| 122 | Port 3310 is open only while a self-test has passed against the clamd process running now. A replaced box gets a new private IP, so update the console's `CARLOS_CLAMD_ADDR` after any `-replace`. The module's `README.md` has the readiness check, four acceptance checks to run before wiring a console, and what a clean verdict does not cover. |
| 123 | |
| 124 | | Variable | Description | |
| 125 | |---|---| |
| 126 | | `name` | Name for the instance, its security group, its IAM role and its instance profile (e.g. `carlos-flagship-scanner`). | |
| 127 | | `ami_id` | Debian arm64 AMI. Pass the console box's own, `module.<host>.ami_id`. Ignored after creation: a newer image never replaces the box. | |
| 128 | | `client_security_group_id` | Security group of the one box allowed to reach clamd: the console's. clamd has no authentication, so this group is the whole fence. | |
| 129 | | `region` | AWS region the box runs in. The box fetches its SSM agent package from that region's bucket. | |
| 130 | | `instance_type` | arm64 instance type. Default `t4g.medium`. A `t4g.small` works only with `concurrent_database_reload = false`. | |
| 131 | | `stream_max_bytes` | clamd's `StreamMaxLength`, in bytes. Default `26214400` (25 MiB), between 8 MiB and 100 MiB. The console's `CARLOS_CLAMD_MAX_BYTES` must equal it. | |
| 132 | | `memory_max` | clamd's cgroup ceiling (systemd `MemoryMax`). Default `3G`. | |
| 133 | | `memory_high` | clamd's throttle point (systemd `MemoryHigh`). Default `2560M`. | |
| 134 | | `concurrent_database_reload` | Keep scanning while new signatures load, at the cost of holding both databases in memory. Default `true`. | |
| 135 | | `subnet_id` | Subnet to launch in. Null (default) is the default subnet of the default VPC, and is refused unless the client security group is in the default VPC. The subnet needs a route out. | |
| 136 | |
| 137 | Outputs: `clamd_addr` (set as the console's `CARLOS_CLAMD_ADDR`), `stream_max_bytes` (set as `CARLOS_CLAMD_MAX_BYTES`), `instance_id`, `security_group_id`. |
| 138 | |
| 139 | ## region-remote-host |
| 140 | |
| 141 | What an off-cloud CARLOS host (customer fleets and tito's hq) |
| 142 | needs from AWS when there is no EC2 instance profile to lean on: a region |
| 143 | data bucket plus a scoped IAM user + access key, limited to this region's |
| 144 | data bucket always, plus the deployment bucket only when |
| 145 | `deployment_bucket_arn` is set — the same wrong-region-fails-at-IAM |
| 146 | residency property `region-host` gives cloud hosts. Fleet-API consumers |
| 147 | (issue #73) omit `deployment_bucket_arn` entirely: the box reaches the |
| 148 | platform's deployment bucket over the console's edge with its tunnel |
| 149 | token, not a direct IAM grant. Consumed by customer fleets (paulca/pi-fleet |
| 150 | is the live example) and the platform's tito hq; the module is reusable |
| 151 | as-is for both. |
| 152 | |
| 153 | | Variable | Description | |
| 154 | |---|---| |
| 155 | | `region_code` | Short region code this host serves (e.g. `pi`, `hq`). | |
| 156 | | `data_bucket_name` | Name of this region's data bucket (created here). | |
| 157 | | `deployment_bucket_arn` | ARN of the existing deployment bucket the host reads releases/status from and writes heartbeats to. | |
| 158 | |
| 159 | Outputs: `bucket_name`, `access_key_id`, `secret_access_key` (sensitive). |
| 160 | |
| 161 | ## Self-hosting from your own repo |
| 162 | |
| 163 | A minimal root module — backend + providers + the two module blocks + a |
| 164 | DNS record. State backend, AWS profile, names, and DNS are yours; the |
| 165 | shapes are these modules. |
| 166 | |
| 167 | ```hcl |
| 168 | terraform { |
| 169 | required_version = ">= 1.10" |
| 170 | backend "s3" { |
| 171 | bucket = "my-tfstate-<account-id>" |
| 172 | key = "carlos/terraform.tfstate" |
| 173 | region = "eu-west-1" |
| 174 | use_lockfile = true |
| 175 | } |
| 176 | required_providers { |
| 177 | aws = { |
| 178 | source = "hashicorp/aws" |
| 179 | version = "~> 6.0" |
| 180 | } |
| 181 | } |
| 182 | } |
| 183 | |
| 184 | provider "aws" { |
| 185 | region = "eu-west-1" |
| 186 | } |
| 187 | |
| 188 | locals { account = "123456789012" } |
| 189 | |
| 190 | module "buckets" { |
| 191 | source = "git::https://github.com/carlosframework/platform.git//infra/modules/deployment-buckets?ref=main" |
| 192 | deployment_bucket_name = "carlos-myco-platform-${local.account}" |
| 193 | data_bucket_name = "carlos-myco-ie-${local.account}" |
| 194 | } |
| 195 | |
| 196 | module "host_ie" { |
| 197 | source = "git::https://github.com/carlosframework/platform.git//infra/modules/region-host?ref=main" |
| 198 | name = "carlos-myco-ie" |
| 199 | sg_description = "carlos myco IE host" |
| 200 | iam_name = "carlos-myco-host-ie" |
| 201 | instance_type = "t4g.small" |
| 202 | root_volume_size = 20 |
| 203 | user_data = file("${path.module}/host-init.sh") |
| 204 | deployment_bucket_arn = module.buckets.deployment_bucket_arn |
| 205 | data_bucket_arn = module.buckets.data_bucket_arn |
| 206 | ssm_parameter_arn = "arn:aws:ssm:eu-west-1:${local.account}:parameter/carlos/*" |
| 207 | } |
| 208 | |
| 209 | resource "aws_route53_record" "ie" { |
| 210 | zone_id = data.aws_route53_zone.mine.zone_id |
| 211 | name = "ie.carlos.example.com" |
| 212 | type = "A" |
| 213 | ttl = 300 |
| 214 | records = [module.host_ie.eip_public_ip] |
| 215 | } |
| 216 | ``` |
| 217 | |
| 218 | Then `tofu init && tofu plan`. Write your own `host-init.sh` — the module |
| 219 | takes its *content*, and the box-side steps it needs to end up performing are |
| 220 | in `modules/region-host/README.md`. |
| 221 | |
| 222 | **Pin `?ref=` to a commit, not to `main`.** The examples above say `main` so |
| 223 | that they keep working as the modules move, and that is exactly the wrong |
| 224 | setting for a live deployment: a routine `tofu apply` would adopt whatever |
| 225 | landed upstream since your last one. With one region host that is a small |
| 226 | risk; with six instantiating the same module it is a fleet-wide change |
| 227 | nobody decided to make. Pin it, and treat bumping the pin as its own |
| 228 | deliberate act: |
| 229 | |
| 230 | ```hcl |
| 231 | source = "git::https://github.com/carlosframework/platform.git//infra/modules/region-host?ref=f975b47cc88a7c033daab6453111298d3d86d401" |
| 232 | ``` |
| 233 | |
| 234 | ## More than one region |
| 235 | |
| 236 | Six regions is the same file six times if you write it that way. One |
| 237 | deployment measured its own: 743 lines across six region files, 118 each, of |
| 238 | which 41 differed — about 65% pure duplication, and the duplicated part |
| 239 | included a value that has to be **different in every file**. |
| 240 | |
| 241 | That value is `sibling_staging_arns`, whose only correctness property is |
| 242 | "every region's data bucket except my own". Maintained by hand it is O(n²) |
| 243 | edits with nothing checking them: adding one region means editing all six, |
| 244 | and the failure it invites — a region staging into its own bucket — is |
| 245 | silent. |
| 246 | |
| 247 | Both problems dissolve if regions are data rather than files. One map, one |
| 248 | `for_each` on each module block, and the mesh as a computed expression: |
| 249 | |
| 250 | ```hcl |
| 251 | locals { |
| 252 | account = "123456789012" |
| 253 | |
| 254 | # The only thing that changes when you add or remove a region. |
| 255 | regions = { |
| 256 | ie = { aws_region = "eu-west-1", instance_type = "t4g.medium" } |
| 257 | uk = { aws_region = "eu-west-2", instance_type = "t4g.small" } |
| 258 | au = { aws_region = "ap-southeast-2", instance_type = "t4g.small" } |
| 259 | } |
| 260 | } |
| 261 | |
| 262 | # One provider alias per region, because a bucket and a box are created in |
| 263 | # the region they live in. This is the one part that stays literal: provider |
| 264 | # blocks cannot be generated. |
| 265 | provider "aws" { |
| 266 | alias = "ie" |
| 267 | region = local.regions.ie.aws_region |
| 268 | } |
| 269 | # ...one per region. |
| 270 | |
| 271 | module "region" { |
| 272 | for_each = local.regions |
| 273 | source = "git::https://github.com/carlosframework/platform.git//infra/modules/region-host?ref=<sha>" |
| 274 | providers = { |
| 275 | aws = aws.${each.key} # see the note below |
| 276 | } |
| 277 | |
| 278 | name = "carlos-myco-${each.key}" |
| 279 | sg_description = "carlos myco ${upper(each.key)} host" |
| 280 | iam_name = "carlos-myco-host-${each.key}" |
| 281 | instance_type = each.value.instance_type |
| 282 | root_volume_size = 20 |
| 283 | user_data = file("${path.module}/host-init.sh") |
| 284 | |
| 285 | deployment_bucket_arn = module.buckets.deployment_bucket_arn |
| 286 | data_bucket_arn = aws_s3_bucket.data[each.key].arn |
| 287 | ssm_parameter_arn = "arn:aws:ssm:${each.value.aws_region}:${local.account}:parameter/carlos/*" |
| 288 | |
| 289 | # The mesh: every region's data bucket except this one's. Computed, so it |
| 290 | # cannot be wrong, and adding a region does not touch this line. |
| 291 | sibling_staging_arns = [ |
| 292 | for code, _ in local.regions : aws_s3_bucket.data[code].arn if code != each.key |
| 293 | ] |
| 294 | } |
| 295 | ``` |
| 296 | |
| 297 | Adding a region becomes a three-line data change instead of six file edits, |
| 298 | and a region staging into its own bucket becomes structurally impossible |
| 299 | rather than something you assert afterwards with a script. |
| 300 | |
| 301 | Two honest caveats. Provider aliases cannot be interpolated — `providers = { |
| 302 | aws = aws.${each.key} }` is not valid HCL — so in practice you either keep |
| 303 | one small module block per region that passes its own provider into a shared |
| 304 | inner module, or generate the root from your own templating. And |
| 305 | `deployment-buckets` makes a deployment+data *pair*, while a deployment has |
| 306 | exactly one deployment bucket, so the per-region data buckets in the sketch |
| 307 | above are plain `aws_s3_bucket` resources with the four settings |
| 308 | `deployment-buckets` would have applied (versioning, encryption, public |
| 309 | access block, ownership controls). |
| 310 | |
| 311 | Neither caveat changes the shape: regions are data, the mesh is computed, |
| 312 | and the thing you edit when the fleet changes is a map. |
| 313 | |
amadan