| 1 | # No-inbound box runbook (private-subnet posture) |
| 2 | |
| 3 | > **STATUS: written for Paul's applies. Nothing in this document has been |
| 4 | > applied, no tofu in it has been written into a `.tf` file, and no live box |
| 5 | > was touched by the branch that added it.** It documents a network shape |
| 6 | > for `carlos sidecars` design §8's `CARLOS_INBOUND=none` boxes; the module |
| 7 | > that would produce it does not exist in this repo yet — authoring it is |
| 8 | > Step 1 below, not a precondition already met. |
| 9 | |
| 10 | A `CARLOS_INBOUND=none` box accepts sidecar work only: no routed instance |
| 11 | traffic, no certs, no tunnel-served upstream, no public status entry. This |
| 12 | runbook covers the **hardest** version of that box — one that sits in a |
| 13 | private subnet with no route to the internet at all, egress only through a |
| 14 | NAT gateway, a security group with zero inbound rules, and no SSM agent, so |
| 15 | the box has no **inbound** surface at all, from any direction, by any |
| 16 | channel — no SSH, no SSM, nothing dialing in. It still has the platform's |
| 17 | own outbound surface: the fleet tunnel and the sidecar's own agent-socket |
| 18 | protocol keep working exactly as they do on any other fleet box (see "The |
| 19 | box env contract" below), and those two are what every verification in |
| 20 | this document ends up reading, because they are the only channels that |
| 21 | exist. |
| 22 | |
| 23 | ## Read this first: do you actually need this? |
| 24 | |
| 25 | **Probably not yet.** Fleet boxes in this estate are *already* |
| 26 | outbound-only dial — `infra/eleven-fleet.md` and `infra/remote-host.md` |
| 27 | describe the shape every existing fleet box runs today: a security group |
| 28 | with no inbound rules beyond what a normal host needs, admin reachability |
| 29 | through SSM (or none, off-cloud), and all platform traffic — heartbeats, |
| 30 | release pulls, the fleet tunnel — dialed **out**. Eleven's boxes in |
| 31 | `titogo` are that shape right now. Design §8 is explicit about this: the |
| 32 | private-subnet posture below is *available when wanted*, not a |
| 33 | prerequisite for running a sidecar. Eleven's first sidecar box needs |
| 34 | **no new network shape at all** — enrol it on the fleet box it already has, |
| 35 | per `infra/eleven-fleet.md`, and stop reading here. |
| 36 | |
| 37 | Reach for this runbook only when outbound-only-at-the-security-group-layer |
| 38 | isn't enough — when the requirement is "this workload must be provably |
| 39 | unreachable even if every SG rule and every credential on the box were |
| 40 | compromised," because there is, at the network layer, no path in to reach |
| 41 | it with. That is a stronger claim than "nothing dials it today," and it |
| 42 | costs a NAT gateway (an hourly + per-GB charge) and the loss of SSM |
| 43 | break-glass access — weigh that before applying anything below. |
| 44 | |
| 45 | ## Preconditions (must be true before Step 1) |
| 46 | |
| 47 | - [ ] **Sidecar members merged and deployed.** The `CARLOS_INBOUND` env |
| 48 | read (`internal/host/agent.go`), the placement half |
| 49 | (`internal/provision/sidecar_reconcile.go`, |
| 50 | `provision.PlacementNoInbound`), and the mint-time refusal on |
| 51 | no-inbound boxes (`internal/provision/reconcile.go`) all need to be |
| 52 | live on the box's `carlos` build — built fresh, not a stale |
| 53 | checkout (recorded trap, `carlos-cli-releases.md`). |
| 54 | - [ ] **The workload already exists as a fleet box**, following |
| 55 | `infra/eleven-fleet.md`'s Ceremony through Step 3 (fleet created, box |
| 56 | attached, `host.env` flipped and verified clean) — this runbook adds |
| 57 | network hardening around a box that already dials the right console |
| 58 | with the right credentials, it does not replace that ceremony. |
| 59 | - [ ] **A decision on which existing VPC/subnet plan hosts this.** The |
| 60 | sketch below assumes an existing VPC (e.g. `titogo`'s) with spare |
| 61 | CIDR room for one new private subnet and either a shared or new NAT |
| 62 | gateway. A from-scratch VPC is a bigger tofu change than this |
| 63 | runbook covers. |
| 64 | - [ ] **The sidecar's own record targets `placement: "no-inbound"`** (or |
| 65 | its app's manifest does) — `carlos sidecars enable -app <app> |
| 66 | -placement no-inbound` (or `carlos sidecars create ... ` under a |
| 67 | manifest already carrying it). Without this the record mints on any |
| 68 | box in the pool, including ones with inbound service, and the box |
| 69 | built below sits idle. |
| 70 | - [ ] **Nobody is depending on SSM Session Manager reaching this box.** |
| 71 | There is no admin channel to it once this ships except the fleet |
| 72 | tunnel and the sidecar's own agent-socket protocol (design §7.1) — |
| 73 | confirm that is acceptable for this workload before Step 3. |
| 74 | |
| 75 | ## The tofu shape |
| 76 | |
| 77 | Four pieces, applied in order because each one's Verify depends on the |
| 78 | last existing. **No module in this repo produces this shape yet** — the |
| 79 | HCL sketches below name the resources it needs, in this repo's existing |
| 80 | naming conventions (`infra/modules/region-host/main.tf`, |
| 81 | `infra/modules/region-remote-host/main.tf`), for whoever authors the |
| 82 | module; they are not a file to copy-paste and apply as-is. |
| 83 | |
| 84 | **1. NAT gateway (skip if one already serves this VPC)** |
| 85 | |
| 86 | A NAT gateway must sit in an existing **public** subnet — one that already |
| 87 | has a route to an internet gateway — even though nothing routes to it from |
| 88 | outside. It is the private subnet's only way out. |
| 89 | |
| 90 | resource "aws_eip" "nat" { |
| 91 | domain = "vpc" |
| 92 | } |
| 93 | |
| 94 | resource "aws_nat_gateway" "sidecar" { |
| 95 | allocation_id = aws_eip.nat.id |
| 96 | subnet_id = <existing public subnet id> |
| 97 | } |
| 98 | |
| 99 | *Verify:* `aws ec2 describe-nat-gateways` (or the tofu plan/apply output) |
| 100 | shows state `available`. This step alone costs money and touches no box — |
| 101 | safe to apply and sit on. |
| 102 | *Abort:* `tofu destroy -target aws_nat_gateway.sidecar -target aws_eip.nat` |
| 103 | before anything below references it. Nothing yet depends on it. |
| 104 | |
| 105 | **2. Private subnet + route table (no IGW route)** |
| 106 | |
| 107 | The defining property: this subnet's route table has **no** `0.0.0.0/0 -> |
| 108 | igw-...` route. Its only default route is to the NAT gateway from Step 1. |
| 109 | |
| 110 | resource "aws_subnet" "sidecar_private" { |
| 111 | vpc_id = <vpc id> |
| 112 | cidr_block = <a free /28 or larger in this VPC's CIDR> |
| 113 | availability_zone = <az the NAT gateway's subnet is in, for cost — cross-AZ NAT traffic bills extra> |
| 114 | } |
| 115 | |
| 116 | resource "aws_route_table" "sidecar_private" { |
| 117 | vpc_id = <vpc id> |
| 118 | route { |
| 119 | cidr_block = "0.0.0.0/0" |
| 120 | nat_gateway_id = aws_nat_gateway.sidecar.id |
| 121 | } |
| 122 | # No route to an aws_internet_gateway, ever — that absence IS the |
| 123 | # posture. Do not add one "temporarily" to debug something; that |
| 124 | # defeats this runbook's entire point for as long as it's there. |
| 125 | } |
| 126 | |
| 127 | resource "aws_route_table_association" "sidecar_private" { |
| 128 | subnet_id = aws_subnet.sidecar_private.id |
| 129 | route_table_id = aws_route_table.sidecar_private.id |
| 130 | } |
| 131 | |
| 132 | *Verify:* `aws ec2 describe-route-tables --route-table-ids |
| 133 | <id>` — confirm the route list contains exactly one `0.0.0.0/0` entry and |
| 134 | its target is the NAT gateway, not `igw-...`. This is the single fact this |
| 135 | whole runbook exists to guarantee; check it by reading the table, not by |
| 136 | trusting the plan. |
| 137 | *Abort:* the subnet and route table have nothing in them yet (Step 3 |
| 138 | hasn't placed an instance) — `tofu destroy -target` both, no downstream |
| 139 | cleanup. |
| 140 | |
| 141 | **3. Security group — zero inbound** |
| 142 | |
| 143 | Stricter than `region-host`'s SG (which still opens 80/443): this box |
| 144 | serves no certs and no tunnel-terminated traffic, so it has nothing to |
| 145 | listen on at all. |
| 146 | |
| 147 | resource "aws_security_group" "sidecar_no_inbound" { |
| 148 | name = "<box name>-no-inbound" |
| 149 | description = "sidecar-only box: no ingress by design" |
| 150 | vpc_id = <vpc id> |
| 151 | # No ingress block. Not "ingress restricted" — no ingress rule |
| 152 | # exists to restrict. Do not add var.extra_ingress-style exceptions |
| 153 | # to this SG; a box that needs an inbound rule is not this class of |
| 154 | # box. |
| 155 | egress { |
| 156 | from_port = 0 |
| 157 | to_port = 0 |
| 158 | protocol = "-1" |
| 159 | cidr_blocks = ["0.0.0.0/0"] |
| 160 | } |
| 161 | } |
| 162 | |
| 163 | *Verify:* `aws ec2 describe-security-groups --group-ids <id>` shows an |
| 164 | empty `IpPermissions` list (ingress) and the single egress rule. |
| 165 | *Abort:* unattached SG, `tofu destroy -target` it directly. |
| 166 | |
| 167 | **4. The instance — no SSM** |
| 168 | |
| 169 | Same AMI/instance-type shape as `infra/modules/region-host`, placed in the |
| 170 | Step 2 subnet with the Step 3 SG, **no public IP**, and — the deliberate |
| 171 | omission — no `aws_iam_role_policy_attachment` for |
| 172 | `AmazonSSMManagedInstanceCore`. Without a route to the internet or a VPC |
| 173 | interface endpoint for SSM (which this runbook does not add — an endpoint |
| 174 | is itself a reachable surface), the SSM agent could not dial its service |
| 175 | even if installed, so leaving the IAM grant off is not extra caution, it |
| 176 | is removing a credential that would do nothing but sit there. **This means |
| 177 | there is no break-glass path onto this box, ever, by anyone, other than |
| 178 | what the sidecar and fleet-tunnel protocols carry.** That is the point, |
| 179 | not an oversight — confirm the last Precondition checkbox again before |
| 180 | applying this step. |
| 181 | |
| 182 | resource "aws_instance" "sidecar" { |
| 183 | ami = <same arm64 Debian AMI region-host resolves> |
| 184 | instance_type = <same class region-host's boxes use> |
| 185 | subnet_id = aws_subnet.sidecar_private.id |
| 186 | vpc_security_group_ids = [aws_security_group.sidecar_no_inbound.id] |
| 187 | associate_public_ip_address = false |
| 188 | iam_instance_profile = <a role scoped like region-remote-host's, |
| 189 | WITHOUT ssm:* and WITHOUT |
| 190 | AmazonSSMManagedInstanceCore> |
| 191 | user_data = <same bootstrap script shape as region-host> |
| 192 | } |
| 193 | |
| 194 | *Verify:* the instance reaches `running` in `describe-instances`, and — the |
| 195 | important negative check — `aws ssm describe-instance-information` does |
| 196 | **not** list it, ever, even minutes later. If it appears there, something |
| 197 | above is wrong (a stray endpoint, a stray IGW route, or the SSM agent |
| 198 | running with a grant it shouldn't have) — treat that as a failed apply, |
| 199 | not a bonus. |
| 200 | *Abort:* terminate the instance (`tofu destroy -target |
| 201 | aws_instance.sidecar`); nothing else in Steps 1-3 depended on it existing. |
| 202 | |
| 203 | ## The box env contract |
| 204 | |
| 205 | `/etc/carlos/host.env` on this box carries the same fleet contract |
| 206 | `infra/eleven-fleet.md` Step 3 stages, **plus** the one new variable that |
| 207 | makes this a no-inbound box: |
| 208 | |
| 209 | AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials |
| 210 | CARLOS_HIBERNATE_BUCKET=<the fleet's data bucket> |
| 211 | CARLOS_FLEET=<fleet name> |
| 212 | CARLOS_FLEET_BOX=<this box's label> |
| 213 | CARLOS_REGION=fleet/<fleet name> |
| 214 | CARLOS_STATUS_PREFIX=fleets/<fleet name>/ |
| 215 | CARLOS_RELEASE_PUBKEY=<the console's release public key> |
| 216 | CARLOS_TUNNEL_URL=https://<the console's reserved tunnel hostname> |
| 217 | CARLOS_TUNNEL_NAME=<fleet name>/<this box's label> |
| 218 | CARLOS_TUNNEL_TOKEN=<from the fleet's add-box call — shown once> |
| 219 | CARLOS_INBOUND=none |
| 220 | |
| 221 | `CARLOS_INBOUND=none` is read wherever `CARLOS_REGION`/`CARLOS_POOLS` are |
| 222 | (`internal/host/agent.go`), folded case/whitespace-insensitively |
| 223 | (`provision.NormalizeInbound`), and — deliberately — any value that is |
| 224 | neither empty, `any`, nor `none` is treated as `none`: an unrecognised |
| 225 | setting is a typo, and the restrictive reading is the safe one. Getting |
| 226 | this var wrong fails closed (a box that should serve nothing accidentally |
| 227 | stays inert) rather than open. |
| 228 | |
| 229 | The NAT gateway makes `CARLOS_TUNNEL_URL`'s dial-out (and the release/ |
| 230 | heartbeat traffic to `CARLOS_HIBERNATE_BUCKET`) work exactly the way it |
| 231 | does on any other fleet box — this box's whole platform surface is still |
| 232 | outbound, it just has no *possible* inbound path underneath it, where an |
| 233 | ordinary fleet box merely has no *configured* one. |
| 234 | |
| 235 | ## Verify the box came up and adopted its sidecars |
| 236 | |
| 237 | This box has no shell, no SSM, and — deliberately — no path into the |
| 238 | platform's own log stream either: `internal/activator/logsink.go`'s |
| 239 | `Event` calls are the only thing that ever writes to |
| 240 | `CARLOS_STATUS_PREFIX`'s log sink, and nothing in `internal/host/agent.go`, |
| 241 | `internal/tunnelclient` (the source of the "tunnel connected" line), or |
| 242 | `internal/provision` (the source of a mint-time log line) calls it. So "go |
| 243 | read a log line" is not an available verb for this box, on-box or |
| 244 | off — every verification below uses a surface the platform already exposes |
| 245 | with no box access of any kind: |
| 246 | |
| 247 | 1. **`carlos sidecars ls -app <app> -long`** — the strongest single check. |
| 248 | `HOST` names a route, `PLACEMENT` reads `no-inbound`, and once the |
| 249 | child has called `/v1/ready` at least once, `VERSION` reads the running |
| 250 | build instead of `unknown` (design §9.2's honesty rule: an absent |
| 251 | version beats an overclaimed one). A reported version proves the whole |
| 252 | chain end to end in one command — the box booted, dialed out, adopted |
| 253 | the record, spawned the child, and the child called back over the |
| 254 | agent socket. This is the check that matters most; do it first. |
| 255 | 2. **`carlos fleets ls -account <acct>`** proves the box is |
| 256 | **registered**, nothing more. Its `STATUS` column is |
| 257 | `registry.FleetBox.Status` — an `active`/`detached` membership flag set |
| 258 | at `add-box` time (`internal/console/fleets.go`), not a heartbeat. |
| 259 | `infra/eleven-fleet.md`'s own Step 2 shows this plainly: it reads |
| 260 | `active` immediately after `add-box`, before the box has ever booted. |
| 261 | Reading `active` here tells you the token exists and hasn't been |
| 262 | detached; it tells you nothing about whether the box has ever dialed |
| 263 | in. Do not stop at this step and conclude the box came up. |
| 264 | 3. **The console fleet page** |
| 265 | (`/a/<account-sqid>/fleets/<fleet name>`) is the one surface that |
| 266 | actually renders heartbeat freshness — `internal/console/fleets.go` |
| 267 | joins each box's registry row to its own heartbeat by label and shows |
| 268 | it Reachable (a fresh heartbeat joined) or gives its Age (a heartbeat |
| 269 | joined but has gone stale) or neither (no heartbeat has ever joined at |
| 270 | all). This is a browser-only check — no CLI verb surfaces it as of this |
| 271 | branch — but it needs no box access, only a signed-in session, so it |
| 272 | still fits this runbook's "no box commands" constraint. |
| 273 | 4. **`aws ec2 get-console-output --instance-id <id>`** if 1 and 3 both |
| 274 | stay stale past a reasonable wait and you need to see the box actually |
| 275 | trying to boot. This is the one channel that needs no reachable surface |
| 276 | on the box at all — it reads whatever the instance wrote to its serial |
| 277 | console (cloud-init/systemd boot log) from the AWS API side, not the |
| 278 | box's. It is the sole operator-side breadcrumb for a box with neither |
| 279 | SSH nor SSM, and the reason to leave the AMI's normal verbose boot |
| 280 | logging on rather than quieting it for this box class. |
| 281 | |
| 282 | There is no honest check for "did the provisioning pass log a mint-time |
| 283 | refusal or success" from off the box — if 1 and 3 both look right but the |
| 284 | record still isn't where you expect, the *Abort* below is the next step, |
| 285 | not a log line to go find. |
| 286 | |
| 287 | *Abort (this section):* if the sidecar never mints here, check placement |
| 288 | before network: `carlos sidecars ls -app <app> -long` showing |
| 289 | `PLACEMENT any` means the record itself never asked for `no-inbound` — fix |
| 290 | the manifest/record, not the box. If placement is right and it still |
| 291 | doesn't mint, there is no way to inspect the running process's actual |
| 292 | environment on this box (no shell, no SSM) — treat `CARLOS_INBOUND` as |
| 293 | unfalsifiable in place. The fix is the same one Rollback summary's last |
| 294 | entry describes for this box class: there is no in-place restage, so |
| 295 | terminate the instance and re-apply Step 4 with the value double-checked |
| 296 | in its user-data, then re-run checks 1 and 3 above against the replacement. |
| 297 | |
| 298 | ## Flipping an EXISTING box to `CARLOS_INBOUND=none` |
| 299 | |
| 300 | **This does not evacuate the instances the box already serves.** Setting |
| 301 | `CARLOS_INBOUND=none` on a box that currently runs routed instances (or |
| 302 | was reachable and had inbound work placed on it) only refuses **new** |
| 303 | routed mints from then on — it is a mint-time check |
| 304 | (`internal/provision/reconcile.go`'s `case !ok:` branch), not a converge- |
| 305 | time or removal-time one. Every route already `Provisioned` on that box |
| 306 | keeps running, keeps converging, keeps being served, exactly as before. |
| 307 | |
| 308 | This is deliberate, not an oversight: making the flag evacuate the box |
| 309 | would mean a single typo in `host.env` — a fat-fingered `CARLOS_INBOUND=none` |
| 310 | meant for a different box, or a stray copy-paste — could take down every |
| 311 | app the box was serving, with no confirmation step and no way to see it |
| 312 | coming. A box that stops *accepting new work* loudly logs every refusal; |
| 313 | a box that *stops serving existing work* on the same typo is a production |
| 314 | incident with the blast radius of whatever happened to be running there. |
| 315 | |
| 316 | **So: drain the box before flipping it, not after.** Move its existing |
| 317 | routed instances off (`carlos apps place` to another target, or whatever |
| 318 | this box's own placement mechanism is) and confirm they've actually |
| 319 | stopped being served here — a `curl -sI --resolve` against each affected |
| 320 | host showing it answered from elsewhere, not just that placement changed |
| 321 | — before staging the new `host.env`. Only once the box carries no routed |
| 322 | work should `CARLOS_INBOUND=none` land on it; at that point it is a |
| 323 | statement of fact about the box, not an instruction to make it one. |
| 324 | |
| 325 | ## Rollback summary |
| 326 | |
| 327 | Every step above has its own Abort; in order of how far you got: |
| 328 | |
| 329 | - Only Steps 1-3 applied, no instance yet: `tofu destroy -target` each |
| 330 | resource in reverse order (SG, route table + association, subnet, NAT |
| 331 | gateway + EIP) — nothing else references them. |
| 332 | - Instance applied, box never adopted anything: `tofu destroy -target |
| 333 | aws_instance.sidecar`, then the network resources as above. |
| 334 | - Box adopted a sidecar, now going away: detach it from the fleet first |
| 335 | (`carlos fleets detach-box -account <acct> <fleet> <box label>` — the |
| 336 | hard cut, per `infra/eleven-fleet.md`'s own Abort for that step) so the |
| 337 | console stops expecting heartbeats from it, then terminate the instance |
| 338 | and tear down the network resources. |
| 339 | - Flipped an existing box and want to revert to accepting inbound work |
| 340 | again: restore the prior `host.env` (drop `CARLOS_INBOUND=none`, or set |
| 341 | it back to `any`), restart the agent unit — this is a converge-time |
| 342 | read, so the very next pass allows new mints again. **This step assumes |
| 343 | the ordinary shell/SSM access an existing fleet box normally has.** It |
| 344 | does NOT apply to a box built via this runbook's own Step 4, which has |
| 345 | neither: there is no in-place way to edit that box's `host.env` at all. |
| 346 | Reverting one of those means replacing it — terminate the instance and |
| 347 | re-apply Step 4 with the desired `host.env` baked into its user-data — |
| 348 | not restaging a file on a running box. It does not retro- |
| 349 | actively place anything that was moved off during the drain; that's a |
| 350 | separate `carlos apps place` back, a deliberate decision, not a side |
| 351 | effect of the env flip. |
| 352 | |
amadan