| new file mode 100644 |
| index 000000000..e16ae23ae |
| --- /dev/null |
| +++ b/infra/no-inbound-box.md |
| @@ -0,0 +1,306 @@ |
| +# No-inbound box runbook (private-subnet posture) |
| + |
| +> **STATUS: written for Paul's applies. Nothing in this document has been |
| +> applied, no tofu in it has been written into a `.tf` file, and no live box |
| +> was touched by the branch that added it.** It documents a network shape |
| +> for `carlos sidecars` design §8's `CARLOS_INBOUND=none` boxes; the module |
| +> that would produce it does not exist in this repo yet — authoring it is |
| +> Step 1 below, not a precondition already met. |
| + |
| +A `CARLOS_INBOUND=none` box accepts sidecar work only: no routed instance |
| +traffic, no certs, no tunnel-served upstream, no public status entry. This |
| +runbook covers the **hardest** version of that box — one that sits in a |
| +private subnet with no route to the internet at all, egress only through a |
| +NAT gateway, a security group with zero inbound rules, and no SSM agent, so |
| +the box has no reachable surface whatsoever, from any direction, by any |
| +channel. |
| + |
| +## Read this first: do you actually need this? |
| + |
| +**Probably not yet.** Fleet boxes in this estate are *already* |
| +outbound-only dial — `infra/eleven-fleet.md` and `infra/remote-host.md` |
| +describe the shape every existing fleet box runs today: a security group |
| +with no inbound rules beyond what a normal host needs, admin reachability |
| +through SSM (or none, off-cloud), and all platform traffic — heartbeats, |
| +release pulls, the fleet tunnel — dialed **out**. Eleven's boxes in |
| +`titogo` are that shape right now. Design §8 is explicit about this: the |
| +private-subnet posture below is *available when wanted*, not a |
| +prerequisite for running a sidecar. Eleven's first sidecar box needs |
| +**no new network shape at all** — enrol it on the fleet box it already has, |
| +per `infra/eleven-fleet.md`, and stop reading here. |
| + |
| +Reach for this runbook only when outbound-only-at-the-security-group-layer |
| +isn't enough — when the requirement is "this workload must be provably |
| +unreachable even if every SG rule and every credential on the box were |
| +compromised," because there is, at the network layer, no path in to reach |
| +it with. That is a stronger claim than "nothing dials it today," and it |
| +costs a NAT gateway (an hourly + per-GB charge) and the loss of SSM |
| +break-glass access — weigh that before applying anything below. |
| + |
| +## Preconditions (must be true before Step 1) |
| + |
| +- [ ] **Sidecar members merged and deployed.** The `CARLOS_INBOUND` env |
| + read (`internal/host/agent.go`), the placement half |
| + (`internal/provision/sidecar_reconcile.go`, |
| + `provision.PlacementNoInbound`), and the mint-time refusal on |
| + no-inbound boxes (`internal/provision/reconcile.go`) all need to be |
| + live on the box's `carlos` build — built fresh, not a stale |
| + checkout (recorded trap, `carlos-cli-releases.md`). |
| +- [ ] **The workload already exists as a fleet box**, following |
| + `infra/eleven-fleet.md`'s Ceremony through Step 3 (fleet created, box |
| + attached, `host.env` flipped and verified clean) — this runbook adds |
| + network hardening around a box that already dials the right console |
| + with the right credentials, it does not replace that ceremony. |
| +- [ ] **A decision on which existing VPC/subnet plan hosts this.** The |
| + sketch below assumes an existing VPC (e.g. `titogo`'s) with spare |
| + CIDR room for one new private subnet and either a shared or new NAT |
| + gateway. A from-scratch VPC is a bigger tofu change than this |
| + runbook covers. |
| +- [ ] **The sidecar's own record targets `placement: "no-inbound"`** (or |
| + its app's manifest does) — `carlos sidecars enable -app <app> |
| + -placement no-inbound` (or `carlos sidecars create ... ` under a |
| + manifest already carrying it). Without this the record mints on any |
| + box in the pool, including ones with inbound service, and the box |
| + built below sits idle. |
| +- [ ] **Nobody is depending on SSM Session Manager reaching this box.** |
| + There is no admin channel to it once this ships except the fleet |
| + tunnel and the sidecar's own agent-socket protocol (design §7.1) — |
| + confirm that is acceptable for this workload before Step 3. |
| + |
| +## The tofu shape |
| + |
| +Four pieces, applied in order because each one's Verify depends on the |
| +last existing. **No module in this repo produces this shape yet** — the |
| +HCL sketches below name the resources it needs, in this repo's existing |
| +naming conventions (`infra/modules/region-host/main.tf`, |
| +`infra/modules/region-remote-host/main.tf`), for whoever authors the |
| +module; they are not a file to copy-paste and apply as-is. |
| + |
| +**1. NAT gateway (skip if one already serves this VPC)** |
| + |
| +A NAT gateway must sit in an existing **public** subnet — one that already |
| +has a route to an internet gateway — even though nothing routes to it from |
| +outside. It is the private subnet's only way out. |
| + |
| + resource "aws_eip" "nat" { |
| + domain = "vpc" |
| + } |
| + |
| + resource "aws_nat_gateway" "sidecar" { |
| + allocation_id = aws_eip.nat.id |
| + subnet_id = <existing public subnet id> |
| + } |
| + |
| +*Verify:* `aws ec2 describe-nat-gateways` (or the tofu plan/apply output) |
| +shows state `available`. This step alone costs money and touches no box — |
| +safe to apply and sit on. |
| +*Abort:* `tofu destroy -target aws_nat_gateway.sidecar -target aws_eip.nat` |
| +before anything below references it. Nothing yet depends on it. |
| + |
| +**2. Private subnet + route table (no IGW route)** |
| + |
| +The defining property: this subnet's route table has **no** `0.0.0.0/0 -> |
| +igw-...` route. Its only default route is to the NAT gateway from Step 1. |
| + |
| + resource "aws_subnet" "sidecar_private" { |
| + vpc_id = <vpc id> |
| + cidr_block = <a free /28 or larger in this VPC's CIDR> |
| + availability_zone = <az the NAT gateway's subnet is in, for cost — cross-AZ NAT traffic bills extra> |
| + } |
| + |
| + resource "aws_route_table" "sidecar_private" { |
| + vpc_id = <vpc id> |
| + route { |
| + cidr_block = "0.0.0.0/0" |
| + nat_gateway_id = aws_nat_gateway.sidecar.id |
| + } |
| + # No route to an aws_internet_gateway, ever — that absence IS the |
| + # posture. Do not add one "temporarily" to debug something; that |
| + # defeats this runbook's entire point for as long as it's there. |
| + } |
| + |
| + resource "aws_route_table_association" "sidecar_private" { |
| + subnet_id = aws_subnet.sidecar_private.id |
| + route_table_id = aws_route_table.sidecar_private.id |
| + } |
| + |
| +*Verify:* `aws ec2 describe-route-tables --route-table-ids |
| +<id>` — confirm the route list contains exactly one `0.0.0.0/0` entry and |
| +its target is the NAT gateway, not `igw-...`. This is the single fact this |
| +whole runbook exists to guarantee; check it by reading the table, not by |
| +trusting the plan. |
| +*Abort:* the subnet and route table have nothing in them yet (Step 3 |
| +hasn't placed an instance) — `tofu destroy -target` both, no downstream |
| +cleanup. |
| + |
| +**3. Security group — zero inbound** |
| + |
| +Stricter than `region-host`'s SG (which still opens 80/443): this box |
| +serves no certs and no tunnel-terminated traffic, so it has nothing to |
| +listen on at all. |
| + |
| + resource "aws_security_group" "sidecar_no_inbound" { |
| + name = "<box name>-no-inbound" |
| + description = "sidecar-only box: no ingress by design" |
| + vpc_id = <vpc id> |
| + # No ingress block. Not "ingress restricted" — no ingress rule |
| + # exists to restrict. Do not add var.extra_ingress-style exceptions |
| + # to this SG; a box that needs an inbound rule is not this class of |
| + # box. |
| + egress { |
| + from_port = 0 |
| + to_port = 0 |
| + protocol = "-1" |
| + cidr_blocks = ["0.0.0.0/0"] |
| + } |
| + } |
| + |
| +*Verify:* `aws ec2 describe-security-groups --group-ids <id>` shows an |
| +empty `IpPermissions` list (ingress) and the single egress rule. |
| +*Abort:* unattached SG, `tofu destroy -target` it directly. |
| + |
| +**4. The instance — no SSM** |
| + |
| +Same AMI/instance-type shape as `infra/modules/region-host`, placed in the |
| +Step 2 subnet with the Step 3 SG, **no public IP**, and — the deliberate |
| +omission — no `aws_iam_role_policy_attachment` for |
| +`AmazonSSMManagedInstanceCore`. Without a route to the internet or a VPC |
| +interface endpoint for SSM (which this runbook does not add — an endpoint |
| +is itself a reachable surface), the SSM agent could not dial its service |
| +even if installed, so leaving the IAM grant off is not extra caution, it |
| +is removing a credential that would do nothing but sit there. **This means |
| +there is no break-glass path onto this box, ever, by anyone, other than |
| +what the sidecar and fleet-tunnel protocols carry.** That is the point, |
| +not an oversight — confirm the last Precondition checkbox again before |
| +applying this step. |
| + |
| + resource "aws_instance" "sidecar" { |
| + ami = <same arm64 Debian AMI region-host resolves> |
| + instance_type = <same class region-host's boxes use> |
| + subnet_id = aws_subnet.sidecar_private.id |
| + vpc_security_group_ids = [aws_security_group.sidecar_no_inbound.id] |
| + associate_public_ip_address = false |
| + iam_instance_profile = <a role scoped like region-remote-host's, |
| + WITHOUT ssm:* and WITHOUT |
| + AmazonSSMManagedInstanceCore> |
| + user_data = <same bootstrap script shape as region-host> |
| + } |
| + |
| +*Verify:* the instance reaches `running` in `describe-instances`, and — the |
| +important negative check — `aws ssm describe-instance-information` does |
| +**not** list it, ever, even minutes later. If it appears there, something |
| +above is wrong (a stray endpoint, a stray IGW route, or the SSM agent |
| +running with a grant it shouldn't have) — treat that as a failed apply, |
| +not a bonus. |
| +*Abort:* terminate the instance (`tofu destroy -target |
| +aws_instance.sidecar`); nothing else in Steps 1-3 depended on it existing. |
| + |
| +## The box env contract |
| + |
| +`/etc/carlos/host.env` on this box carries the same fleet contract |
| +`infra/eleven-fleet.md` Step 3 stages, **plus** the one new variable that |
| +makes this a no-inbound box: |
| + |
| + AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials |
| + CARLOS_HIBERNATE_BUCKET=<the fleet's data bucket> |
| + CARLOS_FLEET=<fleet name> |
| + CARLOS_FLEET_BOX=<this box's label> |
| + CARLOS_REGION=fleet/<fleet name> |
| + CARLOS_STATUS_PREFIX=fleets/<fleet name>/ |
| + CARLOS_RELEASE_PUBKEY=<the console's release public key> |
| + CARLOS_TUNNEL_URL=https://<the console's reserved tunnel hostname> |
| + CARLOS_TUNNEL_NAME=<fleet name>/<this box's label> |
| + CARLOS_TUNNEL_TOKEN=<from the fleet's add-box call — shown once> |
| + CARLOS_INBOUND=none |
| + |
| +`CARLOS_INBOUND=none` is read wherever `CARLOS_REGION`/`CARLOS_POOLS` are |
| +(`internal/host/agent.go`), folded case/whitespace-insensitively |
| +(`provision.NormalizeInbound`), and — deliberately — any value that is |
| +neither empty, `any`, nor `none` is treated as `none`: an unrecognised |
| +setting is a typo, and the restrictive reading is the safe one. Getting |
| +this var wrong fails closed (a box that should serve nothing accidentally |
| +stays inert) rather than open. |
| + |
| +The NAT gateway makes `CARLOS_TUNNEL_URL`'s dial-out (and the release/ |
| +heartbeat traffic to `CARLOS_HIBERNATE_BUCKET`) work exactly the way it |
| +does on any other fleet box — this box's whole platform surface is still |
| +outbound, it just has no *possible* inbound path underneath it, where an |
| +ordinary fleet box merely has no *configured* one. |
| + |
| +## Verify the box came up and adopted its sidecars |
| + |
| +1. `journalctl -u carlos-edge` on the box (if you still have a channel to |
| + read it — see the SSM note above; this may mean reading it once before |
| + Step 4 removes the grant, or reading it from the box's own logs once |
| + they reach the fleet's data bucket) shows "tunnel connected". |
| +2. The fleet page (`/a/<account-sqid>/fleets/<fleet name>`) on the |
| + console shows this box with a fresh heartbeat. |
| +3. `carlos sidecars ls -app <app> -long` shows the sidecar's `HOST` column |
| + naming a route, `PLACEMENT` reading `no-inbound`, and — once it has |
| + called `/v1/ready` at least once — `VERSION` reading the running build |
| + rather than `unknown` (design §9.2's honesty rule: an absent version |
| + beats an overclaimed one, so `unknown` here is not a bug, it just means |
| + the sidecar hasn't reported in yet). |
| +4. The box's own log line confirms placement worked, not just that a |
| + record exists: `provision: minted sidecar route ... on box declaring |
| + CARLOS_INBOUND=none` (or equivalent — check the actual log text at |
| + apply time) in the box's provisioning-pass log output, once the fleet's |
| + serial has moved and the box has polled it. |
| + |
| +*Abort (this section):* if the sidecar never mints here, check placement |
| +before network: `carlos sidecars ls -app <app> -long` showing |
| +`PLACEMENT any` means the record itself never asked for `no-inbound` — fix |
| +the manifest/record, not the box. If placement is right and it still |
| +doesn't mint, confirm `CARLOS_INBOUND=none` actually reached the running |
| +process (`ps`/`/proc/<pid>/environ` from whatever channel you still have, |
| +or a fresh `host.env` re-stage plus agent restart if the value was typoed |
| +on write). |
| + |
| +## Flipping an EXISTING box to `CARLOS_INBOUND=none` |
| + |
| +**This does not evacuate the instances the box already serves.** Setting |
| +`CARLOS_INBOUND=none` on a box that currently runs routed instances (or |
| +was reachable and had inbound work placed on it) only refuses **new** |
| +routed mints from then on — it is a mint-time check |
| +(`internal/provision/reconcile.go`'s `case !ok:` branch), not a converge- |
| +time or removal-time one. Every route already `Provisioned` on that box |
| +keeps running, keeps converging, keeps being served, exactly as before. |
| + |
| +This is deliberate, not an oversight: making the flag evacuate the box |
| +would mean a single typo in `host.env` — a fat-fingered `CARLOS_INBOUND=none` |
| +meant for a different box, or a stray copy-paste — could take down every |
| +app the box was serving, with no confirmation step and no way to see it |
| +coming. A box that stops *accepting new work* loudly logs every refusal; |
| +a box that *stops serving existing work* on the same typo is a production |
| +incident with the blast radius of whatever happened to be running there. |
| + |
| +**So: drain the box before flipping it, not after.** Move its existing |
| +routed instances off (`carlos apps place` to another target, or whatever |
| +this box's own placement mechanism is) and confirm they've actually |
| +stopped being served here — a `curl -sI --resolve` against each affected |
| +host showing it answered from elsewhere, not just that placement changed |
| +— before staging the new `host.env`. Only once the box carries no routed |
| +work should `CARLOS_INBOUND=none` land on it; at that point it is a |
| +statement of fact about the box, not an instruction to make it one. |
| + |
| +## Rollback summary |
| + |
| +Every step above has its own Abort; in order of how far you got: |
| + |
| +- Only Steps 1-3 applied, no instance yet: `tofu destroy -target` each |
| + resource in reverse order (SG, route table + association, subnet, NAT |
| + gateway + EIP) — nothing else references them. |
| +- Instance applied, box never adopted anything: `tofu destroy -target |
| + aws_instance.sidecar`, then the network resources as above. |
| +- Box adopted a sidecar, now going away: detach it from the fleet first |
| + (`carlos fleets detach-box -account <acct> <fleet> <box label>` — the |
| + hard cut, per `infra/eleven-fleet.md`'s own Abort for that step) so the |
| + console stops expecting heartbeats from it, then terminate the instance |
| + and tear down the network resources. |
| +- Flipped an existing box and want to revert to accepting inbound work |
| + again: restore the prior `host.env` (drop `CARLOS_INBOUND=none`, or set |
| + it back to `any`), restart the agent unit — this is a converge-time |
| + read, so the very next pass allows new mints again. It does not retro- |
| + actively place anything that was moved off during the drain; that's a |
| + separate `carlos apps place` back, a deliberate decision, not a side |
| + effect of the env flip. |
amadan