No-inbound box runbook (private-subnet posture)

> STATUS: written for Paul's applies. Nothing in this document has been > applied, no tofu in it has been written into a .tf file, and no live box > was touched by the branch that added it. It documents a network shape > for carlos sidecars design §8's CARLOS_INBOUND=none boxes; the module > that would produce it does not exist in this repo yet — authoring it is > Step 1 below, not a precondition already met.

A CARLOS_INBOUND=none box accepts sidecar work only: no routed instance traffic, no certs, no tunnel-served upstream, no public status entry. This runbook covers the hardest version of that box — one that sits in a private subnet with no route to the internet at all, egress only through a NAT gateway, a security group with zero inbound rules, and no SSM agent, so the box has no inbound surface at all, from any direction, by any channel — no SSH, no SSM, nothing dialing in. It still has the platform's own outbound surface: the fleet tunnel and the sidecar's own agent-socket protocol keep working exactly as they do on any other fleet box (see "The box env contract" below), and those two are what every verification in this document ends up reading, because they are the only channels that exist.

Read this first: do you actually need this?

Probably not yet. Fleet boxes in this estate are already outbound-only dial — infra/eleven-fleet.md and infra/remote-host.md describe the shape every existing fleet box runs today: a security group with no inbound rules beyond what a normal host needs, admin reachability through SSM (or none, off-cloud), and all platform traffic — heartbeats, release pulls, the fleet tunnel — dialed out. Eleven's boxes in titogo are that shape right now. Design §8 is explicit about this: the private-subnet posture below is available when wanted, not a prerequisite for running a sidecar. Eleven's first sidecar box needs no new network shape at all — enrol it on the fleet box it already has, per infra/eleven-fleet.md, and stop reading here.

Reach for this runbook only when outbound-only-at-the-security-group-layer isn't enough — when the requirement is "this workload must be provably unreachable even if every SG rule and every credential on the box were compromised," because there is, at the network layer, no path in to reach it with. That is a stronger claim than "nothing dials it today," and it costs a NAT gateway (an hourly + per-GB charge) and the loss of SSM break-glass access — weigh that before applying anything below.

Preconditions (must be true before Step 1)

• [ ] Sidecar members merged and deployed. The CARLOS_INBOUND env read (internal/host/agent.go), the placement half (internal/provision/sidecar_reconcile.go, provision.PlacementNoInbound), and the mint-time refusal on no-inbound boxes (internal/provision/reconcile.go) all need to be live on the box's carlos build — built fresh, not a stale checkout (recorded trap, carlos-cli-releases.md).
• [ ] The workload already exists as a fleet box, following infra/eleven-fleet.md's Ceremony through Step 3 (fleet created, box attached, host.env flipped and verified clean) — this runbook adds network hardening around a box that already dials the right console with the right credentials, it does not replace that ceremony.
• [ ] A decision on which existing VPC/subnet plan hosts this. The sketch below assumes an existing VPC (e.g. titogo's) with spare CIDR room for one new private subnet and either a shared or new NAT gateway. A from-scratch VPC is a bigger tofu change than this runbook covers.
• [ ] The sidecar's own record targets placement: "no-inbound" (or its app's manifest does) — carlos sidecars enable -app <app> -placement no-inbound (or carlos sidecars create ... under a manifest already carrying it). Without this the record mints on any box in the pool, including ones with inbound service, and the box built below sits idle.
• [ ] Nobody is depending on SSM Session Manager reaching this box. There is no admin channel to it once this ships except the fleet tunnel and the sidecar's own agent-socket protocol (design §7.1) — confirm that is acceptable for this workload before Step 3.

The tofu shape

Four pieces, applied in order because each one's Verify depends on the last existing. No module in this repo produces this shape yet — the HCL sketches below name the resources it needs, in this repo's existing naming conventions (infra/modules/region-host/main.tf, infra/modules/region-remote-host/main.tf), for whoever authors the module; they are not a file to copy-paste and apply as-is.

1. NAT gateway (skip if one already serves this VPC)

A NAT gateway must sit in an existing public subnet — one that already has a route to an internet gateway — even though nothing routes to it from outside. It is the private subnet's only way out.

resource "aws_eip" "nat" { domain = "vpc" }

resource "awsnatgateway" "sidecar" { allocationid = awseip.nat.id subnet_id = <existing public subnet id> }

Verify: aws ec2 describe-nat-gateways (or the tofu plan/apply output) shows state available. This step alone costs money and touches no box — safe to apply and sit on. Abort: tofu destroy -target aws_nat_gateway.sidecar -target aws_eip.nat before anything below references it. Nothing yet depends on it.

2. Private subnet + route table (no IGW route)

The defining property: this subnet's route table has no 0.0.0.0/0 -> igw-... route. Its only default route is to the NAT gateway from Step 1.

resource "awssubnet" "sidecarprivate" { vpcid = <vpc id> cidrblock = <a free /28 or larger in this VPC's CIDR> availability_zone = <az the NAT gateway's subnet is in, for cost — cross-AZ NAT traffic bills extra> }

resource "awsroutetable" "sidecarprivate" { vpcid = <vpc id> route { cidrblock = "0.0.0.0/0" natgatewayid = awsnatgateway.sidecar.id } # No route to an awsinternet_gateway, ever — that absence IS the # posture. Do not add one "temporarily" to debug something; that # defeats this runbook's entire point for as long as it's there. }

resource "awsroutetableassociation" "sidecarprivate" { subnetid = awssubnet.sidecarprivate.id routetableid = awsroutetable.sidecarprivate.id }

Verify: aws ec2 describe-route-tables --route-table-ids <id> — confirm the route list contains exactly one 0.0.0.0/0 entry and its target is the NAT gateway, not igw-.... This is the single fact this whole runbook exists to guarantee; check it by reading the table, not by trusting the plan. Abort: the subnet and route table have nothing in them yet (Step 3 hasn't placed an instance) — tofu destroy -target both, no downstream cleanup.

3. Security group — zero inbound

Stricter than region-host's SG (which still opens 80/443): this box serves no certs and no tunnel-terminated traffic, so it has nothing to listen on at all.

resource "awssecuritygroup" "sidecarnoinbound" { name = "<box name>-no-inbound" description = "sidecar-only box: no ingress by design" vpcid = <vpc id> # No ingress block. Not "ingress restricted" — no ingress rule # exists to restrict. Do not add var.extraingress-style exceptions # to this SG; a box that needs an inbound rule is not this class of # box. egress { fromport = 0 toport = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } }

Verify: aws ec2 describe-security-groups --group-ids <id> shows an empty IpPermissions list (ingress) and the single egress rule. Abort: unattached SG, tofu destroy -target it directly.

4. The instance — no SSM

Same AMI/instance-type shape as infra/modules/region-host, placed in the Step 2 subnet with the Step 3 SG, no public IP, and — the deliberate omission — no aws_iam_role_policy_attachment for AmazonSSMManagedInstanceCore. Without a route to the internet or a VPC interface endpoint for SSM (which this runbook does not add — an endpoint is itself a reachable surface), the SSM agent could not dial its service even if installed, so leaving the IAM grant off is not extra caution, it is removing a credential that would do nothing but sit there. This means there is no break-glass path onto this box, ever, by anyone, other than what the sidecar and fleet-tunnel protocols carry. That is the point, not an oversight — confirm the last Precondition checkbox again before applying this step.

resource "awsinstance" "sidecar" { ami = <same arm64 Debian AMI region-host resolves> instancetype = <same class region-host's boxes use> subnetid = awssubnet.sidecarprivate.id vpcsecuritygroupids = [awssecuritygroup.sidecarnoinbound.id] associatepublicipaddress = false iaminstanceprofile = <a role scoped like region-remote-host's, WITHOUT ssm:* and WITHOUT AmazonSSMManagedInstanceCore> userdata = <same bootstrap script shape as region-host> }

Verify: the instance reaches running in describe-instances, and — the important negative check — aws ssm describe-instance-information does not list it, ever, even minutes later. If it appears there, something above is wrong (a stray endpoint, a stray IGW route, or the SSM agent running with a grant it shouldn't have) — treat that as a failed apply, not a bonus. Abort: terminate the instance (tofu destroy -target aws_instance.sidecar); nothing else in Steps 1-3 depended on it existing.

The box env contract

/etc/carlos/host.env on this box carries the same fleet contract infra/eleven-fleet.md Step 3 stages, plus the one new variable that makes this a no-inbound box:

AWSSHAREDCREDENTIALSFILE=/etc/carlos/credentials CARLOSHIBERNATEBUCKET=<the fleet's data bucket> CARLOSFLEET=<fleet name> CARLOSFLEETBOX=<this box's label> CARLOSREGION=fleet/<fleet name> CARLOSSTATUSPREFIX=fleets/<fleet name>/ CARLOSRELEASEPUBKEY=<the console's release public key> CARLOSTUNNELURL=https://<the console's reserved tunnel hostname> CARLOSTUNNELNAME=<fleet name>/<this box's label> CARLOSTUNNELTOKEN=<from the fleet's add-box call — shown once> CARLOSINBOUND=none

CARLOS_INBOUND=none is read wherever CARLOS_REGION/CARLOS_POOLS are (internal/host/agent.go), folded case/whitespace-insensitively (provision.NormalizeInbound), and — deliberately — any value that is neither empty, any, nor none is treated as none: an unrecognised setting is a typo, and the restrictive reading is the safe one. Getting this var wrong fails closed (a box that should serve nothing accidentally stays inert) rather than open.

The NAT gateway makes CARLOS_TUNNEL_URL's dial-out (and the release/ heartbeat traffic to CARLOS_HIBERNATE_BUCKET) work exactly the way it does on any other fleet box — this box's whole platform surface is still outbound, it just has no possible inbound path underneath it, where an ordinary fleet box merely has no configured one.

Verify the box came up and adopted its sidecars

This box has no shell, no SSM, and — deliberately — no path into the platform's own log stream either: internal/activator/logsink.go's Event calls are the only thing that ever writes to CARLOS_STATUS_PREFIX's log sink, and nothing in internal/host/agent.go, internal/tunnelclient (the source of the "tunnel connected" line), or internal/provision (the source of a mint-time log line) calls it. So "go read a log line" is not an available verb for this box, on-box or off — every verification below uses a surface the platform already exposes with no box access of any kind:

1. carlos sidecars ls -app <app> -long — the strongest single check. HOST names a route, PLACEMENT reads no-inbound, and once the child has called /v1/ready at least once, VERSION reads the running build instead of unknown (design §9.2's honesty rule: an absent version beats an overclaimed one). A reported version proves the whole chain end to end in one command — the box booted, dialed out, adopted the record, spawned the child, and the child called back over the agent socket. This is the check that matters most; do it first.
2. carlos fleets ls -account <acct> proves the box is registered, nothing more. Its STATUS column is registry.FleetBox.Status — an active/detached membership flag set at add-box time (internal/console/fleets.go), not a heartbeat. infra/eleven-fleet.md's own Step 2 shows this plainly: it reads active immediately after add-box, before the box has ever booted. Reading active here tells you the token exists and hasn't been detached; it tells you nothing about whether the box has ever dialed in. Do not stop at this step and conclude the box came up.
3. The console fleet page (/a/<account-sqid>/fleets/<fleet name>) is the one surface that actually renders heartbeat freshness — internal/console/fleets.go joins each box's registry row to its own heartbeat by label and shows it Reachable (a fresh heartbeat joined) or gives its Age (a heartbeat joined but has gone stale) or neither (no heartbeat has ever joined at all). This is a browser-only check — no CLI verb surfaces it as of this branch — but it needs no box access, only a signed-in session, so it still fits this runbook's "no box commands" constraint.
4. aws ec2 get-console-output --instance-id <id> if 1 and 3 both stay stale past a reasonable wait and you need to see the box actually trying to boot. This is the one channel that needs no reachable surface on the box at all — it reads whatever the instance wrote to its serial console (cloud-init/systemd boot log) from the AWS API side, not the box's. It is the sole operator-side breadcrumb for a box with neither SSH nor SSM, and the reason to leave the AMI's normal verbose boot logging on rather than quieting it for this box class.

There is no honest check for "did the provisioning pass log a mint-time refusal or success" from off the box — if 1 and 3 both look right but the record still isn't where you expect, the Abort below is the next step, not a log line to go find.

Abort (this section): if the sidecar never mints here, check placement before network: carlos sidecars ls -app <app> -long showing PLACEMENT any means the record itself never asked for no-inbound — fix the manifest/record, not the box. If placement is right and it still doesn't mint, there is no way to inspect the running process's actual environment on this box (no shell, no SSM) — treat CARLOS_INBOUND as unfalsifiable in place. The fix is the same one Rollback summary's last entry describes for this box class: there is no in-place restage, so terminate the instance and re-apply Step 4 with the value double-checked in its user-data, then re-run checks 1 and 3 above against the replacement.

Flipping an EXISTING box to CARLOS_INBOUND=none

This does not evacuate the instances the box already serves. Setting CARLOS_INBOUND=none on a box that currently runs routed instances (or was reachable and had inbound work placed on it) only refuses new routed mints from then on — it is a mint-time check (internal/provision/reconcile.go's case !ok: branch), not a converge- time or removal-time one. Every route already Provisioned on that box keeps running, keeps converging, keeps being served, exactly as before.

This is deliberate, not an oversight: making the flag evacuate the box would mean a single typo in host.env — a fat-fingered CARLOS_INBOUND=none meant for a different box, or a stray copy-paste — could take down every app the box was serving, with no confirmation step and no way to see it coming. A box that stops accepting new work loudly logs every refusal; a box that stops serving existing work on the same typo is a production incident with the blast radius of whatever happened to be running there.

So: drain the box before flipping it, not after. Move its existing routed instances off (carlos apps place to another target, or whatever this box's own placement mechanism is) and confirm they've actually stopped being served here — a curl -sI --resolve against each affected host showing it answered from elsewhere, not just that placement changed — before staging the new host.env. Only once the box carries no routed work should CARLOS_INBOUND=none land on it; at that point it is a statement of fact about the box, not an instruction to make it one.

Rollback summary

Every step above has its own Abort; in order of how far you got:

• Only Steps 1-3 applied, no instance yet: tofu destroy -target each resource in reverse order (SG, route table + association, subnet, NAT gateway + EIP) — nothing else references them.
• Instance applied, box never adopted anything: tofu destroy -target aws_instance.sidecar, then the network resources as above.
• Box adopted a sidecar, now going away: detach it from the fleet first (carlos fleets detach-box -account <acct> <fleet> <box label> — the hard cut, per infra/eleven-fleet.md's own Abort for that step) so the console stops expecting heartbeats from it, then terminate the instance and tear down the network resources.
• Flipped an existing box and want to revert to accepting inbound work again: restore the prior host.env (drop CARLOS_INBOUND=none, or set it back to any), restart the agent unit — this is a converge-time read, so the very next pass allows new mints again. This step assumes the ordinary shell/SSM access an existing fleet box normally has. It does NOT apply to a box built via this runbook's own Step 4, which has neither: there is no in-place way to edit that box's host.env at all. Reverting one of those means replacing it — terminate the instance and re-apply Step 4 with the desired host.env baked into its user-data — not restaging a file on a running box. It does not retro- actively place anything that was moved off during the drain; that's a separate carlos apps place back, a deliberate decision, not a side effect of the env flip.