amadan
docs: verify a no-inbound box only through channels it actually has
1 file changed,
+56
−24
infra/no-inbound-box.md+56 −24
diff --git a/infra/no-inbound-box.md b/infra/no-inbound-box.md| index e16ae23ae..d35f7cfb0 100644 |
| --- a/infra/no-inbound-box.md |
| +++ b/infra/no-inbound-box.md |
| @@ -12,8 +12,13 @@ traffic, no certs, no tunnel-served upstream, no public status entry. This |
| runbook covers the **hardest** version of that box — one that sits in a |
| private subnet with no route to the internet at all, egress only through a |
| NAT gateway, a security group with zero inbound rules, and no SSM agent, so |
| -the box has no reachable surface whatsoever, from any direction, by any |
| -channel. |
| +the box has no **inbound** surface at all, from any direction, by any |
| +channel — no SSH, no SSM, nothing dialing in. It still has the platform's |
| +own outbound surface: the fleet tunnel and the sidecar's own agent-socket |
| +protocol keep working exactly as they do on any other fleet box (see "The |
| +box env contract" below), and those two are what every verification in |
| +this document ends up reading, because they are the only channels that |
| +exist. |
| ## Read this first: do you actually need this? |
| @@ -229,32 +234,53 @@ ordinary fleet box merely has no *configured* one. |
| ## Verify the box came up and adopted its sidecars |
| -1. `journalctl -u carlos-edge` on the box (if you still have a channel to |
| - read it — see the SSM note above; this may mean reading it once before |
| - Step 4 removes the grant, or reading it from the box's own logs once |
| - they reach the fleet's data bucket) shows "tunnel connected". |
| -2. The fleet page (`/a/<account-sqid>/fleets/<fleet name>`) on the |
| - console shows this box with a fresh heartbeat. |
| -3. `carlos sidecars ls -app <app> -long` shows the sidecar's `HOST` column |
| - naming a route, `PLACEMENT` reading `no-inbound`, and — once it has |
| - called `/v1/ready` at least once — `VERSION` reading the running build |
| - rather than `unknown` (design §9.2's honesty rule: an absent version |
| - beats an overclaimed one, so `unknown` here is not a bug, it just means |
| - the sidecar hasn't reported in yet). |
| -4. The box's own log line confirms placement worked, not just that a |
| - record exists: `provision: minted sidecar route ... on box declaring |
| - CARLOS_INBOUND=none` (or equivalent — check the actual log text at |
| - apply time) in the box's provisioning-pass log output, once the fleet's |
| - serial has moved and the box has polled it. |
| +This box has no shell, no SSM, and — deliberately — no path into the |
| +platform's own log stream either: `internal/activator/logsink.go`'s |
| +`Event` calls are the only thing that ever writes to |
| +`CARLOS_STATUS_PREFIX`'s log sink, and nothing in `internal/host/agent.go`, |
| +`internal/tunnelclient` (the source of the "tunnel connected" line), or |
| +`internal/provision` (the source of a mint-time log line) calls it. So "go |
| +read a log line" is not an available verb for this box, on-box or |
| +off — every verification below uses a surface the platform already exposes |
| +with no box access of any kind: |
| + |
| +1. **`carlos sidecars ls -app <app> -long`** — the strongest single check. |
| + `HOST` names a route, `PLACEMENT` reads `no-inbound`, and once the |
| + child has called `/v1/ready` at least once, `VERSION` reads the running |
| + build instead of `unknown` (design §9.2's honesty rule: an absent |
| + version beats an overclaimed one). A reported version proves the whole |
| + chain end to end in one command — the box booted, dialed out, adopted |
| + the record, spawned the child, and the child called back over the |
| + agent socket. This is the check that matters most; do it first. |
| +2. **`carlos fleets ls -account <acct>`** (or the fleet page, |
| + `/a/<account-sqid>/fleets/<fleet name>`) shows this box `active` with a |
| + fresh heartbeat — proof the tunnel itself is up, independent of whether |
| + any sidecar has adopted yet. Useful to isolate "box never came up" from |
| + "box is up but the record hasn't minted." |
| +3. **`aws ec2 get-console-output --instance-id <id>`** if 1 and 2 both |
| + stay stale past a reasonable wait and you need to see the box actually |
| + trying to boot. This is the one channel that needs no reachable surface |
| + on the box at all — it reads whatever the instance wrote to its serial |
| + console (cloud-init/systemd boot log) from the AWS API side, not the |
| + box's. It is the sole operator-side breadcrumb for a box with neither |
| + SSH nor SSM, and the reason to leave the AMI's normal verbose boot |
| + logging on rather than quieting it for this box class. |
| + |
| +There is no honest check for "did the provisioning pass log a mint-time |
| +refusal or success" from off the box — if 1 and 2 both look right but the |
| +record still isn't where you expect, the *Abort* below is the next step, |
| +not a log line to go find. |
| *Abort (this section):* if the sidecar never mints here, check placement |
| before network: `carlos sidecars ls -app <app> -long` showing |
| `PLACEMENT any` means the record itself never asked for `no-inbound` — fix |
| the manifest/record, not the box. If placement is right and it still |
| -doesn't mint, confirm `CARLOS_INBOUND=none` actually reached the running |
| -process (`ps`/`/proc/<pid>/environ` from whatever channel you still have, |
| -or a fresh `host.env` re-stage plus agent restart if the value was typoed |
| -on write). |
| +doesn't mint, there is no way to inspect the running process's actual |
| +environment on this box (no shell, no SSM) — treat `CARLOS_INBOUND` as |
| +unfalsifiable in place. The fix is the same one Rollback summary's last |
| +entry describes for this box class: there is no in-place restage, so |
| +terminate the instance and re-apply Step 4 with the value double-checked |
| +in its user-data, then re-run checks 1-2 above against the replacement. |
| ## Flipping an EXISTING box to `CARLOS_INBOUND=none` |
| @@ -300,7 +326,13 @@ Every step above has its own Abort; in order of how far you got: |
| - Flipped an existing box and want to revert to accepting inbound work |
| again: restore the prior `host.env` (drop `CARLOS_INBOUND=none`, or set |
| it back to `any`), restart the agent unit — this is a converge-time |
| - read, so the very next pass allows new mints again. It does not retro- |
| + read, so the very next pass allows new mints again. **This step assumes |
| + the ordinary shell/SSM access an existing fleet box normally has.** It |
| + does NOT apply to a box built via this runbook's own Step 4, which has |
| + neither: there is no in-place way to edit that box's `host.env` at all. |
| + Reverting one of those means replacing it — terminate the instance and |
| + re-apply Step 4 with the desired `host.env` baked into its user-data — |
| + not restaging a file on a running box. It does not retro- |
| actively place anything that was moved off during the drain; that's a |
| separate `carlos apps place` back, a deliberate decision, not a side |
| effect of the env flip. |