Console release defences

Status: draft rev 5, 2026-10-06, for Paul's review. Rev 5 applies Astra round 4 (…-r4.md, accept with fixes): rev 4's three fixes were judged sufficient, and two new P2s are fixed. 14 (a non-canary route waiting for a proof stays in the fast loop) is in §4.5, and 15 (readiness follows the effective upstream, Unix socket first) is in §2.4. Rev 4 answered Astra round 3 (…-r3.md, reject). Of the six round-2 findings it re-checked, four were fixed. Two needed more: findings 10 and 11, now in §2.3, where groups key on a non-empty Socket or Addr and every group route is stamped to the bound route's restored version. New finding 13, a promote --retry, is in §2.5. All accepted. Branch design-console-release-defences. Rev 2 answered Astra round 1 (CODEX-FINDINGS-console-release-defences.md, verdict reject, nine findings). All nine were accepted, and each fix is marked (codex N) in the text: 1 §2.3, 2 §2.2, 3 §4.2, 4 §2.3, 5 §4.3, 6 §4.5, 7 §3.2, 8 §5.2 and §6, 9 §6. Rev 3 answers round 2 (…-r2.md, reject: 6 of the 9 fixes sufficient, 3 new findings). All six are accepted. Round 1's 6, 7 and 9 are redone: a separate world-readable policy file instead of host.env, a per-route program history for every route kind, and direct-bucket mode for the rehearsal. The new findings 10 and 11 (rollback scoped to the process group, registry stamps restored, refusal for every host in the group) are in §2.3, and 12 (a binding mismatch keeps propagating the restart's error) is in §2.2.

0. Why

On 2026-10-06 the flagship console (console.carloku.com, dashboard.carloku.com, platform.carlosframework.com) answered 502 for 24 minutes, 09:30 to 09:54 UTC.

What happened, confirmed from the box journal and the bucket:

• At 09:30:15 a console release named v0.24.2 was shipped to apps/bab/console/releases/v0.24.2/ and promoted to canary/console at 09:30:30. Its one artifact was the public carlos-linux-arm64 from the v0.24.2 GitHub release (sha256 bc53cfc5…, matching that release's SHA256SUMS.txt): the CLI, not ./cmd/console. The manifest says built_by: carlos v0.24.2. It was a hand step, not roll-console.sh, which builds ./cmd/console and names versions by git sha.
• The README's release recipe never mentions the console, and the public release carries no console artifact, so whoever rolled v0.24.2 had nothing correct to ship.
• The root stage pass staged it, repointed all three live links, and ran systemctl try-restart console.service. console serve is not a carlos verb, so the new process printed the CLI's usage and exited 2 after 55 ms. try-restart had already returned 0 (the unit is Type=simple), so adoptHost's existing rollback branch (internal/release/adopt.go, the rerr != nil arm: poison, repoint back, start) never ran.
• systemd restarted the unit five times in one second, hit its start limit and left it failed. The live links named the bad build; nothing looked again.
• Nothing alerted. A person noticed.

Recovery was a promote of the previous build f4478b15 back to canary/console, then systemctl reset-failed and start over SSM: the repoint alone does not revive a unit systemd has given up on.

1. Rulings

These were decided with Paul on 2026-10-06 and are not reopened here.

1. Four defences, in this order of value: (D1) a box rolls back a named unit that will not come up; (D2) carlos ship and adoption refuse a binary whose Go main package differs from the app's previous release; (D3) the flagship console adopts only builds proven on dev; (D4) process: the release publishes a console artifact and the recipe rolls it dev first.
2. No new AWS account. Dev (account 640012953448, platform.dev.carlosframework.com) already is the separate test deployment. The failure was that it was skipped. D3 makes skipping it impossible for the flagship console without root on the console box.
3. D1 and D2 are on for every deployment and add no steps. D3 is off unless a deployment turns it on. An operator who does not want the testing ceremony (Tito, a self-hoster) configures nothing and adopts exactly as before, still covered by D1 and D2. There is no per-promote bypass flag: the bypass is local and explicit, as for bake windows (internal/release/bake.go).
4. Enforcement lives on the box that adopts, not in the carlos client. Anyone holding a deployment's release key can promote anything, and today's promote was a hand step, so a client-side check would not have fired.
5. Uptime alerting from outside the box (would have paged within a minute) belongs to #192, not this spec.
6. Scope is named-unit routes (registry.Route.Unit, today only console.service) for D1 and D3. Box binaries keep carlos system update's own gate and rollback. Exec children and carlos-app@ tenants are out of scope for D1; D2 applies to every binary app.

Threat model. These are defences against mistakes. A compromised release key or bucket is the signing chain's and the bake window's problem. Where a mechanism below trusts an unsigned manifest field, that is why.

2. D1: roll back a named unit that will not come up

2.1 The gap

makeRestartRoute's first arm (internal/host/agent.go) returns systemctlTryRestartUnit(rt.Unit), whose error is only systemctl's. For a Type=simple unit that is "the start job was queued", which says nothing about whether the new binary runs. adoptHost already does the right thing when the restart callback returns an error: it poisons the version for the pass, repoints back to cur, restores capability drop-ins, and starts the unit on the restored build (WithStartUnit). D1 makes the named-unit arm return an honest error.

2.2 The binding comes first (codex 2)

The console is one unit behind several hostnames, and adoption is per host in ORDER BY host (internal/registry/registry.go:800). Each adoptHost repoints its own live/<host> and then restarts. The unit runs whichever link its ExecStart names: on flagship that is live/platform.carlosframework.com, which sorts last of the three. A restart fired from an earlier host runs the old build. The console-as-an-app spec already requires that the one Unit-stamped route be the host ExecStart names (2026-08-23 design §4c), and roll-console.sh's final systemctl restart is what has hidden it when it isn't.

So D1 judges a restart only when the binding is verified. Before the settle check, the named-unit arm reads ExecStart (systemctl show -p ExecStart <unit>). Its executable path must be live/<rt.Host> for this route. If it is not, the arm logs binding: <unit> runs live/<X>, but the unit stamp is on <rt.Host>; restart not checked, does today's plain try-restart, and returns its error unchanged, as the arm does today (internal/host/agent.go:3107), so a failed try-restart still drives the existing rollback (codex 12). Only the settle check is skipped. That is no worse than today. Rollout step 1 (§6) verifies the binding on every console box and restamps where it is wrong, and roll-console.sh keeps its final restart until the binding check passes on that box.

2.3 The settle check

After systemctl try-restart <unit> returns 0 on a verified binding, the named-unit arm waits for the unit to settle:

• Poll every 500 ms, up to 30 s (CARLOS_UNIT_SETTLE, a duration, floor 5 s), reading ActiveState, InvocationID, MainPID and NRestarts from systemctl show.
• The first poll that finds the unit active with an InvocationID different from the one before the restart fixes the new invocation. Everything after is judged against it, never against the previous invocation's counters: systemd resets NRestarts on a manual start, so a before-and-after comparison misreads healthy restarts (codex 4).
• Healthy when, within the new invocation, ActiveState=active, the MainPID's /proc/<pid>/exe resolves into the new version's directory, and the route's readiness check passes (§2.4). Return nil immediately. The console takes about 8 s, so a healthy restart costs about 8 s, not 30.
• Failed as soon as ActiveState is failed, or the InvocationID changes again (systemd restarted it), or NRestarts within the new invocation goes above zero. Return an error carrying the state and the last 20 lines of journalctl -u <unit> -o cat.
• Failed at the deadline if the unit never satisfied "healthy". That includes a process that is still running but never became ready.

A unit that was not running before the restart (an operator stopped it) is not checked: try-restart started nothing, and that stays a no-op.

The check runs inside the root stage pass and makes it up to 30 s longer (about 8 s for a healthy console). A stage request written while that pass runs past the relay's 15 s window waits for the next trigger: the known gap recorded with the deploy-pickup fix. Only the bound route (§2.2) carries the Unit stamp, so there is one checked restart per unit per pass. Hosts that sort after it skip a rejected version through poisoned, and hosts before it are repointed by the rollback below.

On a failure the existing adoptHost branch then repoints to cur. The rollback must replace whatever is running, not merely start (codex 1). A deadline failure can leave the rejected build running and unready, and systemctl start does nothing to a running unit (internal/host/agent.go:3208). So for a named unit that D1 rejected, the rollback runs systemctl reset-failed <unit>, which clears a start limit, then systemctl restart <unit>. Every other caller of makeStartRoute keeps start. The rollback gets the same settle check against cur's directory. Its failure is logged as "rollback did not come up either", the one case that still needs a person (§2.6 alerts it).

The process group (codex 10, 11). Hosts that sort before the bound host may already have repointed their links to the rejected version and had their registry versions stamped (SetVersions, internal/release/adopt.go:1203), and the edge's X-Carlos-Version comes from that stamp (internal/edge/edge.go:752). The rollback covers exactly the routes that share the rejected process. A route's upstream is its Socket if set, else its Addr (Socket wins, internal/registry/registry.go:40). The group is the bound route plus every unstamped route of the same app with the same non-empty upstream. A bound route with no upstream at all is a group of one, and two empty upstreams never match (codex 10, round 3). A route of the same app with a different upstream is a different process, and the rollback does not touch it, whatever version it runs.

Every route in the group is served by the one process, so after a rollback each of them is serving whatever the bound route rolled back to. On rollback, every group route whose live link names the rejected version is repointed to the bound route's restored version, and its registry versions are set to match with SetVersions. This applies whether the route moved in this pass or in an earlier, interrupted one (codex 11, round 3). The stamp then names what is actually answering. The refusal record (§2.5) is written for every host in the group, so no sibling re-adopts the rejected build on the next pass.

2.4 Readiness

In order:

Readiness follows the route's effective upstream: Socket if set, else Addr. Socket wins, as it does at the edge (internal/registry/registry.go:40), and named units may be socket-backed (unitRail, cli.go:1624) (codex 15, round 4). In order:

1. The route's own health probe (registry.Route.Health, the probe-config JSON instances already use), sent as HTTP to the effective upstream: over the Unix socket for a socket-backed route, over TCP to Addr otherwise.
2. Else, for a route with an upstream: a connect to it succeeds (Unix or TCP).
3. Else: the unit stays active within one invocation for 5 s.

The three console routes get {"path":"/healthz","status":200} as part of rollout (§6). The console's /healthz (internal/console/server.go) is unauthenticated and answers as soon as the listener is up.

2.5 Refusing the build until something changes

poisoned lives for one pass. Without more, the next pass, and the agent's 2 s fast loop asking for a root stage pass every 15 s, would re-adopt the same bad build and bounce the console each time.

When D1 rolls a route back, the root pass writes a refusal record to the box's bake directory (BakeDir, under the data dir, which the sandboxed agent can read):

{"schema":1,"host":"console.carloku.com","app":"console","version":"v0.24.2",
 "sha256":"bc53cfc5…","platform":"linux-arm64",
 "pointer_promoted_at":"2026-10-06T09:30:30Z","refused_at":"…","reason":"…"}

adoptHost, on every pass, root or agent, before ensureStaged and before stageBlocked can relay: if the channel pointer names this sha256 for this platform and the pointer's PromotedAt equals the record's, print refused: <version> did not come up on this box at <time> (<reason>); retry with: carlos promote --retry … and return. A pointer that moves to different bytes clears it, and so does a pointer re-signed with a new PromotedAt.

The retry needs a new flag (codex 13, round 3). Promoting the version a channel already names is a successful no-op that does not rewrite the pointer (PromoteOnLadder, internal/release/release.go:910), so a plain re-promote leaves PromotedAt unchanged and the refusal standing. carlos promote --retry <version> <channel> re-signs the pointer with a fresh PromotedAt even when it already names <version>. It refuses unless it does already name <version>, so it can only renew a promotion, never change one. The API promote door takes the same retry field. Both reach it through one PromoteOnLadder option, so the ladder, scope and authorization checks run exactly as for any promote.

The record keys on the artifact sha256 read from the manifest, not on the version name, because names get reused.

2.6 Being seen

• A rollback writes a line naming both versions, the reason and the journal tail to the root pass's output (journal).
• The agent raises an alert through its existing sinks (alertsink.go: SNS, which works on flagship, and CARLOS_ALERT_WEBHOOK) the first time it reads a new refusal record: subject console.carloku.com: v0.24.2 did not start, rolled back to f4478b15. "Rollback did not come up either" alerts with a louder subject. The plan decides whether this joins the metrics Event shape or a small sibling; it must not wait on #192.
• carlos status --live shows a refused host with the refusal's version and reason.

3. D2: refuse the wrong program

3.1 At ship

For a binary artifact, carlos ship reads the Go build info with debug/buildinfo.ReadFile. This works on a stripped (-s -w) binary built for another architecture without running it: a stripped arm64 carlos read on amd64 gives path github.com/carlosframework/platform, and a console build gives github.com/carlosframework/platform/cmd/console.

• The manifest gains program (string, the build info's main package path), additive to schema 1 like capabilities and label: omitted when empty, ignored by old binaries. A non-Go binary has no build info and no program. With several platform artifacts, every one must report the same path, or ship refuses.
• Ship reads the app's latest release's manifest. If both name a program and they differ, ship refuses:

`` this artifact is github.com/carlosframework/platform; console's releases are github.com/carlosframework/platform/cmd/console. If the app really is changing program, ship again with --program-change. ``

• --program-change ships anyway and records "program_change": true in the manifest.
• No previous release, a previous release without program, or a non-Go artifact: ship as today. The check starts with the next release of each app.
• The console's API ship door (internal/console/apicli.go, the release.Ship caller) makes the same check server-side, so API-mode ships cannot skip it and direct-bucket ships (cli.go) do it client-side. Both call one function in internal/release.

"Latest" is the newest release by manifest built_at under apps/<acct>/<app>/releases/. The plan may use the channel heads instead if listing every release is too slow; either catches today's case.

3.2 At adoption

program is an unsigned manifest field, and an older client writes none. So the box checks the bytes themselves. In adoptHost, after ensureStaged and before the capability converge and PointLive: read build info from the staged new binary and from the binary cur names. If both are readable and their main package paths differ, and the new manifest does not carry program_change: true, refuse:

console.carloku.com: v0.24.2 is github.com/carlosframework/platform, the live
build is …/cmd/console — not pointed live (ship with --program-change if this
is intended)

and write a refusal record (§2.5) so the agent does not relay it again. This alone would have stopped today's outage on the box, with no service interruption, even with the bad manifest already promoted.

Going back across a deliberate change (codex 7). After an intended A→B change, promoting an old A release back would be refused, because A's immutable manifest has no program_change. So each route keeps a program history in the bake directory: every main package path the box has pointed live for that route, recorded at PointLive for every route kind (exec, carlos-app@, named unit). The check passes when the staged binary's program is in the route's history. Going back to A is therefore accepted on any route that ran A here, and a program the route has never run still needs --program-change. A route with no history yet, such as a box upgraded to this release, seeds it from the live binary on its first pass.

4. D3: the flagship console adopts only builds proven on dev

4.1 Shape

• A proving deployment (dev) mints a signed proof once a build has run healthily on its console for a bake period.
• An operator carries the proof to the target deployment's bucket. It is signed, so carrying needs no trust and no access between the AWS accounts.
• A requiring deployment (flagship) adopts a build of a listed app only with a proof for those exact bytes from a prover it trusts, or a build it has itself run healthily before.

4.2 The proof

{"schema":1,"app":"console","platform":"linux-arm64",
 "sha256":"e47fcc1e…","version":"f4478b15",
 "prover":"dev","active_since":"…","proven_at":"…",
 "sig":"<base64 ed25519 over the canonical JSON of every other field>"}

It binds app, platform and artifact sha256. version is informational. The target compares sha256, so flagship must run the same bytes dev ran (§5.2), and a version name reused for different bytes has no proof.

Every sha256 in D3 is hashed from bytes, never copied from a manifest (codex 3). A version name can be reused after pruning (internal/release/release.go:632). An already-staged binary skips the download check (internal/release/adopt.go:1317), and backfillManifest fetches whatever manifest the bucket now holds for that name without comparing it to the staged file (internal/release/adopt.go:1402). So a manifest's sha256 can name bytes the box is not running. Minting hashes the running process's executable (/proc/<MainPID>/exe). The requiring box hashes the staged file it is about to point live, on a cache hit too, and the healthy history records the hash of the executable that passed the settle check.

The proving key is a fresh ed25519 key, separate from both release keys, held only by the proving console box as a root-only file. carlos proof keygen writes the private half and prints the public half.

4.3 Minting on dev

Dev's console box gets a prove block in its proof policy file (§4.5): apps console, prover dev, after 15m, and the private key at /etc/carlos/proving.key (root 0600).

A root pass mints a proof for a named-unit route of a listed app when all of these hold at once:

• the route's binding is verified (§2.2), and the unit is active with an ActiveEnterTimestamp at least after ago and the same InvocationID throughout;
• S is the sha256 of the bytes of /proc/<MainPID>/exe, and the live link resolves to that same file;
• the route's readiness check (§2.4) passes now;
• no refusal record names S;
• no proof for S exists yet in dev's bucket.

It writes apps/<acct>/<app>/proofs/<sha256>/<prover>.json to dev's own bucket: the same key shape the target uses (§4.4), so proof get and proof put move one object between two buckets under one name.

Minting needs root (the key), and root passes run hourly or when the agent asks. So the agent asks: on its fast loop, when a listed app's route has been live and active for after with no proof in the bucket, it requests a root pass through the existing stage-request relay. It keeps asking every 2 minutes until it sees the proof in the bucket (codex 5). The relay can miss a request written while carlos-stage.service is already running (internal/host/agent.go:1203), and a mint write can fail. A proof is then normally minted within a minute or two of the bake period ending, and at worst within the hourly pass.

Dev's bucket layout and IAM already let the box write under its own prefix; the plan confirms the exact prefix the box role may write.

4.4 Carrying it

• carlos proof get --app console --sha256 <S>, against the proving deployment (dev env), prints the proof JSON, or fails "not proven yet: <S> has been live on dev since <t>, proven after <15m>".
• carlos proof put --app console <file>, against the target (flagship env), checks the JSON's shape and writes it to apps/<acct>/<app>/proofs/<sha256>/<prover>.json. It cannot check the signature without the target's trust config, and need not: the box does.

roll-console.sh runs both (§5.2).

4.5 Requiring on flagship

Flagship's console box gets a require block in its proof policy file, naming app console, prover dev and dev's public key. The file's shape and how it is read are below.

In adoptHost, for a route whose app is listed, the check runs twice:

1. A cheap gate, after reading the new manifest and before ensureStaged, so an unproven build is neither downloaded nor relayed to root. The box takes the manifest's sha256 for its platform and looks for apps/<acct>/<app>/proofs/<sha256>/<prover>.json. It continues if the proof parses, its signature verifies against the configured key, and its app, platform and sha256 match.
2. The binding check, after ensureStaged and before PointLive. The box hashes the staged file and requires that hash to equal the proof's sha256, or to be in the healthy history. A mismatch means the staged bytes are not the proven ones (a reused name, a stale cache). The box refuses the version with a refusal record (§2.5) naming both hashes.

It also proceeds, with no proof, if the sha256 is in the box's own healthy history: a record in the bake directory of builds this box has run well. A root pass adds a sha256 when D1's settle check passes for it, and also when it finds any named-unit route's live build meeting §4.3's first three mint conditions (bound, active in one invocation for the policy's after, the hashed exe, ready; 15m when there is no policy file). The second rule is what puts the build already live when the requirement is switched on into the history without a restart. So rolling back to a build flagship ran before, as this morning's recovery did, never waits on dev.

Otherwise it prints console.carloku.com: v… (sha256 …) is not proven on dev; staying on <cur> and returns. This is not a refusal record: the box looks again, so the build adopts soon after its proof arrives. The agent's fast loop runs every 2 s, so a missing proof is cached per (route, sha256) for 30 s: at most two bucket GETs a minute per listed route while waiting, none once the build is live.

A waiting route stays in the fast loop (codex 14, round 4). fastMembers (internal/host/agent.go:1550) includes canary routes on every tick, but drops a non-canary route as soon as it has a live version. So a non-canary route waiting for a proof would be rechecked only by the next serial bump or the hourly pass. While a listed route has an outstanding "not proven" verdict for its channel head, fastMembers keeps it as a member, and the 30 s cache above bounds what that costs. The console is on canary/console and is always a member; this is for any other listed app.

carlos status --live shows "waiting for a dev proof" for that host.

Where the setting lives (codex 6). The agent gets host.env through EnvironmentFile only when it starts (internal/host/units/carlos-edge.service:18), while each root pass starts fresh (carlos-stage.service:30). An edit would reach one process and not the other, and restarting carlos-edge takes the box's edge down. host.env is also root-only 0600 (infra/modules/region-host/README.md:23), so the agent cannot read it itself.

So the policy is not in host.env. It lives in its own file, /etc/carlos/proof-policy.json, root-owned, mode 0644. It holds no secrets: app names, the prover name, public keys and durations:

{"schema":1,
 "prove":{"apps":["console"],"prover":"dev","after":"15m",
          "key_file":"/etc/carlos/proving.key"},
 "require":[{"app":"console","prover":"dev","key":"<base64 ed25519 public key>"}]}

Both the agent and root read it at the start of every pass. An absent file means D3 is off. An unreadable or unparseable file means D3 is on and waiting for every named-unit route, since the box cannot tell which apps were listed. It logs this every pass, because a typo must not open the gate. key_file names the root-only private key, and only root opens it.

Break glass: delete the require entry (or the file) as root, then touch the stage-request path (or run carlos adopt --stage-only) so a root pass adopts at once. The agent's next tick also sees the line gone. While the requirement is off for an app on a box that has required a proof for it before (a marker the root pass writes when it reads a require entry), the box logs once an hour that the requirement is off. A box that never had the requirement, such as Tito's or a self-hoster's, stays quiet. Nothing in the bucket or the CLI can switch it off.

4.6 Who sees none of this

A deployment with no proof policy file: no proof is minted, read or required, and adoption is unchanged except for D1 and D2. Tito, self-hosters and every customer app on carloku.com are in this position.

5. D4: process

5.1 The release carries a console

The release recipe (README "Releasing the platform") builds console-linux-arm64 and console-linux-amd64 from the tag with go build -trimpath -ldflags "-s -w -X …version.Build=vX.Y.Z" ./cmd/console, lists them in SHA256SUMS.txt (signed as today), and publishes them with the other files. Step 4's "eight files" becomes ten.

5.2 The console roll recipe

carloku-infrastructure/scripts/roll-console.sh changes:

• A dev target with the dev coordinates (profile kass, bucket carlos-dev-640012953448, console box i-05498eada6f0afd23, platform.dev.carlosframework.com). The dev coordinates move here from platform-infrastructure, because flagship's rule now depends on dev.
• --artifact <file>: roll exactly this file instead of building. A release's console roll uses the public console-linux-arm64, verified against the signed SHA256SUMS.txt, so dev and flagship run the bytes the public got. Without --artifact it builds once with -trimpath and keeps the file for the next target.
• flagship runs proof get against dev first and refuses to ship if there is none, naming how long dev has had the build. With a proof it ships, promotes, runs proof put and verifies as today.
• The adopt and restart SSM steps go once the box's binding is verified (§2.2): the script reads the binding line from the box first, and keeps its final systemctl restart on any box where the binding is wrong. Otherwise adoption restarts the unit and D1 checks it. The /proc/<pid>/exe verification stays.
• A --bootstrap flag for the first flagship roll only skips the proof step, for the one roll that installs D3 before dev can mint (§6). It prints that it is skipping. It does not exist on the box: flagship's requirement is switched on only after that roll.

The recipe's console step reads: roll the release's console to dev, wait for the proof (15 minutes), roll it to flagship. It says in so many words that the public carlos-linux-* binary is never a console artifact. D2 enforces that now, but the recipe should not lean on it.

6. Rollout

Order matters: the box binary is what checks, so it goes first.

1. Fix the bindings. On dev's and flagship's console boxes, check that the one Unit-stamped route is the host ExecStart names, and restamp where it isn't (2026-08-23 design §4c). Flagship was stamped on console.carloku.com while ExecStart names live/platform.carlosframework.com; confirm the current state on the box.
2. Ship D1, D2 and D3 in one platform release. With no config, D3 is inert.
3. Set the health probe on the console routes on dev and flagship.
4. Roll the release's boxes (carlos system update) on dev and flagship.
5. On dev: run carlos proof keygen on the console box, write its proof policy file with the prove block, and roll the console to dev with the new recipe. That roll is the first live check of D1: watch the journal for the settle line.
6. Rehearse on dev (codex 9): - Ship the public carlos-linux-arm64 as a console release using a v0.24.1 or older client in direct-bucket mode (CARLOS_DEPLOYMENT_BUCKET set, no --console), which is how today's bad ship went out. That client has no ship-side check and writes no program. API mode would hit the console's server-side check instead (codex 9, round 2). Promote it and confirm that the box's D2 check refuses it, with no restart. - Ship a console built to exit 2 on start, from a real ./cmd/console so D2 passes. Confirm D1 rolls back within 30 s, writes a refusal, raises the alert, stays refused after a plain re-promote, and retries after carlos promote --retry. - Ship a console that runs but never answers /healthz. Confirm the deadline rollback replaces the running process. - Delete all three releases.
7. Wait for dev's proof of the build rolled in step 5.
8. Roll that build to flagship. The recipe carries the proof (codex 8). If the flagship console must move before dev can mint, use --bootstrap once.
9. On flagship: write the proof policy file with the require block and dev's public key. The live build is unaffected, because the requirement gates moves only, and it enters the healthy history 15 minutes later (§4.5).
10. Every later console roll goes through dev, with the proof carried.

The proof policy files are not delivered by tofu user_data, because a whole-root apply restarts every box (memory: carloku-infra-split). Write them over SSM and record their contents in carloku-infrastructure's runbook (for flagship) and platform-infrastructure's (for dev). The plan checks whether they have a managed home they should go in instead.

7. Testing

• D1: makeRestartRoute's named-unit arm with a fake systemctl show sequence: healthy after N polls; failed; InvocationID changed after the new invocation was fixed; NRestarts reset to 0 by the manual restart and the unit healthy (must pass); deadline with the process still running; not running before (no check); ExecStart naming another host's link (no check, plain try-restart). A D1 rollback runs reset-failed then restart, and replaces a still-running rejected process. Other start callers keep start. A binding mismatch whose try-restart fails still returns the error and rolls back. adoptHost with a failing settle: every group route whose link names the rejected version is repointed and stamped to the bound route's restored version, including one advanced by an earlier interrupted pass; groups key on Socket before Addr; two routes with empty upstreams are never grouped; a same-app route on a different upstream is left alone; the refusal record is written for every host in the group; the next pass, agent or root, skips without relaying.
• Readiness: a socket-backed named unit with a configured probe passes over its Unix socket; a route with both Socket and Addr is probed on the socket only.
• Retry: a plain re-promote of the refused version is still a no-op and the refusal stands; promote --retry through the CLI (direct mode) and the API door re-signs with a fresh PromotedAt and clears it; --retry against a channel naming a different version is refused.
• D2: buildinfo read from tiny stripped binaries with two different main packages, cross-built for linux-arm64 by a test helper at test time (no binaries checked in). Ship refuses, accepts with --program-change, accepts non-Go, accepts a first release. The API door and the CLI call the same function. adoptHost refuses a mismatched staged binary before PointLive and writes a refusal. It accepts a program in the route's program history (rollback across a deliberate change) on an exec route, a carlos-app@ route and a named-unit route, and seeds the history from the live binary on a route that has none.
• D3: proof canonical encoding round-trip and signature; mint conditions each false in turn, including a live link and a running exe that hash differently; adoptHost with the requirement set: no proof, bad signature, wrong sha256, wrong app, wrong prover, valid proof, healthy history; a valid proof for the manifest's sha256 with a staged cache hit whose bytes hash differently (refused at the binding check). The policy file changes between passes and the agent sees the change without a restart; an unparseable policy file keeps the gate closed. The agent re-requests a mint until the proof appears. A non-canary listed route with a live version stays in fastMembers while it waits for a proof, and leaves once it adopts. No requirement set: no bucket GET is made (assert on the fake store).
• The dev rehearsal in §6 step 6 is the end-to-end test. Mocks cannot show that systemd's start-limit state and reset-failed behave as written.

8. Not in this spec

• External uptime alerting: #192.
• D1 for exec children and carlos-app@ units. Those restarts already have other failure handling, and a wrong-program ship to them is D2's job.
• Making carlos system update require dev proofs for box binaries. Its own gate covers box binaries, and the proof shape extends to it if wanted later.
• Deleting the bad apps/bab/console/releases/v0.24.2 release. That is an operator clean-up, independent of this work; D2's box check makes it inert once rolled.