amadan
spec: console release defences rev 3, Astra round 2 answered
Round 2 (committed unedited) found 6 of rev 2's 9 fixes sufficient and raised 3 new findings; all accepted. The proof policy moves to its own world-readable file, since host.env is root 0600. D2 keeps a per-route program history for every route kind. The rehearsal uses direct-bucket mode. D1's rollback is scoped to the routes sharing the rejected process, restores each one's own previous version and registry stamp, and refuses the build for every host in the group. A binding mismatch keeps today's error propagation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+121
−52
docs/superpowers/specs/2026-10-06-console-release-defences-design.md+100 −52docs/superpowers/specs/CODEX-FINDINGS-console-release-defences-r2.md+21 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-06-console-release-defences-design.md b/docs/superpowers/specs/2026-10-06-console-release-defences-design.md| index 8d4f39d84..e98eb06df 100644 |
| --- a/docs/superpowers/specs/2026-10-06-console-release-defences-design.md |
| +++ b/docs/superpowers/specs/2026-10-06-console-release-defences-design.md |
| @@ -1,10 +1,17 @@ |
| # Console release defences |
| -Status: draft rev 2, 2026-10-06. Branch `design-console-release-defences`. |
| -Rev 2 answers Astra round 1 (`CODEX-FINDINGS-console-release-defences.md`, |
| +Status: draft rev 3, 2026-10-06. Branch `design-console-release-defences`. |
| +Rev 2 answered Astra round 1 (`CODEX-FINDINGS-console-release-defences.md`, |
| verdict reject, nine findings). All nine were accepted, and each fix is marked |
| **(codex N)** in the text: 1 §2.3, 2 §2.2, 3 §4.2, 4 §2.3, 5 §4.3, 6 §4.5, |
| -7 §3.2, 8 §5.2 and §6, 9 §6. |
| +7 §3.2, 8 §5.2 and §6, 9 §6. Rev 3 answers round 2 (`…-r2.md`, reject: 6 of |
| +the 9 fixes sufficient, 3 new findings). All six are accepted. Round 1's 6, |
| +7 and 9 are redone: a separate world-readable policy file instead of |
| +host.env, a per-route program history for every route kind, and |
| +direct-bucket mode for the rehearsal. The new findings 10 and 11 (rollback |
| +scoped to the process group, registry stamps restored, refusal for every host |
| +in the group) are in §2.3, and 12 (a binding mismatch keeps propagating the |
| +restart's error) is in §2.2. |
| ## 0. Why |
| @@ -101,7 +108,10 @@ settle check, the named-unit arm reads `ExecStart` (`systemctl show -p |
| ExecStart <unit>`). Its executable path must be `live/<rt.Host>` for this |
| route. If it is not, the arm logs `binding: <unit> runs live/<X>, but the unit |
| stamp is on <rt.Host>; restart not checked`, does today's plain try-restart, |
| -and returns nil. That is no worse than today. Rollout step 1 (§6) verifies the |
| +and returns **its error unchanged**, as the arm does today |
| +(`internal/host/agent.go:3107`), so a failed `try-restart` still drives the |
| +existing rollback (codex 12). Only the settle check is skipped. That is no |
| +worse than today. Rollout step 1 (§6) verifies the |
| binding on every console box and restamps where it is wrong, and |
| `roll-console.sh` keeps its final restart until the binding check passes on |
| that box. |
| @@ -152,11 +162,21 @@ gets the same settle check against `cur`'s directory. Its failure is logged |
| as "rollback did not come up either", the one case that still needs a person |
| (§2.6 alerts it). |
| -Hosts earlier in the pass may already have repointed their links to the |
| -rejected version (they sort before the bound host). The rollback repoints |
| -every route on this box whose live link names the rejected version of the same |
| -app back to `cur`. That keeps their `X-Carlos-Version` honest, which today |
| -depends on alphabetical order (2026-08-23 design §4c). |
| +**The process group (codex 10, 11).** Hosts that sort before the bound host |
| +may already have repointed their links to the rejected version and had their |
| +registry versions stamped (`SetVersions`, `internal/release/adopt.go:1203`), |
| +and the edge's `X-Carlos-Version` comes from that stamp |
| +(`internal/edge/edge.go:752`). The rollback covers exactly the routes that |
| +share the rejected **process**: the bound route plus every unstamped route of |
| +the same app whose `Addr` equals the bound route's `Addr`. A route of the same |
| +app with a different `Addr` is a different process, and the rollback does not |
| +touch it, whatever version it runs. |
| + |
| +For each route in the group that this pass moved, the pass remembers the |
| +version it moved from. On rollback each one is repointed to **its own** |
| +previous version and its registry versions are restored with `SetVersions`. |
| +The refusal record (§2.5) is written for **every host in the group**, so no |
| +sibling re-adopts the rejected build on the next pass. |
| ### 2.4 Readiness |
| @@ -269,10 +289,14 @@ interruption, even with the bad manifest already promoted. |
| **Going back across a deliberate change (codex 7).** After an intended A→B |
| change, promoting an old A release back would be refused, because A's |
| -immutable manifest has no `program_change`. So the check is skipped when the |
| -staged binary's sha256 is in this box's healthy history (§4.5): a build that |
| -ran well here before is never a wrong program. The history is kept for every |
| -named-unit route, not only D3-listed apps, so D2 can use it everywhere. |
| +immutable manifest has no `program_change`. So each route keeps a **program |
| +history** in the bake directory: every main package path the box has pointed |
| +live for that route, recorded at `PointLive` for every route kind (exec, |
| +`carlos-app@`, named unit). The check passes when the staged binary's program |
| +is in the route's history. Going back to A is therefore accepted on any route |
| +that ran A here, and a program the route has never run still needs |
| +`--program-change`. A route with no history yet, such as a box upgraded to |
| +this release, seeds it from the live binary on its first pass. |
| ## 4. D3: the flagship console adopts only builds proven on dev |
| @@ -317,15 +341,15 @@ writes the private half and prints the public half. |
| ### 4.3 Minting on dev |
| -Dev's console box host.env gains `CARLOS_PROVE_APPS=console`, |
| -`CARLOS_PROVER_NAME=dev`, `CARLOS_PROVING_KEY_FILE=/etc/carlos/proving.key` |
| -(root 0600) and `CARLOS_PROVE_AFTER=15m`. |
| +Dev's console box gets a `prove` block in its proof policy file (§4.5): |
| +apps `console`, prover `dev`, `after` 15m, and the private key at |
| +`/etc/carlos/proving.key` (root 0600). |
| A root pass mints a proof for a named-unit route of a listed app when all of |
| these hold at once: |
| - the route's binding is verified (§2.2), and the unit is `active` with an |
| - `ActiveEnterTimestamp` at least `CARLOS_PROVE_AFTER` ago and the same |
| + `ActiveEnterTimestamp` at least `after` ago and the same |
| `InvocationID` throughout; |
| - S is the sha256 of the bytes of `/proc/<MainPID>/exe`, and the live link |
| resolves to that same file; |
| @@ -337,7 +361,7 @@ It writes `proofs/<app>/<sha256>.json` to dev's own bucket. |
| Minting needs root (the key), and root passes run hourly or when the agent |
| asks. So the agent asks: on its fast loop, when a listed app's route has been |
| -live and active for `CARLOS_PROVE_AFTER` with no proof in the bucket, it |
| +live and active for `after` with no proof in the bucket, it |
| requests a root pass through the existing stage-request relay. It **keeps |
| asking every 2 minutes until it sees the proof in the bucket** (codex 5). The |
| relay can miss a request written while `carlos-stage.service` is already |
| @@ -362,9 +386,9 @@ the plan confirms the exact prefix the box role may write. |
| ### 4.5 Requiring on flagship |
| -Flagship's console box host.env gains |
| -`CARLOS_REQUIRE_PROOF=console:dev:<base64 ed25519 public key>`, a |
| -comma-separated list of `app:prover:key` entries. |
| +Flagship's console box gets a `require` block in its proof policy file, |
| +naming app `console`, prover `dev` and dev's public key. The file's shape and |
| +how it is read are below. |
| In `adoptHost`, for a route whose app is listed, the check runs twice: |
| @@ -384,8 +408,8 @@ It also proceeds, with no proof, if the sha256 is in the box's own **healthy |
| history**: a record in the bake directory of builds this box has run well. A |
| root pass adds a sha256 when D1's settle check passes for it, and also when it |
| finds any named-unit route's live build meeting §4.3's first three mint |
| -conditions (bound, active in one invocation for `CARLOS_PROVE_AFTER`, the |
| -hashed exe, ready; default 15m when unset). The second rule is what puts the build already live when |
| +conditions (bound, active in one invocation for the policy's `after`, the |
| +hashed exe, ready; 15m when there is no policy file). The second rule is what puts the build already live when |
| the requirement is switched on into the history without a restart. So rolling |
| back to a build flagship ran before, as this morning's recovery did, never |
| waits on dev. |
| @@ -399,17 +423,32 @@ once the build is live. |
| `carlos status --live` shows "waiting for a dev proof" for that host. |
| -**Where the setting is read (codex 6).** The agent gets host.env through |
| +**Where the setting lives (codex 6).** The agent gets host.env through |
| `EnvironmentFile` only when it starts (`internal/host/units/carlos-edge.service:18`), |
| while each root pass starts fresh (`carlos-stage.service:30`). An edit would |
| reach one process and not the other, and restarting `carlos-edge` takes the |
| -box's edge down. So the four `PROVE` variables and `CARLOS_REQUIRE_PROOF` are |
| -**read from the file `/etc/carlos/host.env` at the start of every pass** by |
| -both the agent and root, not from the process environment. A small parser |
| -reads only those keys. The file is root-owned and world-readable today; the |
| -plan confirms that the sandboxed agent can read it. |
| +box's edge down. host.env is also root-only 0600 |
| +(`infra/modules/region-host/README.md:23`), so the agent cannot read it |
| +itself. |
| -**Break glass:** remove the `CARLOS_REQUIRE_PROOF` line from host.env as root, |
| +So the policy is **not in host.env**. It lives in its own file, |
| +`/etc/carlos/proof-policy.json`, root-owned, mode 0644. It holds no secrets: |
| +app names, the prover name, public keys and durations: |
| + |
| +```json |
| +{"schema":1, |
| + "prove":{"apps":["console"],"prover":"dev","after":"15m", |
| + "key_file":"/etc/carlos/proving.key"}, |
| + "require":[{"app":"console","prover":"dev","key":"<base64 ed25519 public key>"}]} |
| +``` |
| + |
| +Both the agent and root read it at the start of every pass. An absent file |
| +means D3 is off. An unreadable or unparseable file means D3 is **on and |
| +waiting** for every named-unit route, since the box cannot tell which apps |
| +were listed. It logs this every pass, because a typo must not open the gate. `key_file` names the root-only |
| +private key, and only root opens it. |
| + |
| +**Break glass:** delete the `require` entry (or the file) as root, |
| then `touch` the stage-request path (or run `carlos adopt --stage-only`) so a |
| root pass adopts at once. The agent's next tick also sees the line gone. |
| While the requirement is off on a box that has a healthy history for an app, |
| @@ -418,7 +457,7 @@ or the CLI can switch it off. |
| ### 4.6 Who sees none of this |
| -A deployment with neither variable set: no proof is minted, read or required, |
| +A deployment with no proof policy file: no proof is minted, read or required, |
| and adoption is unchanged except for D1 and D2. Tito, self-hosters and every |
| customer app on carloku.com are in this position. |
| @@ -475,14 +514,17 @@ Order matters: the box binary is what checks, so it goes first. |
| 2. Ship D1, D2 and D3 in one platform release. With no config, D3 is inert. |
| 3. Set the health probe on the console routes on dev and flagship. |
| 4. Roll the release's boxes (`carlos system update`) on dev and flagship. |
| -5. **On dev:** run `carlos proof keygen` on the console box, set the four |
| - `PROVE` variables, and roll the console to dev with the new recipe. That |
| +5. **On dev:** run `carlos proof keygen` on the console box, write its proof |
| + policy file with the `prove` block, and roll the console to dev with the |
| + new recipe. That |
| roll is the first live check of D1: watch the journal for the settle line. |
| 6. **Rehearse on dev** (codex 9): |
| - Ship the public `carlos-linux-arm64` as a console release using a |
| - **v0.24.1 or older client**, which has no ship-side check and writes no |
| - `program`. Promote it and confirm that the box's D2 check refuses it, |
| - with no restart. |
| + **v0.24.1 or older client in direct-bucket mode** (`CARLOS_DEPLOYMENT_BUCKET` |
| + set, no `--console`), which is how today's bad ship went out. That |
| + client has no ship-side check and writes no `program`. API mode would |
| + hit the console's server-side check instead (codex 9, round 2). Promote |
| + it and confirm that the box's D2 check refuses it, with no restart. |
| - Ship a console built to exit 2 on start, from a real `./cmd/console` so |
| D2 passes. Confirm D1 rolls back within 30 s, writes a refusal, raises |
| the alert, and retries after a re-promote. |
| @@ -493,15 +535,17 @@ Order matters: the box binary is what checks, so it goes first. |
| 8. **Roll that build to flagship.** The recipe carries the proof (codex 8). |
| If the flagship console must move before dev can mint, use `--bootstrap` |
| once. |
| -9. **On flagship:** set `CARLOS_REQUIRE_PROOF` with dev's public key. The |
| - live build is unaffected, because the requirement gates moves only, and it |
| - enters the healthy history 15 minutes later (§4.5). |
| +9. **On flagship:** write the proof policy file with the `require` block and |
| + dev's public key. The live build is unaffected, because the requirement |
| + gates moves only, and it enters the healthy history 15 minutes later |
| + (§4.5). |
| 10. Every later console roll goes through dev, with the proof carried. |
| -host.env on the flagship box is not delivered by tofu `user_data` for this; |
| -whole-root applies restart every box (memory: carloku-infra-split). Set the |
| -line over SSM and record it in carloku-infrastructure's runbook. The plan |
| -checks whether host.env has a managed home it should go in instead. |
| +The proof policy files are not delivered by tofu `user_data`, because a |
| +whole-root apply restarts every box (memory: carloku-infra-split). Write them |
| +over SSM and record their contents in carloku-infrastructure's runbook (for |
| +flagship) and platform-infrastructure's (for dev). The plan checks whether |
| +they have a managed home they should go in instead. |
| ## 7. Testing |
| @@ -512,26 +556,30 @@ checks whether host.env has a managed home it should go in instead. |
| not running before (no check); `ExecStart` naming another host's link (no |
| check, plain try-restart). A D1 rollback runs `reset-failed` then |
| `restart`, and replaces a still-running rejected process. Other start |
| - callers keep `start`. `adoptHost` with a failing settle: repoints back the |
| - bound host and every earlier host whose link names the rejected version, |
| - writes the refusal record; the next pass, agent or root, skips without |
| + callers keep `start`. A binding mismatch whose `try-restart` fails still |
| + returns the error and rolls back. `adoptHost` with a failing settle: |
| + repoints back the bound host and every earlier same-`Addr` route of the |
| + app, each to its own previous version with its registry versions restored; |
| + leaves a same-app route on a different `Addr` alone; writes the refusal |
| + record for every host in the group; the next pass, agent or root, skips without |
| relaying; a pointer with a new `PromotedAt` clears it. |
| - **D2:** `buildinfo` read from tiny stripped binaries with two different |
| main packages, cross-built for linux-arm64 by a test helper at test time |
| (no binaries checked in). Ship refuses, accepts with |
| `--program-change`, accepts non-Go, accepts a first release. The API door |
| and the CLI call the same function. `adoptHost` refuses a mismatched staged |
| - binary before `PointLive` and writes a refusal. It accepts a mismatch whose |
| - staged sha256 is in the healthy history (rollback across a deliberate |
| - change). |
| + binary before `PointLive` and writes a refusal. It accepts a program in the |
| + route's program history (rollback across a deliberate change) on an exec |
| + route, a `carlos-app@` route and a named-unit route, and seeds the history |
| + from the live binary on a route that has none. |
| - **D3:** proof canonical encoding round-trip and signature; mint conditions |
| each false in turn, including a live link and a running exe that hash |
| differently; `adoptHost` with the requirement set: no proof, bad signature, |
| wrong sha256, wrong app, wrong prover, valid proof, healthy history; a |
| valid proof for the manifest's sha256 with a staged cache hit whose bytes |
| - hash differently (refused at the binding check). The policy is read from a |
| - host.env file that changes between passes, and the agent sees the change |
| - without a restart. The agent re-requests a mint until the proof appears. No |
| + hash differently (refused at the binding check). The policy file changes |
| + between passes and the agent sees the change without a restart; an |
| + unparseable policy file keeps the gate closed. The agent re-requests a mint until the proof appears. No |
| requirement set: no bucket GET is made (assert on the fake store). |
| - **The dev rehearsal in §6 step 6** is the end-to-end test. Mocks cannot show |
| that systemd's start-limit state and `reset-failed` behave as written. |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-console-release-defences-r2.md b/docs/superpowers/specs/CODEX-FINDINGS-console-release-defences-r2.md| new file mode 100644 |
| index 000000000..b3a6ace90 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-console-release-defences-r2.md |
| @@ -0,0 +1,21 @@ |
| +| Round 1 finding | Sufficient? | Why | |
| +|---|---|---| |
| +| 1. Rejected process survives rollback | Yes | §2.3 requires `reset-failed` followed by `restart`, then checks the restored executable. | |
| +| 2. Shared-unit binding rejects healthy builds | Yes | §2.2 requires the unit stamp on the `ExecStart` host and verifies that binding before checking the restart. | |
| +| 3. Proof certifies bytes that never ran | Yes | §4.2 hashes the running executable when minting and the actual staged file when adopting, including cache hits. | |
| +| 4. Incorrect `NRestarts` baseline | Yes | §2.3 checks the new invocation and its counter instead of comparing counters across the manual restart. | |
| +| 5. Lost proof trigger | Yes | §4.3 retries every two minutes until publication is observed. | |
| +| 6. Running agent retains old proof policy | No | §4.5 assumes readable `host.env`, but the documented setup is [root-owned `0600`]( /home/paulca/amadan.net/carlos/platform-design-console-release-defences/infra/modules/region-host/README.md:23); the agent needs an accessible policy source. | |
| +| 7. D2 prevents rollback across program changes | No | §3.2 records history only for named-unit routes; ordinary exec and `carlos-app@` apps still cannot roll back across an intentional program change. | |
| +| 8. Proof required before producer configured | Yes | §6 configures dev and obtains its proof before the flagship roll. | |
| +| 9. Rehearsal bypasses D2 | No | An old client in API mode still encounters §3.1’s server-side ship check; the rehearsal must explicitly select direct-bucket shipping to reach adoption’s check. | |
| + |
| +New findings at `8535d250`: |
| + |
| +10. **P1 — Rollback can repoint unrelated, healthy instances.** [§2.3](/home/paulca/amadan.net/carlos/platform-design-console-release-defences/docs/superpowers/specs/2026-10-06-console-release-defences-design.md:155) requires repointing **every route** running the rejected version of the same app to the failing host’s `cur`. Those routes can belong to separate processes and follow different channels ([registry.go:69](/home/paulca/amadan.net/carlos/platform-design-console-release-defences/internal/registry/registry.go:69)). For example, one instance already runs V2 successfully while another moves V1→V2 and fails its environment-specific readiness check. The prescribed rollback also moves the healthy instance’s link to V1, without restoring its own prior state or coordinating its process. Restrict rollback to the routes sharing the affected process, and preserve each affected route’s previous state. |
| + |
| +11. **P2 — Shared-host rollback leaves incorrect version headers and permits immediate re-adoption.** [§2.3](/home/paulca/amadan.net/carlos/platform-design-console-release-defences/docs/superpowers/specs/2026-10-06-console-release-defences-design.md:155) claims repointing earlier hosts keeps `X-Carlos-Version` honest. However, [`PointLive`](/home/paulca/amadan.net/carlos/platform-design-console-release-defences/internal/release/versions.go:267) changes only a symlink; earlier hosts have already received their registry stamps through [`SetVersions`](/home/paulca/amadan.net/carlos/platform-design-console-release-defences/internal/release/adopt.go:1203), and the edge obtains the header from that [registry value](/home/paulca/amadan.net/carlos/platform-design-console-release-defences/internal/edge/edge.go:752). Moreover, §2.5 specifies a host-shaped refusal for the failed bound route, leaving the other hosts free to re-adopt the rejected version on the next pass. Require restoration of the affected registry stamps and refusal coverage for all hosts sharing the rolled-back process. |
| + |
| +12. **P2 — Binding fallback discards restart errors that currently trigger recovery.** [§2.2](/home/paulca/amadan.net/carlos/platform-design-console-release-defences/docs/superpowers/specs/2026-10-06-console-release-defences-design.md:99) says a binding mismatch performs plain `try-restart` and returns nil. Today, the named-unit arm [returns the command’s error](/home/paulca/amadan.net/carlos/platform-design-console-release-defences/internal/host/agent.go:3107), which activates [rollback](/home/paulca/amadan.net/carlos/platform-design-console-release-defences/internal/release/adopt.go:1103). If `try-restart` fails on a mismatched binding, rev 2 would suppress recovery and stamp the deployment successful, potentially leaving the unit down. Skip only the new settle check in this fallback; preserve the existing command-error propagation. |
| + |
| +Verdict: reject. |
| \ No newline at end of file |