amadan
spec: instance update settings rev 4 — answers Astra S2 (digest-bound promotion records, build state beside the replica, roll ordering, request replay)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+143
−46
docs/superpowers/specs/2026-10-07-instance-update-settings-design.md+104 −46docs/superpowers/specs/CODEX-FINDINGS-SETTINGS-S2.md+39 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-07-instance-update-settings-design.md b/docs/superpowers/specs/2026-10-07-instance-update-settings-design.md| index c9e4042dc..486491912 100644 |
| --- a/docs/superpowers/specs/2026-10-07-instance-update-settings-design.md |
| +++ b/docs/superpowers/specs/2026-10-07-instance-update-settings-design.md |
| @@ -1,6 +1,6 @@ |
| # Instance update settings: auto-update, channel, major hold, pin |
| -Status: draft rev 3, 2026-10-07. Split out of the zero-downtime deploys |
| +Status: draft rev 4, 2026-10-07. Split out of the zero-downtime deploys |
| spec (`2026-10-07-zero-downtime-deploys-design.md`) at its rev 6. Rev 1 was |
| a design with console-signed per-instance directives. Astra's round 6 |
| (`CODEX-FINDINGS-ZERO-DOWNTIME-R6.md`) rejected it with 6 findings. |
| @@ -13,7 +13,10 @@ directives, and root applies the settings itself (§4). §7 maps the six |
| findings. Astra closed all six on rev 2 and raised five new ones |
| (`CODEX-FINDINGS-SETTINGS-S1.md`). Rev 3 answers them by making **root |
| the only judge of eligibility**, recording only facts on the record, and |
| -adding a durable signed record of every promoted build (§4.1). |
| +adding a durable signed record of every promoted build (§4.1). Astra's S2 |
| +review of rev 3 (`CODEX-FINDINGS-SETTINGS-S2.md`) closed two of those five |
| +and left three partial. Rev 4 answers those three and one new finding |
| +(§10). |
| ## 1. Why |
| @@ -109,11 +112,19 @@ proves nothing. **Root installs build V of app A only if it can verify |
| that V was authorized for A:** |
| - a signed channel pointer of A names V as its head or in its history |
| (today's check); or |
| -- a **signed promotion record** for (account, A, V). This is new: every |
| - promotion writes `releases/<acct>/<app>/promoted/<version>`, signed with |
| - the key that signs pointers. Promotion is the act that authorizes a |
| - build, so this makes the authorization durable after V falls out of the |
| - pointer's ten-entry history. |
| +- a **signed promotion record** for (account, A, V, **manifest digest**). |
| + This is new: every promotion writes |
| + `releases/<acct>/<app>/promoted/<version>`, signed with the key that |
| + signs pointers. The digest is the sha256 of the canonical manifest bytes, |
| + which carry each artifact's sha256. |
| + |
| +Promotion is the act that authorizes a build, so the record keeps the |
| +authorization valid after V falls out of the pointer's ten-entry history. |
| +Root checks that the manifest it is about to stage hashes to the record's |
| +digest before staging, and staging's artifact check then binds the bytes. |
| +A version name reused after pruning has a different manifest, so an old |
| +promotion record can't authorize its new bytes, even if the record is |
| +replayed. |
| So a pin must name a build that has been **promoted** to some channel of |
| the app. A build that was shipped but never promoted is refused, both by |
| @@ -121,21 +132,29 @@ the console when the pin is set and by root. A build already staged and |
| verified on the box can be recovered there without verifying again. That |
| is the zero-downtime spec's retained identity. |
| -### 4.2 The record carries facts, not choices |
| +### 4.2 Facts the platform keeps |
| -| Field | Written by | Meaning | |
| -|-------|------------|---------| |
| -| settings (§3.1) | people, through the console and API | the preference | |
| -| `install_request` | the console, on "Update now" | an id that only ever increases. Consumed once by root (§4.3) | |
| -| `held_build` | the console, from what boxes report | the build a held instance (`manual`, or held at major N) last had installed. Used only to restore it on a box with nothing installed | |
| +| Fact | Where | Written by | Meaning | |
| +|------|-------|------------|---------| |
| +| settings (§3.1) | provision record | people, through the console and API | the preference | |
| +| `install_request` | provision record | the console, on "Update now" | an id that only ever increases | |
| +| **build state** | an object beside the instance's replica in the bucket | the agent, under the instance's lease | see below | |
| -`held_build` comes from the instance's installed build in box status. That |
| -report is trusted the way settings are (§2 decision 5): at worst a forged |
| -report picks another authorized build of the same app. It is refreshed |
| -whenever a held instance's reported installed build changes, including |
| -right after an install request completes. **Turning `manual` on records |
| -the build installed now, not the channel head**, so it never moves the |
| -instance. |
| +The **build state** object holds three things: |
| +- the **installed build**; |
| +- the **last install request root resolved**, with the build it resolved |
| + to; |
| +- whether that request has **completed**. |
| + |
| +The agent rewrites the object each time root stamps an install or resolves |
| +a request (§4.3 rule 2), before reporting either as done. It is trusted |
| +the way settings are (§2 decision 5). A forged object can at most choose |
| +another authorized build of the same app. |
| + |
| +The object replaces rev 3's status-derived `held_build`. It is written in |
| +step with the install itself, not on the five-minute status cycle, so |
| +turning `manual` on can't record a build the instance has already moved |
| +past. |
| ### 4.3 The rules |
| @@ -145,20 +164,32 @@ build its box holds as failed unless it has already served that build. |
| 1. **`pin` set:** install the pinned build, if §4.1 authorizes it. Bake |
| doesn't apply, because a person chose it. |
| -2. **A pending `install_request`** (its id is above the id root last |
| - consumed for this host): install the channel head **that this box's own |
| - bake check says is eligible**, or the current head if the request asked |
| - for `FastTrack`. `manual` and `major` are ignored for this one move. |
| - Root then records the consumed id, root-owned under |
| - `/opt/carlos/consumed/<host>`. That happens after the stamp, so an |
| - interrupted pass simply runs the request again. If the head is still |
| - baking, the request stays pending and the reason is reported. |
| +2. **An `install_request` R above the last one completed** (taken from the |
| + box's own record of consumed requests, or else from the build state |
| + object): |
| + - **If R has already been resolved to a build T,** install T. That |
| + covers a pass that was interrupted, and a request picked up on |
| + another box after a move. |
| + - **Otherwise**, resolve R now: take the channel head that this box's |
| + own bake check says is eligible, or the current head if the request |
| + asked for `FastTrack`. Write {R, T} to the root-owned file |
| + `/opt/carlos/consumed/<host>` before staging. The agent mirrors it |
| + into the build state. |
| + - **Completion:** after the stamp, mark R completed, both locally and in |
| + the build state. |
| + - `manual` and `major` are ignored for this one move. |
| + - **If the head is still baking**, R stays unresolved and the reason is |
| + reported. |
| 3. **`manual`, or `major = N` with a head whose semver tag is absent or has |
| another major:** don't move. |
| - - **With nothing installed on this box** (a move, a restore or a fresh |
| - box), install `held_build`, if §4.1 authorizes it. Bake doesn't |
| - apply: the instance already ran that build. |
| - - **With no `held_build` either**, the instance is new and gets the |
| + - **Nothing installed on this box** (a move, a restore or a fresh box): |
| + install the build state's installed build, if §4.1 authorizes it. Bake |
| + doesn't apply, because the instance already ran that build. |
| + - **No build state but the instance has a replica** (it has data, so it |
| + isn't new): its selection is **unknown**. Install nothing, report |
| + *build unknown: use Update now or set a pin*, and leave the instance |
| + down rather than guess. |
| + - **Neither build state nor replica:** the instance is new and gets the |
| eligible channel head. |
| 4. **Otherwise:** today's adoption, unchanged. |
| @@ -167,12 +198,23 @@ The console refuses "Update now" on a pinned instance, naming the pin. |
| ### 4.4 Retention |
| Bucket pruning (`internal/release/prune.go`, called from |
| -`internal/host/retention.go`) adds to its protected set, for the app, every |
| -build that a provision record names as a `pin` or a `held_build`. It also |
| +`internal/host/retention.go`) adds to its protected set, for the app: |
| +- every build a provision record names as a `pin`; |
| +- every build any build state object names, whether installed or as a |
| + resolved request. It also |
| keeps the signed promotion record of every protected build. If the |
| -housekeeper can't read every record of the app, it prunes nothing for that |
| -app on that pass. That is the same refusal it makes for an incomplete |
| -protected set today. |
| +housekeeper can't read every record and build state object of the app, it |
| +prunes nothing for that app on that pass. That is the same refusal it |
| +makes for an incomplete protected set today. |
| + |
| +**Order of the roll.** The housekeeper duty moves between boxes |
| +(`internal/host/agent.go`), and a box from before this feature protects |
| +only its own installed builds. So: |
| +- The console **refuses to enable** `pin`, `manual` or `major` on any |
| + instance until every box in the deployment reports a binary that |
| + protects them. |
| +- Before rolling any box back below that version, an operator **turns |
| + retention off** for the deployment. The roll docs carry both steps. |
| ### 4.5 Rollback, named units, trust |
| @@ -185,7 +227,7 @@ version, and the roll docs say so. |
| settings on them. |
| **Trust, stated plainly.** |
| -- Settings, `install_request` and `held_build` aren't signed. A compromised |
| +- Settings, `install_request` and the build state object aren't signed. A compromised |
| agent, or a forged record or status report, can make an instance run a |
| different **authorized** build of its own app. That is the accepted |
| model (§2 decision 5). |
| @@ -237,24 +279,31 @@ build whatever these settings say. |
| - **`release.Target`:** a table test over: |
| - pin; |
| - a pending, consumed or interrupted `install_request`; |
| - - manual, with and without an installed build, and with and without |
| - `held_build`; |
| + - manual, with and without an installed build, with and without a |
| + build state object, and with and without a replica; |
| - major, with a matching, other or untagged head; |
| - bake windows from the app's pipeline, and `FastTrack`; |
| - a failure-held build. |
| - **Authorization:** a fresh never-promoted version with a matching |
| manifest is refused for a pin. A promoted build that has left the |
| pointer's history is accepted through its signed promotion record. |
| -- **Retention:** pinned and `held_build` builds survive pruning. An |
| - unreadable record set prunes nothing for the app. |
| +- **Retention:** pinned builds and builds named by build state survive |
| + pruning. An unreadable record or build state prunes nothing for the app. |
| + The console refuses the settings while any box is on an older binary. |
| +- **Promotion binding:** a reused version name with a replayed promotion |
| + record and a different manifest is refused before staging. |
| +- **Request replay:** a completed request isn't run again on a destination |
| + box. A request interrupted after resolving installs the same T. An |
| + existing instance with a replica but no build state is held as unknown, |
| + never moved to the head. |
| - **Root pass:** |
| - - a manual or major-held instance on a fresh box installs `held_build`, |
| - not the head; |
| + - a manual or major-held instance on a fresh box installs the build |
| + state's installed build, not the head; |
| - an instance held at major N stays put while the channel moves; |
| - an install request waits on this box's bake, and is consumed exactly |
| once. |
| -- **Console:** turning on `manual` records the installed build, never the |
| - channel head. "Update now" bumps the request id. Every §3.3 refusal is tested from the |
| +- **Console:** "Update now" bumps the request id, and the console never |
| + chooses a build. Every §3.3 refusal is tested from the |
| console, the CLI and the instance-bound credential. |
| - **Rollback:** a box whose root predates this feature shows *settings |
| not enforced*. |
| @@ -268,3 +317,12 @@ build whatever these settings say. |
| | 3 | Retention deletes pinned or held builds | §4.4: pins and `held_build`s are protected across the app's records; an incomplete read prunes nothing | |
| | 4 | An install request can't be told apart from turning on manual | §4.2–4.3: a separate `install_request` id, consumed once by root and re-run if interrupted | |
| | 5 | The console can't judge bake | §4.3 rule 2: root's own bake check decides; the request waits while the build is baking; recovery installs don't bake | |
| + |
| +## 10. Astra S2 findings (rev 3), answered by rev 4 |
| + |
| +| # | Finding | Answer | |
| +|---|---------|--------| |
| +| S1-1 (partial), 1 | A durable promotion authorizes reused names | §4.1: the promotion record binds the manifest digest; root checks it before staging | |
| +| S1-2 (partial), 2 | Status-derived `held_build` can roll a manual instance back | §4.2: build state written under the lease in step with each install, replacing `held_build`; §4.3: unknown selection is held, never moved to the head | |
| +| S1-3 (partial), 3 | Older housekeepers delete protected builds | §4.4: the console refuses the settings until every box can protect them; retention goes off before any rollback | |
| +| 4 | A completed install request runs again after a move | §4.3 rule 2: {R, T} and completion persist locally and in the build state; resolved requests replay to the same T | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-SETTINGS-S2.md b/docs/superpowers/specs/CODEX-FINDINGS-SETTINGS-S2.md| new file mode 100644 |
| index 000000000..3735f22d3 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-SETTINGS-S2.md |
| @@ -0,0 +1,39 @@ |
| +REJECT |
| + |
| +S1-1: PARTIAL — Signed evidence is required, but durable authorization remains vulnerable to version-name reuse; finding 1. |
| +S1-2: PARTIAL — `held_build` supplies a recovery target, but asynchronous status can preserve the wrong one; finding 2. |
| +S1-3: PARTIAL — Updated pruners protect records, but an older housekeeper can still delete protected builds; finding 3. |
| +S1-4: CLOSED — A distinct request distinguishes “Update now” from enabling manual mode. |
| +S1-5: CLOSED — Root’s own bake clock decides request eligibility. |
| + |
| +1. **HIGH — §4.1: Durable promotion authorization can authorize replacement bytes that were never promoted.** |
| + |
| + **Code:** [internal/release/release.go:632](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/release.go:632) explicitly permits different content under a pruned version name. [internal/release/adopt.go:1421](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:1421) checks artifact bytes against the unsigned manifest. Hosts have deployment-bucket write access at [infra/modules/region-host/main.tf:166](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/infra/modules/region-host/main.tf:166). |
| + |
| + **Failure:** V was legitimately promoted, then leaves pointer history and is pruned. A compromised box uploads arbitrary bytes and a matching manifest under V, restores V’s previously valid signed promotion record if necessary, and requests V. The record authorizes `(account, app, V)`; every specified check passes on a fresh box. The installed bytes were never authorized. This exceeds the accepted freedom to select another authorized build. |
| + |
| + **Smallest fix:** Bind the promotion record to the manifest digest, including artifact digests, and verify that binding before staging. Deleting the record during pruning is insufficient because its signature can be replayed. |
| + |
| +2. **HIGH — §§4.2–4.3: Status-derived `held_build` can roll a manual instance back during recovery.** |
| + |
| + **Code:** [internal/status/fleet.go:111](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/status/fleet.go:111) specifies five-minute status publication plus jitter. [internal/console/apiinstances.go:383](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/console/apiinstances.go:383) reads the reported version; [internal/registry/registry.go:1720](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/registry/registry.go:1720) stamps installation independently. |
| + |
| + **Failure:** H last reported A. Automatic adoption installs B, which starts serving. Before the next report, the owner enables manual mode; the console records reported A as `held_build`, while B correctly remains installed. The box then fails. On its replacement, rule 3 installs A against B’s database. This can cause an outage after a migration incompatible with A. Missing status is also unsafe: rule 3 treats an existing instance with no recorded build as new and installs the head. |
| + |
| + **Smallest fix:** Have root capture and durably acknowledge the held selection when applying manual mode or completing a held instance’s installation. Recovery must distinguish “never initialized” from “selection unknown”; unknown must not select the head. No stronger authentication is needed under decision 5. |
| + |
| +3. **HIGH — §§4.4–4.5: Rolling upgrades and rollback leave a destructive retention path on older boxes.** |
| + |
| + **Code:** [internal/host/agent.go:2721](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/agent.go:2721) makes retention a transferable housekeeper duty. [internal/host/retention.go:33](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/retention.go:33) protects only that box’s installed/previous versions before invoking shared-store pruning. |
| + |
| + **Failure:** Upgraded box X hosts H pinned to old, untagged A. Older box Y hosts the same app on recent builds and holds—or later acquires—the housekeeper lease. Once A leaves history and the retention window, Y deletes it despite H’s provision-record pin. H’s settings remain enforced on X, so §4.5’s per-instance warning does not expose the problem. H’s next move or restore fails. |
| + |
| + **Smallest fix:** Require every eligible housekeeper to be upgraded before enabling these settings. Disable affected-app retention before rolling any eligible housekeeper back to an incompatible version. State this ordering explicitly in §4.5. |
| + |
| +4. **MEDIUM — §4.3 rule 2: A completed install request executes again after a move or restore.** |
| + |
| + **Code:** [internal/provision/reconcile.go:394](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/provision/reconcile.go:394) constructs the destination route from the provision record. [internal/release/versions.go:290](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/versions.go:290) reports no installed version without a local live link. Neither transfers the proposed `/opt/carlos/consumed/<host>` marker. |
| + |
| + **Failure:** Manual H completes request 7, installing B on X; `held_build=B` is fully reported. The channel later advances to eligible C. H moves to fresh box Y, where no consumed marker exists. Rule 2 treats request 7 as pending and installs C, overriding manual mode before recovery can select B. Separately, interruption after stamping B but before recording consumption can repeat the request against a later head. |
| + |
| + **Smallest fix:** Persist the request’s resolved target and completion with instance state that survives relocation. Replays of one request must finish the same selection; completed requests must remain completed on another box. |
| \ No newline at end of file |