amadan
spec: zero-downtime deploys rev 6 — console-signed per-instance directives replace the agent-root request machinery (Astra round 5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+258
−312
docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md+219 −312docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R5.md+39 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md| index ec0cf7cfd..43581f887 100644 |
| --- a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| +++ b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| @@ -1,11 +1,19 @@ |
| # Zero-downtime deploys: overlap swaps, update settings, paced rollout |
| -Status: draft rev 5, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| +Status: draft rev 6, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| (per-app restart pace, first-route health step) feed into this. Astra |
| rejected rev 1 with 17 findings (`CODEX-FINDINGS-ZERO-DOWNTIME.md`) and |
| rev 2 with 12 new ones (`CODEX-FINDINGS-ZERO-DOWNTIME-R2.md`), and |
| rev 3 with 7 new ones and 3 partials (`CODEX-FINDINGS-ZERO-DOWNTIME-R3.md`), |
| -and rev 4 with 3 new ones and 2 partials (`CODEX-FINDINGS-ZERO-DOWNTIME-R4.md`). §16 maps every finding to its answer. |
| +rev 4 with 3 new ones and 2 partials (`CODEX-FINDINGS-ZERO-DOWNTIME-R4.md`), |
| +and rev 5 with 4 new ones and 1 partial (`CODEX-FINDINGS-ZERO-DOWNTIME-R5.md`). |
| + |
| +Rev 6 replaces the box-side machinery that rounds 2–5 kept breaking. That |
| +was the agent's exact repoint requests, root's install ledger, the |
| +agent–root handshake and carried build state. In their place the console |
| +computes **signed per-instance directives** (§5), and fallback never needs |
| +root (§8.2). Answers in earlier rounds that pointed at the removed |
| +machinery are superseded by §5 and §8. §16 maps every finding to its answer. |
| ## 1. Why |
| @@ -72,8 +80,10 @@ These were settled with Paul on 2026-10-06/07 and are not reopened here. |
| - **Serving build**: the build of the generation currently taking new |
| connections. It can differ from the installed build while a rollout |
| queues, checks health, or falls back. |
| -- **Desired build**: what target resolution (§5) says the instance should |
| - be on now. |
| +- **Desired build**: the build the instance should serve now. That is its |
| + installed build, which root put there from its channel or its directive |
| + (§5), unless a failure record holds that build, in which case it is the |
| + good build (§8.2). |
| - **Generation**: one spawn of one instance, numbered upwards for the life |
| of the activator entry. |
| - **Attempt**: one try at moving one instance to a new generation. It has |
| @@ -141,62 +151,106 @@ runs in `overlap`. |
| their app. |
| - A refused change names the setting, and says the app keeps it for itself. |
| -## 5. Target resolution |
| - |
| -There is **one resolver**, `release.Desired(record, policy, heads, |
| -failures, good)`. Only the agent runs it. For exec routes on a box where |
| -the protocol is active (§7.11), root no longer decides anything. It only |
| -carries out the agent's exact per-host requests, and checks each one |
| -against an authorization it can verify itself (§7.7). No other path reads |
| -a channel pointer to choose a build for those routes: not the root stage |
| -pass, not the hourly adopt loop, and not first-time materialization. |
| - |
| -The resolver works in two steps. |
| - |
| -**Candidate.** |
| -- If `pin` is set, the candidate is the pinned build, at the pin's |
| - revision (each pin write bumps it). |
| -- Otherwise the candidate is the head of the instance's `channel`, after |
| - bake holds (`internal/release/bake.go`) and `FastTrack`, exactly as |
| - today. |
| - |
| -**Holds.** The first hold that applies makes the desired build the |
| -installed build: |
| -1. `updates = manual`, unless the candidate came from a pin. Setting a pin |
| - is itself a request to install it, so it moves the instance even in |
| - `manual`. Bake holds don't apply to a pin. |
| -2. A failure record matches the candidate for this instance (§8.4). That |
| - includes a pinned candidate at the same pin revision. A failed pin stays |
| - held until the pin is set again or the build is retried. |
| -3. `major` is set and the candidate is not pinned, and the head's tag is |
| - absent or has another major. The instance is *held at major N*. |
| -4. **Bootstrap.** The instance has no good build, its installed build is |
| - *untested*, meaning no failure record matches it (§8.5), and the |
| - candidate didn't come from a pin. The instance stays where it is until |
| - a generation has served there. This way an instance that predates this |
| - feature establishes a baseline before any cold move. |
| - |
| - Hold 4 doesn't apply when the installed build has a failure record. An |
| - instance whose only build never started moves to any other authorized |
| - candidate, and that build's first wake is its canary. |
| - |
| -An instance is **new** when it has no installed build, no good build and |
| -no carried build state (§8.6). A new instance resolves to the candidate, |
| -ignoring `manual`. An instance arriving by a move or a restore is not new. |
| -It starts from its carried state. |
| - |
| -If no hold applies, the desired build is the candidate. |
| - |
| -**Installing by hand** is refused on a pinned instance, naming the pin. |
| -Otherwise it moves the instance to its channel head, ignoring `manual`, |
| -`major`, bake holds and hold 4, and clears the instance's failure hold |
| -(§8.4). |
| - |
| -**Which routes this covers.** Settings, the resolver and the scheduler |
| -apply to exec routes only. Named-unit routes, the console's own route |
| -included, keep root's existing channel-driven adoption and its privileged |
| -restart, unchanged (§13). The console refuses update settings on a |
| -named-unit route, saying they aren't supported there yet. |
| +## 5. Who decides which build an instance installs |
| + |
| +**An instance's build is decided by the console, which owns its settings |
| +and policy, and is verified by root**, by the same signature scheme that |
| +already protects channel pointers. The box never makes this decision, so a |
| +sandboxed agent can't steer root, and nothing depends on the agent and |
| +root agreeing. |
| + |
| +### 5.1 Two ways a build is chosen |
| + |
| +- **Channel-following instances** have every setting at its default: |
| + `auto`, the app's channel, no `major`, no `pin`. They are adopted |
| + exactly as today. Root reads the signed channel pointer, applies bake |
| + holds (`internal/release/bake.go`) and `FastTrack`, stages the build and |
| + repoints. Most instances are like this, and their install path doesn't |
| + change. |
| +- **Directed instances** have any non-default setting, or a pending |
| + install by hand. For each of these the console keeps a **signed install |
| + directive** on the instance's provision record. Root installs exactly |
| + the directive's build and ignores the channel for that host. |
| + |
| +### 5.2 The directive |
| + |
| +A directive is signed with the key that signs channel pointers. It binds: |
| +- account, app and host; |
| +- the host's **incarnation** (`hostclaim`), so a directive for a deleted |
| + and recreated host is void; |
| +- the build; |
| +- a **revision**, which only ever increases for each host. |
| + |
| +When a pin is set, the directive's revision is the **pin revision**. |
| + |
| +**What root checks.** Before acting on a directive, root: |
| +1. verifies the signature, the account, app and host, and the |
| + incarnation it sees for the host; |
| +2. checks that the revision is not lower than its **high-water mark** for |
| + that host. The mark is kept root-owned at |
| + `/opt/carlos/directives/<host>`, which the agent can't write. |
| + |
| +Then it stages the build (download and sha256 check, as `ensureStaged` |
| +does), repoints, stamps, and raises the mark. Unsigned, mismatched or |
| +replayed directives are refused and reported in box status. The agent |
| +copies the record's directive into the registry, as reconcile copies other |
| +record fields. Because of the signature, the agent can carry the |
| +directive but can't forge one. |
| + |
| +### 5.3 How the console computes a directive |
| + |
| +The console recomputes an instance's directive whenever its settings, its |
| +app's policy, or its channel head change. Promotions already go through |
| +the console. Take the first rule that applies: |
| + |
| +1. **`pin` set:** the pinned build. Setting a pin is itself an install |
| + request, so it applies even in `manual`. No bake hold applies. |
| +2. **`updates = manual`:** the build the instance has installed now, as |
| + last reported in box status. This never moves the instance on its own. |
| + A new instance with nothing installed gets the channel head. |
| +3. **`major` set, and the channel head's tag is absent or has another |
| + major:** the installed build. The instance is *held at major N*. |
| +4. **Otherwise:** the channel head, after bake holds computed from the |
| + promotion time with `DefaultHolds`, and `FastTrack`. |
| + |
| +Box status is only a report. **For rules 2 and 3, the console accepts a |
| +reported build only if the app's signed history shows it was shipped to |
| +the instance's channel, or it was pinned for that host.** So a lying box |
| +can at most choose among builds already authorized for that instance. |
| + |
| +**Installing by hand** on a pinned instance is refused, naming the pin. |
| +Otherwise it issues a directive to the channel head, ignoring `manual`, |
| +`major` and bake holds. When an instance's settings return to default and |
| +its directive names the channel head, the console drops the directive, and |
| +the instance goes back to following the channel. |
| + |
| +### 5.4 Which routes this covers |
| + |
| +Settings, directives and the scheduler apply to exec routes only. |
| +Named-unit routes, the console's own route included, keep today's root |
| +adoption and its privileged restart unchanged (§13). The console refuses |
| +update settings on a named-unit route, saying they aren't supported there |
| +yet. |
| + |
| +### 5.5 Rollback and moves |
| + |
| +**A box rolled back to a binary from before directives** ignores them and |
| +adopts every host from its channel. So `manual`, pins and major holds stop |
| +being enforced there. The console compares each directed instance's |
| +reported installed build with its directive, flags any mismatch on the |
| +instance and in the roll docs, and enforcement resumes with the next |
| +forward roll. |
| + |
| +**Moves and restores need no state carried with them.** The directive |
| +lives on the console-owned record, so the destination installs exactly |
| +what the source was told to install. |
| + |
| +**Replay.** On a box that has never applied a directive for a host, an |
| +older directive for the same incarnation could be replayed by someone |
| +with write access to the record. It could only install a build that host |
| +was previously directed to. The high-water mark refuses anything older |
| +from the first application on, and the console's mismatch check flags |
| +it. |
| ## 6. Rollout and pacing |
| @@ -210,17 +264,17 @@ channels or pins form different cohorts and never take each other's |
| targets. |
| It is fed by: |
| -- the agent's pass, which runs the resolver for each route after a serial |
| - bump, a config sync or a reconcile; |
| +- the agent's pass, which compares each route's installed build, stamped |
| + by root (§5), with the build its current generation serves; |
| - `carlos restart` requests, which arrive through `restart-gen.json`; |
| - config-revision changes; |
| - installs by hand and retries. |
| ### 6.2 Order within a cohort |
| -1. **Sleeping instances move at once.** The installed build is repointed |
| - (§7.7), and they boot it on their next wake, which is a first-wake |
| - canary (§6.4). There is no process, so there is nothing to pace. |
| +1. **Sleeping instances move at once.** Root installs their new build (§5), |
| + and they boot it on their next wake, which is a first-wake canary |
| + (§6.4). There is no process, so there is nothing to pace. |
| 2. **Awake instances swap.** The first swap in a cohort is its **canary**. |
| It must pass the health check and then stay up for a **10 s settle**. No |
| other swap in that cohort starts until the canary settles. After that, |
| @@ -452,74 +506,18 @@ generation, and its observers and sweeps learn the new unit name. The old |
| template stays until no box runs it, so a rolled-back binary still has the |
| unit it expects. |
| -### 7.7 Repoints go through an exact per-host request |
| - |
| -The sandboxed agent can't write `/opt`. Today it asks root through a marker |
| -that carries only a timestamp, and the root pass reads the channel head |
| -itself. So a repoint to any other build is impossible, and root can |
| -override `manual`, a pin or a failure hold. |
| - |
| -The agent now writes **one request file per host** under its data dir: |
| -`stage-requests/<host>.json`. The file holds the destination build, the |
| -installed build it expects to replace, the host's incarnation, the attempt |
| -identity and a sequence number. **The agent writes or replaces a request |
| -only while holding the host's guard**, using a temp file and a rename, so |
| -a newer request supersedes an older one. Root does its re-check and commit |
| -under the same guard (below). So a request can't be replaced between |
| -root's sequence check and its repoint. Root becomes an executor and |
| -decides nothing (§5). |
| - |
| -**Authorization root checks for itself.** A request is only as trustworthy |
| -as the sandboxed agent that wrote it, so root accepts destination B for |
| -host H only when it can verify one of these: |
| -- B is the head of H's channel, or in that channel's history, as read |
| - through the signed pointer (`release.Channel`, which verifies the |
| - pointer's signature). That covers channel moves, installs by hand, |
| - retries and fallback, because a good build was installed from that |
| - pointer. |
| -- A **console-signed pin authorization** binds account, app, H, B and the |
| - pin revision. The console signs it with the key that signs channel |
| - pointers when a pin is set. It travels on the provision record. |
| -- B is in **root's own install ledger** for H. Whenever root installs a |
| - build after verifying one of the two authorizations above, it keeps that |
| - signed evidence in a root-owned directory, `/opt/carlos/auth/<host>/`, |
| - which the agent can't write. Fallback to a good build is then accepted |
| - even after the channel has moved more than ten releases on (history is |
| - capped at ten, `internal/release/release.go`), after a channel change, |
| - or after the pin that authorized it has been replaced. Ledger entries |
| - are pruned along with the version sweep: an entry goes when its build |
| - is no longer pinned for that host (§8.5). |
| - |
| -A request that fails this check is refused, and the refusal is written as |
| -its outcome. Being a shipped build of the right app is not enough. |
| - |
| -**Execution under the host's guard.** For each request, root: |
| -1. stages B if needed (download and sha256 check, as `ensureStaged` does); |
| -2. takes the host's guard (`hostguard`, shared with the agent); |
| -3. re-reads the request file and checks it still carries the sequence it |
| - started with. A superseded request is dropped, and the newer one is |
| - handled in its turn; |
| -4. checks that the live symlink still names the expected build and that |
| - the incarnation matches; |
| -5. repoints; |
| -6. writes `SetVersions` with an expected-current condition; |
| -7. writes `stage-results/<host>.json` with that sequence and the outcome; |
| -8. releases the guard. |
| - |
| -The agent consumes only an outcome whose sequence matches its current |
| -request. If root dies between the repoint and the registry write, the next |
| -pass sees the symlink already at B with the request still pending, finishes |
| -the registry write, and reports the outcome. |
| - |
| -**The trigger is retransmitted.** `carlos-stage.path` uses `PathChanged` |
| -(`internal/host/units/carlos-stage.path`), so a marker touched while a |
| -pass is running is missed. While any request lacks an outcome with its |
| -sequence, the agent touches the marker again every `stageRequestEvery` |
| -(15 s), through the existing throttle. It also rescans the directory when |
| -it starts, so a restart doesn't lose pending requests. |
| - |
| -Whether root acts on the request files at all is decided by the protocol |
| -handshake (§7.11). |
| +### 7.7 Root's part: staging and installing |
| + |
| +Root's job is unchanged in kind. It stages and repoints channel-following |
| +hosts from the signed pointer, and directed hosts from their verified |
| +directive (§5). Swaps exec staged builds by path (§7.5 step 3), so they |
| +don't depend on when root repoints. A swap waits only for its build to be |
| +staged. Fallback execs the staged good build directly and needs nothing |
| +from root (§8.2). **The agent never asks root for a specific build.** The |
| +stage relay from #231 still triggers root passes, and still carries only a |
| +timestamp. The agent touches it again every `stageRequestEvery` while any |
| +exec route's channel head or directive isn't installed, so a touch missed |
| +during a running pass is repeated. |
| ### 7.8 A `hold` swap |
| @@ -558,65 +556,6 @@ contract (§2 decision 1) covers them the same way: |
| so the stopping generation is the current one, and its writes apply as |
| today. |
| -### 7.11 The agent–root protocol handshake |
| - |
| -The agent and root run the same binary, but at different times. Root runs |
| -`/opt/carlos/carlos` fresh on every stage pass. The agent keeps running |
| -whatever version it started with until it restarts. During a roll, or |
| -after a rollback, the two can disagree. The new behaviour is on only while |
| -both sides agree, and the agreement is tied to **identity, not liveness**: |
| -nothing expires on a timer. |
| - |
| -**Agent session.** When the agent starts, its first act is to write |
| -`stage-protocol.agent` in its data dir. The stamp names protocol 2 and |
| -the agent's **session**: its PID and the process start time from |
| -`/proc/<pid>/stat`. |
| - |
| -**Root picks its mode on every pass, from the live agent.** |
| -1. Root reads the main PID of `carlos-edge.service`. |
| -2. It runs that process's own executable (`/proc/<pid>/exe`) with |
| - `stage-protocol`. A binary from before this feature doesn't have the |
| - verb. |
| - - **The live agent's binary lacks protocol 2,** or no agent is running: |
| - **legacy mode**, which is today's channel-driven adoption. An old |
| - agent expects root to adopt from the channel. |
| - - **The live agent's binary has protocol 2:** **executor mode** for exec |
| - routes. Root carries out requests only if the stamp's session matches |
| - the live PID and start time. If it doesn't match (the agent has just |
| - restarted and hasn't stamped yet), root **installs nothing** for exec |
| - routes on that pass and waits for the next one. It never falls back |
| - to legacy for a live protocol-2 agent. |
| -3. Every executor-mode pass writes `stage-protocol.root`, which names the |
| - session it served. The file is in a root-owned place the agent can |
| - read. |
| -4. Named-unit routes always get legacy adoption (§5). |
| - |
| -**The agent enforces settings only when both hold:** |
| -- the root stamp names its own session; |
| -- the binary root will run next, `/opt/carlos/carlos`, has protocol 2. |
| - |
| -The agent checks the binary with `stage-protocol` whenever that file's |
| -inode or mtime changes. If the binary is replaced by one without protocol |
| -2 (a rollback), the agent stops writing requests and stops reporting |
| -settings as enforced. That's before the replaced binary's first root pass |
| -could run legacy adoption. Until both conditions hold, the agent keeps |
| -today's behaviour, converging onto whatever root installs with |
| -stop-then-wake, and the console shows the box as *settings not yet |
| -enforced*. |
| - |
| -**Why there's no expiry.** A session ends only when its agent process |
| -ends, and root sees that directly from the live PID. A hung agent leaves |
| -root in executor mode with nothing to carry out: installs stall, which is |
| -the safe direction. The previous session can't be revoked without its |
| -process ending, and a new session must be acknowledged before it enforces |
| -anything. |
| - |
| -**Rollback** below protocol 2 puts both sides back on today's behaviour. |
| -Root adopts from the channel again, so `manual` and pins stop being |
| -enforced on that box. The agent reports this, the console shows it for |
| -each affected instance, and the roll docs name it. The overlap swap is |
| -also gated by the box flag (§15), independently. |
| - |
| ## 8. Failure and rollback |
| ### 8.1 A failed overlap swap |
| @@ -633,9 +572,11 @@ the cause: |
| 1. A failure record for the build is written (§8.4). |
| 2. The cohort's queued attempts onto that build are cancelled. Its |
| attempts already past spawn are abandoned at their next check (§7.4). |
| - 3. Every instance in the cohort with that build installed that has never |
| - served it, sleeping instances included, gets an exact repoint (§7.7) |
| - back to its **good build**. |
| + 3. From then on, every wake of an instance in the cohort whose installed |
| + build is held by the record **boots its good build directly** from the |
| + staged path (§8.2). The symlink is left where root put it. No repoint |
| + is needed and none is asked for. Sleeping instances that were moved |
| + cold are covered this way too. |
| 4. Instances that have already served the build stay on it. |
| The halt is at the first failure, whether that's the canary or a later |
| @@ -688,20 +629,19 @@ If B fails: |
| 2. The good build is spawned from its staged path. Exec needs no write to |
| `/opt`. |
| 3. The waiting requests are served by it. |
| -4. The cohort-failure transition runs (§8.1). That writes the record, |
| - cancels the cohort's queued attempts, and repoints every never-served |
| - install of B in the cohort, this instance included, to its good build. |
| +4. The cohort-failure transition runs (§8.1). |
| -**When the wake picks a build,** if the installed build is held by a |
| -failure record, the wake boots the good build directly and asks for the |
| -repoint, instead of booting the failed build again. That is a second line |
| -of defence for a sleeper whose repoint hasn't landed yet. |
| +**When the wake picks a build:** if the installed build is held by a |
| +failure record, the wake boots the good build directly instead of the |
| +installed one. That is how every instance in a failed cohort keeps |
| +serving until a newer build, a retry or an install by hand clears the |
| +hold (§8.4). |
| -An instance with no good build has nothing to fall back to and fails as a |
| -wake fails today. §5's hold 4 means that only happens on an instance's |
| -very first build, and the next authorized build is free to replace it. In |
| -`hold` mode none of this falls back (§6.4). The settle rule for first wakes |
| -is in §6.4. |
| +An instance with no good build has nothing to fall back to, and fails as a |
| +wake fails today. That happens on an instance's very first build, during |
| +the bootstrap (§8.5), and on a box an instance has just moved to, until it |
| +has served there. In `hold` mode none of this falls back (§6.4). The settle |
| +rule for first wakes is in §6.4. |
| ### 8.3 A failed `hold` swap |
| @@ -713,19 +653,21 @@ cohort halts, and the deploy watch says so, naming `carlos rollback`. |
| A failure record names an **(app, build, box) scope**. It is stored in the |
| agent's data dir, so it survives agent restarts, and it is reported in box |
| -status. Resolver rule 3 holds that build for every instance of the app on |
| -that box. Clearing: |
| +status. While it stands, the scheduler makes no attempt onto that build for |
| +any instance of the app on that box, and wakes boot the good build instead |
| +(§8.2). Clearing: |
| -- **A newer build on an instance's channel** gives that instance a |
| - different head, so the record no longer matches it. The record stays for |
| - any instance still pointed at the failed build. |
| +- **A newer installed build** for an instance, from its channel or its |
| + directive, no longer matches the record. The record stays for any |
| + instance still on the failed build. |
| - **`carlos instances update <host>`** clears it for that host only, and |
| the next attempt runs as a canary. |
| -- **A pinned candidate** is matched by build and pin revision. Setting the |
| +- **A pinned build** is matched by build and directive revision. Setting the |
| pin again bumps the revision and authorizes one more attempt, for that |
| instance only. |
| - **`carlos deploy --retry`** writes an app-wide retry request with its own |
| - id into the release store. Each box clears its record for that build when |
| + id into the release store. It is signed like a channel pointer, so a box |
| + can't forge one for another. Each box clears its record for that build when |
| it sees a retry id newer than the record. This is how "deploy the same |
| build again" retries: promoting a build that's already current is a no-op |
| (`internal/release/release.go`) and can't be the signal. |
| @@ -741,64 +683,29 @@ ready (§6.5). It is written with an expected-current condition. Fallback |
| uses it, and `PrevVersion` keeps its old meaning. |
| **Bootstrap.** The column starts empty on every existing route, and nothing |
| -is inferred from `Version` or `PrevVersion`. |
| - |
| -Root seeds its install ledger, on its first executor-mode pass, with every |
| -exec host's installed build for which it can verify evidence now: the |
| -build is the signed head of the host's channel or in that channel's |
| -history. An installed build too old for that has no ledger entry. |
| - |
| -If that build later becomes the good build and a fallback needs it, the |
| -fallback still **serves**. The agent execs the staged good build directly |
| -(§8.2), and every wake picks the good build while the installed one is |
| -failure-held. Only root's repoint of the symlink is refused, and that is |
| -reported. The next authorized install puts it right. On the first start of an agent |
| +is inferred from `Version` or `PrevVersion`. On the first start of an agent |
| that has this feature, every instance that is awake and healthy records |
| -its serving build as good. A sleeping instance stays on its installed |
| -build (§5, hold 4) until its next wake serves that build and records it. |
| -Only after that can it move cold. Any instance that predates the feature |
| -therefore has a good build before it can need one. The version |
| -sweep (`internal/host/versionsweep.go`) adds to its pins `GoodVersion` and |
| -every build named by a live generation or a pending attempt (current, |
| -successor, retiring, spawning), which the activator reports. |
| - |
| -### 8.6 Build state travels with the instance |
| - |
| -The installed build, the good build and root's signed evidence for both |
| -(§7.7) are box-local. Without more, an instance that moves or is restored |
| -on a box that has never hosted it would look new, and would skip `manual` |
| -or a pin. |
| - |
| -So the agent keeps a small **build-state object** beside the instance's |
| -replica in the bucket, under the instance's own prefix, which the lease |
| -already guards. It holds: |
| -- the installed build; |
| -- the good build; |
| -- the pin revision it was installed under, if any; |
| -- the signed evidence, meaning the channel pointer or the pin |
| - authorization, that authorized each of those builds. |
| - |
| -The agent rewrites the object whenever any of these change, under the |
| -lease. A box only writes it while it holds the lease, so the object can't |
| -be written by a box that doesn't own the instance. |
| - |
| -**On the destination**, the agent reads the object before the instance's |
| -first wake there. The resolver treats its installed build as installed, |
| -and its good build as good. Root verifies the carried evidence like any |
| -other request (§7.7), because a signed pointer or pin authorization |
| -verifies the same anywhere. Then root installs the carried installed build, |
| -not the channel head. Its ledger gains the carried entries. A carried |
| -build whose evidence fails verification is refused and reported, and the |
| -instance then waits for an authorized install. It does not become new. |
| - |
| -The same applies to a restore onto a new box after a box is lost, and to |
| -moves (`internal/host/movespass.go`). |
| - |
| -### 8.7 Other cases |
| +its serving build as good. Every other instance records one the first time |
| +it serves. Until then it has no fallback, which is today's behaviour, not |
| +a regression. |
| + |
| +**The version sweep** (`internal/host/versionsweep.go`) adds to its pins |
| +`GoodVersion` and every build named by a live generation or a pending |
| +attempt: current, successor, retiring and spawning. The activator reports |
| +those. |
| + |
| +**Bucket retention** (`internal/host/retention.go`) adds to its protected |
| +set every build named by a directive on any provision record in the |
| +deployment. Moves and restores of directed instances install those builds |
| +from the bucket. If the housekeeper can't list the records, it prunes |
| +nothing that pass, just as it declines on any incomplete protected set |
| +today. |
| + |
| +### 8.6 Other cases |
| - **The agent restarts in the middle of a swap.** `Shutdown` cancels and |
| joins the whole operation set (§7.4). The next wake boots the installed |
| - build. Failure records and request files are on disk. |
| + build. Failure records are on disk. |
| - **The lease is lost in the middle of a swap.** Every generation is |
| stopped through the barrier before the lease state is acted on. |
| - **A crash after the switch and the settle** is an ordinary crash, with |
| @@ -882,14 +789,13 @@ keep timer names stable across the two builds. Otherwise deploy with |
| - The console checks only credentials it minted (§4.4). Bucket objects |
| under box prefixes and box-minted admin tokens are never authority for a |
| settings write. |
| -- Per-host request files are written by the agent and read by root, so root |
| - authorizes each one itself (§7.7). Root accepts only a build that a |
| - signed channel pointer names, as its head or in its history, for the |
| - host's own channel, or one that a console-signed pin authorization names |
| - for that host. A compromised agent can therefore choose among builds the |
| - operator has authorized for that host. It can't pick an unauthorized |
| - build of the same app. Ordinary adoption can already move between those |
| - builds, through rollback, so this adds no reach. |
| +- Root installs only what a signed channel pointer or a signed directive |
| + names (§5). The agent carries directives but can't forge them, and root |
| + never runs anything the agent controls. The replay residual is stated in |
| + §5.5. |
| +- The console accepts a box-reported installed build into a directive |
| + only if the app's signed history or a pin authorized it for that |
| + instance (§5.3). |
| - `pin` must name a shipped build of the same app. The console resolves it |
| against the app's manifests, never as a path. |
| @@ -911,44 +817,32 @@ keep timer names stable across the two builds. Otherwise deploy with |
| good build, supersession and re-checks, attempt-identity dedupe, the |
| restart and config failure paths, and config acknowledgement only after |
| a switch. |
| -- **Resolver:** a table test of §5 over pin, manual, failure, major, bake |
| - holds and untagged heads. The test asserts that the root pass and the |
| - agent resolve identically from the same inputs. |
| -- **Root requests:** an exact repoint, a refused repoint when the expected |
| - build doesn't match, validation refusals, an old root binary ignoring |
| - the directory, and `SetVersions` conditions. |
| +- **Directive computation (console):** a table test of §5.3 over pin, |
| + manual, major, bake holds and untagged heads. Reported builds outside |
| + the instance's signed history are refused. Directives are dropped when |
| + settings return to default. |
| +- **Directive verification (root):** bad signatures, the wrong host or |
| + incarnation, a replay below the high-water mark, and an old root binary |
| + ignoring directives are each refused or reported. Channel-following |
| + hosts adopt exactly as today. |
| - **Containment:** per-generation children, the parent-limit raise and |
| restore with `limitsOK`, retiring one generation without touching the |
| other, and parent stop evidence. Include a memory-hog successor that |
| overruns its child limit and is OOM-killed while the incumbent keeps |
| serving. |
| -- **Root authorization and execution:** |
| - - refuses an unauthorized shipped build; |
| - - accepts a pin authorization, a history entry, and a ledger entry for a |
| - good build that has left the channel's ten-entry history; |
| - - handles a request replaced mid-download, with the agent's write and |
| - root's commit both under the guard; |
| - - recovers when root dies between the repoint and the registry write; |
| - - resends the marker while a request has no outcome, including across an |
| - agent restart. |
| -- **Handshake:** every pairing of new and old agent with new and old root, |
| - including a legacy pass already running when the agent declares itself |
| - and a rollback below protocol 2, leaves exactly one decider per exec |
| - route. |
| - **Scheduled work:** a successor's writes apply immediately with the |
| fence. A failed successor's acknowledged registrations survive. A |
| retiring generation's writes are refused after the switch. |
| - **`hold` carried everywhere:** a `hold` build that fails after exec, on a |
| first wake, a settle or a worker replacement, never falls back. |
| -- **Bootstrap recovery:** an instance whose only build never started moves |
| - to the next authorized build. |
| -- **Handshake identity:** an agent restart between root passes suspends |
| - installs; it never triggers legacy adoption. A rolled-back binary makes |
| - the agent stop enforcing before root's next pass. No timer ends a |
| - session. |
| -- **Carried build state:** a `manual` instance moved to a fresh box comes up |
| - on its carried build, not the channel head. A forged or unverifiable |
| - carried build is refused, and the instance doesn't become new. |
| +- **Failure without a repoint:** after a cohort failure, wakes of |
| + sleepers moved cold boot the good build directly while the symlink still |
| + names the failed build. A newer build, a retry or an install by hand |
| + lifts the hold. |
| +- **Moves of directed instances:** a `manual` instance moved to a fresh |
| + box installs its directive's build, not the channel head. |
| +- **Retention:** directive builds are protected deployment-wide. An |
| + unlistable record set prunes nothing. |
| - **Config activation:** a failed config swap keeps delivered ≠ active, |
| retries with backoff even though the file is unchanged, and acknowledges |
| only once the revision is active. |
| @@ -990,7 +884,7 @@ take a third kind of attempt for them, and #230 stays open for that spec. |
| Until that follow-up, named-unit routes, the console's own route included, |
| keep **today's adoption unchanged**. Root reads their channel, stages, |
| repoints, restarts the unit with its rollback-on-failed-restart, and |
| -stamps the result. Nothing in this spec touches that path (§5, §7.11). |
| +stamps the result. Nothing in this spec touches that path (§5.4). |
| ## 14. Out of scope |
| @@ -998,9 +892,10 @@ stamps the result. Nothing in this spec touches that path (§5, §7.11). |
| - Keeping instances up across an agent restart or box roll. |
| - A configurable health path, or a settle on swaps after the canary. |
| - Instance-only console principals (§4.4). |
| -- The level-triggered stage marker noted on #231. §7.7 resends the |
| - marker while any request has no outcome, which closes the gap without |
| - changing the unit. |
| +- The level-triggered stage marker noted on #231. The agent touches the |
| + marker again every `stageRequestEvery` (15 s) while any exec route's |
| + channel head or directive isn't installed. That narrows the gap without |
| + a unit change (§7.7). |
| ## 15. Rollout of this feature |
| @@ -1009,11 +904,13 @@ The box-side swap lands **dormant** behind a box flag, |
| uses today's stop-then-wake. It is proven on dev and then on |
| `bab/adopt-probe`. |
| -The resolver, the per-host requests and the handshake are not behind the |
| -flag. With every setting at its default, they produce today's result. |
| +Directives aren't behind the flag. An instance with default settings has |
| +none and is adopted exactly as today. A box whose root binary predates |
| +directives shows *settings not enforced* for its directed instances (§5.5). |
| -The console's settings and policy can ship first. Until a box has |
| -completed the handshake, it shows *not yet enforced*. |
| +**Order:** boxes first, then the console's settings, policy and |
| +directives. That way no directive is issued before the boxes can honour |
| +it. |
| The changelog and the deploy docs tell implementers about the migrations |
| rule (§9.5) before the flag goes on across the fleet. From then on, every |
| @@ -1084,3 +981,13 @@ exec app swaps in `overlap` unless it has chosen `hold`. |
| | R3-4 (partial) | Existing installs have no ledger | §8.5 bootstrap: the ledger is seeded from verifiable evidence; an unverifiable good build still serves through direct exec, and only the repoint is refused | |
| | 2 | A move skips `manual` | §5 definition of new; §8.6 build state carried beside the replica under the lease, with signed evidence | |
| | 3 | Config is never retried when the file is unchanged | §8.1: delivered and active revisions; a pending config attempt with backoff until active | |
| + |
| +### Round 5 |
| + |
| +| # | Finding | Answer | |
| +|---|---------|--------| |
| +| R3-3 (partial) | Agent restarts re-enable legacy adoption | Superseded: root's mode no longer depends on the agent. Directed hosts follow signed directives and the rest follow their channel (§5) | |
| +| 1 | Root executes an agent-controlled binary | Superseded: no handshake; root never runs anything the agent controls (§5, §11) | |
| +| 2 | A lease doesn't authenticate carried state | Superseded: no carried state; directives are console-signed and bound to host and incarnation (§5.2, §5.5) | |
| +| 3 | No agent means legacy adoption | Superseded, as for R3-3 | |
| +| 4 | Retention can delete a directed build | §8.5: directive builds are protected deployment-wide; an unlistable record set prunes nothing | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R5.md b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R5.md| new file mode 100644 |
| index 000000000..b7db3f872 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R5.md |
| @@ -0,0 +1,39 @@ |
| +REJECT |
| + |
| +R4-1: CLOSED — Session identity removes stamp expiry. |
| +R4-2: CLOSED — Carried installation state distinguishes relocation from a new instance. |
| +R4-3: CLOSED — Delivered/active revisions provide an independent retry trigger. |
| +R3-3 partial: PARTIAL — The revised handshake still permits legacy adoption during an agent restart; finding 3 below. |
| +R3-4 partial: CLOSED — Ledger seeding and the explicitly stated direct-exec fallback address the bootstrap gap. |
| + |
| +1. **HIGH — §7.11: Root executes an executable controlled by the unprivileged agent.** |
| + |
| + **Code:** [internal/host/units/carlos-edge.service:28](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/units/carlos-edge.service:28) runs the agent as `carlos`, with writable `/data` at line 37. [internal/host/units/carlos-stage.service:34](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/units/carlos-stage.service:34) explicitly runs staging as root. |
| + |
| + **Failure:** A compromised agent writes an executable under `/data` and replaces itself with it using `execve`. Its PID and process start time remain unchanged, and systemd still identifies it as the main process. The next root pass executes `/proc/<pid>/exe stage-protocol`, running the attacker’s executable as root. Neither session matching nor the proposed release-authorization checks prevent this. |
| + |
| + **Smallest fix:** Determine protocol support without executing agent-controlled code as root. Identify the opened executable against authenticated, root-owned platform release metadata; treat an unknown executable as a refusal. Keep executable identity and session checks bound to the same observation. |
| + |
| +2. **HIGH — §8.6: A lease does not authenticate the carried installation state.** |
| + |
| + **Code:** [internal/lease/lease.go:114](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/lease/lease.go:114) acquires a separate `lock.json`; it does not restrict other object writes. [infra/modules/region-host/main.tf:166](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/infra/modules/region-host/main.tf:166) grants writes throughout the region’s data bucket. The signed [ChannelPointer at internal/release/release.go:116](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/release.go:116) binds account, app, channel and builds—not a host’s installation or its writer. |
| + |
| + **Failure:** H is manual on A. Another compromised box with bucket access replaces H’s build-state object with `installed=B` and a genuine signed pointer authorizing B for the same app and channel. On relocation or recovery, the destination accepts the signature and treats B as already installed. Manual resolution therefore preserves B and boots it without H’s owner requesting installation. The signature authenticates the release; it does not authenticate the assertion that H installed it. The statement that a non-owner cannot write this object is false. |
| + |
| + **Smallest fix:** Authenticate the selected installation separately from release authorization. Use console-owned installation state or a verifiable installation receipt binding host, incarnation, build and revision, with ownership and replay checks. A channel signature inside an otherwise writable object is insufficient. |
| + |
| +3. **HIGH — §7.11: “No agent running” re-enables legacy adoption during an ordinary restart.** |
| + |
| + **Code:** [internal/host/units/carlos-edge.service:23](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/units/carlos-edge.service:23) specifies `Restart=always` and `RestartSec=1`. Legacy adoption selects the channel head at [internal/release/adopt.go:943](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:943) and repoints at [line 1072](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:1072). |
| + |
| + **Failure:** A protocol-2 agent maintains manual H at A while its channel names B. The agent exits. During the restart interval, a stage or hourly pass observes no main PID and, exactly as §7.11 requires, selects legacy mode and installs B. The replacement protocol-2 agent subsequently establishes its session; manual resolution now holds the already-installed B. No platform rollback occurred. This also contradicts §12’s requirement that an agent restart suspend installations rather than trigger legacy adoption. |
| + |
| + **Smallest fix:** Persist executor mode across agent absence. Once activated, a missing or changing session must suspend exec installations. Returning to legacy mode requires an explicit, serialized downgrade transition that drains adoption work. |
| + |
| +4. **MEDIUM — §§4.1, 8.5–8.6: Bucket retention can delete a build required by a manual or pinned instance.** |
| + |
| + **Code:** [internal/host/retention.go:32](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/retention.go:32) protects only the pruning box’s registry `Version` and `PrevVersion`. [internal/release/prune.go:20](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/prune.go:20) otherwise protects recent releases, channel history and tags. A fresh destination requires the manifest and artifact at [internal/release/adopt.go:1398](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:1398). |
| + |
| + **Failure:** With retention enabled, H remains on an old, untagged build A on one box. Another box becomes housekeeper and has only newer builds in its registry. Once A leaves channel history and the retention window, that box deletes A from the release bucket. H continues from its local staged copy, but a later move or box-loss recovery cannot install A. The carried signature remains valid, yet its artifact is gone. §8.5 changes only the local version sweep. |
| + |
| + **Smallest fix:** Extend bucket retention’s protection set to installation, pin and good-build state across the deployment, including carried state needed for recovery. Refuse pruning when that protection set cannot be established. |
| \ No newline at end of file |