amadan
spec: zero-downtime deploys rev 5 — answers Astra round 4 (3 new, 2 partial)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+176
−38
docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md+140 −38docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R4.md+36 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md| index 8606a4ce4..ec0cf7cfd 100644 |
| --- a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| +++ b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| @@ -1,10 +1,11 @@ |
| # Zero-downtime deploys: overlap swaps, update settings, paced rollout |
| -Status: draft rev 4, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| +Status: draft rev 5, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| (per-app restart pace, first-route health step) feed into this. Astra |
| rejected rev 1 with 17 findings (`CODEX-FINDINGS-ZERO-DOWNTIME.md`) and |
| rev 2 with 12 new ones (`CODEX-FINDINGS-ZERO-DOWNTIME-R2.md`), and |
| -rev 3 with 7 new ones and 3 partials (`CODEX-FINDINGS-ZERO-DOWNTIME-R3.md`). §16 maps every finding to its answer. |
| +rev 3 with 7 new ones and 3 partials (`CODEX-FINDINGS-ZERO-DOWNTIME-R3.md`), |
| +and rev 4 with 3 new ones and 2 partials (`CODEX-FINDINGS-ZERO-DOWNTIME-R4.md`). §16 maps every finding to its answer. |
| ## 1. Why |
| @@ -179,8 +180,10 @@ installed build: |
| instance whose only build never started moves to any other authorized |
| candidate, and that build's first wake is its canary. |
| -An instance with neither an installed nor a good build is new. It resolves |
| -to the candidate, ignoring `manual`. |
| +An instance is **new** when it has no installed build, no good build and |
| +no carried build state (§8.6). A new instance resolves to the candidate, |
| +ignoring `manual`. An instance arriving by a move or a restore is not new. |
| +It starts from its carried state. |
| If no hold applies, the desired build is the candidate. |
| @@ -560,34 +563,59 @@ contract (§2 decision 1) covers them the same way: |
| The agent and root run the same binary, but at different times. Root runs |
| `/opt/carlos/carlos` fresh on every stage pass. The agent keeps running |
| whatever version it started with until it restarts. During a roll, or |
| -after a rollback, the two can disagree. The new behaviour turns on only |
| -when both sides agree. |
| - |
| -**The stamps.** |
| -- The agent writes `stage-protocol.agent` in its data dir, naming protocol |
| - 2, with the time. |
| -- Root writes `stage-protocol.root` in a root-owned place the agent can |
| - read, naming protocol 2, the agent stamp it saw, and the time. |
| - |
| -**What each side does.** |
| -- **Root** runs as an executor (§7.7) for exec routes only while the agent |
| - stamp says 2 and is fresher than `CARLOS_HIBERNATE_IDLE`. Otherwise it |
| - runs today's channel-driven adoption. Named-unit routes always get |
| - today's adoption (§5). |
| -- **The agent** enforces update settings and requests exact repoints only |
| - once the root stamp says 2 *and* names the agent stamp it is answering. |
| - Only a root pass that started after the agent declared itself can have |
| - written that, so no legacy pass that began earlier is still running. |
| - carlos-stage.service runs one pass at a time. |
| -- **Until both sides agree**, the agent keeps today's behaviour: it |
| - converges onto whatever root installs, with stop-then-wake. The console |
| - shows the box as *settings not yet enforced*. |
| - |
| -**Rollback.** If the binary is rolled back below protocol 2, both sides go |
| -back to today's behaviour. Root adopts from the channel again, so `manual` |
| -and pins stop being enforced on that box. The console shows that state for |
| -each affected instance, and the roll docs name it. The overlap swap itself |
| -is also gated by the box flag (§15), independently. |
| +after a rollback, the two can disagree. The new behaviour is on only while |
| +both sides agree, and the agreement is tied to **identity, not liveness**: |
| +nothing expires on a timer. |
| + |
| +**Agent session.** When the agent starts, its first act is to write |
| +`stage-protocol.agent` in its data dir. The stamp names protocol 2 and |
| +the agent's **session**: its PID and the process start time from |
| +`/proc/<pid>/stat`. |
| + |
| +**Root picks its mode on every pass, from the live agent.** |
| +1. Root reads the main PID of `carlos-edge.service`. |
| +2. It runs that process's own executable (`/proc/<pid>/exe`) with |
| + `stage-protocol`. A binary from before this feature doesn't have the |
| + verb. |
| + - **The live agent's binary lacks protocol 2,** or no agent is running: |
| + **legacy mode**, which is today's channel-driven adoption. An old |
| + agent expects root to adopt from the channel. |
| + - **The live agent's binary has protocol 2:** **executor mode** for exec |
| + routes. Root carries out requests only if the stamp's session matches |
| + the live PID and start time. If it doesn't match (the agent has just |
| + restarted and hasn't stamped yet), root **installs nothing** for exec |
| + routes on that pass and waits for the next one. It never falls back |
| + to legacy for a live protocol-2 agent. |
| +3. Every executor-mode pass writes `stage-protocol.root`, which names the |
| + session it served. The file is in a root-owned place the agent can |
| + read. |
| +4. Named-unit routes always get legacy adoption (§5). |
| + |
| +**The agent enforces settings only when both hold:** |
| +- the root stamp names its own session; |
| +- the binary root will run next, `/opt/carlos/carlos`, has protocol 2. |
| + |
| +The agent checks the binary with `stage-protocol` whenever that file's |
| +inode or mtime changes. If the binary is replaced by one without protocol |
| +2 (a rollback), the agent stops writing requests and stops reporting |
| +settings as enforced. That's before the replaced binary's first root pass |
| +could run legacy adoption. Until both conditions hold, the agent keeps |
| +today's behaviour, converging onto whatever root installs with |
| +stop-then-wake, and the console shows the box as *settings not yet |
| +enforced*. |
| + |
| +**Why there's no expiry.** A session ends only when its agent process |
| +ends, and root sees that directly from the live PID. A hung agent leaves |
| +root in executor mode with nothing to carry out: installs stall, which is |
| +the safe direction. The previous session can't be revoked without its |
| +process ending, and a new session must be acknowledged before it enforces |
| +anything. |
| + |
| +**Rollback** below protocol 2 puts both sides back on today's behaviour. |
| +Root adopts from the channel again, so `manual` and pins stop being |
| +enforced on that box. The agent reports this, the console shows it for |
| +each affected instance, and the roll docs name it. The overlap swap is |
| +also gated by the box flag (§15), independently. |
| ## 8. Failure and rollback |
| @@ -614,10 +642,22 @@ the cause: |
| swap. |
| - **`restart`, `config`.** No build changes, and no failure record is |
| written. The current generation keeps serving the build it was serving. |
| - The attempt is reported failed. A config revision is acknowledged |
| - (`internal/appconfig/syncexec.go`) only after a successful switch, so a |
| - failed activation stays unacknowledged and is retried by config sync's |
| - existing rules. |
| + The attempt is reported failed. |
| + |
| + **Config is tracked as two revisions per instance: delivered and |
| + active.** Delivery is config sync writing the file. Activation is a |
| + generation started with that file becoming current. A revision is |
| + acknowledged (`internal/appconfig/syncexec.go`) only when it becomes |
| + active. |
| + |
| + Config sync returns early when the rendered file hasn't changed |
| + (`syncexec.go`), so it can't be what retries. Instead, while delivered ≠ |
| + active, the scheduler keeps a pending `config` attempt for the instance |
| + and retries it with backoff, from 1 minute up to 30 minutes. Status shows |
| + *config delivered, not active*, with the last failure. A newer delivered |
| + revision replaces the pending attempt. A sleeping instance needs no |
| + attempt: its next wake starts with the delivered file, and that makes it |
| + active. |
| ### 8.2 A failed first wake (`overlap` mode) |
| @@ -701,7 +741,18 @@ ready (§6.5). It is written with an expected-current condition. Fallback |
| uses it, and `PrevVersion` keeps its old meaning. |
| **Bootstrap.** The column starts empty on every existing route, and nothing |
| -is inferred from `Version` or `PrevVersion`. On the first start of an agent |
| +is inferred from `Version` or `PrevVersion`. |
| + |
| +Root seeds its install ledger, on its first executor-mode pass, with every |
| +exec host's installed build for which it can verify evidence now: the |
| +build is the signed head of the host's channel or in that channel's |
| +history. An installed build too old for that has no ledger entry. |
| + |
| +If that build later becomes the good build and a fallback needs it, the |
| +fallback still **serves**. The agent execs the staged good build directly |
| +(§8.2), and every wake picks the good build while the installed one is |
| +failure-held. Only root's repoint of the symlink is refused, and that is |
| +reported. The next authorized install puts it right. On the first start of an agent |
| that has this feature, every instance that is awake and healthy records |
| its serving build as good. A sleeping instance stays on its installed |
| build (§5, hold 4) until its next wake serves that build and records it. |
| @@ -711,7 +762,39 @@ sweep (`internal/host/versionsweep.go`) adds to its pins `GoodVersion` and |
| every build named by a live generation or a pending attempt (current, |
| successor, retiring, spawning), which the activator reports. |
| -### 8.6 Other cases |
| +### 8.6 Build state travels with the instance |
| + |
| +The installed build, the good build and root's signed evidence for both |
| +(§7.7) are box-local. Without more, an instance that moves or is restored |
| +on a box that has never hosted it would look new, and would skip `manual` |
| +or a pin. |
| + |
| +So the agent keeps a small **build-state object** beside the instance's |
| +replica in the bucket, under the instance's own prefix, which the lease |
| +already guards. It holds: |
| +- the installed build; |
| +- the good build; |
| +- the pin revision it was installed under, if any; |
| +- the signed evidence, meaning the channel pointer or the pin |
| + authorization, that authorized each of those builds. |
| + |
| +The agent rewrites the object whenever any of these change, under the |
| +lease. A box only writes it while it holds the lease, so the object can't |
| +be written by a box that doesn't own the instance. |
| + |
| +**On the destination**, the agent reads the object before the instance's |
| +first wake there. The resolver treats its installed build as installed, |
| +and its good build as good. Root verifies the carried evidence like any |
| +other request (§7.7), because a signed pointer or pin authorization |
| +verifies the same anywhere. Then root installs the carried installed build, |
| +not the channel head. Its ledger gains the carried entries. A carried |
| +build whose evidence fails verification is refused and reported, and the |
| +instance then waits for an authorized install. It does not become new. |
| + |
| +The same applies to a restore onto a new box after a box is lost, and to |
| +moves (`internal/host/movespass.go`). |
| + |
| +### 8.7 Other cases |
| - **The agent restarts in the middle of a swap.** `Shutdown` cancels and |
| joins the whole operation set (§7.4). The next wake boots the installed |
| @@ -859,6 +942,16 @@ keep timer names stable across the two builds. Otherwise deploy with |
| first wake, a settle or a worker replacement, never falls back. |
| - **Bootstrap recovery:** an instance whose only build never started moves |
| to the next authorized build. |
| +- **Handshake identity:** an agent restart between root passes suspends |
| + installs; it never triggers legacy adoption. A rolled-back binary makes |
| + the agent stop enforcing before root's next pass. No timer ends a |
| + session. |
| +- **Carried build state:** a `manual` instance moved to a fresh box comes up |
| + on its carried build, not the channel head. A forged or unverifiable |
| + carried build is refused, and the instance doesn't become new. |
| +- **Config activation:** a failed config swap keeps delivered ≠ active, |
| + retries with backoff even though the file is unchanged, and acknowledges |
| + only once the revision is active. |
| - **Generation identity:** a stop, then a crash, then a wake never reuses a |
| generation number, and an aliased cache entry from the earlier entry |
| misses. |
| @@ -982,3 +1075,12 @@ exec app swaps in `overlap` unless it has chosen `hold`. |
| | 5 | Generation numbers repeat across entries | §7.1: one counter for the whole process | |
| | 6 | Discarded candidate timers strand work | §7.10: successor writes apply immediately; timers are shared state under the overlap contract | |
| | 7 | Named-unit adoption has no path | §5, §13: named-unit routes keep today's root adoption unchanged | |
| + |
| +### Round 4 |
| + |
| +| # | Finding | Answer | |
| +|---|---------|--------| |
| +| R3-3 (partial), 1 | Stamp expiry restores legacy adoption | §7.11: session is the live agent's PID and start time; root's mode comes from the live agent's binary; a mismatch suspends installs; nothing expires; the agent watches `/opt/carlos/carlos` for a rollback | |
| +| R3-4 (partial) | Existing installs have no ledger | §8.5 bootstrap: the ledger is seeded from verifiable evidence; an unverifiable good build still serves through direct exec, and only the repoint is refused | |
| +| 2 | A move skips `manual` | §5 definition of new; §8.6 build state carried beside the replica under the lease, with signed evidence | |
| +| 3 | Config is never retried when the file is unchanged | §8.1: delivered and active revisions; a pending config attempt with backoff until active | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R4.md b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R4.md| new file mode 100644 |
| index 000000000..64cb3470b |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R4.md |
| @@ -0,0 +1,36 @@ |
| +REJECT |
| + |
| +R3-1: CLOSED — Hold mode governs every fallback path. |
| +R3-2: CLOSED — A failed bootstrap build can receive its replacement. |
| +R3-3: PARTIAL — The new handshake permits disagreement after stamp expiry; finding 1 below. |
| +R3-4: PARTIAL — Existing installations receive `GoodVersion` without a defined way to populate their authorization ledger. |
| +R3-5: CLOSED — Generation numbers cannot repeat within the edge process. |
| +R3-6: CLOSED — Acknowledged successor timer writes survive failure. |
| +R3-7: CLOSED — Named-unit adoption is explicitly preserved. |
| +R2-4 partial: CLOSED — Request replacement takes the shared guard. |
| +R1-4 partial: CLOSED — Root’s sequence check and commit exclude request replacement. |
| +R1-11 partial: CLOSED — Deadline arithmetic accounts for preliminary work, with timeout behavior explicitly defined. |
| + |
| +1. **HIGH — §7.11: Stamp expiry restores root adoption while the agent still considers settings enforced.** |
| + |
| + **Code:** [internal/activator/activator.go:117](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:117) defaults the idle duration to 15 minutes. Legacy adoption selects the channel head at [internal/release/adopt.go:942](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:942) and repoints at [:1072](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:1072). |
| + |
| + **Failure:** Root acknowledges agent stamp S, and an instance remains at A under `updates=manual`. Once S exceeds the idle duration, root must resume channel-driven adoption and install B. The agent’s acceptance condition still passes: the root acknowledgement names S and protocol 2. No renewal or revocation procedure is specified. A subsequent wake runs B while the agent still reports enforcement. Adding periodic renewal alone leaves the same failure during a delayed heartbeat. |
| + |
| + **Smallest fix:** Separate session identity from liveness. Expired liveness must suspend installations while retaining executor mode. Returning to legacy adoption requires a coordinated transition that revokes the agent’s authority and drains pending work. |
| + |
| +2. **MEDIUM — §§4.1, 5, 8.5, 10: Moving an instance silently defeats `manual`.** |
| + |
| + **Code:** [internal/host/movespass.go:480](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/movespass.go:480) mirrors the database replica; [:493](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/movespass.go:493) completes the residence change. Destination reconciliation constructs a route without installed-version history at [internal/provision/reconcile.go:394](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/provision/reconcile.go:394). |
| + |
| + **Failure:** H runs A with `updates=manual`; its channel advances to B. H moves to a box that has never hosted it. Its settings survive in the provision record, but its installed build and `GoodVersion` remain source-local. The destination satisfies §5’s definition of “new”—neither installed nor good—and explicitly ignores `manual`, installing B against H’s existing database without an install or pin request. |
| + |
| + **Smallest fix:** Preserve the selected installed build and good-build identity, with destination-verifiable authorization, through relocation. Distinguish a rematerialized instance from a genuinely new instance before applying the bootstrap exception. |
| + |
| +3. **MEDIUM — §8.1: Config sync does not retry an unacknowledged activation when the delivered file is unchanged.** |
| + |
| + **Code:** [internal/appconfig/syncexec.go:1280](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/appconfig/syncexec.go:1280) returns immediately when rendered config matches the file. The file is installed before the restart callback at [:1302](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/appconfig/syncexec.go:1302) and [:1312](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/appconfig/syncexec.go:1312). |
| + |
| + **Failure:** Config revision C is written, but its successor fails health checking. The incumbent keeps serving the previous config and C remains unacknowledged, as specified. Subsequent config syncs see identical file contents and return without requesting another activation. §8.1’s reliance on “config sync’s existing rules” leaves an otherwise healthy, continuously awake instance on old config indefinitely. |
| + |
| + **Smallest fix:** Track delivered and successfully activated revisions separately. Retain or recreate the pending config attempt until activation succeeds, even when the file contents already match. |
| \ No newline at end of file |