amadan
spec: zero-downtime deploys rev 10 — answers Astra round 9 (artifact-bound swap policy, worker recycles keep the serving build)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+83
−11
docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md+61 −11docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R9.md+22 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md| index 77b9f9058..7e09dc282 100644 |
| --- a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| +++ b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| @@ -1,6 +1,6 @@ |
| # Zero-downtime deploys: overlap swaps and paced rollout |
| -Status: draft rev 9, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| +Status: draft rev 10, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| (per-app restart pace, first-route health step) feed into this. Astra |
| rejected rev 1 with 17 findings, rev 2 with 12 new ones, rev 3 with 7, rev 4 |
| with 3 and rev 5 with 4. The findings files are |
| @@ -110,16 +110,34 @@ sets it when the release is shipped. An attempt onto build B runs in |
| `hold` if either the app policy or B's manifest says `hold`. Otherwise it |
| runs in `overlap`. |
| -**B's demand is read from its canonical manifest** in the release store. |
| -That manifest is content-addressed, and its sha is named by the signed |
| -pointer. The box's staged copy isn't trusted for this, because staging |
| -re-marshals the parsed struct (`internal/release/adopt.go`), so a root |
| -binary older than this field writes copies without it. Staging also |
| -changes to keep the manifest's raw bytes, so copies staged from now on are |
| -lossless. |
| - |
| -**If the canonical manifest can't be read, the attempt runs in `hold`.** |
| -Missing policy data never permits an overlap. |
| +**B's demand is read from its canonical manifest** in the release store, |
| +not from the box's staged copy. Staging re-marshals the parsed struct |
| +(`internal/release/adopt.go`), so a root binary older than this field |
| +writes copies without it. Staging also changes to keep the manifest's raw |
| +bytes, so copies staged from now on are lossless. |
| + |
| +**The policy must describe the bytes that will run.** The signed pointer |
| +names a version, not a manifest hash (`ChannelPointer`, |
| +`internal/release/release.go`). A version name can be shipped again with |
| +different content after pruning. And root reuses an existing staged |
| +directory without comparing it with the canonical artifact |
| +(`adopt.go`'s staging). So before any attempt onto B: |
| +- **The agent compares the staged executable's sha256** with the |
| + canonical manifest's artifact digest for the box's platform. |
| +- **On a match**, the manifest's `swap` applies to those bytes. |
| +- **On a mismatch**, the attempt is refused as a staging error, with no |
| + failure record against B. It is reported, and the stage relay is |
| + touched. |
| +- **Root's staging** checks a reused staged directory's digest against |
| + the canonical manifest and restages on a mismatch, so the next pass |
| + repairs it. |
| + |
| +A good build that is still staged was verified when it last ran, and |
| +fallback runs those same bytes (§8.2). |
| + |
| +**If the canonical manifest can't be read, or the digests can't be |
| +compared, the attempt runs in `hold`.** Missing or unverifiable policy |
| +data never permits an overlap. |
| ## 5. Which build an instance installs |
| @@ -446,6 +464,24 @@ a `release` attempt** with the scheduler, instead of respawning straight |
| away, so pacing, the canary and failure records apply to sidecars as to |
| any host. |
| +**Automatic recycling never changes a worker's build.** Two paths respawn |
| +a worker today by executing the live symlink |
| +(`backend_exec.go`'s worker start): |
| +- crash recovery (`respawnWorkerLocked` from the crash path); |
| +- an always-on worker reaching `max_run`. |
| + |
| +Both now respawn the worker's **serving build**, through its immutable |
| +staged path, inside the operation set (§7.4). Only the paced `release` |
| +attempt moves a worker to a different installed build. |
| + |
| +If a recycle falls due while a `release` attempt for that host is queued |
| +and the scheduler would admit it now (it is the canary, or the canary has |
| +settled and a concurrency slot is free), the two **coalesce**: the attempt |
| +runs instead, and there is one spawn. Otherwise the recycle runs on the |
| +serving build, and the attempt keeps its place in the queue. A serving |
| +build is never held by a failure record, because it has already served |
| +on that host (§8.4). |
| + |
| For a sidecar, the **installed build** is what root stamped. The |
| **serving build** is the version the running worker reports. These are |
| kept apart: the worker-reported version is recorded as serving and no |
| @@ -731,6 +767,13 @@ keep timer names stable across the two builds. Otherwise deploy with |
| - **Swap mode source:** a staged copy without `swap`, from an older root, |
| can't produce an overlap for a `hold` build. An unreadable canonical |
| manifest runs in `hold`. |
| +- **Artifact binding:** a re-shipped version name with different bytes, |
| + plus an old staged copy, refuses the attempt as a staging error, and |
| + root restages. An unverifiable digest runs in `hold`. |
| +- **Worker recycling:** a crash or `max_run` recycle during another |
| + worker's canary respawns the serving build, not the installed one. |
| + `swap_concurrency=1` holds across recycles. A recycle that is admissible |
| + coalesces with the queued attempt. |
| - **Sidecars:** `SidecarAdoptable` admits sidecars, and `Adoptable`'s |
| other callers are unchanged. A channel advance is staged, repointed and |
| then queued as a paced `release` attempt. A new sidecar materializes. |
| @@ -929,3 +972,10 @@ answer for each. |
| |---|---------|--------| |
| | R7-2 (open) | Sidecars have no channel resolution | §7.9: a `SidecarAdoptable` leg in root's pass, with the same signed selection, bake, staging and materialization; the drift check then queues a paced attempt | |
| | R7-5 (partial), 1 | An idle scheduler reports done before B is observed | §9.1: completion per targeted instance, tied to the watched build: pending until B's install is observed, then a final state | |
| + |
| +### Round 9 |
| + |
| +| # | Finding | Answer | |
| +|---|---------|--------| |
| +| 1 | Canonical policy can describe different bytes from the staged executable | §4.2: compare the staged digest with the canonical artifact digest before any attempt; a mismatch is a staging error; root restages on a mismatch; unverifiable → `hold`. Corrects the rev-8 claim that the pointer names a manifest hash | |
| +| 2 | Worker crash and `max_run` respawns bypass pacing | §7.9: recycles respawn the serving build by immutable path inside the operation set, and coalesce with an admissible queued attempt | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R9.md b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R9.md| new file mode 100644 |
| index 000000000..343afa3a7 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R9.md |
| @@ -0,0 +1,22 @@ |
| +REJECT |
| + |
| +R7-2: **CLOSED** — §7.9 specifies a separate sidecar adoption and materialization leg. |
| +Round-8 finding 1: **CLOSED** — §9.1 requires observation of the watched build’s installation before completion. |
| + |
| +No split/removal regression found in the earlier fixes for fallback without repointing, channel-installed failure holds, cold sleepers, `GoodVersion` versus `PrevVersion`, or stage-trigger retransmission. |
| + |
| +1. **HIGH — §§4.2, 7.5, 8.5: The canonical swap policy can describe different bytes from the staged executable.** |
| + |
| + **Code:** [`internal/release/release.go:103`](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/release.go:103) defines a signed pointer containing a version name, **not a manifest hash**. [`release.go:632`](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/release.go:632) explicitly permits different content under that name after pruning. [`internal/release/adopt.go:1374`](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:1374) accepts an existing staged executable and manifest without fetching or comparing the canonical artifact. |
| + |
| + **Failure:** A box retains an old executable B whose release demanded `hold`, potentially pinned as another instance’s good build. Bucket retention removes B; B is subsequently re-shipped with different bytes and an overlap-compatible manifest, then promoted. Root reuses the old staged executable. The scheduler reads the new canonical manifest and permits overlap, executing the old, hold-requiring binary beside the incumbent. Reading canonical metadata and preserving raw staged metadata do not establish that they describe the same executable. |
| + |
| + **Smallest fix:** Bind the attempt’s policy to the actual staged artifact: compare its digest with the canonical manifest’s artifact digest before permitting overlap, and refuse mismatches. Correct the claimed signed-manifest-hash contract; it does not exist today. |
| + |
| +2. **HIGH — §§6.1–6.3, 7.9, 10: Automatic worker recycling remains a route around paced adoption.** |
| + |
| + **Code:** Crash recovery calls `respawnWorkerLocked` directly at [`internal/activator/activator.go:1816`](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:1816). An always-on worker reaching `max_run` does the same at [`activator.go:2570`](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:2570). Both reach [`internal/activator/backend_exec.go:1379`](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/backend_exec.go:1379), which executes the live symlink. These branches precede and bypass the drift check that §7.9 explicitly moves into the scheduler. |
| + |
| + **Failure:** Root installs B across several workers while they still run A. One worker starts B’s canary. Another reaches `max_run` during that settle and follows ordinary recycling, stopping A and executing installed B before the canary permits it. Several workers can therefore run B concurrently despite `swap_concurrency=1`. The scheduler inputs and attempt causes do not assign automatic recycling a target or define its relationship to the queued release. |
| + |
| + **Smallest fix:** Explicitly cover crash and `max_run` respawns. Recover the captured serving build through an immutable path, respecting failure holds and the operation barrier; leave adoption of a different installed build to the paced release attempt. Define how recovery coalesces with that queued attempt. |
| \ No newline at end of file |