amadan
spec: zero-downtime deploys rev 3 — answers Astra round 2 (12 new, 2 partial)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+364
−100
docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md+281 −100docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R2.md+83 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md| index 519a9fc67..a5410e08e 100644 |
| --- a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| +++ b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| @@ -1,9 +1,10 @@ |
| # Zero-downtime deploys: overlap swaps, update settings, paced rollout |
| -Status: draft rev 2, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| -(per-app restart pace, first-route health step) feed into this. Rev 1 was |
| -rejected by Astra with 17 findings |
| -(`CODEX-FINDINGS-ZERO-DOWNTIME.md`). §15 maps each finding to its answer. |
| +Status: draft rev 3, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| +(per-app restart pace, first-route health step) feed into this. Astra |
| +rejected rev 1 with 17 findings (`CODEX-FINDINGS-ZERO-DOWNTIME.md`) and |
| +rev 2 with 12 new ones (`CODEX-FINDINGS-ZERO-DOWNTIME-R2.md`), leaving two |
| +round-1 answers partial. §15 maps every finding to its answer. |
| ## 1. Why |
| @@ -141,32 +142,45 @@ runs in `overlap`. |
| ## 5. Target resolution |
| -There is **one resolver**, `release.Desired(record, policy, heads, failures)`. |
| -Every path that installs a build calls it: the agent's pass, the root stage |
| -pass (`carlos adopt --stage-only`), the hourly adopt loop, and first-time |
| -materialization. The only exception is an explicit per-host request (§7.7). |
| -No path reads a channel pointer to make its own choice. |
| - |
| -For each instance, take the first rule that applies: |
| - |
| -1. **`pin` set:** the desired build is the pinned build. Setting a pin is |
| - itself an install request: the instance moves to the pinned build |
| - whether `updates` is `auto` or `manual`, and no bake hold applies, |
| - because a person chose it. |
| -2. **`updates = manual`:** the desired build is the installed build. An |
| - instance with no installed build yet resolves as if `auto`. |
| -3. **A failure record holds the channel head for this instance's scope** |
| - (§8.4): the desired build is the installed build. |
| -4. **`major` is set, and the head's tag is absent or has another major:** |
| - the desired build is the installed build, and the instance is *held at |
| - major N*. |
| -5. **Otherwise:** the head of the instance's `channel`, after bake holds |
| - (`internal/release/bake.go`) and `FastTrack`, exactly as today. |
| +There is **one resolver**, `release.Desired(record, policy, heads, |
| +failures, good)`. Only the agent runs it. Root no longer decides anything. |
| +It only carries out the agent's exact per-host requests, and checks each |
| +one against an authorization it can verify itself (§7.7). No other path |
| +reads a channel pointer to make its own choice: not the root stage pass, |
| +not the hourly adopt loop, and not first-time materialization. |
| + |
| +The resolver works in two steps. |
| + |
| +**Candidate.** |
| +- If `pin` is set, the candidate is the pinned build, at the pin's |
| + revision (each pin write bumps it). |
| +- Otherwise the candidate is the head of the instance's `channel`, after |
| + bake holds (`internal/release/bake.go`) and `FastTrack`, exactly as |
| + today. |
| + |
| +**Holds.** The first hold that applies makes the desired build the |
| +installed build: |
| +1. `updates = manual`, unless the candidate came from a pin. Setting a pin |
| + is itself a request to install it, so it moves the instance even in |
| + `manual`. Bake holds don't apply to a pin. |
| +2. A failure record matches the candidate for this instance (§8.4). That |
| + includes a pinned candidate at the same pin revision. A failed pin stays |
| + held until the pin is set again or the build is retried. |
| +3. `major` is set and the candidate is not pinned, and the head's tag is |
| + absent or has another major. The instance is *held at major N*. |
| +4. **The instance has no good build yet but does have an installed build** |
| + (§8.5). It stays where it is until a generation has served there. |
| + Instances that predate this feature therefore establish a baseline |
| + before any cold move. |
| + |
| +An instance with neither an installed nor a good build is new. It resolves |
| +to the candidate, ignoring `manual`. |
| + |
| +If no hold applies, the desired build is the candidate. |
| **Installing by hand** is refused on a pinned instance, naming the pin. |
| Otherwise it moves the instance to its channel head, ignoring `manual`, |
| -`major` and bake holds. It also clears a failure record for that instance |
| -only (§8.4). |
| +`major` and bake holds, and clears the instance's failure hold (§8.4). |
| ## 6. Rollout and pacing |
| @@ -222,12 +236,31 @@ restart request id, scheduler revision). The causes are `release`, |
| ### 6.4 First-wake canary |
| A wake onto a build the instance has never served is a canary for that |
| -build. It gets its own budget (§8.2). A settle doesn't apply to the |
| -request that is waiting. Instead, if the new generation exits within 10 s |
| -of becoming healthy, that counts as a failure of the build: the |
| -failure record is written (§8.4), and the next wake uses the good build. |
| - |
| -### 6.5 Across boxes |
| +build, with its own budget (§8.2). The waiting request doesn't sit through |
| +a settle. Instead, an **unexpected** exit within 10 s of becoming healthy |
| +counts as a failure of the build: it goes through the cohort-failure |
| +transition (§8.1), and the next wake uses the good build. An intentional |
| +stop during those 10 s cancels the observation and records nothing. That |
| +covers a deferred restart or config attempt, a hibernate, a move or park, |
| +lease loss and `Shutdown`. The same rule applies to the canary settle in |
| +§6.2. |
| + |
| +### 6.5 Processes without a socket |
| + |
| +Workers and native sidecar members have no HTTP health check. For them: |
| + |
| +- **Ready** means the process is still running 10 s after it started, for |
| + every attempt, not only the canary. |
| +- **A clean exit** within those 10 s counts as success for a one-shot |
| + process. Any other exit counts as failure. |
| +- **The good build** for such a host is set once it is ready. |
| +- **No incumbent survives a stop-first replacement.** So a failed |
| + `release`, `install` or `retry` attempt is treated as a failed first |
| + wake: the host's good build is spawned in its place, and the cohort |
| + fails (§8.1). A failed `restart` or `config` attempt respawns the build |
| + it replaced. |
| + |
| +### 6.6 Across boxes |
| Each box runs its own schedulers and canaries. A halt on one box does not |
| stop the others. The deploy watch reports it (§9.1). A fleet-wide canary is |
| @@ -247,20 +280,22 @@ generation `n`: |
| | `<socket>.control` | `<generation socket>.control` | |
| The registry keeps the canonical path as the route's identity. The |
| -activator owns the mapping from host to **current generation socket**, and |
| -every component that dials an app socket asks it, through one function, |
| -`activator.AppSocket(host)`: |
| - |
| -- the edge proxy (§7.2); |
| -- scheduled delivery (`internal/host/schedule_wiring.go`), which today |
| - builds its client from `rt.Socket`; |
| -- the next-due probe at stop (`backend_exec.go`'s `NextDue`), which must |
| - ask the **retiring** generation while that generation is draining and the |
| - current one otherwise; |
| -- version probes and health checks; |
| -- anything else that reads `rt.Socket` to dial the app. The plan lists |
| - every reader of `rt.Socket` and assigns each one to the canonical path or |
| - to `AppSocket`. |
| +activator owns the generations, and there are two ways to address one: |
| + |
| +- **`activator.AppSocket(host)`** names the **current** generation's |
| + socket. It is for traffic: the edge proxy (§7.2), scheduled delivery |
| + (`internal/host/schedule_wiring.go`, which today builds its client from |
| + `rt.Socket`), and the next-due probe at a hibernating stop |
| + (`backend_exec.go`'s `NextDue`). Hibernation joins any successor first |
| + (§7.4), so at that point the current generation is the only one. |
| +- **A generation handle** (socket, control socket, build, cgroup child) is |
| + returned by the spawn and held by whoever started it. Health checks, |
| + settle observation and retirement use the handle, never `AppSocket`. |
| + The successor isn't current until the switch, so checking its health |
| + through `AppSocket` would probe the old process. |
| + |
| +The plan lists every reader of `rt.Socket` and assigns each one to the |
| +canonical path, to `AppSocket`, or to a handle. |
| `prepare()` deletes only its own generation's socket path, never a |
| sibling's. The app still just receives `--socket <path>` and |
| @@ -276,11 +311,20 @@ generation stops being current, its idle connections are closed. |
| `dialUnixWithRetry` keeps its ENOENT/ECONNREFUSED retry, and each retry |
| asks `AppSocket` again. |
| -**The edge cache gets a per-host generation.** A request records the |
| -host's cache generation when it starts. When it finishes, its response is |
| -stored only if the generation hasn't changed. The switch purges the cache |
| -and bumps the generation, so a response finishing late on a draining |
| -process never fills the cache after the purge. |
| +**The edge cache is tagged with the serving route and its generation.** |
| +The cache key stays the requested hostname. Custom domains, wildcards and |
| +path-routed front doors all differ from the instance host |
| +(`internal/edge/edge.go`, the routing-attribution comment). Each entry |
| +records the **resolved serving route** (the instance host) and that |
| +route's generation when the request started. |
| +- **Insertion** is refused if the serving route's generation has changed |
| + since the request started. A response that finishes late on a draining |
| + process never fills the cache after the switch. |
| +- **Lookup** treats an entry as a miss if its serving route's generation is |
| + no longer current. Every alias of the route is covered without listing |
| + the aliases. |
| +- The switch bumps the route's generation, and the existing purge then |
| + reclaims the space. |
| **`X-Carlos-Version` reports the build of the generation that served the |
| request**, not the registry's `Version`. Status and the deploy watch show |
| @@ -292,14 +336,22 @@ The host's cgroup becomes a **parent that holds no processes**, with one |
| child cgroup per generation: `carlos-exec@<host>/g<n>`, or the nested |
| instance slice's equivalent. |
| -- The tier's memory limit is set on the parent. During an overlap it is |
| - raised to twice `memory.max`. It goes back once the retiring generation's |
| - child is empty. |
| +- **Each generation child gets the tier's full limits**: `memory.max`, |
| + `memory.high`, `memory.swap.max`, and `memory.oom.group=1`. Those are |
| + today's per-host values, moved down a level. A successor that overruns |
| + is OOM-killed alone, and the incumbent is untouched. A successor that |
| + passes its health check already fits the tier, so it still fits when the |
| + overlap ends. |
| +- **The parent** gets `memory.max`, `memory.high` and `memory.swap.max` at |
| + twice the tier while two children exist, and at the tier otherwise, with |
| + `memory.oom.group=0`. It is a backstop for the instance as a whole, not |
| + the boundary a single generation dies at. The raise happens before the |
| + successor spawns. The restore happens after the retiring child is empty. |
| - The claim-mode reservation (`PrepareStrict`, |
| `internal/cgroup/strict_linux.go`) moves from one per host to one per |
| generation child. The parent's identity and limits are prepared once per |
| - entry, and its expected-limit record is updated through the same API that |
| - raises and restores the limit. That way `limitsOK` never sees a limit it |
| + entry. Its expected-limit record is updated through the same API that |
| + raises and restores the limit, so `limitsOK` never sees a limit it |
| didn't set. |
| - `KillAndEmpty` on a generation acts only on that generation's child. |
| Retiring the old generation never touches the successor. |
| @@ -378,23 +430,54 @@ itself. So a repoint to any other build is impossible, and root can |
| override `manual`, a pin or a failure hold. |
| The agent now writes **one request file per host** under its data dir: |
| -`stage-requests/<host>.json`. Each holds the destination build, the |
| -installed build it expects to replace, the attempt identity, and a |
| -sequence number. The existing marker still triggers the root pass. The root |
| -pass then: |
| - |
| -- for each request file: stages the destination if needed, and repoints |
| - only if the live symlink still names the expected build. It writes the |
| - outcome to `stage-results/<host>.json`, which the agent reads; |
| -- for every other host: runs the shared resolver (§5), so its own choices |
| - can't disagree with the agent's; |
| -- writes `SetVersions` with an expected-current condition, so a stale |
| - outcome can't overwrite a newer install. |
| - |
| -Request files are validated before root acts on them: the host must exist |
| -in the registry, the build must be a staged or stageable build of that |
| -host's app, and the path components are checked. A box running an older |
| -root binary ignores the directory and behaves as today. |
| +`stage-requests/<host>.json`. The file holds the destination build, the |
| +installed build it expects to replace, the host's incarnation, the attempt |
| +identity and a sequence number. It is rewritten in place, so a newer |
| +request supersedes an older one. Root becomes an executor and decides |
| +nothing (§5). |
| + |
| +**Authorization root checks for itself.** A request is only as trustworthy |
| +as the sandboxed agent that wrote it, so root accepts destination B for |
| +host H only when it can verify one of these: |
| +- B is the head of H's channel, or in that channel's history, as read |
| + through the signed pointer (`release.Channel`, which verifies the |
| + pointer's signature). That covers channel moves, installs by hand, |
| + retries and fallback, because a good build was installed from that |
| + pointer. |
| +- A **console-signed pin authorization** binds account, app, H, B and the |
| + pin revision. The console signs it with the key that signs channel |
| + pointers when a pin is set. It travels on the provision record. |
| + |
| +A request that fails this check is refused, and the refusal is written as |
| +its outcome. Being a shipped build of the right app is not enough. |
| + |
| +**Execution under the host's guard.** For each request, root: |
| +1. stages B if needed (download and sha256 check, as `ensureStaged` does); |
| +2. takes the host's guard (`hostguard`, shared with the agent); |
| +3. re-reads the request file and checks it still carries the sequence it |
| + started with. A superseded request is dropped, and the newer one is |
| + handled in its turn; |
| +4. checks that the live symlink still names the expected build and that |
| + the incarnation matches; |
| +5. repoints; |
| +6. writes `SetVersions` with an expected-current condition; |
| +7. writes `stage-results/<host>.json` with that sequence and the outcome; |
| +8. releases the guard. |
| + |
| +The agent consumes only an outcome whose sequence matches its current |
| +request. If root dies between the repoint and the registry write, the next |
| +pass sees the symlink already at B with the request still pending, finishes |
| +the registry write, and reports the outcome. |
| + |
| +**The trigger is retransmitted.** `carlos-stage.path` uses `PathChanged` |
| +(`internal/host/units/carlos-stage.path`), so a marker touched while a |
| +pass is running is missed. While any request lacks an outcome with its |
| +sequence, the agent touches the marker again every `stageRequestEvery` |
| +(15 s), through the existing throttle. It also rescans the directory when |
| +it starts, so a restart doesn't lose pending requests. |
| + |
| +A box whose root binary predates this feature ignores the directory and |
| +behaves as it does today. |
| ### 7.8 A `hold` swap |
| @@ -407,8 +490,31 @@ held requests are served once it is healthy. |
| Worker processes and native sidecar members (`Kind=sidecar`) serve no |
| requests, and two copies could do the same work twice. They keep |
| -stop-then-start, on the respawn-on-same-entry path. They exec the attempt's |
| -staged build, and the scheduler paces them like any other attempt. |
| +stop-then-start, on the respawn-on-same-entry path, and exec the attempt's |
| +staged build. The scheduler paces them with the readiness and failure rules |
| +in §6.5. |
| + |
| +### 7.10 Scheduled work during an overlap |
| + |
| +Timers are written by host and name (`internal/schedule/control.go`), so |
| +two generations could overwrite each other. The rule is that only the |
| +current generation's schedule writes take effect: |
| + |
| +- **A successor's** `PUT` and `DELETE` on its control socket are recorded |
| + against its generation, and nothing is applied before the switch. At the |
| + switch they are applied in order, under the entry lock. A successor that |
| + fails has its record discarded, so a failed candidate never changes the |
| + live schedule. |
| +- **A successor's reads** (`GET /schedule`) see the live schedule with its |
| + own pending writes on top, so a boot-time "assert my timers" call can |
| + work out what it needs to change. |
| +- **A retiring generation's writes** are refused with 409 once the switch |
| + has happened. Its reassert-at-shutdown behaviour |
| + (`backend_exec.go`'s stop path) exists for hibernation, where the stopping |
| + process is the last one. In a swap the successor owns the schedule. |
| +- A **hibernating stop** is unchanged. It joins any successor first (§7.4), |
| + so the stopping generation is the current one, and its writes apply as |
| + today. |
| ## 8. Failure and rollback |
| @@ -420,11 +526,19 @@ switch, or, as a canary, dies within the settle. The successor is stopped |
| generation keeps serving. No request notices. What happens next depends on |
| the cause: |
| -- **`release`, `install`, `retry`.** A failure record is written (§8.4). |
| - The cohort halts on that box at the first failure, whether that's the |
| - canary or a later swap. Swapped instances stay on the new build. Every |
| - instance in the cohort that the rollout moved but that has never served |
| - the build gets an exact repoint (§7.7) back to its **good build**. |
| +- **`release`, `install`, `retry`.** These go through the **cohort-failure |
| + transition**, which is the same for a failed swap, a failed first wake |
| + (§8.2) and a failed socketless replacement (§6.5): |
| + 1. A failure record for the build is written (§8.4). |
| + 2. The cohort's queued attempts onto that build are cancelled. Its |
| + attempts already past spawn are abandoned at their next check (§7.4). |
| + 3. Every instance in the cohort with that build installed that has never |
| + served it, sleeping instances included, gets an exact repoint (§7.7) |
| + back to its **good build**. |
| + 4. Instances that have already served the build stay on it. |
| + |
| + The halt is at the first failure, whether that's the canary or a later |
| + swap. |
| - **`restart`, `config`.** No build changes, and no failure record is |
| written. The current generation keeps serving the build it was serving. |
| The attempt is reported failed. A config revision is acknowledged |
| @@ -434,20 +548,33 @@ the cause: |
| ### 8.2 A failed first wake (`overlap` mode) |
| -A first wake onto build B runs inside a **first-wake budget**: 30 s for B to |
| -become healthy, then up to 25 s to boot the good build. Requests waiting on |
| -that wake are held for up to 55 s, in place of `wakeWaitBound`'s 30 s, and |
| -only for first wakes onto a new build. If B fails: |
| +A first wake onto build B runs inside a **first-wake budget** of 55 s: |
| +- up to 30 s for B to become healthy; |
| +- up to 2 s to kill it; |
| +- up to 23 s to boot the good build. |
| + |
| +Requests waiting on that wake are held for up to 55 s instead of |
| +`wakeWaitBound`'s 30 s, but only for first wakes onto a new build. If B |
| +fails: |
| -1. B's generation is stopped and its child cgroup emptied. |
| +1. B's generation is killed outright with `cgroup.kill` on its child and |
| + then waited on until empty. There's no SIGTERM drain, because it never |
| + became healthy and has served nothing. |
| 2. The good build is spawned from its staged path. Exec needs no write to |
| `/opt`. |
| 3. The waiting requests are served by it. |
| -4. A failure record for B is written (§8.4), and an exact repoint to the |
| - good build is requested (§7.7). |
| +4. The cohort-failure transition runs (§8.1). That writes the record, |
| + cancels the cohort's queued attempts, and repoints every never-served |
| + install of B in the cohort, this instance included, to its good build. |
| -An instance with no good build (it has never served anything) fails as a |
| -wake fails today. The settle rule for first wakes is §6.4. |
| +**When the wake picks a build,** if the installed build is held by a |
| +failure record, the wake boots the good build directly and asks for the |
| +repoint, instead of booting the failed build again. That is a second line |
| +of defence for a sleeper whose repoint hasn't landed yet. |
| + |
| +An instance with no good build has nothing to fall back to and fails as a |
| +wake fails today. Rule 4 of §5 means that only happens on an instance's |
| +very first build. The settle rule for first wakes is in §6.4. |
| ### 8.3 A failed `hold` swap |
| @@ -467,6 +594,9 @@ that box. Clearing: |
| any instance still pointed at the failed build. |
| - **`carlos instances update <host>`** clears it for that host only, and |
| the next attempt runs as a canary. |
| +- **A pinned candidate** is matched by build and pin revision. Setting the |
| + pin again bumps the revision and authorizes one more attempt, for that |
| + instance only. |
| - **`carlos deploy --retry`** writes an app-wide retry request with its own |
| id into the release store. Each box clears its record for that build when |
| it sees a retry id newer than the record. This is how "deploy the same |
| @@ -479,8 +609,17 @@ that box. Clearing: |
| `GoodVersion` is a new registry column. It is set when a generation on |
| build B becomes current after passing its health check. For a canary, that |
| -means after the settle. It is written with an expected-current condition. |
| -Fallback uses it, and `PrevVersion` keeps its old meaning. The version |
| +means after the settle; for a host with no socket, it means when it is |
| +ready (§6.5). It is written with an expected-current condition. Fallback |
| +uses it, and `PrevVersion` keeps its old meaning. |
| + |
| +**Bootstrap.** The column starts empty on every existing route, and nothing |
| +is inferred from `Version` or `PrevVersion`. On the first start of an agent |
| +that has this feature, every instance that is awake and healthy records |
| +its serving build as good. A sleeping instance stays on its installed |
| +build (§5, hold 4) until its next wake serves that build and records it. |
| +Only after that can it move cold. Any instance that predates the feature |
| +therefore has a good build before it can need one. The version |
| sweep (`internal/host/versionsweep.go`) adds to its pins `GoodVersion` and |
| every build named by a live generation or a pending attempt (current, |
| successor, retiring, spawning), which the activator reports. |
| @@ -571,10 +710,14 @@ and remove in a later one, or deploy with `--swap hold`, or set the app to |
| - The console checks only credentials it minted (§4.4). Bucket objects |
| under box prefixes and box-minted admin tokens are never authority for a |
| settings write. |
| -- Per-host request files are agent-written and root-read, so root |
| - validates them (§7.7). A compromised agent can only ask root to install a |
| - build of that host's own app that has already been shipped. That is no |
| - more reach than the agent's own adoption has today. |
| +- Per-host request files are written by the agent and read by root, so root |
| + authorizes each one itself (§7.7). Root accepts only a build that a |
| + signed channel pointer names, as its head or in its history, for the |
| + host's own channel, or one that a console-signed pin authorization names |
| + for that host. A compromised agent can therefore choose among builds the |
| + operator has authorized for that host. It can't pick an unauthorized |
| + build of the same app. Ordinary adoption can already move between those |
| + builds, through rollback, so this adds no reach. |
| - `pin` must name a shipped build of the same app. The console resolves it |
| against the app's manifests, never as a path. |
| @@ -604,7 +747,23 @@ and remove in a later one, or deploy with `--swap hold`, or set the app to |
| the directory, and `SetVersions` conditions. |
| - **Containment:** per-generation children, the parent-limit raise and |
| restore with `limitsOK`, retiring one generation without touching the |
| - other, and parent stop evidence. |
| + other, and parent stop evidence. Include a memory-hog successor that |
| + overruns its child limit and is OOM-killed while the incumbent keeps |
| + serving. |
| +- **Root authorization and execution:** refusing an unauthorized shipped |
| + build, accepting a pin authorization and a history entry, superseding a |
| + request mid-download, recovering when root dies between the repoint and |
| + the registry write, and resending the marker while a request has no |
| + outcome, including across an agent restart. |
| +- **Scheduled work:** successor writes are deferred and then applied at the |
| + switch, discarded on failure, and visible to the successor's own reads. |
| + Retiring writes are refused after the switch. |
| +- **Lifecycle versus settle:** an intentional stop during the settle records |
| + no failure. |
| +- **Cache aliases:** a custom domain, a wildcard and a path-routed request |
| + each miss after the switch. |
| +- **Bootstrap:** an upgraded box with sleeping instances and an empty |
| + `GoodVersion` doesn't move them cold until a wake has served. |
| - **Nested:** the generation template's ID split, the shared slice and data |
| mounts, and per-generation executable mounts. |
| - **Fallback:** the first-wake budget, waiting requests served by the good |
| @@ -637,10 +796,13 @@ take a third kind of attempt for them, and #230 stays open for that spec. |
| - Keeping instances up across an agent restart or box roll. |
| - A configurable health path, or a settle on swaps after the canary. |
| - Instance-only console principals (§4.4). |
| -- The level-triggered stage marker noted on #231. §7.7's request files |
| - don't depend on it. |
| +- The level-triggered stage marker noted on #231. §7.7 resends the |
| + marker while any request has no outcome, which closes the gap without |
| + changing the unit. |
| + |
| +## 15. Findings and their answers |
| -## 15. Rev 1 findings and their answers |
| +### Round 1 |
| | # | Finding | Answer | |
| |---|---------|--------| |
| @@ -661,3 +823,22 @@ take a third kind of attempt for them, and #230 stays open for that spec. |
| | 15 | The version header reports the install | §7.2: the serving generation's build; §9.1 shows three builds | |
| | 16 | Pin semantics contradict | §5 rule 1: a pin is an install request; `manual` with no install resolves as `auto` | |
| | 17 | No instance-only principal | §4.4: none is invented; tenants act through their app | |
| + |
| +### Round 2 |
| + |
| +| # | Finding | Answer | |
| +|---|---------|--------| |
| +| R1-4 (partial) | Root doesn't check that a request has been superseded | §7.7: sequence re-read under the host guard, then incarnation and expected-build checks | |
| +| R1-11 (partial) | No time to kill the candidate | §8.2: 30 + 2 + 23 s, with `cgroup.kill` and no drain | |
| +| 1 | Exact requests bypass release authorization | §5: root decides nothing; §7.7: a signed pointer's head or history, or a console-signed pin authorization; §11 | |
| +| 2 | A candidate's OOM kills the incumbent | §7.3: full tier limits and `oom.group` per generation child; the parent is a backstop with `oom.group=0` | |
| +| 3 | `AppSocket` can't address the successor | §7.1: generation handles for health, settle and retirement | |
| +| 4 | Root commits a superseded request | §7.7: steps 2–8 under the guard, sequence-matched outcomes, crash recovery | |
| +| 5 | A failed pin bypasses the failure hold | §5 hold 2 at the pin revision; §8.4 | |
| +| 6 | A first-wake failure leaves sibling sleepers on the failed build | §8.1 cohort-failure transition for every failure kind; §8.2 wake-time selection of the good build | |
| +| 7 | `GoodVersion` has no bootstrap | §5 hold 4; §8.5 bootstrap | |
| +| 8 | Both generations write the same schedule | §7.10: successor writes deferred until the switch, retiring writes refused | |
| +| 9 | Worker readiness is undefined | §6.5 | |
| +| 10 | Intentional stops during the settle count as failures | §6.4: lifecycle stops cancel the observation | |
| +| 11 | The cache needs the serving-route identity | §7.2: entries tagged with the serving route and its generation, checked on insert and lookup | |
| +| 12 | Request files still depend on a lost trigger | §7.7: the marker is resent while a request has no outcome; §14 | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R2.md b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R2.md| new file mode 100644 |
| index 000000000..4d5b493ab |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME-R2.md |
| @@ -0,0 +1,83 @@ |
| +REJECT |
| + |
| +Round-1 answers: |
| + |
| +1. CLOSED — Console-minted instance credentials replace box-prefix authority; writes wait for that dependency. |
| +2. CLOSED — Generation children address reservation and retirement isolation. |
| +3. CLOSED — The operation set explicitly covers outstanding spawns, retirees and terminal stops. |
| +4. PARTIAL — Exact destinations are specified, but root does not validate request supersession before committing. |
| +5. CLOSED — Both layouts execute immutable staged builds. |
| +6. CLOSED — `GoodVersion`, generation pins and conditional writes address the original install-history problem. |
| +7. CLOSED — Host identity and generation identity are separated throughout the nested layout. |
| +8. CLOSED — Scheduled delivery and next-due probes are explicitly included in the socket-consumer migration. |
| +9. CLOSED — Attempt causes, deduplication and config acknowledgement are distinguished. |
| +10. CLOSED — Targets are per instance, with separate cohorts. |
| +11. PARTIAL — The 55-second budget allocates 30 seconds plus 25 seconds, leaving no time for candidate termination. |
| +12. CLOSED — Explicit retry identifiers replace reliance on an unchanged promotion. |
| +13. CLOSED — Named-unit rollout is explicitly deferred. |
| +14. CLOSED — Generation-checked insertion addresses responses completing after a purge. |
| +15. CLOSED — The header must identify the generation that served the request. |
| +16. CLOSED — Setting a pin explicitly requests installation, including in manual mode. |
| +17. CLOSED — The unsupported instance-only human principal is removed. |
| + |
| +New findings: |
| + |
| +1. **HIGH — §§5, 7.7, 11: Exact installation requests bypass the existing release-authorization chain.** |
| + **Code:** [release.go:43](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/release.go:43) makes manifests unsigned; [release.go:1047](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/release.go:1047) authenticates the channel pointer; [adopt.go:1336](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:1336) explicitly identifies that signature as part of the staging trust chain. |
| + **Failure:** A compromised agent requests a shipped but never-authorized build. Root’s specified checks—existing host, matching app, stageable build—accept it without the signed authorization required by ordinary adoption. §11’s “no more reach” claim is false. |
| + **Smallest fix:** Require root-verifiable installation authorization binding account, app, host, build and request revision. Explicit pins may authorize any shipped build, but the console must sign that authorization; fallback needs retained authenticated authorization. |
| + |
| +2. **HIGH — §§7.3, 8.1: A candidate’s memory failure can kill the incumbent.** |
| + **Code:** [strict_linux.go:219](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/cgroup/strict_linux.go:219) sets `memory.oom.group=1`; [cgroup.go:329](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/cgroup/cgroup.go:329) explains that this kills the whole family. |
| + **Failure:** Separate generation children still share the parent’s OOM boundary. B exceeds the doubled parent limit and kills healthy A too. Alternatively, B passes health while exceeding its normal tier, then dies when the parent limit is restored. Raising only `memory.max` also leaves the existing `memory.high` throttling threshold unchanged. |
| + **Smallest fix:** Specify per-generation memory limits and OOM isolation, plus coordinated parent `max`, `high` and swap limits. Require a memory-hog candidate test that preserves the incumbent. |
| + |
| +3. **HIGH — §§7.1, 7.5: `AppSocket(host)` cannot identify the successor being health-checked.** |
| + **Code:** [backend_exec.go:989](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/backend_exec.go:989) probes the spawned process’s socket; [backend_exec.go:1455](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/backend_exec.go:1455) considers an HTTP response below 500 healthy. |
| + **Failure:** §7.1 requires health checks to use the current-socket resolver, but §7.5 keeps A current until B passes health. Following both instructions probes healthy A and can approve a broken B. The same host-only interface cannot express the explicitly required retiring-generation next-due query. |
| + **Smallest fix:** Keep `AppSocket(host)` for current-generation traffic. Give health checks and retirement probes an explicit generation handle containing that generation’s socket and build. |
| + |
| +4. **HIGH — §§6.3, 7.7: Root can commit a superseded request even when its expected-build check passes.** |
| + **Code:** [versions.go:267](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/versions.go:267) performs an unconditional symlink replacement; installation and registry stamping occur separately at [adopt.go:1072](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:1072) and [adopt.go:1258](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:1258). |
| + **Failure:** Root reads A→B, sequence 1, and starts downloading. The agent replaces it with A→C, sequence 2, while A remains installed. Root’s specified expected-A check passes and it installs obsolete B. A sleeping instance can boot B; the swap’s newest-attempt check does not protect installation. Merely recording sequence numbers does not prevent this. |
| + **Smallest fix:** Revalidate the current request sequence, policy/attempt revision and host incarnation under the shared host lock immediately before repointing. Define sequence-specific acknowledgement and consumption, including recovery between filesystem and registry writes. |
| + |
| +5. **HIGH — §§5, 8.4: A failed pin bypasses the failure hold indefinitely.** |
| + **Code:** [adopt.go:943](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:943) reads the installed version and skips adoption only when it equals the destination. |
| + **Failure:** An instance serving A sets `pin=B`. B fails, and rollback restores A. The next resolver call takes pin rule 1 before failure rule 3 and selects B again, without a new pin change or retry request. The supposedly halted instance repeatedly reinstalls the failed build. |
| + **Smallest fix:** Treat changing a pin as a one-time installation authorization. Subsequently suppress that failed pinned target until its pin revision changes or an explicit retry authorizes another attempt. |
| + |
| +6. **HIGH — §§5, 6.4, 8.2: A first-wake failure leaves sibling sleepers targeting the rejected build.** |
| + **Code:** [adopt.go:883](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:883) explicitly permits cold installation without health checking; [activator.go:1282](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:1282) subsequently starts that installed build. |
| + **Failure:** Sleeping instances H1 and H2 both install B, with good build A. H1’s first wake rejects B. §8.2 restores only H1 and writes the box-wide failure record. For H2, resolver rule 3 returns its installed build—still B—so H2 boots the build already rejected on that box. The cohort-wide restoration in §8.1 applies only to failed overlap swaps. |
| + **Smallest fix:** Route first-wake failures through the same cohort-failure transition: cancel queued B attempts and restore never-served B installations, or explicitly make wake selection choose the good build when the installed build is held. |
| + |
| +7. **MEDIUM — §8.5: Existing instances have no defined `GoodVersion` initialization.** |
| + **Code:** [registry.go:60](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/registry/registry.go:60) stores only installed and previous versions; [adopt.go:883](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:883) allows sleeping instances to advance without running. |
| + **Failure:** An existing sleeper previously served A. After the platform upgrade, its new `GoodVersion` column is empty. Root installs B before another wake; B fails, and the instance takes §8.2’s “never served anything” failure path despite having a working predecessor. |
| + **Smallest fix:** Define migration/bootstrap behavior before cold installations advance. Establish a confirmed baseline through serving evidence or a successful baseline wake; do not infer success from `Version` or `PrevVersion`. |
| + |
| +8. **MEDIUM — §§7.1, 7.5, 9.3: Both generations can mutate the same scheduled-work registrations.** |
| + **Code:** [control.go:126](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/schedule/control.go:126) writes timers by host/name; [control.go:153](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/schedule/control.go:153) deletes them without generation checks. [backend_exec.go:1532](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/backend_exec.go:1532) deliberately permits timer reassertion during shutdown. |
| + **Failure:** B registers its new timer and becomes current. Draining A then reasserts an obsolete registration or deletes that name, overwriting B’s schedule. A candidate that eventually fails can likewise alter A’s live schedule before switching. Separate control-socket paths do not separate this authority. |
| + **Smallest fix:** Define generation ownership for schedule mutations: stage candidate changes until commitment, discard failed-candidate changes, and fence or reconcile retiring-generation writes. Preserve terminal-stop registration explicitly. |
| + |
| +9. **MEDIUM — §§6.2, 7.9, 8.1–8.5: Worker canary, success and failure rules are undefined.** |
| + **Code:** [backend_exec.go:1344](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/backend_exec.go:1344) specifies no worker socket or health wait; [activator.go:1780](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:1780) distinguishes clean one-shot completion from crashes and always-on exits. |
| + **Failure:** Workers are paced “like any other attempt,” but cohort advancement and `GoodVersion` require HTTP health. Applying that requirement stalls worker cohorts; treating successful exec as health lets immediately crashing workers advance them. Their stop-first replacement also has no surviving incumbent to satisfy §8.1. |
| + **Smallest fix:** Specify worker readiness and canary criteria, clean one-shot completion semantics, `GoodVersion` establishment, and replacement-spawn failure behavior. |
| + |
| +10. **MEDIUM — §6.4: Intentional stops during first-wake settle are classified as build failures.** |
| + **Code:** [activator.go:998](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:998) services deferred restart/config requests immediately after wake; [activator.go:2924](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:2924) intentionally stops awake entries during shutdown. |
| + **Failure:** B becomes healthy, then a requested restart, move or agent shutdown terminates it within ten seconds. §6.4 says any such exit fails the build, creating a box-wide failure hold for a healthy release. |
| + **Smallest fix:** Distinguish unexpected exit from lifecycle cancellation. Intentional stops cancel the settle observation without recording a build failure. |
| + |
| +11. **MEDIUM — §7.2: Cache generations need the serving-route identity, not just the requested hostname.** |
| + **Code:** [edge.go:1187](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/edge/edge.go:1187) documents that routing attribution and cache hostname differ; [edge.go:1662](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/edge/edge.go:1662) keys responses by requested hostname. [cache.go:464](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/edge/cache.go:464) purges only that hostname’s entries. |
| + **Failure:** A route switches builds, but responses cached under its custom domain, wildcard hostname or path-routing front door survive a purge of the instance hostname. A generation tracked by requested hostname likewise misses the serving instance’s switch. |
| + **Smallest fix:** Associate cached responses with the resolved serving route and generation, and validate that association on both lookup and insertion. Explicitly cover aliases, wildcards and path-routed requests. |
| + |
| +12. **MEDIUM — §§7.7, 14: Durable request files still depend on a potentially lost stage trigger.** |
| + **Code:** [carlos-stage.path:22](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/units/carlos-stage.path:22) uses `PathChanged`; [agent.go:1322](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/agent.go:1322) documents that changes during an active stage service do not trigger another run. |
| + **Failure:** Root has already scanned requests when a new request arrives and the marker changes. The active service finishes without processing it. Persisting the request prevents its loss, but does not schedule another pass; installation can wait for the hourly timer, contrary to §14’s claimed independence from the marker issue. |
| + **Smallest fix:** Specify bounded retransmission of the marker while a request lacks a matching outcome, including after agent restart. This can preserve the decision to defer a level-triggered marker. |
| \ No newline at end of file |