| index 43581f887..d88f8fa81 100644 |
| --- a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| +++ b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| @@ -1,19 +1,18 @@ |
| -# Zero-downtime deploys: overlap swaps, update settings, paced rollout |
| +# Zero-downtime deploys: overlap swaps and paced rollout |
| |
| -Status: draft rev 6, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| +Status: draft rev 7, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| (per-app restart pace, first-route health step) feed into this. Astra |
| -rejected rev 1 with 17 findings (`CODEX-FINDINGS-ZERO-DOWNTIME.md`) and |
| -rev 2 with 12 new ones (`CODEX-FINDINGS-ZERO-DOWNTIME-R2.md`), and |
| -rev 3 with 7 new ones and 3 partials (`CODEX-FINDINGS-ZERO-DOWNTIME-R3.md`), |
| -rev 4 with 3 new ones and 2 partials (`CODEX-FINDINGS-ZERO-DOWNTIME-R4.md`), |
| -and rev 5 with 4 new ones and 1 partial (`CODEX-FINDINGS-ZERO-DOWNTIME-R5.md`). |
| - |
| -Rev 6 replaces the box-side machinery that rounds 2–5 kept breaking. That |
| -was the agent's exact repoint requests, root's install ledger, the |
| -agent–root handshake and carried build state. In their place the console |
| -computes **signed per-instance directives** (§5), and fallback never needs |
| -root (§8.2). Answers in earlier rounds that pointed at the removed |
| -machinery are superseded by §5 and §8. §16 maps every finding to its answer. |
| +rejected rev 1 with 17 findings, rev 2 with 12 new ones, rev 3 with 7, rev 4 |
| +with 3 and rev 5 with 4. The findings files are |
| +`CODEX-FINDINGS-ZERO-DOWNTIME*.md` beside this spec. |
| + |
| +Rev 6's 6 findings all concerned per-instance update settings: manual, |
| +pin, major and in-app control. So rev 7 **splits those out** into |
| +`2026-10-07-instance-update-settings-design.md`, which starts from them. |
| +This spec keeps the swap, pacing, canary, fallback and swap policy. Here |
| +every instance follows its channel exactly as today. §16 maps every |
| +finding to its answer. Answers about directives and settings now live in |
| +the settings spec. |
| |
| ## 1. Why |
| |
| @@ -45,12 +44,10 @@ These were settled with Paul on 2026-10-06/07 and are not reopened here. |
| it. It means stop-then-start with requests held. It exists for apps or |
| releases whose migrations can't share the database with the previous |
| build. |
| -3. **Instances get Eleven's update model:** auto-update on or off, a |
| - channel, a major-version hold and an exact pin. The app implementer sets |
| - the defaults and decides how much of that each instance may change. |
| -4. **Instance settings are changed from the console and CLI, and by the app |
| - itself**, within the app's policy. Only the implementer can change the |
| - policy. |
| +3. **Instances get Eleven's update model.** That is in the instance update |
| + settings spec. |
| +4. **Instance settings are changed from the console, the CLI and the app.** |
| + That is in the settings spec too. |
| 5. **The app policy lives in app settings.** A release can still demand |
| `hold` (`carlos deploy --swap hold`), and that demand is stored with the |
| release. |
| @@ -81,7 +78,7 @@ These were settled with Paul on 2026-10-06/07 and are not reopened here. |
| connections. It can differ from the installed build while a rollout |
| queues, checks health, or falls back. |
| - **Desired build**: the build the instance should serve now. That is its |
| - installed build, which root put there from its channel or its directive |
| + installed build, which root put there from its channel as it does today |
| (§5), unless a failure record holds that build, in which case it is the |
| good build (§8.2). |
| - **Generation**: one spawn of one instance, numbered upwards for the life |
| @@ -92,36 +89,20 @@ These were settled with Paul on 2026-10-06/07 and are not reopened here. |
| - **Cohort**: on one box, the instances of one app whose attempts share a |
| desired build. The canary rule (§6.2) applies per cohort. |
| |
| -## 4. Settings and policy |
| +## 4. Swap policy |
| |
| -### 4.1 Instance settings |
| - |
| -The settings live on the instance's provision record. The console owns the |
| -record, and boxes read it through reconcile as they do today. |
| - |
| -| Setting | Values | Meaning | |
| -|-----------|----------------------------|---------| |
| -| `updates` | `auto` / `manual` | With `manual`, nothing installs on its own. The installed build stays put, across sleep and wake, until someone installs or pins. | |
| -| `channel` | an existing channel | What the instance follows. It already exists per instance (`Record.Channel`). | |
| -| `major` | empty or an integer ≥ 0 | Follow the channel, but hold a release whose semver tag is absent or has another major. It is only accepted with `stable`, because only stable promotions cut tags (`stableTag`, `internal/release/release.go`). | |
| -| `pin` | empty or a build version | Be on exactly this build (§5). It must name a build the app has shipped. | |
| - |
| -There is no "download automatically". A box fetches a build once and every |
| -instance on it shares it, so an instance has nothing of its own to |
| -download. |
| - |
| -### 4.2 App policy |
| +### 4.1 App policy |
| |
| | Setting | Values | Default | Meaning | |
| |--------------------|--------|---------|---------| |
| | `swap` | `overlap` / `hold` | `overlap` | How an awake instance moves to a new generation. | |
| | `swap_concurrency` | integer ≥ 1 | 1 | How many swaps of this app may run at once on one box, after a cohort's canary. | |
| -| `instance_control` | a subset of {`updates`, `channel`, `major`, `pin`} | empty | The settings the instance itself may change. | |
| -| instance defaults | one value per §4.1 setting | `auto`, the app's channel, empty, empty | Used where an instance has never set the value explicitly. | |
| |
| -An empty `instance_control` keeps every existing app as it is today. |
| +These are app settings, changed by app members with write access in the |
| +console or CLI (today's `apiAccountApp` gate). They reach boxes with the |
| +app's config, as other app settings do. |
| |
| -### 4.3 The release's demand |
| +### 4.2 The release's demand |
| |
| `Manifest` gains `swap` (`"hold"` or empty). It is additive to schema 1, |
| like `Label`, and absent from older manifests. `carlos deploy --swap hold` |
| @@ -129,128 +110,16 @@ sets it when the release is shipped. An attempt onto build B runs in |
| `hold` if either the app policy or B's manifest says `hold`. Otherwise it |
| runs in `overlap`. |
| |
| -### 4.4 Who may change what |
| - |
| -- **App members** with write access, in the console or CLI, may set any |
| - instance setting and the app policy. This uses today's `apiAccountApp` |
| - gate. |
| -- **The instance itself** may change the settings named in |
| - `instance_control`, and install by hand if `updates` is named. It does |
| - this by calling the console API with an **instance-bound credential**, |
| - which is #155's step 6: a token that names one instance, minted by the |
| - console and checked by the console. The console can check a credential it |
| - minted. It cannot check `CARLOS_ADMIN_TOKEN`, which the box mints |
| - (`internal/activator/admintoken.go`), nor a bucket object written under a |
| - box prefix, which is only a pseudonym (`internal/edge/storedoor.go`). So |
| - neither of those is used for writes. |
| -- The control socket offers **read only**: `GET /updates` answers from the |
| - box's copy of the record (§9.3). In-app writes ship once #155 step 6 has |
| - landed. Until then an app can show its update state but not change it. |
| -- No console principal has "access to this instance but not its app", so |
| - this spec doesn't invent one. Tenants without Carloku accounts act through |
| - their app. |
| -- A refused change names the setting, and says the app keeps it for itself. |
| - |
| -## 5. Who decides which build an instance installs |
| - |
| -**An instance's build is decided by the console, which owns its settings |
| -and policy, and is verified by root**, by the same signature scheme that |
| -already protects channel pointers. The box never makes this decision, so a |
| -sandboxed agent can't steer root, and nothing depends on the agent and |
| -root agreeing. |
| - |
| -### 5.1 Two ways a build is chosen |
| - |
| -- **Channel-following instances** have every setting at its default: |
| - `auto`, the app's channel, no `major`, no `pin`. They are adopted |
| - exactly as today. Root reads the signed channel pointer, applies bake |
| - holds (`internal/release/bake.go`) and `FastTrack`, stages the build and |
| - repoints. Most instances are like this, and their install path doesn't |
| - change. |
| -- **Directed instances** have any non-default setting, or a pending |
| - install by hand. For each of these the console keeps a **signed install |
| - directive** on the instance's provision record. Root installs exactly |
| - the directive's build and ignores the channel for that host. |
| - |
| -### 5.2 The directive |
| - |
| -A directive is signed with the key that signs channel pointers. It binds: |
| -- account, app and host; |
| -- the host's **incarnation** (`hostclaim`), so a directive for a deleted |
| - and recreated host is void; |
| -- the build; |
| -- a **revision**, which only ever increases for each host. |
| - |
| -When a pin is set, the directive's revision is the **pin revision**. |
| - |
| -**What root checks.** Before acting on a directive, root: |
| -1. verifies the signature, the account, app and host, and the |
| - incarnation it sees for the host; |
| -2. checks that the revision is not lower than its **high-water mark** for |
| - that host. The mark is kept root-owned at |
| - `/opt/carlos/directives/<host>`, which the agent can't write. |
| - |
| -Then it stages the build (download and sha256 check, as `ensureStaged` |
| -does), repoints, stamps, and raises the mark. Unsigned, mismatched or |
| -replayed directives are refused and reported in box status. The agent |
| -copies the record's directive into the registry, as reconcile copies other |
| -record fields. Because of the signature, the agent can carry the |
| -directive but can't forge one. |
| - |
| -### 5.3 How the console computes a directive |
| - |
| -The console recomputes an instance's directive whenever its settings, its |
| -app's policy, or its channel head change. Promotions already go through |
| -the console. Take the first rule that applies: |
| - |
| -1. **`pin` set:** the pinned build. Setting a pin is itself an install |
| - request, so it applies even in `manual`. No bake hold applies. |
| -2. **`updates = manual`:** the build the instance has installed now, as |
| - last reported in box status. This never moves the instance on its own. |
| - A new instance with nothing installed gets the channel head. |
| -3. **`major` set, and the channel head's tag is absent or has another |
| - major:** the installed build. The instance is *held at major N*. |
| -4. **Otherwise:** the channel head, after bake holds computed from the |
| - promotion time with `DefaultHolds`, and `FastTrack`. |
| - |
| -Box status is only a report. **For rules 2 and 3, the console accepts a |
| -reported build only if the app's signed history shows it was shipped to |
| -the instance's channel, or it was pinned for that host.** So a lying box |
| -can at most choose among builds already authorized for that instance. |
| - |
| -**Installing by hand** on a pinned instance is refused, naming the pin. |
| -Otherwise it issues a directive to the channel head, ignoring `manual`, |
| -`major` and bake holds. When an instance's settings return to default and |
| -its directive names the channel head, the console drops the directive, and |
| -the instance goes back to following the channel. |
| - |
| -### 5.4 Which routes this covers |
| - |
| -Settings, directives and the scheduler apply to exec routes only. |
| +## 5. Which build an instance installs |
| + |
| +**Unchanged.** Root adopts every exec route from its channel exactly as it |
| +does today, with the signed pointer, bake holds and `FastTrack`. It stages, |
| +repoints and stamps. This spec changes how the installed build is |
| +**swapped in**, not which build is installed. Per-instance settings that |
| +change *which* build (manual, pin, major) are in the settings spec. |
| + |
| Named-unit routes, the console's own route included, keep today's root |
| -adoption and its privileged restart unchanged (§13). The console refuses |
| -update settings on a named-unit route, saying they aren't supported there |
| -yet. |
| - |
| -### 5.5 Rollback and moves |
| - |
| -**A box rolled back to a binary from before directives** ignores them and |
| -adopts every host from its channel. So `manual`, pins and major holds stop |
| -being enforced there. The console compares each directed instance's |
| -reported installed build with its directive, flags any mismatch on the |
| -instance and in the roll docs, and enforcement resumes with the next |
| -forward roll. |
| - |
| -**Moves and restores need no state carried with them.** The directive |
| -lives on the console-owned record, so the destination installs exactly |
| -what the source was told to install. |
| - |
| -**Replay.** On a box that has never applied a directive for a host, an |
| -older directive for the same incarnation could be replayed by someone |
| -with write access to the record. It could only install a build that host |
| -was previously directed to. The high-water mark refuses anything older |
| -from the first application on, and the console's mismatch check flags |
| -it. |
| +adoption and its privileged restart unchanged (§13). |
| |
| ## 6. Rollout and pacing |
| |
| @@ -260,7 +129,7 @@ Each box runs one rollout scheduler per (account, app). Each scheduler runs |
| in its own goroutine, off the agent's tick, so a slow rollout never delays |
| the tick or another app. The scheduler holds **a desired build per |
| instance**, never one build per app. Instances of one app on different |
| -channels or pins form different cohorts and never take each other's |
| +channels form different cohorts and never take each other's |
| targets. |
| |
| It is fed by: |
| @@ -508,15 +377,14 @@ unit it expects. |
| |
| ### 7.7 Root's part: staging and installing |
| |
| -Root's job is unchanged in kind. It stages and repoints channel-following |
| -hosts from the signed pointer, and directed hosts from their verified |
| -directive (§5). Swaps exec staged builds by path (§7.5 step 3), so they |
| +Root's job is unchanged. It stages and repoints from the signed pointer |
| +(§5). Swaps exec staged builds by path (§7.5 step 3), so they |
| don't depend on when root repoints. A swap waits only for its build to be |
| staged. Fallback execs the staged good build directly and needs nothing |
| from root (§8.2). **The agent never asks root for a specific build.** The |
| stage relay from #231 still triggers root passes, and still carries only a |
| timestamp. The agent touches it again every `stageRequestEvery` while any |
| -exec route's channel head or directive isn't installed, so a touch missed |
| +exec route's channel head isn't installed, so a touch missed |
| during a running pass is repeated. |
| |
| ### 7.8 A `hold` swap |
| @@ -657,14 +525,11 @@ status. While it stands, the scheduler makes no attempt onto that build for |
| any instance of the app on that box, and wakes boot the good build instead |
| (§8.2). Clearing: |
| |
| -- **A newer installed build** for an instance, from its channel or its |
| - directive, no longer matches the record. The record stays for any |
| +- **A newer installed build** for an instance, from its channel, no longer |
| + matches the record. The record stays for any |
| instance still on the failed build. |
| -- **`carlos instances update <host>`** clears it for that host only, and |
| +- **`carlos instances retry <host>`** clears it for that host only, and |
| the next attempt runs as a canary. |
| -- **A pinned build** is matched by build and directive revision. Setting the |
| - pin again bumps the revision and authorizes one more attempt, for that |
| - instance only. |
| - **`carlos deploy --retry`** writes an app-wide retry request with its own |
| id into the release store. It is signed like a channel pointer, so a box |
| can't forge one for another. Each box clears its record for that build when |
| @@ -694,13 +559,6 @@ a regression. |
| attempt: current, successor, retiring and spawning. The activator reports |
| those. |
| |
| -**Bucket retention** (`internal/host/retention.go`) adds to its protected |
| -set every build named by a directive on any provision record in the |
| -deployment. Moves and restores of directed instances install those builds |
| -from the bucket. If the housekeeper can't list the records, it prunes |
| -nothing that pass, just as it declines on any incomplete protected set |
| -today. |
| - |
| ### 8.6 Other cases |
| |
| - **The agent restarts in the middle of a swap.** `Shutdown` cancels and |
| @@ -718,40 +576,22 @@ strings here describe meaning, not proposed copy. |
| |
| ### 9.1 CLI |
| |
| -- `carlos apps set <app> --swap overlap|hold --swap-concurrency N |
| - --instance-control updates,channel,major,pin --default-updates auto|manual |
| - --default-channel C --default-major N`. |
| -- `carlos instances set <host> --updates … --channel … --major … --pin …`, |
| - where an empty value clears the setting. |
| -- `carlos instances update <host>` installs now (§5) and clears the |
| - instance's failure hold. |
| -- `carlos deploy --swap hold` stores the release's demand (§4.3). |
| +- `carlos apps set <app> --swap overlap|hold --swap-concurrency N`. |
| +- `carlos deploy --swap hold` stores the release's demand (§4.2). |
| `carlos deploy --retry` retries a build that failed (§8.4). |
| +- `carlos instances retry <host>` clears one instance's failure hold. |
| - `carlos deploy`'s watch follows the schedulers and reports per box: how |
| many instances have swapped, which failed and why, and which stayed on |
| their good build. It ends when every box is done or halted, or at its |
| existing deadline. A box counts as done when every instance's serving |
| build is its desired build. |
| -- `carlos instances list` shows installed, serving and desired builds and, |
| - when they differ, the reason: manual, pinned, held at major N, failed |
| - build, or rolling out. |
| +- `carlos instances list` shows the installed and serving builds and, when |
| + they differ, the reason: failed build or rolling out. |
| |
| ### 9.2 Console |
| |
| -App settings gain an Updates section with the §4.2 policy. Each instance |
| -page gains an Updates section with the §4.1 settings, editable by app |
| -members, and "Update now" when the instance is behind. |
| - |
| -### 9.3 Instance API and control socket |
| - |
| -- **Console API:** `GET` and `PUT /api/v1/instances/<host>/updates` and |
| - `POST …/updates/install`. They take app-member credentials, or the |
| - instance-bound credential for that host (§4.4) limited to |
| - `instance_control`. The instance-bound half ships with #155 step 6. |
| -- **Control socket:** `GET /updates`, mounted beside `schedule.Mount`'s |
| - verbs with the same bearer and fixed-host rule. It returns the box's copy |
| - of the settings, what the instance may change, and the installed, |
| - serving and desired builds with the reason they differ. |
| +App settings gain the §4.1 swap policy. The instance page shows the |
| +installed and serving builds, and any failure hold, with a retry action. |
| |
| ### 9.4 Events |
| |
| @@ -786,18 +626,13 @@ keep timer names stable across the two builds. Otherwise deploy with |
| |
| ## 11. Security |
| |
| -- The console checks only credentials it minted (§4.4). Bucket objects |
| - under box prefixes and box-minted admin tokens are never authority for a |
| - settings write. |
| -- Root installs only what a signed channel pointer or a signed directive |
| - names (§5). The agent carries directives but can't forge them, and root |
| - never runs anything the agent controls. The replay residual is stated in |
| - §5.5. |
| -- The console accepts a box-reported installed build into a directive |
| - only if the app's signed history or a pin authorized it for that |
| - instance (§5.3). |
| -- `pin` must name a shipped build of the same app. The console resolves it |
| - against the app's manifests, never as a path. |
| +- Root installs only what a signed channel pointer names, as today (§5). |
| + The agent never asks root for a specific build, and root never runs |
| + anything the agent controls. Fallback execs a staged build the agent |
| + could already run (§8.2). |
| +- `carlos deploy --retry` is signed like a channel pointer, so one box |
| + can't forge a retry that clears another box's failure records (§8.4). |
| +- Swap policy is an app setting, changed only by app members. |
| |
| ## 12. Testing |
| |
| @@ -817,14 +652,6 @@ keep timer names stable across the two builds. Otherwise deploy with |
| good build, supersession and re-checks, attempt-identity dedupe, the |
| restart and config failure paths, and config acknowledgement only after |
| a switch. |
| -- **Directive computation (console):** a table test of §5.3 over pin, |
| - manual, major, bake holds and untagged heads. Reported builds outside |
| - the instance's signed history are refused. Directives are dropped when |
| - settings return to default. |
| -- **Directive verification (root):** bad signatures, the wrong host or |
| - incarnation, a replay below the high-water mark, and an old root binary |
| - ignoring directives are each refused or reported. Channel-following |
| - hosts adopt exactly as today. |
| - **Containment:** per-generation children, the parent-limit raise and |
| restore with `limitsOK`, retiring one generation without touching the |
| other, and parent stop evidence. Include a memory-hog successor that |
| @@ -839,10 +666,6 @@ keep timer names stable across the two builds. Otherwise deploy with |
| sleepers moved cold boot the good build directly while the symlink still |
| names the failed build. A newer build, a retry or an install by hand |
| lifts the hold. |
| -- **Moves of directed instances:** a `manual` instance moved to a fresh |
| - box installs its directive's build, not the channel head. |
| -- **Retention:** directive builds are protected deployment-wide. An |
| - unlistable record set prunes nothing. |
| - **Config activation:** a failed config swap keeps delivered ≠ active, |
| retries with backoff even though the file is unchanged, and acknowledges |
| only once the revision is active. |
| @@ -853,8 +676,9 @@ keep timer names stable across the two builds. Otherwise deploy with |
| no failure. |
| - **Cache aliases:** a custom domain, a wildcard and a path-routed request |
| each miss after the switch. |
| -- **Bootstrap:** an upgraded box with sleeping instances and an empty |
| - `GoodVersion` doesn't move them cold until a wake has served. |
| +- **Bootstrap:** on an upgraded box, awake healthy instances record their |
| + good build at agent start. Sleeping ones record it on their first |
| + healthy serve, and have no fallback until then. |
| - **Nested:** the generation template's ID split, the shared slice and data |
| mounts, and per-generation executable mounts. |
| - **Fallback:** the first-wake budget, waiting requests served by the good |
| @@ -884,17 +708,18 @@ take a third kind of attempt for them, and #230 stays open for that spec. |
| Until that follow-up, named-unit routes, the console's own route included, |
| keep **today's adoption unchanged**. Root reads their channel, stages, |
| repoints, restarts the unit with its rollback-on-failed-restart, and |
| -stamps the result. Nothing in this spec touches that path (§5.4). |
| +stamps the result. Nothing in this spec touches that path (§5). |
| |
| ## 14. Out of scope |
| |
| - A fleet-wide canary (one box first). |
| - Keeping instances up across an agent restart or box roll. |
| - A configurable health path, or a settle on swaps after the canary. |
| -- Instance-only console principals (§4.4). |
| +- Per-instance update settings (manual, pin, major, in-app control). They |
| + are in `2026-10-07-instance-update-settings-design.md`. |
| - The level-triggered stage marker noted on #231. The agent touches the |
| marker again every `stageRequestEvery` (15 s) while any exec route's |
| - channel head or directive isn't installed. That narrows the gap without |
| + channel head isn't installed. That narrows the gap without |
| a unit change (§7.7). |
| |
| ## 15. Rollout of this feature |
| @@ -904,13 +729,7 @@ The box-side swap lands **dormant** behind a box flag, |
| uses today's stop-then-wake. It is proven on dev and then on |
| `bab/adopt-probe`. |
| |
| -Directives aren't behind the flag. An instance with default settings has |
| -none and is adopted exactly as today. A box whose root binary predates |
| -directives shows *settings not enforced* for its directed instances (§5.5). |
| - |
| -**Order:** boxes first, then the console's settings, policy and |
| -directives. That way no directive is issued before the boxes can honour |
| -it. |
| +**Order:** boxes first, then the console's swap policy settings. |
| |
| The changelog and the deploy docs tell implementers about the migrations |
| rule (§9.5) before the flag goes on across the fleet. From then on, every |
| @@ -991,3 +810,11 @@ exec app swaps in `overlap` unless it has chosen `hold`. |
| | 2 | A lease doesn't authenticate carried state | Superseded: no carried state; directives are console-signed and bound to host and incarnation (§5.2, §5.5) | |
| | 3 | No agent means legacy adoption | Superseded, as for R3-3 | |
| | 4 | Retention can delete a directed build | §8.5: directive builds are protected deployment-wide; an unlistable record set prunes nothing | |
| + |
| +### Round 6 |
| + |
| +All 6 findings concern per-instance update settings: status-derived |
| +manual targets, leaving directed mode, superseded directives, bake |
| +windows, and when to recompute. They moved with those settings to |
| +`2026-10-07-instance-update-settings-design.md` §7, with a proposed |
| +answer for each. |
amadan