amadan
spec: zero-downtime deploys rev 2 — answers Astra round 1 (17 findings)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+647
−409
docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md+561 −409docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME.md+86 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md| index b9ce295e5..519a9fc67 100644 |
| --- a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| +++ b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| @@ -1,7 +1,9 @@ |
| # Zero-downtime deploys: overlap swaps, update settings, paced rollout |
| -Status: draft for review, 2026-10-07. Discussions #231 (deploy pickup) and |
| -#230 (per-app restart pace, first-route health step) feed into this. |
| +Status: draft rev 2, 2026-10-07. Discussions #231 (deploy pickup) and #230 |
| +(per-app restart pace, first-route health step) feed into this. Rev 1 was |
| +rejected by Astra with 17 findings |
| +(`CODEX-FINDINGS-ZERO-DOWNTIME.md`). §15 maps each finding to its answer. |
| ## 1. Why |
| @@ -9,503 +11,653 @@ Paul's ruling, 2026-10-05: "When someone ships a new release, running |
| instances should pick up the new release immediately unless they're |
| configured not to, and should swap in the new version with zero-downtime." |
| -Pickup is fixed. A promotion now reaches an awake instance in 20–25 s on |
| -the flagship (hotfix `f3f6cf73`, #231). The swap is not fixed. Every path |
| -that moves an awake instance to a new build is stop-then-wake: the old |
| -process gets SIGTERM and up to 20 s to drain, new requests are held for up |
| -to 30 s (`wakeWaitBound`), and the new build boots on the next request. |
| -That is a pause on every deploy, and longer than a pause when the new build |
| -is slow to boot or fails. |
| +Pickup is fixed: a promotion reaches an awake instance in 20–25 s on the |
| +flagship (hotfix `f3f6cf73`, #231). The swap is not. Every path that moves |
| +an awake instance to a new build is stop-then-wake. The old process gets |
| +SIGTERM and up to 20 s to drain, new requests are held for up to 30 s |
| +(`wakeWaitBound`), and the new build boots on the next request. That is a |
| +pause on every deploy, and an outage when the new build is slow or fails. |
| -#230 adds a second problem. A pass that retargets many routes of one app |
| -on one box restarts them back to back, with no pace and no health step, so |
| -a build that boots and then dies reaches every tenant within seconds. |
| +#230 adds the second problem. A pass that retargets many routes of one app |
| +on one box restarts them back to back, with no pace and no health step. A |
| +build that boots and then dies reaches every tenant within seconds. |
| ## 2. Decisions taken in brainstorming |
| These were settled with Paul on 2026-10-06/07 and are not reopened here. |
| -1. **Overlap is the default.** The new process boots beside the old one |
| - and must pass its health check before new requests go to it. The old one |
| - then drains. Both share the instance's SQLite database for the overlap. |
| - So the new build's boot migrations run while the old build still serves, |
| - and apps must keep migrations backward-compatible. |
| -2. **`hold` is the per-app alternative**, and a single release can ask for |
| - it too. It is stop-then-start with requests held. It is for apps or |
| - releases whose migrations can't be shared with the previous build. |
| -3. **Instances get Eleven's update model.** Auto-update on or off, a |
| - channel, a major-version hold and an exact pin. The app implementer |
| - decides the defaults and how much of that model each instance may change. |
| -4. **Instance settings can be changed from the console and CLI, and by the |
| - app itself** over its control socket, within the app's policy. The app |
| - policy can be changed only by the implementer. |
| +1. **Overlap is the default.** The new process boots beside the old one and |
| + must pass its health check before new requests reach it; then the old |
| + one drains. Both processes share the instance's SQLite database during |
| + the overlap, so the new build's boot migrations run while the old build |
| + is still serving. Apps must keep migrations backward-compatible. |
| +2. **`hold` is the per-app alternative**, and a single release can demand |
| + it. It means stop-then-start with requests held. It exists for apps or |
| + releases whose migrations can't share the database with the previous |
| + build. |
| +3. **Instances get Eleven's update model:** auto-update on or off, a |
| + channel, a major-version hold and an exact pin. The app implementer sets |
| + the defaults and decides how much of that each instance may change. |
| +4. **Instance settings are changed from the console and CLI, and by the app |
| + itself**, within the app's policy. Only the implementer can change the |
| + policy. |
| 5. **The app policy lives in app settings.** A release can still demand |
| `hold` (`carlos deploy --swap hold`), and that demand is stored with the |
| release. |
| -6. **Pacing is per app, per box: one at a time by default, and the first |
| - swap is a canary.** The concurrency is an app setting. |
| +6. **Pacing is per app per box: one swap at a time by default, and the |
| + first swap is a canary.** The concurrency is an app setting. |
| 7. **Every first wake onto a new build is its own canary.** Sleeping |
| - instances move straight away. A failed first wake falls back to the |
| - previous build in `overlap` mode. |
| -8. **Mechanism: two sockets, and the edge switches.** Each spawn gets its |
| - own socket path, and the route's current socket moves after the health |
| - check. The rename trick was rejected because runtimes unlink their socket |
| - path on exit and would delete the successor's socket. Socket activation was |
| - rejected because only apps written for it would work. |
| -9. **Scope.** Exec instances, in both the flat and the nested layout, get |
| - overlap. Worker and sidecar processes keep stop-then-start. Named-unit |
| - routes can't overlap, because the unit belongs to the implementer, so they |
| - get the paced canary rollout and `hold`. Static sites already swap |
| - atomically. |
| + instances move to the new build straight away. In `overlap` mode, a |
| + failed first wake falls back to the last build that worked. |
| +8. **The mechanism is two sockets, with the edge switching between them.** |
| + Each spawn gets its own socket, and the instance's current socket moves |
| + only after the health check. A rename trick was rejected because runtimes |
| + unlink their socket path on exit. Socket activation was rejected because |
| + only apps written for it would work. |
| +9. **Scope.** Exec instances, in both the flat and nested layouts, overlap. |
| + Worker and sidecar processes keep stop-then-start. Static sites already |
| + swap atomically. Named-unit routes are in §13. |
| ## 3. Terms |
| -- **Build**: one shipped release version of an app (`Manifest.Version`, a |
| - short sha). |
| -- **Installed build**: the build an instance's live symlink names, which |
| - every wake boots. The registry stamps it as `Version`, with |
| - `PrevVersion` beside it. |
| -- **Target build**: the build the update settings say the instance should |
| - be on now (§5). |
| -- **Generation**: one spawn of one instance, numbered from 1, increasing |
| - for the life of the activator entry. |
| -- **Swap**: moving an awake instance from one generation to the next. |
| -- **Rollout**: the work on one box of moving every instance of one app to |
| - a new target build. |
| +- **Build**: one shipped version of an app (`Manifest.Version`). |
| +- **Installed build**: what the instance's live symlink names. It is stamped |
| + as registry `Version`, with `PrevVersion` beside it. |
| +- **Good build**: the last build that served for this instance after |
| + passing its health check (§8.5). This is new, and separate from |
| + `PrevVersion`, which records install history whether or not the build |
| + ever ran. |
| +- **Serving build**: the build of the generation currently taking new |
| + connections. It can differ from the installed build while a rollout |
| + queues, checks health, or falls back. |
| +- **Desired build**: what target resolution (§5) says the instance should |
| + be on now. |
| +- **Generation**: one spawn of one instance, numbered upwards for the life |
| + of the activator entry. |
| +- **Attempt**: one try at moving one instance to a new generation. It has |
| + an identity (§6.3). |
| +- **Swap**: an attempt applied to an awake instance. |
| +- **Cohort**: on one box, the instances of one app whose attempts share a |
| + desired build. The canary rule (§6.2) applies per cohort. |
| ## 4. Settings and policy |
| ### 4.1 Instance settings |
| -These are stored on the instance's provision record. The console owns the |
| -record, and boxes read it through the reconcile path as they do today. |
| +The settings live on the instance's provision record. The console owns the |
| +record, and boxes read it through reconcile as they do today. |
| -| Setting | Values | Default from policy | Meaning | |
| -|-----------|----------------------------|---------------------|---------| |
| -| `updates` | `auto` / `manual` | `auto` | With `manual`, nothing is installed automatically. The installed build stays put, across sleep and wake, until someone installs. | |
| -| `channel` | an existing channel name | the app's channel | The channel the instance follows. This is the per-route channel that exists today, now also settable on the record. | |
| -| `major` | empty or a non-negative integer | empty | Follow the channel, but hold a release whose semver tag has a different major (§5). It is accepted only with the `stable` channel, because only stable promotions cut semver tags (`stableTag`, `internal/release/release.go`). Setting it on any other channel is refused, with that reason. | |
| -| `pin` | empty or a build version | empty | Stay on exactly this build. It beats `major` and `channel`. It must name a build the app has shipped. | |
| +| Setting | Values | Meaning | |
| +|-----------|----------------------------|---------| |
| +| `updates` | `auto` / `manual` | With `manual`, nothing installs on its own. The installed build stays put, across sleep and wake, until someone installs or pins. | |
| +| `channel` | an existing channel | What the instance follows. It already exists per instance (`Record.Channel`). | |
| +| `major` | empty or an integer ≥ 0 | Follow the channel, but hold a release whose semver tag is absent or has another major. It is only accepted with `stable`, because only stable promotions cut tags (`stableTag`, `internal/release/release.go`). | |
| +| `pin` | empty or a build version | Be on exactly this build (§5). It must name a build the app has shipped. | |
| There is no "download automatically". A box fetches a build once and every |
| -instance on it shares it, so an instance has nothing of its own to download. |
| +instance on it shares it, so an instance has nothing of its own to |
| +download. |
| ### 4.2 App policy |
| -These are app settings, owned by the implementer. |
| - |
| | Setting | Values | Default | Meaning | |
| |--------------------|--------|---------|---------| |
| -| `swap` | `overlap` / `hold` | `overlap` | How an awake instance moves to a new build. | |
| -| `swap_concurrency` | integer ≥ 1 | 1 | How many instances of this app may swap at once on one box, after the canary. | |
| -| `instance_control` | a subset of {`updates`, `channel`, `major`, `pin`} | empty | Which instance settings the instance side may change. | |
| -| instance defaults | one value per §4.1 setting | as in §4.1 | Applied to new instances. They also apply to an existing instance whose setting has never been set explicitly. | |
| +| `swap` | `overlap` / `hold` | `overlap` | How an awake instance moves to a new generation. | |
| +| `swap_concurrency` | integer ≥ 1 | 1 | How many swaps of this app may run at once on one box, after a cohort's canary. | |
| +| `instance_control` | a subset of {`updates`, `channel`, `major`, `pin`} | empty | The settings the instance itself may change. | |
| +| instance defaults | one value per §4.1 setting | `auto`, the app's channel, empty, empty | Used where an instance has never set the value explicitly. | |
| -An empty `instance_control` keeps every existing app as it is today: |
| -instances change nothing until the implementer allows it. |
| +An empty `instance_control` keeps every existing app as it is today. |
| ### 4.3 The release's demand |
| -`Manifest` gains `swap` (`"hold"` or empty). Like `Label` and `Notes`, it |
| -is additive to schema 1 and absent from older manifests. `carlos deploy |
| ---swap hold` sets it at ship time. The effective mode for a swap onto a |
| -build is `hold` if either the app policy or that build's manifest says |
| -`hold`, and `overlap` otherwise. Because the manifest is immutable, the |
| -demand applies on every box, to every instance that ever moves onto that |
| -build, including instances that pick it up later by hand. |
| +`Manifest` gains `swap` (`"hold"` or empty). It is additive to schema 1, |
| +like `Label`, and absent from older manifests. `carlos deploy --swap hold` |
| +sets it when the release is shipped. An attempt onto build B runs in |
| +`hold` if either the app policy or B's manifest says `hold`. Otherwise it |
| +runs in `overlap`. |
| ### 4.4 Who may change what |
| -- **App members** with write access may set any instance setting on any |
| - instance of the app, and the app policy. |
| -- **The instance side** may change only the settings named in |
| - `instance_control`. That is console users with access to the instance but |
| - not to its app, and the app itself over the control socket (§9.3). A |
| - refused change names the setting and says the app keeps it for itself. |
| -- Installing by hand (§9) follows the same rule as `updates`. An instance |
| - side that may not change `updates` may not install by hand either. |
| +- **App members** with write access, in the console or CLI, may set any |
| + instance setting and the app policy. This uses today's `apiAccountApp` |
| + gate. |
| +- **The instance itself** may change the settings named in |
| + `instance_control`, and install by hand if `updates` is named. It does |
| + this by calling the console API with an **instance-bound credential**, |
| + which is #155's step 6: a token that names one instance, minted by the |
| + console and checked by the console. The console can check a credential it |
| + minted. It cannot check `CARLOS_ADMIN_TOKEN`, which the box mints |
| + (`internal/activator/admintoken.go`), nor a bucket object written under a |
| + box prefix, which is only a pseudonym (`internal/edge/storedoor.go`). So |
| + neither of those is used for writes. |
| +- The control socket offers **read only**: `GET /updates` answers from the |
| + box's copy of the record (§9.3). In-app writes ship once #155 step 6 has |
| + landed. Until then an app can show its update state but not change it. |
| +- No console principal has "access to this instance but not its app", so |
| + this spec doesn't invent one. Tenants without Carloku accounts act through |
| + their app. |
| +- A refused change names the setting, and says the app keeps it for itself. |
| ## 5. Target resolution |
| -For each instance, given the app's channel heads and the instance's |
| -settings, apply these rules in order: |
| - |
| -1. `pin` set: the target is the pinned build. |
| -2. `updates = manual`: the target is the installed build (no move). |
| -3. Otherwise, take the head of the instance's `channel`. If `major` is set |
| - and the head's semver tag is absent, or has a different major, the |
| - target is the installed build and the instance is *held at major N*. |
| -4. Otherwise the target is the channel head. |
| - |
| -Bake holds (`internal/release/bake.go`) apply to rule 4 exactly as today. |
| -`FastTrack` waives them for one promotion. Rules 1 and 2 never move an |
| -instance on their own. Installing by hand on a pinned instance is refused, |
| -with the pin named; clearing the pin comes first. Otherwise, installing by |
| -hand sets the installed build to the channel head, as rule 4 would if |
| -`updates` were `auto`, and ignores `major`. That is the point of the major |
| -hold: a person decides. |
| - |
| -A pin to a build that is not staged on the box is staged as adoption |
| -stages anything else. |
| +There is **one resolver**, `release.Desired(record, policy, heads, failures)`. |
| +Every path that installs a build calls it: the agent's pass, the root stage |
| +pass (`carlos adopt --stage-only`), the hourly adopt loop, and first-time |
| +materialization. The only exception is an explicit per-host request (§7.7). |
| +No path reads a channel pointer to make its own choice. |
| + |
| +For each instance, take the first rule that applies: |
| + |
| +1. **`pin` set:** the desired build is the pinned build. Setting a pin is |
| + itself an install request: the instance moves to the pinned build |
| + whether `updates` is `auto` or `manual`, and no bake hold applies, |
| + because a person chose it. |
| +2. **`updates = manual`:** the desired build is the installed build. An |
| + instance with no installed build yet resolves as if `auto`. |
| +3. **A failure record holds the channel head for this instance's scope** |
| + (§8.4): the desired build is the installed build. |
| +4. **`major` is set, and the head's tag is absent or has another major:** |
| + the desired build is the installed build, and the instance is *held at |
| + major N*. |
| +5. **Otherwise:** the head of the instance's `channel`, after bake holds |
| + (`internal/release/bake.go`) and `FastTrack`, exactly as today. |
| + |
| +**Installing by hand** is refused on a pinned instance, naming the pin. |
| +Otherwise it moves the instance to its channel head, ignoring `manual`, |
| +`major` and bake holds. It also clears a failure record for that instance |
| +only (§8.4). |
| ## 6. Rollout and pacing |
| -### 6.1 Shape |
| - |
| -When a release reaches a box, through a serial bump and `fullPass`, the |
| -agent resolves a target for every route of the app (§5) and starts or |
| -updates the app's **rollout** on that box. Each app's rollout runs in its |
| -own goroutine, off the agent's tick. That follows the lesson of the |
| -converge leg (`fix-converge-own-loop`): a slow rollout never delays the |
| -tick, and never delays another app's rollout. |
| - |
| -A rollout on one box has a target build, a queue of awake instances still |
| -to swap, and a state: running, done or halted. A newer target for the same |
| -app replaces the queue. Instances already swapped to the superseded build |
| -are simply re-queued for the new one. |
| - |
| -### 6.2 Order |
| - |
| -1. **Sleeping instances move at once.** Their installed build is |
| - repointed, through the root stage relay when the agent can't write |
| - `/opt`, and they boot it on their next wake. There is no process, so |
| - there is nothing to pace. |
| -2. **Awake instances swap in queue order.** The first swap of the rollout is |
| - the **canary**. It must pass the health check and then stay up for a |
| - **10 s settle**, which catches the build that boots and then dies. No |
| - other awake instance of the app on that box swaps until the canary |
| - settles. After that, up to `swap_concurrency` swaps run at once. A |
| - non-canary swap needs only the health check. |
| -3. A route the agent can't swap because it is waking or stopping keeps its |
| - queue place. It is retried when it reaches a stable state. A swap that |
| - lands during a wake is deferred the way `bounceAfterWake` is today. |
| - |
| -### 6.3 First-wake canary |
| - |
| -Every first wake of an instance onto a build it hasn't run before counts as |
| -a canary for that build on that box. In `overlap` mode, a first wake that |
| -fails its health check falls back (§8.2). In `hold` mode it does not. |
| - |
| -### 6.4 Restarts and config changes use the rollout |
| - |
| -`carlos restart`'s request leg (`restartRequestLeg`/`bounceExecRoutes`, |
| -`internal/host/restart.go`) and the config-change bounce |
| -(`appconfig/syncexec.go`) stop bouncing every awake instance in one pass. |
| -They queue a rollout with the instance's own installed build as the |
| -target, so swap, pacing and canary apply to them unchanged. That is #230 |
| -item 1, for releases and restarts alike. |
| +### 6.1 The scheduler |
| + |
| +Each box runs one rollout scheduler per (account, app). Each scheduler runs |
| +in its own goroutine, off the agent's tick, so a slow rollout never delays |
| +the tick or another app. The scheduler holds **a desired build per |
| +instance**, never one build per app. Instances of one app on different |
| +channels or pins form different cohorts and never take each other's |
| +targets. |
| + |
| +It is fed by: |
| +- the agent's pass, which runs the resolver for each route after a serial |
| + bump, a config sync or a reconcile; |
| +- `carlos restart` requests, which arrive through `restart-gen.json`; |
| +- config-revision changes; |
| +- installs by hand and retries. |
| + |
| +### 6.2 Order within a cohort |
| + |
| +1. **Sleeping instances move at once.** The installed build is repointed |
| + (§7.7), and they boot it on their next wake, which is a first-wake |
| + canary (§6.4). There is no process, so there is nothing to pace. |
| +2. **Awake instances swap.** The first swap in a cohort is its **canary**. |
| + It must pass the health check and then stay up for a **10 s settle**. No |
| + other swap in that cohort starts until the canary settles. After that, |
| + up to `swap_concurrency` swaps run at once across the app's cohorts on |
| + the box. A swap after the canary needs only the health check. |
| +3. An instance that is waking or stopping keeps its place in the queue and |
| + is tried again once it is stable. An attempt that lands during a wake is |
| + deferred, as `bounceAfterWake` is today. |
| + |
| +### 6.3 Attempt identity, and what supersedes what |
| + |
| +An attempt is identified by (host, cause, desired build, config revision, |
| +restart request id, scheduler revision). The causes are `release`, |
| +`install`, `retry`, `restart` and `config`. |
| + |
| +- **Dedupe** compares the whole identity. A `restart` or `config` attempt |
| + onto the build already serving is real work, never a no-op. This keeps |
| + `BounceAdopted`'s exclusion of restart and config requests |
| + (`internal/activator/activator.go`). Only a `release` attempt whose |
| + desired build is already serving is dropped. |
| +- **Supersession is per instance.** A newer desired build for an instance |
| + replaces its queued attempt. An attempt already past spawn finishes or is |
| + cancelled at the entry's barrier (§7.4). Before switching, the swap |
| + re-checks that its attempt is still the instance's newest; if not, it |
| + abandons the successor instead of switching. |
| +- Restart and config attempts for one instance coalesce: the newest config |
| + revision and restart id cover the older ones. |
| + |
| +### 6.4 First-wake canary |
| + |
| +A wake onto a build the instance has never served is a canary for that |
| +build. It gets its own budget (§8.2). A settle doesn't apply to the |
| +request that is waiting. Instead, if the new generation exits within 10 s |
| +of becoming healthy, that counts as a failure of the build: the |
| +failure record is written (§8.4), and the next wake uses the good build. |
| ### 6.5 Across boxes |
| -Each box runs its own rollout and its own canary. A halt on one box does |
| -not stop the others, but the deploy watch reports it (§9.1). A fleet-wide |
| -canary, with one box first, is out of scope (§12). |
| +Each box runs its own schedulers and canaries. A halt on one box does not |
| +stop the others. The deploy watch reports it (§9.1). A fleet-wide canary is |
| +out of scope (§14). |
| ## 7. The swap |
| -### 7.1 Generations and sockets |
| +### 7.1 Generations and app-socket consumers |
| The activator entry gains a generation counter. Each spawn takes the next |
| -generation `n`, and its paths carry it: |
| +generation `n`: |
| -| Today | With generations | |
| -|-------|------------------| |
| -| `/run/carlos/<host>.sock` | `/run/carlos/<host>.<n>.sock` | |
| -| `/run/carlos/instances/<id>/app.sock` | `/run/carlos/instances/<id>/app.<n>.sock` | |
| +| Canonical (registry) | Generation path | |
| +|----------------------|-----------------| |
| +| `/run/carlos/<host>.sock` | `/run/carlos/<host>.g<n>.sock` | |
| +| `/run/carlos/instances/<id>/app.sock` | `/run/carlos/instances/<id>/app.g<n>.sock` | |
| | `<socket>.control` | `<generation socket>.control` | |
| -The app still receives `--socket <path>` and `CARLOS_CONTROL_SOCKET`. |
| -Nothing changes for the app. `prepare()` deletes only the socket path of the |
| -generation it is preparing, never a sibling generation's. |
| - |
| -### 7.2 The edge looks up the socket per connection |
| - |
| -Today the route's proxy is built against one fixed socket path |
| -(`unixProxy(t.Socket)`, `internal/edge/edge.go`). It becomes a dial |
| -function that asks the activator for the host's **current socket** each |
| -time it opens a connection. That is a map read under the entry's lock. The |
| -transport keys its idle pool per socket path, so a pooled keep-alive to |
| -generation `n` is never reused for `n+1`. When a generation stops being |
| -current, its idle pooled connections are closed. `dialUnixWithRetry`'s |
| -ENOENT/ECONNREFUSED retry stays: a dial that raced a switch retries |
| -against the new current socket. Routes that aren't activator-managed keep |
| -their fixed socket. |
| - |
| -### 7.3 An overlap swap, step by step |
| - |
| -1. The entry gains a **successor**. It keeps its current process, lease and |
| - replicator. The replicator and the lease belong to the agent, not to |
| - either process, as `respawnWorkerLocked` already relies on. Both |
| - processes open the same database file. Litestream requires WAL mode, and |
| - in WAL mode SQLite coordinates processes on one machine through its |
| - locks and `-shm`. |
| -2. During the overlap the instance's cgroup memory limit is raised to |
| - **twice** its tier's `memory.max`. Both generations run in the same |
| - `carlos-exec@<host>` cgroup, or the same nested instance cgroup, so |
| - cgroup-wide accounting and stop evidence cover both. |
| -3. The successor spawns from the target build, with the generation-`n+1` |
| - paths, and must pass the existing health check (`GET /` below 500 within |
| - `spawnHealthTimeout`, 30 s). A canary must then also survive the 10 s |
| - settle. |
| -4. **Switch.** Under the entry lock, the current socket becomes the |
| - successor's, the entry's `spawnVersion` becomes the target build, and |
| - the host's edge cache is purged. The purge moves from spawn time to |
| - switch time. New connections now reach the successor. |
| -5. The old generation gets SIGTERM. Its in-flight requests finish under its |
| - own graceful shutdown, with `actStopTimeout` (20 s) before SIGKILL, as |
| - today. Long-lived connections (websockets, SSE) close when it exits. Its |
| - control socket and exit records are removed. |
| -6. When the old generation has exited, the memory limit goes back to the |
| - tier's `memory.max` and the swap is done. The event is in §9.4. |
| -7. A legacy co-spawned sidecar (`<host>-sidecar.env`) is then restarted by |
| - stop-then-start onto the new build. |
| - |
| -There is one successor at a time per entry. A swap requested while one is |
| -in flight is queued behind it. |
| - |
| -### 7.4 The nested layout |
| - |
| -The root control service's start verb gains a generation argument and |
| -starts `carlos-instance@<id>.<n>.service`, a second instance of the same |
| -template beside the first. The launch file, log sockets and termination |
| -proof (unit state plus an empty cgroup) are per generation. Both units are |
| -placed in the instance's cgroup subtree, so the instance-level limit and |
| -stop evidence see both. Raising and restoring the limit for the overlap |
| -goes through the same control service. |
| - |
| -### 7.5 A `hold` swap |
| - |
| -This is stop-then-start on the existing path. The old generation stops, |
| -with requests held by `begin()` up to `wakeWaitBound`. The new generation |
| -then boots. It no longer has to wait for the next request: the swap boots |
| -it straight away, so requests held across it are served as soon as it is |
| -healthy. |
| - |
| -### 7.6 Workers, sidecar members, named units, static sites |
| - |
| -- **Worker processes and native sidecar members** (`Kind=sidecar`) serve |
| - no requests, and two copies could do the same job twice. They keep |
| - stop-then-start, with the respawn-on-same-entry path, and are paced by |
| - the rollout like any other swap. |
| -- **Named-unit routes** (`-unit`) get the rollout's pacing and canary, with |
| - `try-restart` as the swap and `WaitHealthyVersion` as the health check |
| - (#230 item 2). There is no overlap and no fallback, as in `hold`. #230 |
| - item 3, `-unit` at provisioning, is not part of this spec. |
| -- **Static sites** are unchanged. Their swap is already atomic |
| - (`internal/edge/static.go`). |
| +The registry keeps the canonical path as the route's identity. The |
| +activator owns the mapping from host to **current generation socket**, and |
| +every component that dials an app socket asks it, through one function, |
| +`activator.AppSocket(host)`: |
| + |
| +- the edge proxy (§7.2); |
| +- scheduled delivery (`internal/host/schedule_wiring.go`), which today |
| + builds its client from `rt.Socket`; |
| +- the next-due probe at stop (`backend_exec.go`'s `NextDue`), which must |
| + ask the **retiring** generation while that generation is draining and the |
| + current one otherwise; |
| +- version probes and health checks; |
| +- anything else that reads `rt.Socket` to dial the app. The plan lists |
| + every reader of `rt.Socket` and assigns each one to the canonical path or |
| + to `AppSocket`. |
| + |
| +`prepare()` deletes only its own generation's socket path, never a |
| +sibling's. The app still just receives `--socket <path>` and |
| +`CARLOS_CONTROL_SOCKET`. |
| + |
| +### 7.2 Edge |
| + |
| +The edge builds an activator-managed route's proxy around a dial function |
| +that calls `AppSocket(host)` for each new connection, instead of |
| +`unixProxy(t.Socket)`. The transport's idle pool is keyed per socket path, |
| +so a keep-alive to generation `n` is never reused for `n+1`. When a |
| +generation stops being current, its idle connections are closed. |
| +`dialUnixWithRetry` keeps its ENOENT/ECONNREFUSED retry, and each retry |
| +asks `AppSocket` again. |
| + |
| +**The edge cache gets a per-host generation.** A request records the |
| +host's cache generation when it starts. When it finishes, its response is |
| +stored only if the generation hasn't changed. The switch purges the cache |
| +and bumps the generation, so a response finishing late on a draining |
| +process never fills the cache after the purge. |
| + |
| +**`X-Carlos-Version` reports the build of the generation that served the |
| +request**, not the registry's `Version`. Status and the deploy watch show |
| +installed, desired and serving builds separately. |
| + |
| +### 7.3 Containment |
| + |
| +The host's cgroup becomes a **parent that holds no processes**, with one |
| +child cgroup per generation: `carlos-exec@<host>/g<n>`, or the nested |
| +instance slice's equivalent. |
| + |
| +- The tier's memory limit is set on the parent. During an overlap it is |
| + raised to twice `memory.max`. It goes back once the retiring generation's |
| + child is empty. |
| +- The claim-mode reservation (`PrepareStrict`, |
| + `internal/cgroup/strict_linux.go`) moves from one per host to one per |
| + generation child. The parent's identity and limits are prepared once per |
| + entry, and its expected-limit record is updated through the same API that |
| + raises and restores the limit. That way `limitsOK` never sees a limit it |
| + didn't set. |
| +- `KillAndEmpty` on a generation acts only on that generation's child. |
| + Retiring the old generation never touches the successor. |
| +- Stop evidence (`internal/stopevidence`) is collected on the **parent**: |
| + it is empty only when every generation is. |
| + |
| +### 7.4 The entry's barrier |
| + |
| +The entry owns one **operation set**: spawning, current, successor and |
| +retiring generations. Each is tracked until its termination is proven. |
| + |
| +- **Every terminal stop cancels and joins every member of the set** before |
| + the final checkpoint, `finishStop`, the replicator close and the lease |
| + release. That covers hibernation, lease loss, `Shutdown`, maintenance's |
| + `FenceAndStop`, moves and parks. If termination can't be proven |
| + (`ErrStillRunning`), ownership and the lease are kept, as today. |
| +- `Shutdown` covers entries that are waking or swapping. Today it skips |
| + waking entries. |
| +- Before a spawn and before a switch, the attempt re-checks, under the |
| + host's guard (`hostguard.With`), that the entry is still awake, that the |
| + lease is still held, that its attempt is still the newest (§6.3), and |
| + that no claim operation holds the guard. If any check fails, the |
| + successor is abandoned. |
| +- A swap never starts while the host's guard is held for an operation that |
| + needs stop evidence. |
| + |
| +### 7.5 An overlap swap, step by step |
| + |
| +1. The attempt passes the §7.4 checks and joins the operation set as the |
| + successor. The lease and replicator belong to the agent and stay put, as |
| + `respawnWorkerLocked` already relies on. Both generations open the same |
| + database. Litestream requires WAL, and in WAL mode SQLite coordinates |
| + processes on one machine through its locks and `-shm`. |
| +2. The parent's memory limit is raised (§7.3). |
| +3. The successor execs **the attempt's build, by its immutable staged |
| + path** (`/opt/carlos/versions/<app>/<build>/…`), never through the live |
| + symlink. The nested launch file and executable mount name the same |
| + staged path. The build the generation runs is therefore the build the |
| + attempt chose, whatever root does to the symlink meanwhile. |
| +4. The successor must pass the health check: `GET /` below 500 within |
| + `spawnHealthTimeout` (30 s). A canary must then survive the 10 s settle. |
| +5. **Switch.** The §7.4 checks are made again. Then, under the entry lock, |
| + `AppSocket` moves to the successor, the entry's serving build becomes |
| + the attempt's build, and the cache generation is bumped and purged. The |
| + successor becomes current, and the old generation becomes retiring. |
| +6. The retiring generation gets SIGTERM and finishes its in-flight requests |
| + under its own graceful shutdown, with `actStopTimeout` (20 s) before |
| + SIGKILL. Long-lived connections close when it exits. Its control socket |
| + and exit records are removed once its child cgroup is empty. |
| +7. The parent's limit is restored, the good build is updated (§8.5), and |
| + the attempt is done. |
| +8. A legacy co-spawned sidecar (`<host>-sidecar.env`) is restarted by |
| + stop-then-start onto the same staged build. |
| + |
| +There is one successor per entry at a time. |
| + |
| +### 7.6 The nested layout |
| + |
| +The unit template separates the **host ID** from the **generation**. Today |
| +`carlos-instance@.service` passes `%I` as the instance ID and derives the |
| +slice, data mounts and executable mounts from it, so `<id>.<n>` would |
| +address a different instance. The new template, |
| +`carlos-instance-gen@<id>:<n>.service`, has the trampoline split the |
| +instance string. The slice and the data mounts are keyed by host ID only. |
| +The launch file, the executable mount, the log sockets and exit records are |
| +keyed by generation. The root control service's start verb takes the |
| +generation, and its observers and sweeps learn the new unit name. The old |
| +template stays until no box runs it, so a rolled-back binary still has the |
| +unit it expects. |
| + |
| +### 7.7 Repoints go through an exact per-host request |
| + |
| +The sandboxed agent can't write `/opt`. Today it asks root through a marker |
| +that carries only a timestamp, and the root pass reads the channel head |
| +itself. So a repoint to any other build is impossible, and root can |
| +override `manual`, a pin or a failure hold. |
| + |
| +The agent now writes **one request file per host** under its data dir: |
| +`stage-requests/<host>.json`. Each holds the destination build, the |
| +installed build it expects to replace, the attempt identity, and a |
| +sequence number. The existing marker still triggers the root pass. The root |
| +pass then: |
| + |
| +- for each request file: stages the destination if needed, and repoints |
| + only if the live symlink still names the expected build. It writes the |
| + outcome to `stage-results/<host>.json`, which the agent reads; |
| +- for every other host: runs the shared resolver (§5), so its own choices |
| + can't disagree with the agent's; |
| +- writes `SetVersions` with an expected-current condition, so a stale |
| + outcome can't overwrite a newer install. |
| + |
| +Request files are validated before root acts on them: the host must exist |
| +in the registry, the build must be a staged or stageable build of that |
| +host's app, and the path components are checked. A box running an older |
| +root binary ignores the directory and behaves as today. |
| + |
| +### 7.8 A `hold` swap |
| + |
| +This is stop-then-start on the existing path, run through the barrier. The |
| +old generation stops while `begin()` holds requests, up to `wakeWaitBound`. |
| +The new generation boots straight away from the attempt's staged build, and |
| +held requests are served once it is healthy. |
| + |
| +### 7.9 Workers and sidecar members |
| + |
| +Worker processes and native sidecar members (`Kind=sidecar`) serve no |
| +requests, and two copies could do the same work twice. They keep |
| +stop-then-start, on the respawn-on-same-entry path. They exec the attempt's |
| +staged build, and the scheduler paces them like any other attempt. |
| ## 8. Failure and rollback |
| ### 8.1 A failed overlap swap |
| -A failed swap is one where the successor fails its health check, exits |
| -before the switch, or (as canary) dies within the settle. The successor is |
| -stopped, the old generation keeps serving, the memory limit is restored, |
| -and no request has seen anything. Then: |
| - |
| -- **The rollout halts for that app on that box, at the first failure**, |
| - whether the canary or a later swap. |
| -- Instances already swapped stay on the new build. The failed instance, |
| - and every awake and sleeping instance the rollout moved but hasn't |
| - successfully run, are repointed to their `PrevVersion` (through the stage |
| - relay when needed). |
| -- The box records **failed build B for app A**, in the agent's data dir |
| - beside the stage marker, so it survives an agent restart. Target |
| - resolution treats B as held for A on that box until the record is |
| - cleared. The record clears on a newer target build, `carlos restart`, |
| - installing by hand, or a redeploy of B, meaning a promotion that names B |
| - again. |
| +A swap fails when the successor fails the health check, exits before the |
| +switch, or, as a canary, dies within the settle. The successor is stopped |
| +(its child cgroup emptied), the limit is restored, and the current |
| +generation keeps serving. No request notices. What happens next depends on |
| +the cause: |
| + |
| +- **`release`, `install`, `retry`.** A failure record is written (§8.4). |
| + The cohort halts on that box at the first failure, whether that's the |
| + canary or a later swap. Swapped instances stay on the new build. Every |
| + instance in the cohort that the rollout moved but that has never served |
| + the build gets an exact repoint (§7.7) back to its **good build**. |
| +- **`restart`, `config`.** No build changes, and no failure record is |
| + written. The current generation keeps serving the build it was serving. |
| + The attempt is reported failed. A config revision is acknowledged |
| + (`internal/appconfig/syncexec.go`) only after a successful switch, so a |
| + failed activation stays unacknowledged and is retried by config sync's |
| + existing rules. |
| ### 8.2 A failed first wake (`overlap` mode) |
| -When a first wake onto build B fails its health check, the entry spawns the |
| -instance's `PrevVersion` directly from its versions directory. The agent |
| -can exec staged builds; it just can't write `/opt`. The request waiting on |
| -the wake is served by that build. The agent then asks root to point the |
| -live symlink back, and records B as failed for the app (§8.1), which halts |
| -the rollout. The version sweep already treats `PrevVersion` as a pin |
| -(`internal/host/versionsweep.go`), so the previous build is still on the |
| -box. An instance with no `PrevVersion` (its first build ever) has nothing to |
| -fall back to and fails as a wake fails today. |
| +A first wake onto build B runs inside a **first-wake budget**: 30 s for B to |
| +become healthy, then up to 25 s to boot the good build. Requests waiting on |
| +that wake are held for up to 55 s, in place of `wakeWaitBound`'s 30 s, and |
| +only for first wakes onto a new build. If B fails: |
| + |
| +1. B's generation is stopped and its child cgroup emptied. |
| +2. The good build is spawned from its staged path. Exec needs no write to |
| + `/opt`. |
| +3. The waiting requests are served by it. |
| +4. A failure record for B is written (§8.4), and an exact repoint to the |
| + good build is requested (§7.7). |
| + |
| +An instance with no good build (it has never served anything) fails as a |
| +wake fails today. The settle rule for first wakes is §6.4. |
| ### 8.3 A failed `hold` swap |
| Nothing falls back automatically, because a `hold` build may have migrated |
| -the database past what the previous build can read. The instance is down, |
| -the rollout halts, and the deploy watch says so, naming `carlos rollback` |
| -as the way out. |
| - |
| -### 8.4 Other cases |
| - |
| -- **The agent restarts during a swap.** `Shutdown` stops every child, both |
| - generations included, as today. The next wake boots the installed build. |
| - The failed-build record survives on disk. |
| -- **The lease is lost during a swap.** Both generations stop, through the |
| - existing lost-lease path. |
| -- **The new generation crashes after the switch and the settle.** That is an |
| - ordinary crash, handled by the existing exit and respawn path. It is not |
| - a failed swap, and nothing rolls back. |
| -- **A successor that never gets healthy** within `spawnHealthTimeout` (30 s) |
| - is a failed swap. |
| +the database past what the good build can read. The instance is down, the |
| +cohort halts, and the deploy watch says so, naming `carlos rollback`. |
| + |
| +### 8.4 Failure records |
| + |
| +A failure record names an **(app, build, box) scope**. It is stored in the |
| +agent's data dir, so it survives agent restarts, and it is reported in box |
| +status. Resolver rule 3 holds that build for every instance of the app on |
| +that box. Clearing: |
| + |
| +- **A newer build on an instance's channel** gives that instance a |
| + different head, so the record no longer matches it. The record stays for |
| + any instance still pointed at the failed build. |
| +- **`carlos instances update <host>`** clears it for that host only, and |
| + the next attempt runs as a canary. |
| +- **`carlos deploy --retry`** writes an app-wide retry request with its own |
| + id into the release store. Each box clears its record for that build when |
| + it sees a retry id newer than the record. This is how "deploy the same |
| + build again" retries: promoting a build that's already current is a no-op |
| + (`internal/release/release.go`) and can't be the signal. |
| +- `carlos restart` doesn't clear failure records. It restarts the serving |
| + build. |
| + |
| +### 8.5 The good build |
| + |
| +`GoodVersion` is a new registry column. It is set when a generation on |
| +build B becomes current after passing its health check. For a canary, that |
| +means after the settle. It is written with an expected-current condition. |
| +Fallback uses it, and `PrevVersion` keeps its old meaning. The version |
| +sweep (`internal/host/versionsweep.go`) adds to its pins `GoodVersion` and |
| +every build named by a live generation or a pending attempt (current, |
| +successor, retiring, spawning), which the activator reports. |
| + |
| +### 8.6 Other cases |
| + |
| +- **The agent restarts in the middle of a swap.** `Shutdown` cancels and |
| + joins the whole operation set (§7.4). The next wake boots the installed |
| + build. Failure records and request files are on disk. |
| +- **The lease is lost in the middle of a swap.** Every generation is |
| + stopped through the barrier before the lease state is acted on. |
| +- **A crash after the switch and the settle** is an ordinary crash, with |
| + ordinary recovery. Nothing rolls back. |
| ## 9. Surfaces |
| -All member-facing wording below goes through copy review before it ships. |
| -The strings here are placeholders for meaning, not proposed copy. |
| +All member-facing wording goes through copy review before it ships. The |
| +strings here describe meaning, not proposed copy. |
| ### 9.1 CLI |
| - `carlos apps set <app> --swap overlap|hold --swap-concurrency N |
| --instance-control updates,channel,major,pin --default-updates auto|manual |
| --default-channel C --default-major N`. |
| -- `carlos instances set <host> --updates auto|manual --channel C --major N |
| - --pin V`, with an empty value to clear. Limited by §4.4. |
| -- `carlos instances update <host>`: install now (§5). |
| -- `carlos deploy --swap hold`: stores the demand in the manifest (§4.3). |
| -- `carlos deploy`'s watch follows the rollout and reports per box: how many |
| - instances have swapped, which ones failed and why, and which stayed on |
| - the previous build. It finishes when every box is done or halted, or when |
| - its existing deadline passes. |
| -- `carlos instances list` shows each instance's installed build and, when |
| - it is behind its channel, why: manual, pinned, held at major N, failed |
| +- `carlos instances set <host> --updates … --channel … --major … --pin …`, |
| + where an empty value clears the setting. |
| +- `carlos instances update <host>` installs now (§5) and clears the |
| + instance's failure hold. |
| +- `carlos deploy --swap hold` stores the release's demand (§4.3). |
| + `carlos deploy --retry` retries a build that failed (§8.4). |
| +- `carlos deploy`'s watch follows the schedulers and reports per box: how |
| + many instances have swapped, which failed and why, and which stayed on |
| + their good build. It ends when every box is done or halted, or at its |
| + existing deadline. A box counts as done when every instance's serving |
| + build is its desired build. |
| +- `carlos instances list` shows installed, serving and desired builds and, |
| + when they differ, the reason: manual, pinned, held at major N, failed |
| build, or rolling out. |
| ### 9.2 Console |
| -- App settings gain an Updates section with the §4.2 policy. |
| -- Each instance page gains an Updates section with the §4.1 settings. A |
| - setting the app keeps for itself is shown read-only, labelled as set by |
| - the app. "Update now" appears when the instance is behind. |
| +App settings gain an Updates section with the §4.2 policy. Each instance |
| +page gains an Updates section with the §4.1 settings, editable by app |
| +members, and "Update now" when the instance is behind. |
| -### 9.3 Control socket |
| +### 9.3 Instance API and control socket |
| -These are three verbs mounted beside `schedule.Mount`'s, with the same |
| -bearer (`CARLOS_ADMIN_TOKEN`) and the same fixed-host rule: the host comes |
| -from the socket, never from the request. |
| - |
| -- `GET /updates` returns the instance's settings, which of them it may |
| - change, its installed build, its target build and, when they differ, why. |
| -- `PUT /updates` changes settings named in `instance_control`. Anything |
| - else is refused with 403 and the setting's name. |
| -- `POST /updates/install` installs now, if `updates` is in |
| - `instance_control`. |
| - |
| -**How a change reaches the console.** The agent doesn't own the record, so |
| -it writes the request as an object under its own box's status prefix in |
| -the deployment bucket. That is the channel `statusLoop` already writes |
| -through. The console applies a request only when all of these hold: |
| - |
| -- the instance is placed on the box whose prefix carries it (the box can |
| - vouch only for what it hosts); |
| -- the change is allowed by the app policy at the time it is applied; |
| -- the request is newer than the last one it applied for that instance. |
| - |
| -`PUT` and `POST` answer 202 with the pending change. `GET /updates` shows |
| -it as pending until the record carries it. The plan must confirm how |
| -quickly the console reads box status. If no loop reads it within a |
| -minute, the plan adds one for this prefix. It must not add a box→console |
| -HTTP path. |
| +- **Console API:** `GET` and `PUT /api/v1/instances/<host>/updates` and |
| + `POST …/updates/install`. They take app-member credentials, or the |
| + instance-bound credential for that host (§4.4) limited to |
| + `instance_control`. The instance-bound half ships with #155 step 6. |
| +- **Control socket:** `GET /updates`, mounted beside `schedule.Mount`'s |
| + verbs with the same bearer and fixed-host rule. It returns the box's copy |
| + of the settings, what the instance may change, and the installed, |
| + serving and desired builds with the reason they differ. |
| ### 9.4 Events |
| -The new events are swapping, swapped (with the old generation's drain |
| -time), swap failed (still serving the previous build), and rollout halted |
| -(app, box, build, reason). The bounce event stops saying "(carlos restart)" |
| -for everything. It names its cause: a new release, a config change, or |
| -`carlos restart`. |
| +Swapping, swapped (with the retiring generation's drain time), swap failed |
| +(still serving build X), cohort halted (app, box, build, reason), and |
| +fallback served. The bounce event stops saying "(carlos restart)" for every |
| +cause. It names the attempt's cause instead. |
| ### 9.5 Docs |
| -The deploy docs gain the migrations rule for `overlap`: add in one |
| -release, remove in a later one. Otherwise deploy with `--swap hold`, or set |
| -the app to `hold`. |
| +The deploy docs gain the migrations rule for `overlap`: add in one release |
| +and remove in a later one, or deploy with `--swap hold`, or set the app to |
| +`hold`. |
| ## 10. Interactions |
| -- **Stage relay (#231).** Repoints still go through root when the agent |
| - can't write `/opt`. The rollout queues a swap only after the instance's |
| - installed build is the target, so it waits for root. It never spawns a |
| - successor from a build that isn't staged. Fallback (§8.2) is the one |
| - exception: it execs an already-staged previous build before root has |
| - pointed back. |
| - **The converge leg and sweep** (`ConvergeAdopted`, `Drifted`, |
| - `ConvergeGrace`). These become the rollout's entry points instead of |
| - bouncing directly. `BounceAdopted`'s rule ("nothing to do if the child |
| - already runs the build the symlink names") becomes the rollout's |
| - dedupe. |
| -- **Host lifecycle claim** (`hostclaim`, `hostguard`, `stopevidence`). |
| - Admission belongs to the agent's replicator, which a swap keeps, so a swap |
| - is not a new admission. Stop evidence requires that nothing holds the |
| - database open and that the cgroup is empty. Both generations sit in the |
| - instance's cgroup, so an operation that needs stop evidence (move, park, |
| - maintenance's `FenceAndStop`) must first end any swap in flight, by |
| - stopping both generations, and then collect evidence as today. A swap |
| - must not start while the host's guard is held for such an operation. |
| -- **Channels and `carlos route --channel`.** `channel` on the record |
| - becomes the source the box reads. The per-route registry column stays as |
| - the box's copy. |
| -- **Bake holds** are unchanged (§5). |
| -- **The version sweep** already keeps `Version` and `PrevVersion`. A pinned |
| - build is the installed build, so it is kept too. |
| -- **Agent restarts** still stop every child. Keeping instances up across an |
| - agent restart is out of scope (§12). It is why a box roll still cycles |
| - instances. |
| + `ConvergeGrace`) stop bouncing directly. They feed the scheduler. |
| +- **`carlos restart`** (`restartRequestLeg`, `bounceExecRoutes`) and **the |
| + config bounce** feed the scheduler with `restart` and `config` attempts. |
| + Their pacing comes from §6, which gives #230 item 1 for exec instances. |
| +- **Host lifecycle claim.** Admission belongs to the agent's replicator, |
| + which a swap keeps, so a successor is not a new admission. Stop evidence |
| + is taken on the cgroup parent after the barrier has joined every |
| + generation (§7.3, §7.4). |
| +- **The version sweep:** see §8.5. |
| +- **Agent restarts** still stop every child. Keeping instances up across |
| + an agent restart is out of scope (§14). It is why a box roll still |
| + cycles instances. |
| ## 11. Security |
| -- The control-socket verbs add **reach, not privilege**, under the same |
| - bound `schedule.Mount` documents. A same-uid sibling that reads another |
| - host's admin token can change that instance's settings, but only within |
| - `instance_control`, and only the way that instance's own app could. |
| - Per-instance uids remain the real fix. |
| -- The console never trusts the bucket request for tenancy. It checks |
| - placement and policy when it applies the request (§9.3). See the |
| - "bucket records are not a tenancy boundary" lesson. |
| -- `pin` must name a build of the same app. The console resolves it against |
| - the app's shipped manifests, never as a path. |
| - |
| -## 12. Out of scope |
| +- The console checks only credentials it minted (§4.4). Bucket objects |
| + under box prefixes and box-minted admin tokens are never authority for a |
| + settings write. |
| +- Per-host request files are agent-written and root-read, so root |
| + validates them (§7.7). A compromised agent can only ask root to install a |
| + build of that host's own app that has already been shipped. That is no |
| + more reach than the agent's own adoption has today. |
| +- `pin` must name a shipped build of the same app. The console resolves it |
| + against the app's manifests, never as a path. |
| + |
| +## 12. Testing |
| + |
| +- **Activator:** the overlap order (the successor is healthy before the |
| + switch, no new connections reach the retiring generation, in-flight |
| + requests finish), abandonment of a failed successor, the canary settle, |
| + `hold`, and the barrier against every terminal stop. Use the spawn-count |
| + assertion from #231's test trap, because `Owns` stays true while an entry |
| + drains. |
| +- **Edge:** per-connection `AppSocket`, the per-socket idle pool, a dial |
| + that races a switch, the cache-generation guard, and the version header |
| + from the serving generation. |
| +- **Every `rt.Socket` reader** assigned in §7.1 has a test that dials the |
| + current generation. |
| +- **Scheduler:** per-instance desired builds across cohorts, canary gating, |
| + `swap_concurrency`, halting on the first failure, repoint-back to the |
| + good build, supersession and re-checks, attempt-identity dedupe, the |
| + restart and config failure paths, and config acknowledgement only after |
| + a switch. |
| +- **Resolver:** a table test of §5 over pin, manual, failure, major, bake |
| + holds and untagged heads. The test asserts that the root pass and the |
| + agent resolve identically from the same inputs. |
| +- **Root requests:** an exact repoint, a refused repoint when the expected |
| + build doesn't match, validation refusals, an old root binary ignoring |
| + the directory, and `SetVersions` conditions. |
| +- **Containment:** per-generation children, the parent-limit raise and |
| + restore with `limitsOK`, retiring one generation without touching the |
| + other, and parent stop evidence. |
| +- **Nested:** the generation template's ID split, the shared slice and data |
| + mounts, and per-generation executable mounts. |
| +- **Fallback:** the first-wake budget, waiting requests served by the good |
| + build, an exit within the settle, and an instance with no good build. |
| +- **Concurrency:** an invariant simulation of attempts against wakes, |
| + stops, hibernation, lease loss, `Shutdown`, claim operations and root |
| + repoints, at high seed counts, with mutation checks. The invariants: |
| + never two current generations; never a lease released while any |
| + generation runs; never a generation running a build its attempt didn't |
| + choose; never a release-cause failure leaving an instance without a good |
| + build it can boot. |
| +- **Flagship probe:** on `bab/adopt-probe` under continuous load, deploys |
| + in `overlap` show zero failed requests and no request held longer than |
| + a normal response. |
| + |
| +## 13. Named-unit routes: a follow-up |
| + |
| +Pacing named-unit routes (#230 items 1–2) needs two things this spec |
| +doesn't build. First, root performs their restarts today, during adoption, |
| +so the paced attempt has to own a privileged restart. Second, the health |
| +check has to work over TCP as well as Unix sockets: `WaitHealthyVersion` is |
| +socket-only and expects `/api/version` (`internal/release/versions.go`), |
| +while address-backed routes have no socket. The scheduler (§6) is built to |
| +take a third kind of attempt for them, and #230 stays open for that spec. |
| +#230 item 3 (`-unit` at provisioning) is separate again. |
| + |
| +## 14. Out of scope |
| - A fleet-wide canary (one box first). |
| - Keeping instances up across an agent restart or box roll. |
| -- A configurable health path, or a settle on non-canary swaps. |
| -- #230 item 3 (`-unit` at provisioning). |
| -- The level-triggered stage marker (`PathExists`) noted on #231. |
| - |
| -## 13. Testing |
| - |
| -- **Activator.** Overlap ordering: the successor is healthy before the |
| - switch, there are no new connections to the old generation after it, and |
| - in-flight requests on the old generation finish. A failed successor |
| - leaves the old generation serving. The canary settle catches a |
| - boot-then-exit. `hold` holds requests and serves them from the new |
| - generation. Use the spawn-count assertion from #231's test trap: `Owns` |
| - stays true while an entry drains. |
| -- **Edge.** Per-connection socket lookup; the idle pool is keyed per |
| - socket; a dial racing a switch lands on the new socket. |
| -- **Rollout.** Canary gating, `swap_concurrency`, halt on first failure, |
| - repoint-back of the unswapped, a superseding target, the failed-build |
| - record surviving a restart, and its clear conditions. |
| -- **Target resolution.** A table test of §5 over the four settings, bake |
| - holds and untagged heads. |
| -- **Fallback.** A first wake onto a failing build serves the waiting |
| - request from `PrevVersion` and records the failure. |
| -- **Policy.** Every §4.4 refusal, from the console, the CLI and the control |
| - socket. |
| -- **Concurrency.** A simulation of swaps against wakes, stops, lease loss, |
| - agent shutdown and stop-evidence operations, with mutation checks. This |
| - follows the invariant-simulation practice; example tests alone have |
| - missed interleavings here before. |
| -- **Flagship probe.** On `bab/adopt-probe`, deploys under continuous load |
| - must show zero failed requests and no request held longer than one normal |
| - response time in `overlap`. |
| - |
| -## 14. Rollout of this feature |
| - |
| -Box-side changes land dormant behind a box flag (`CARLOS_SWAP=overlap`), |
| -defaulting to today's stop-then-wake, and are proven on dev and then on |
| -`bab/adopt-probe`. The console's settings and policy can ship before the |
| -flag flips, because target resolution with every setting at its default |
| -gives today's result. When the flag turns on, every app gets `overlap` |
| -unless it has set `hold`. The changelog and the deploy docs tell |
| -implementers about the migrations rule before then. |
| +- A configurable health path, or a settle on swaps after the canary. |
| +- Instance-only console principals (§4.4). |
| +- The level-triggered stage marker noted on #231. §7.7's request files |
| + don't depend on it. |
| + |
| +## 15. Rev 1 findings and their answers |
| + |
| +| # | Finding | Answer | |
| +|---|---------|--------| |
| +| 1 | A status prefix does not authenticate a box | §4.4, §9.3, §11: in-app writes go through the console API with an instance-bound credential (#155 step 6); the control socket is read-only | |
| +| 2 | Claim containment refuses a second generation | §7.3: per-generation child cgroups, parent limits, per-generation reservation | |
| +| 3 | No cancel-and-join contract | §7.4: the operation set, the barrier, re-checks under the guard, `Shutdown` covers waking entries | |
| +| 4 | Root repoints to the channel head | §5's single resolver; §7.7's exact per-host requests | |
| +| 5 | The successor execs the mutable symlink | §7.5 step 3: the immutable staged path, for flat and nested | |
| +| 6 | `PrevVersion` isn't last-known-good | §8.5: `GoodVersion`, sweep pins, conditional writes | |
| +| 7 | `.n` changes the nested instance ID | §7.6: a generation template that splits host ID from generation | |
| +| 8 | Scheduled work dials the old socket | §7.1: `AppSocket` for every app-socket consumer | |
| +| 9 | Restart and config can't share release dedupe and rollback | §6.3 attempt identity; §8.1 cause-specific failure; config acknowledged after the switch | |
| +| 10 | One target per app | §6.1: per-instance desired builds and cohorts | |
| +| 11 | The first-wake budget doesn't fit `wakeWaitBound` | §8.2: a 55 s first-wake budget; §6.4 settle rule | |
| +| 12 | Redeploying the same build is a no-op | §8.4: `carlos deploy --retry`, and host-scoped clearing | |
| +| 13 | Named-unit health check | §13: a follow-up spec | |
| +| 14 | Old responses refill the cache after the purge | §7.2: a per-host cache generation | |
| +| 15 | The version header reports the install | §7.2: the serving generation's build; §9.1 shows three builds | |
| +| 16 | Pin semantics contradict | §5 rule 1: a pin is an install request; `manual` with no install resolves as `auto` | |
| +| 17 | No instance-only principal | §4.4: none is invented; tenants act through their app | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME.md b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME.md| new file mode 100644 |
| index 000000000..2076391f0 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-ZERO-DOWNTIME.md |
| @@ -0,0 +1,86 @@ |
| +REJECT |
| + |
| +1. **HIGH — §§9.3, 11: A status prefix does not authenticate a box.** |
| + **Evidence:** [internal/edge/storedoor.go:495](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/edge/storedoor.go:495) explicitly calls the box segment an unverifiable pseudonym; `aclWrite` at line 502 permits writes throughout the caller’s fleet prefix. |
| + **Failure:** Box A writes an update request under box B’s prefix for an instance hosted by B. The proposed placement check passes, granting A authority beyond its hosted instances. |
| + **Smallest fix:** Authenticate the submitting box independently of the object key, and bind requests to the current host incarnation and placement. Define replay protection across placement changes; neither a caller-chosen prefix nor sequence number establishes authority. |
| + |
| +2. **HIGH — §§7.3, 10: Claim-mode containment cannot support the proposed generation retirement.** |
| + **Evidence:** [internal/cgroup/strict_linux.go:199](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/cgroup/strict_linux.go:199) refuses a second reservation, and line 215 requires an empty host cgroup. [internal/activator/claimwake.go:311](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/claimwake.go:311) stops a claimed process with `KillAndEmpty` on that containment. |
| + **Failure:** Reusing claim-mode spawn refuses the successor. Bypassing that refusal and sharing the existing containment makes retiring either generation kill both. Changing limits also conflicts with the prepared containment’s expected-limit checks. |
| + **Smallest fix:** Specify host-owned containment with separately stoppable generation children, while retaining aggregate host stop evidence. Define how overlap limit changes update the containment’s validated limits. |
| + |
| +3. **HIGH — §§7.3, 8.4, 10: The swap lacks a cancellation-and-join contract for every process it creates.** |
| + **Evidence:** [internal/activator/activator.go:2120](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:2120) waits for worker respawn completion before releasing the lease. `finishStop` closes replication at line 2363. `Shutdown` explicitly skips waking entries at line 2929. |
| + **Failure:** Hibernation, lease loss, or shutdown stops the current process while successor creation remains outstanding. Cleanup releases the lease, then the successor finishes spawning against the database. After switching, the retiring process creates the same risk if only “current” and “successor” remain tracked. The assertion that shutdown already stops every child is false for in-flight wakes. |
| + **Smallest fix:** Define one entry-owned operation barrier covering spawning, current, successor, and retiring generations. Every terminal stop must cancel and join it before checkpoint/release; unproven termination must retain ownership. Require guarded revalidation immediately before spawn and switch. |
| + |
| +4. **HIGH — §§5, 8.1–8.2, 10: The root relay cannot perform the requested fallback or pinned adoption as described.** |
| + **Evidence:** [internal/host/agent.go:1341](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/agent.go:1341) writes only a timestamp to the stage marker. The root pass calls `release.Pass` at line 3820; [internal/release/adopt.go:942](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:942) takes its destination from the channel pointer. |
| + **Failure:** B fails, the agent requests a repoint to A, and root reads the unchanged channel head B. Root reinstalls B. The same independent path can override manual settings, pins, and failed-build holds. |
| + **Smallest fix:** Specify a durable, validated per-host adoption request containing the exact destination and expected prior state. Make every adoption path, including hourly root adoption and materialization, honor the same settings and failure state. |
| + |
| +5. **HIGH — §§7.3, 10: Waiting for the target symlink does not bind the successor to that target.** |
| + **Evidence:** [internal/activator/backend_exec.go:925](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/backend_exec.go:925) documents the readlink-to-exec race; execution still uses the live symlink. The nested launch also uses that symlink at line 1085. |
| + **Failure:** A swap selects overlap build B. Root repoints to C before execution, and the successor actually boots C—possibly a release demanding `hold`—while A still serves. The health result is then attributed to B. |
| + **Smallest fix:** Bind each attempt to an immutable staged executable and manifest identity, including the nested executable mount. Revalidate the rollout revision before switching; never infer the executed build from a mutable symlink. |
| + |
| +6. **HIGH — §§6.1, 8.1–8.2, 10: `PrevVersion` is not a last-known-good build.** |
| + **Evidence:** [internal/release/adopt.go:1258](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:1258) stamps the previous symlink destination, regardless of whether it ran. [internal/host/versionsweep.go:59](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/versionsweep.go:59) pins only registry `Version` and `PrevVersion`, in addition to live links. |
| + **Failure:** An instance serves A while root adopts B and then C. Its registry becomes C/B although B never ran. C fails and fallback selects B; A can eventually be swept despite still serving. A superseded B failure can also overwrite C’s installation because `SetVersions` has no expected-version condition. |
| + **Smallest fix:** Persist a confirmed fallback build separately from installation history. Pin running, draining, candidate, and fallback builds. Condition rollback writes on the attempt’s rollout revision and expected installed build. |
| + |
| +7. **HIGH — §7.4: Adding `.n` to the existing unit instance changes the instance directory and parent slice.** |
| + **Evidence:** [internal/host/units/carlos-instance@.service:20](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/units/carlos-instance@.service:20) passes `%I` as the instance ID; lines 33, 47, and 54 derive the slice, database-directory mounts, and executable mounts from it. |
| + **Failure:** Starting `carlos-instance@<id>.<n>.service` with the same template addresses `<id>.<n>` directories and a different slice. It neither shares the intended database directory nor falls under the original instance limit and stop evidence. |
| + **Smallest fix:** Specify separate host-ID and generation addressing throughout the template, trampoline, executable mounts, launch files, observers, and root sweep. Keep shared data and the parent slice keyed solely by host ID. |
| + |
| +8. **HIGH — §§7.1–7.2: Generation sockets leave scheduled work dialing the old address.** |
| + **Evidence:** [internal/host/schedule_wiring.go:377](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/schedule_wiring.go:377) constructs scheduled delivery from `rt.Socket`. [internal/activator/backend_exec.go:771](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/backend_exec.go:771) uses the same field for `NextDue`; probe failure records no next wake in `activator.go:2219`. |
| + **Failure:** Public requests work through the new socket resolver, but scheduled POSTs dial the obsolete socket. On hibernation, the next-due probe fails and clears the instance’s wake schedule, potentially leaving it asleep indefinitely. |
| + **Smallest fix:** Require generation-aware routing for every app-socket consumer, including scheduled delivery, next-due queries, and version probes. Keep canonical registry layout paths separate from generation-specific spawn paths. |
| + |
| +9. **HIGH — §§6.4, 8.1, 10: Release deduplication and rollback cannot apply unchanged to restarts and config changes.** |
| + **Evidence:** [internal/activator/activator.go:1474](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:1474) explicitly excludes restart/config requests from version deduplication. [internal/appconfig/syncexec.go:1312](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/appconfig/syncexec.go:1312) treats successful callback return as restart completion and acknowledges it at line 1341. |
| + **Failure:** A same-build restart is discarded by the proposed dedupe. Alternatively, a config attempt fails and §8.1 repoints the healthy current release to an older `PrevVersion`, although no release change was requested. Merely enqueueing the callback also produces a premature config acknowledgement. |
| + **Smallest fix:** Include cause, restart request ID, and config revision in attempt identity. Define cause-specific failure handling and acknowledge config activation only after the corresponding successful switch. |
| + |
| +10. **HIGH — §§5–6, 8.1: One target build per app cannot represent the permitted instance settings.** |
| + **Evidence:** [internal/provision/provision.go:44](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/provision/provision.go:44) already stores channel per instance. [internal/host/restart.go:100](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/host/restart.go:100) identifies apps by account/app pairs. |
| + **Failure:** Two instances of one app follow different channels or pins. Resolving B for one and C for the other repeatedly replaces the single app rollout, or moves an instance to another instance’s target. A “newer target” on one channel can also clear another channel’s failed-build hold. |
| + **Smallest fix:** Specify an account/app scheduler containing per-instance desired targets and separately identified canary cohorts. Define failure and supersession scope without collapsing these into one app-wide build. |
| + |
| +11. **MEDIUM — §§6.3, 8.2: First-wake fallback cannot meet the stated waiting-request behavior with existing deadlines.** |
| + **Evidence:** [internal/activator/activator.go:725](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/activator.go:725) starts one `wakeWaitBound` timer covering the entire wait. [internal/activator/backend_exec.go:34](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/activator/backend_exec.go:34) gives the candidate health check 30 seconds. |
| + **Failure:** B consumes its health timeout; the waiting request expires before stopping B and booting A. Also, §6.3 calls first wakes canaries but specifies failure only at health checking, leaving the 10-second settle undefined for this path. |
| + **Smallest fix:** Define a total first-wake budget with time reserved for cleanup and fallback, plus explicit settle behavior and the point at which waiting requests are released. |
| + |
| +12. **MEDIUM — §8.1: Redeploying the same build does not produce the event needed to clear its failure record.** |
| + **Evidence:** [internal/release/release.go:910](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/release.go:910) makes promotion to an already-current version a no-op, returning without rewriting the pointer. |
| + **Failure:** B remains the channel head after a box rejects it. Redeploying B leaves its promotion identity unchanged, so the box cannot distinguish the promised retry from an ordinary reread. Separately, clearing an app-wide failure through one host’s update or scoped restart permits unrelated hosts to retry B. |
| + **Smallest fix:** Add an explicit retry-attempt identifier and define its scope. A host-scoped action must authorize that host’s retry; an app-wide retry needs an app-wide request. |
| + |
| +13. **MEDIUM — §7.6: The named-unit health check does not support the route shape it must verify.** |
| + **Evidence:** [internal/release/versions.go:319](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/versions.go:319) implements `WaitHealthyVersion` solely over a Unix socket and requires `/api/version` JSON. [internal/provision/provision.go:69](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/provision/provision.go:69) states that address-backed routes derive no socket. |
| + **Failure:** A healthy TCP-backed named unit restarts successfully but its canary always fails the proposed verifier. Root adoption also currently performs its own named-unit restart before the agent can pace it. |
| + **Smallest fix:** Specify transport-aware health/version verification and the supported application contract. Route privileged restart execution through the paced attempt, rather than leaving root adoption as an independent restart owner. |
| + |
| +14. **MEDIUM — §7.3 step 4: Purging at switch does not prevent old responses from repopulating the cache.** |
| + **Evidence:** [internal/edge/edge.go:1682](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/edge/edge.go:1682) caches after proxy completion. [internal/edge/cache.go:464](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/edge/cache.go:464) only removes entries currently present. |
| + **Failure:** An A request starts before the switch, B becomes current and purges the cache, then A’s draining response completes and is cached. Later requests receive A’s content throughout its cache lifetime. |
| + **Smallest fix:** Capture a cache generation for each request and reject insertion if the host’s generation changed before completion. |
| + |
| +15. **MEDIUM — §§3, 7.3, 9.1: The version header reports installation rather than the generation serving the response.** |
| + **Evidence:** [internal/edge/edge.go:760](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/edge/edge.go:760) builds the header wrapper from registry `Version`; root stamps that field during adoption at `internal/release/adopt.go:1258`. |
| + **Failure:** Root installs B while A still serves during queueing, health checking, or a failed canary. Successful responses claim B, including fallback responses before root repoints. Existing consumers of `X-Carlos-Version` obtain false deployment evidence. |
| + **Smallest fix:** Stamp activator-managed responses from the generation selected for that request. Keep installed, desired, and serving versions distinct in status and deploy-watch completion. |
| + |
| +16. **MEDIUM — §§4.1, 5: Pin installation semantics contradict themselves.** |
| + **Evidence:** [internal/release/adopt.go:961](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/release/adopt.go:961) skips only when installed equals destination; otherwise adoption stages and moves toward the destination. |
| + **Failure:** An instance on A sets `pin=B`. Rule 1 resolves B and the spec requires staging it, but the following paragraph says rule 1 never moves an instance. Installing by hand is then refused because the pin exists. A planner must choose between violating “manual installs nothing automatically” and making a different-build pin impossible to install. |
| + **Smallest fix:** State whether setting a pin is itself an explicit installation request, and define its interaction with manual mode and bake holds. Specify initialization when a manual instance has no installed build. |
| + |
| +17. **MEDIUM — §§4.4, 9.2: The instance-only console principal has no defined authorization model.** |
| + **Evidence:** [internal/console/apiinstances.go:81](/home/paulca/amadan.net/carlos/platform-design-zero-downtime-deploys/internal/console/apiinstances.go:81) delegates instance authorization to `apiAccountApp`; it does not establish the proposed “access to this instance but not its app” relationship. |
| + **Failure:** Reusing the existing gate denies the promised instance-only users. Relaxing it without defining host-scoped grants risks granting them access to sibling instances or app policy. |
| + **Smallest fix:** Define how instance access is granted, stored, revoked, and checked, and map human and service principals to each permitted operation. Keep app-policy authorization separate from instance-control authorization. |
| \ No newline at end of file |