amadan
spec: zero-downtime deploys — overlap swaps, update settings, paced rollout
Draft for review. Folds in #230 (per-app restart pace, first-route health step) and follows #231 (deploy pickup).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+511
−0
docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md+511 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md| new file mode 100644 |
| index 000000000..b9ce295e5 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md |
| @@ -0,0 +1,511 @@ |
| +# Zero-downtime deploys: overlap swaps, update settings, paced rollout |
| + |
| +Status: draft for review, 2026-10-07. Discussions #231 (deploy pickup) and |
| +#230 (per-app restart pace, first-route health step) feed into this. |
| + |
| +## 1. Why |
| + |
| +Paul's ruling, 2026-10-05: "When someone ships a new release, running |
| +instances should pick up the new release immediately unless they're |
| +configured not to, and should swap in the new version with zero-downtime." |
| + |
| +Pickup is fixed. A promotion now reaches an awake instance in 20–25 s on |
| +the flagship (hotfix `f3f6cf73`, #231). The swap is not fixed. Every path |
| +that moves an awake instance to a new build is stop-then-wake: the old |
| +process gets SIGTERM and up to 20 s to drain, new requests are held for up |
| +to 30 s (`wakeWaitBound`), and the new build boots on the next request. |
| +That is a pause on every deploy, and longer than a pause when the new build |
| +is slow to boot or fails. |
| + |
| +#230 adds a second problem. A pass that retargets many routes of one app |
| +on one box restarts them back to back, with no pace and no health step, so |
| +a build that boots and then dies reaches every tenant within seconds. |
| + |
| +## 2. Decisions taken in brainstorming |
| + |
| +These were settled with Paul on 2026-10-06/07 and are not reopened here. |
| + |
| +1. **Overlap is the default.** The new process boots beside the old one |
| + and must pass its health check before new requests go to it. The old one |
| + then drains. Both share the instance's SQLite database for the overlap. |
| + So the new build's boot migrations run while the old build still serves, |
| + and apps must keep migrations backward-compatible. |
| +2. **`hold` is the per-app alternative**, and a single release can ask for |
| + it too. It is stop-then-start with requests held. It is for apps or |
| + releases whose migrations can't be shared with the previous build. |
| +3. **Instances get Eleven's update model.** Auto-update on or off, a |
| + channel, a major-version hold and an exact pin. The app implementer |
| + decides the defaults and how much of that model each instance may change. |
| +4. **Instance settings can be changed from the console and CLI, and by the |
| + app itself** over its control socket, within the app's policy. The app |
| + policy can be changed only by the implementer. |
| +5. **The app policy lives in app settings.** A release can still demand |
| + `hold` (`carlos deploy --swap hold`), and that demand is stored with the |
| + release. |
| +6. **Pacing is per app, per box: one at a time by default, and the first |
| + swap is a canary.** The concurrency is an app setting. |
| +7. **Every first wake onto a new build is its own canary.** Sleeping |
| + instances move straight away. A failed first wake falls back to the |
| + previous build in `overlap` mode. |
| +8. **Mechanism: two sockets, and the edge switches.** Each spawn gets its |
| + own socket path, and the route's current socket moves after the health |
| + check. The rename trick was rejected because runtimes unlink their socket |
| + path on exit and would delete the successor's socket. Socket activation was |
| + rejected because only apps written for it would work. |
| +9. **Scope.** Exec instances, in both the flat and the nested layout, get |
| + overlap. Worker and sidecar processes keep stop-then-start. Named-unit |
| + routes can't overlap, because the unit belongs to the implementer, so they |
| + get the paced canary rollout and `hold`. Static sites already swap |
| + atomically. |
| + |
| +## 3. Terms |
| + |
| +- **Build**: one shipped release version of an app (`Manifest.Version`, a |
| + short sha). |
| +- **Installed build**: the build an instance's live symlink names, which |
| + every wake boots. The registry stamps it as `Version`, with |
| + `PrevVersion` beside it. |
| +- **Target build**: the build the update settings say the instance should |
| + be on now (§5). |
| +- **Generation**: one spawn of one instance, numbered from 1, increasing |
| + for the life of the activator entry. |
| +- **Swap**: moving an awake instance from one generation to the next. |
| +- **Rollout**: the work on one box of moving every instance of one app to |
| + a new target build. |
| + |
| +## 4. Settings and policy |
| + |
| +### 4.1 Instance settings |
| + |
| +These are stored on the instance's provision record. The console owns the |
| +record, and boxes read it through the reconcile path as they do today. |
| + |
| +| Setting | Values | Default from policy | Meaning | |
| +|-----------|----------------------------|---------------------|---------| |
| +| `updates` | `auto` / `manual` | `auto` | With `manual`, nothing is installed automatically. The installed build stays put, across sleep and wake, until someone installs. | |
| +| `channel` | an existing channel name | the app's channel | The channel the instance follows. This is the per-route channel that exists today, now also settable on the record. | |
| +| `major` | empty or a non-negative integer | empty | Follow the channel, but hold a release whose semver tag has a different major (§5). It is accepted only with the `stable` channel, because only stable promotions cut semver tags (`stableTag`, `internal/release/release.go`). Setting it on any other channel is refused, with that reason. | |
| +| `pin` | empty or a build version | empty | Stay on exactly this build. It beats `major` and `channel`. It must name a build the app has shipped. | |
| + |
| +There is no "download automatically". A box fetches a build once and every |
| +instance on it shares it, so an instance has nothing of its own to download. |
| + |
| +### 4.2 App policy |
| + |
| +These are app settings, owned by the implementer. |
| + |
| +| Setting | Values | Default | Meaning | |
| +|--------------------|--------|---------|---------| |
| +| `swap` | `overlap` / `hold` | `overlap` | How an awake instance moves to a new build. | |
| +| `swap_concurrency` | integer ≥ 1 | 1 | How many instances of this app may swap at once on one box, after the canary. | |
| +| `instance_control` | a subset of {`updates`, `channel`, `major`, `pin`} | empty | Which instance settings the instance side may change. | |
| +| instance defaults | one value per §4.1 setting | as in §4.1 | Applied to new instances. They also apply to an existing instance whose setting has never been set explicitly. | |
| + |
| +An empty `instance_control` keeps every existing app as it is today: |
| +instances change nothing until the implementer allows it. |
| + |
| +### 4.3 The release's demand |
| + |
| +`Manifest` gains `swap` (`"hold"` or empty). Like `Label` and `Notes`, it |
| +is additive to schema 1 and absent from older manifests. `carlos deploy |
| +--swap hold` sets it at ship time. The effective mode for a swap onto a |
| +build is `hold` if either the app policy or that build's manifest says |
| +`hold`, and `overlap` otherwise. Because the manifest is immutable, the |
| +demand applies on every box, to every instance that ever moves onto that |
| +build, including instances that pick it up later by hand. |
| + |
| +### 4.4 Who may change what |
| + |
| +- **App members** with write access may set any instance setting on any |
| + instance of the app, and the app policy. |
| +- **The instance side** may change only the settings named in |
| + `instance_control`. That is console users with access to the instance but |
| + not to its app, and the app itself over the control socket (§9.3). A |
| + refused change names the setting and says the app keeps it for itself. |
| +- Installing by hand (§9) follows the same rule as `updates`. An instance |
| + side that may not change `updates` may not install by hand either. |
| + |
| +## 5. Target resolution |
| + |
| +For each instance, given the app's channel heads and the instance's |
| +settings, apply these rules in order: |
| + |
| +1. `pin` set: the target is the pinned build. |
| +2. `updates = manual`: the target is the installed build (no move). |
| +3. Otherwise, take the head of the instance's `channel`. If `major` is set |
| + and the head's semver tag is absent, or has a different major, the |
| + target is the installed build and the instance is *held at major N*. |
| +4. Otherwise the target is the channel head. |
| + |
| +Bake holds (`internal/release/bake.go`) apply to rule 4 exactly as today. |
| +`FastTrack` waives them for one promotion. Rules 1 and 2 never move an |
| +instance on their own. Installing by hand on a pinned instance is refused, |
| +with the pin named; clearing the pin comes first. Otherwise, installing by |
| +hand sets the installed build to the channel head, as rule 4 would if |
| +`updates` were `auto`, and ignores `major`. That is the point of the major |
| +hold: a person decides. |
| + |
| +A pin to a build that is not staged on the box is staged as adoption |
| +stages anything else. |
| + |
| +## 6. Rollout and pacing |
| + |
| +### 6.1 Shape |
| + |
| +When a release reaches a box, through a serial bump and `fullPass`, the |
| +agent resolves a target for every route of the app (§5) and starts or |
| +updates the app's **rollout** on that box. Each app's rollout runs in its |
| +own goroutine, off the agent's tick. That follows the lesson of the |
| +converge leg (`fix-converge-own-loop`): a slow rollout never delays the |
| +tick, and never delays another app's rollout. |
| + |
| +A rollout on one box has a target build, a queue of awake instances still |
| +to swap, and a state: running, done or halted. A newer target for the same |
| +app replaces the queue. Instances already swapped to the superseded build |
| +are simply re-queued for the new one. |
| + |
| +### 6.2 Order |
| + |
| +1. **Sleeping instances move at once.** Their installed build is |
| + repointed, through the root stage relay when the agent can't write |
| + `/opt`, and they boot it on their next wake. There is no process, so |
| + there is nothing to pace. |
| +2. **Awake instances swap in queue order.** The first swap of the rollout is |
| + the **canary**. It must pass the health check and then stay up for a |
| + **10 s settle**, which catches the build that boots and then dies. No |
| + other awake instance of the app on that box swaps until the canary |
| + settles. After that, up to `swap_concurrency` swaps run at once. A |
| + non-canary swap needs only the health check. |
| +3. A route the agent can't swap because it is waking or stopping keeps its |
| + queue place. It is retried when it reaches a stable state. A swap that |
| + lands during a wake is deferred the way `bounceAfterWake` is today. |
| + |
| +### 6.3 First-wake canary |
| + |
| +Every first wake of an instance onto a build it hasn't run before counts as |
| +a canary for that build on that box. In `overlap` mode, a first wake that |
| +fails its health check falls back (§8.2). In `hold` mode it does not. |
| + |
| +### 6.4 Restarts and config changes use the rollout |
| + |
| +`carlos restart`'s request leg (`restartRequestLeg`/`bounceExecRoutes`, |
| +`internal/host/restart.go`) and the config-change bounce |
| +(`appconfig/syncexec.go`) stop bouncing every awake instance in one pass. |
| +They queue a rollout with the instance's own installed build as the |
| +target, so swap, pacing and canary apply to them unchanged. That is #230 |
| +item 1, for releases and restarts alike. |
| + |
| +### 6.5 Across boxes |
| + |
| +Each box runs its own rollout and its own canary. A halt on one box does |
| +not stop the others, but the deploy watch reports it (§9.1). A fleet-wide |
| +canary, with one box first, is out of scope (§12). |
| + |
| +## 7. The swap |
| + |
| +### 7.1 Generations and sockets |
| + |
| +The activator entry gains a generation counter. Each spawn takes the next |
| +generation `n`, and its paths carry it: |
| + |
| +| Today | With generations | |
| +|-------|------------------| |
| +| `/run/carlos/<host>.sock` | `/run/carlos/<host>.<n>.sock` | |
| +| `/run/carlos/instances/<id>/app.sock` | `/run/carlos/instances/<id>/app.<n>.sock` | |
| +| `<socket>.control` | `<generation socket>.control` | |
| + |
| +The app still receives `--socket <path>` and `CARLOS_CONTROL_SOCKET`. |
| +Nothing changes for the app. `prepare()` deletes only the socket path of the |
| +generation it is preparing, never a sibling generation's. |
| + |
| +### 7.2 The edge looks up the socket per connection |
| + |
| +Today the route's proxy is built against one fixed socket path |
| +(`unixProxy(t.Socket)`, `internal/edge/edge.go`). It becomes a dial |
| +function that asks the activator for the host's **current socket** each |
| +time it opens a connection. That is a map read under the entry's lock. The |
| +transport keys its idle pool per socket path, so a pooled keep-alive to |
| +generation `n` is never reused for `n+1`. When a generation stops being |
| +current, its idle pooled connections are closed. `dialUnixWithRetry`'s |
| +ENOENT/ECONNREFUSED retry stays: a dial that raced a switch retries |
| +against the new current socket. Routes that aren't activator-managed keep |
| +their fixed socket. |
| + |
| +### 7.3 An overlap swap, step by step |
| + |
| +1. The entry gains a **successor**. It keeps its current process, lease and |
| + replicator. The replicator and the lease belong to the agent, not to |
| + either process, as `respawnWorkerLocked` already relies on. Both |
| + processes open the same database file. Litestream requires WAL mode, and |
| + in WAL mode SQLite coordinates processes on one machine through its |
| + locks and `-shm`. |
| +2. During the overlap the instance's cgroup memory limit is raised to |
| + **twice** its tier's `memory.max`. Both generations run in the same |
| + `carlos-exec@<host>` cgroup, or the same nested instance cgroup, so |
| + cgroup-wide accounting and stop evidence cover both. |
| +3. The successor spawns from the target build, with the generation-`n+1` |
| + paths, and must pass the existing health check (`GET /` below 500 within |
| + `spawnHealthTimeout`, 30 s). A canary must then also survive the 10 s |
| + settle. |
| +4. **Switch.** Under the entry lock, the current socket becomes the |
| + successor's, the entry's `spawnVersion` becomes the target build, and |
| + the host's edge cache is purged. The purge moves from spawn time to |
| + switch time. New connections now reach the successor. |
| +5. The old generation gets SIGTERM. Its in-flight requests finish under its |
| + own graceful shutdown, with `actStopTimeout` (20 s) before SIGKILL, as |
| + today. Long-lived connections (websockets, SSE) close when it exits. Its |
| + control socket and exit records are removed. |
| +6. When the old generation has exited, the memory limit goes back to the |
| + tier's `memory.max` and the swap is done. The event is in §9.4. |
| +7. A legacy co-spawned sidecar (`<host>-sidecar.env`) is then restarted by |
| + stop-then-start onto the new build. |
| + |
| +There is one successor at a time per entry. A swap requested while one is |
| +in flight is queued behind it. |
| + |
| +### 7.4 The nested layout |
| + |
| +The root control service's start verb gains a generation argument and |
| +starts `carlos-instance@<id>.<n>.service`, a second instance of the same |
| +template beside the first. The launch file, log sockets and termination |
| +proof (unit state plus an empty cgroup) are per generation. Both units are |
| +placed in the instance's cgroup subtree, so the instance-level limit and |
| +stop evidence see both. Raising and restoring the limit for the overlap |
| +goes through the same control service. |
| + |
| +### 7.5 A `hold` swap |
| + |
| +This is stop-then-start on the existing path. The old generation stops, |
| +with requests held by `begin()` up to `wakeWaitBound`. The new generation |
| +then boots. It no longer has to wait for the next request: the swap boots |
| +it straight away, so requests held across it are served as soon as it is |
| +healthy. |
| + |
| +### 7.6 Workers, sidecar members, named units, static sites |
| + |
| +- **Worker processes and native sidecar members** (`Kind=sidecar`) serve |
| + no requests, and two copies could do the same job twice. They keep |
| + stop-then-start, with the respawn-on-same-entry path, and are paced by |
| + the rollout like any other swap. |
| +- **Named-unit routes** (`-unit`) get the rollout's pacing and canary, with |
| + `try-restart` as the swap and `WaitHealthyVersion` as the health check |
| + (#230 item 2). There is no overlap and no fallback, as in `hold`. #230 |
| + item 3, `-unit` at provisioning, is not part of this spec. |
| +- **Static sites** are unchanged. Their swap is already atomic |
| + (`internal/edge/static.go`). |
| + |
| +## 8. Failure and rollback |
| + |
| +### 8.1 A failed overlap swap |
| + |
| +A failed swap is one where the successor fails its health check, exits |
| +before the switch, or (as canary) dies within the settle. The successor is |
| +stopped, the old generation keeps serving, the memory limit is restored, |
| +and no request has seen anything. Then: |
| + |
| +- **The rollout halts for that app on that box, at the first failure**, |
| + whether the canary or a later swap. |
| +- Instances already swapped stay on the new build. The failed instance, |
| + and every awake and sleeping instance the rollout moved but hasn't |
| + successfully run, are repointed to their `PrevVersion` (through the stage |
| + relay when needed). |
| +- The box records **failed build B for app A**, in the agent's data dir |
| + beside the stage marker, so it survives an agent restart. Target |
| + resolution treats B as held for A on that box until the record is |
| + cleared. The record clears on a newer target build, `carlos restart`, |
| + installing by hand, or a redeploy of B, meaning a promotion that names B |
| + again. |
| + |
| +### 8.2 A failed first wake (`overlap` mode) |
| + |
| +When a first wake onto build B fails its health check, the entry spawns the |
| +instance's `PrevVersion` directly from its versions directory. The agent |
| +can exec staged builds; it just can't write `/opt`. The request waiting on |
| +the wake is served by that build. The agent then asks root to point the |
| +live symlink back, and records B as failed for the app (§8.1), which halts |
| +the rollout. The version sweep already treats `PrevVersion` as a pin |
| +(`internal/host/versionsweep.go`), so the previous build is still on the |
| +box. An instance with no `PrevVersion` (its first build ever) has nothing to |
| +fall back to and fails as a wake fails today. |
| + |
| +### 8.3 A failed `hold` swap |
| + |
| +Nothing falls back automatically, because a `hold` build may have migrated |
| +the database past what the previous build can read. The instance is down, |
| +the rollout halts, and the deploy watch says so, naming `carlos rollback` |
| +as the way out. |
| + |
| +### 8.4 Other cases |
| + |
| +- **The agent restarts during a swap.** `Shutdown` stops every child, both |
| + generations included, as today. The next wake boots the installed build. |
| + The failed-build record survives on disk. |
| +- **The lease is lost during a swap.** Both generations stop, through the |
| + existing lost-lease path. |
| +- **The new generation crashes after the switch and the settle.** That is an |
| + ordinary crash, handled by the existing exit and respawn path. It is not |
| + a failed swap, and nothing rolls back. |
| +- **A successor that never gets healthy** within `spawnHealthTimeout` (30 s) |
| + is a failed swap. |
| + |
| +## 9. Surfaces |
| + |
| +All member-facing wording below goes through copy review before it ships. |
| +The strings here are placeholders for meaning, not proposed copy. |
| + |
| +### 9.1 CLI |
| + |
| +- `carlos apps set <app> --swap overlap|hold --swap-concurrency N |
| + --instance-control updates,channel,major,pin --default-updates auto|manual |
| + --default-channel C --default-major N`. |
| +- `carlos instances set <host> --updates auto|manual --channel C --major N |
| + --pin V`, with an empty value to clear. Limited by §4.4. |
| +- `carlos instances update <host>`: install now (§5). |
| +- `carlos deploy --swap hold`: stores the demand in the manifest (§4.3). |
| +- `carlos deploy`'s watch follows the rollout and reports per box: how many |
| + instances have swapped, which ones failed and why, and which stayed on |
| + the previous build. It finishes when every box is done or halted, or when |
| + its existing deadline passes. |
| +- `carlos instances list` shows each instance's installed build and, when |
| + it is behind its channel, why: manual, pinned, held at major N, failed |
| + build, or rolling out. |
| + |
| +### 9.2 Console |
| + |
| +- App settings gain an Updates section with the §4.2 policy. |
| +- Each instance page gains an Updates section with the §4.1 settings. A |
| + setting the app keeps for itself is shown read-only, labelled as set by |
| + the app. "Update now" appears when the instance is behind. |
| + |
| +### 9.3 Control socket |
| + |
| +These are three verbs mounted beside `schedule.Mount`'s, with the same |
| +bearer (`CARLOS_ADMIN_TOKEN`) and the same fixed-host rule: the host comes |
| +from the socket, never from the request. |
| + |
| +- `GET /updates` returns the instance's settings, which of them it may |
| + change, its installed build, its target build and, when they differ, why. |
| +- `PUT /updates` changes settings named in `instance_control`. Anything |
| + else is refused with 403 and the setting's name. |
| +- `POST /updates/install` installs now, if `updates` is in |
| + `instance_control`. |
| + |
| +**How a change reaches the console.** The agent doesn't own the record, so |
| +it writes the request as an object under its own box's status prefix in |
| +the deployment bucket. That is the channel `statusLoop` already writes |
| +through. The console applies a request only when all of these hold: |
| + |
| +- the instance is placed on the box whose prefix carries it (the box can |
| + vouch only for what it hosts); |
| +- the change is allowed by the app policy at the time it is applied; |
| +- the request is newer than the last one it applied for that instance. |
| + |
| +`PUT` and `POST` answer 202 with the pending change. `GET /updates` shows |
| +it as pending until the record carries it. The plan must confirm how |
| +quickly the console reads box status. If no loop reads it within a |
| +minute, the plan adds one for this prefix. It must not add a box→console |
| +HTTP path. |
| + |
| +### 9.4 Events |
| + |
| +The new events are swapping, swapped (with the old generation's drain |
| +time), swap failed (still serving the previous build), and rollout halted |
| +(app, box, build, reason). The bounce event stops saying "(carlos restart)" |
| +for everything. It names its cause: a new release, a config change, or |
| +`carlos restart`. |
| + |
| +### 9.5 Docs |
| + |
| +The deploy docs gain the migrations rule for `overlap`: add in one |
| +release, remove in a later one. Otherwise deploy with `--swap hold`, or set |
| +the app to `hold`. |
| + |
| +## 10. Interactions |
| + |
| +- **Stage relay (#231).** Repoints still go through root when the agent |
| + can't write `/opt`. The rollout queues a swap only after the instance's |
| + installed build is the target, so it waits for root. It never spawns a |
| + successor from a build that isn't staged. Fallback (§8.2) is the one |
| + exception: it execs an already-staged previous build before root has |
| + pointed back. |
| +- **The converge leg and sweep** (`ConvergeAdopted`, `Drifted`, |
| + `ConvergeGrace`). These become the rollout's entry points instead of |
| + bouncing directly. `BounceAdopted`'s rule ("nothing to do if the child |
| + already runs the build the symlink names") becomes the rollout's |
| + dedupe. |
| +- **Host lifecycle claim** (`hostclaim`, `hostguard`, `stopevidence`). |
| + Admission belongs to the agent's replicator, which a swap keeps, so a swap |
| + is not a new admission. Stop evidence requires that nothing holds the |
| + database open and that the cgroup is empty. Both generations sit in the |
| + instance's cgroup, so an operation that needs stop evidence (move, park, |
| + maintenance's `FenceAndStop`) must first end any swap in flight, by |
| + stopping both generations, and then collect evidence as today. A swap |
| + must not start while the host's guard is held for such an operation. |
| +- **Channels and `carlos route --channel`.** `channel` on the record |
| + becomes the source the box reads. The per-route registry column stays as |
| + the box's copy. |
| +- **Bake holds** are unchanged (§5). |
| +- **The version sweep** already keeps `Version` and `PrevVersion`. A pinned |
| + build is the installed build, so it is kept too. |
| +- **Agent restarts** still stop every child. Keeping instances up across an |
| + agent restart is out of scope (§12). It is why a box roll still cycles |
| + instances. |
| + |
| +## 11. Security |
| + |
| +- The control-socket verbs add **reach, not privilege**, under the same |
| + bound `schedule.Mount` documents. A same-uid sibling that reads another |
| + host's admin token can change that instance's settings, but only within |
| + `instance_control`, and only the way that instance's own app could. |
| + Per-instance uids remain the real fix. |
| +- The console never trusts the bucket request for tenancy. It checks |
| + placement and policy when it applies the request (§9.3). See the |
| + "bucket records are not a tenancy boundary" lesson. |
| +- `pin` must name a build of the same app. The console resolves it against |
| + the app's shipped manifests, never as a path. |
| + |
| +## 12. Out of scope |
| + |
| +- A fleet-wide canary (one box first). |
| +- Keeping instances up across an agent restart or box roll. |
| +- A configurable health path, or a settle on non-canary swaps. |
| +- #230 item 3 (`-unit` at provisioning). |
| +- The level-triggered stage marker (`PathExists`) noted on #231. |
| + |
| +## 13. Testing |
| + |
| +- **Activator.** Overlap ordering: the successor is healthy before the |
| + switch, there are no new connections to the old generation after it, and |
| + in-flight requests on the old generation finish. A failed successor |
| + leaves the old generation serving. The canary settle catches a |
| + boot-then-exit. `hold` holds requests and serves them from the new |
| + generation. Use the spawn-count assertion from #231's test trap: `Owns` |
| + stays true while an entry drains. |
| +- **Edge.** Per-connection socket lookup; the idle pool is keyed per |
| + socket; a dial racing a switch lands on the new socket. |
| +- **Rollout.** Canary gating, `swap_concurrency`, halt on first failure, |
| + repoint-back of the unswapped, a superseding target, the failed-build |
| + record surviving a restart, and its clear conditions. |
| +- **Target resolution.** A table test of §5 over the four settings, bake |
| + holds and untagged heads. |
| +- **Fallback.** A first wake onto a failing build serves the waiting |
| + request from `PrevVersion` and records the failure. |
| +- **Policy.** Every §4.4 refusal, from the console, the CLI and the control |
| + socket. |
| +- **Concurrency.** A simulation of swaps against wakes, stops, lease loss, |
| + agent shutdown and stop-evidence operations, with mutation checks. This |
| + follows the invariant-simulation practice; example tests alone have |
| + missed interleavings here before. |
| +- **Flagship probe.** On `bab/adopt-probe`, deploys under continuous load |
| + must show zero failed requests and no request held longer than one normal |
| + response time in `overlap`. |
| + |
| +## 14. Rollout of this feature |
| + |
| +Box-side changes land dormant behind a box flag (`CARLOS_SWAP=overlap`), |
| +defaulting to today's stop-then-wake, and are proven on dev and then on |
| +`bab/adopt-probe`. The console's settings and policy can ship before the |
| +flag flips, because target resolution with every setting at its default |
| +gives today's result. When the flag turns on, every app gets `overlap` |
| +unless it has set `hold`. The changelog and the deploy docs tell |
| +implementers about the migrations rule before then. |