amadan
zdd-6 final fix wave 1: W7 roll notes
The unsigned host-retry log is on the owed list as a flag-on blocker (Astra plan 6 R1 #5, plan Q4). Gates F and G repeat Gate E's owed-list guard; the E and F approval lines name the probe's app actions; the deploy-watch bullet states the new fallback rule and tells a refusing box from a declined one; the Q3 note and the probe's control caution follow W6; the claims-check line is impersonal; moved line anchors re-checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+68
−39
docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md+68 −39
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md| index 595ccf46b..d2caa4fa3 100644 |
| --- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| +++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| @@ -80,6 +80,16 @@ dormant and `CARLOS_SWAP` must stay unset until everything below is closed. |
| `bab/adopt-probe` under load with zero failed requests, then the fleet. |
| (This file.) |
| +### Added by plan 6's final review (not in #272) |
| + |
| +This item comes from Astra's final review of this branch, round 1, not from |
| +the discussion. The controller parked it as a flag-on blocker, so it belongs |
| +on #272 too. |
| + |
| +- Host retries ride the unsigned restart-scope log; a box that can write the |
| + bucket can clear another box's failure hold (Astra plan 6 R1 #5, plan Q4); |
| + sign retry entries or accept the risk in writing. |
| + |
| ### Reply: four claims-on gaps |
| Plan 5 was merged dormant as `d905a67`, after merging main d6f66ef3 (#268 |
| @@ -98,11 +108,11 @@ claims on, or close all four:** |
| 4. Successful claimed wakes cannot record their good build or serving |
| version. |
| -I did not find a claims check in `swapArming` |
| -(`internal/host/rollout.go:145-158`), which is where the arming decision is |
| -made. I looked only there, so this is not proof that nothing declines |
| -elsewhere. Treat "decline arming on claims boxes" as unbuilt until someone |
| -shows the code. |
| +No claims check was found in `swapArming` |
| +(`internal/host/rollout.go:145-156`), which is where the arming decision is |
| +made. Only that function was read, so this is not proof that nothing |
| +declines elsewhere. Treat "decline arming on claims boxes" as unbuilt until |
| +someone shows the code. |
| ### Reply: wiring caution (settings plan 2, merged as `70f8119`) |
| @@ -142,47 +152,59 @@ review, and only Paul closes the discussion's items. |
| `internal/console/apisidecars.go:121-130` and `sidecars_cli.go:368-378`. |
| - **The swap-policy door is person-only.** `PUT |
| /api/cli/apps/{account}/{app}/swap` is registered with the zero role set |
| - (`internal/console/authcli.go:858`, reasoning at |
| + (`internal/console/authcli.go:863`, reasoning at |
| `internal/console/apiswap.go:1-8`). A signed-in member can run |
| `carlos apps set`. A CI token or any credential the app itself holds |
| cannot. If CI needs to set the policy, that is a one-line role change and |
| Paul's call. |
| -- **The first non-default swap policy creates `config.json` and causes one |
| - config delivery (plan Q3).** Setting the defaults (`overlap`, 1) on an app |
| - with no `config.json` writes nothing (`internal/appconfig/swap.go:47-56`). |
| - The first `hold` or concurrency above 1 creates the file, and boxes then |
| - see a config change once. After that, config delivery restarts an instance |
| - only when the rendered bytes change. In the probe this means the |
| - `--swap hold` control may restart `adopt-probe` once, a few seconds after |
| - the command and before the `hold` load phase has had its 20 s head start. |
| - Do not read that restart as part of the control's result. Q3 stays open with |
| - Paul. |
| +- **The first non-default swap policy creates `config.json` but restarts |
| + nothing (plan Q3, Astra plan 6 R1 #1).** Setting the defaults (`overlap`, |
| + 1) on an app with no `config.json` writes nothing |
| + (`internal/appconfig/swap.go:47-57`). The first `hold` or concurrency above |
| + 1 creates the file with no vars in it. Config sync leaves an instance's |
| + config file absent when its bundle renders nothing and it has no instance |
| + overlay (`internal/appconfig/syncexec.go:1397`), so no instance restarts, |
| + no config revision reaches the release scheduler and no config ack is |
| + written. That is the state every never-configured app's instances are in. |
| + An instance that already has a config file converges on its bytes as |
| + before. This answers Q3's restart concern; closing Q3 is Paul's call. |
| - **A host retry rides the unsigned restart-scope log (plan Q4).** |
| `carlos instances retry` asks the box to retry through the restart-scope |
| log. That log is not signed, and the box's `RetryHost` |
| (`internal/rollout/manager.go:501-515`) trusts any request that names a host |
| on that box. Only the app's record and the rollout report gate it, at the |
| - console, not at the box. Q4 stays open with Paul. |
| + console, not at the box. It is on the owed list above as a flag-on |
| + blocker. |
| - **`FailureBook` is not rendered anywhere.** The box publishes it |
| (`internal/status/snapshot.go:268-275`) but no `carlos` surface or console |
| page shows it. It is parked in #272. What `carlos` shows for a failed or |
| held build comes from the per-host rollout report instead. |
| -- **The deploy watch polls the instance list every 5 s.** It does not poll |
| - every 2 s. It falls back to polling the URL (today's `watchVersion`) only |
| - after a complete list with no reports for the targeted instances, or after |
| - 3 failed first lists. A truncated list never triggers the fallback, and a |
| - box that refuses or declines is named, not URL-polled (declined boxes fall |
| - back to today's URL watch). The constants are `deployWatchListEvery`, |
| - `firstListTries` and `deployWatchPollEvery` |
| - (`deploywatch.go:85-88`). A deploy against a flaky console can therefore |
| - wait up to about 15 s longer before it falls back. |
| +- **The deploy watch reads the instance list every 5 s, not every 2 s.** It |
| + falls back to polling the URL (today's `watchVersion`) only on evidence |
| + that no targeted instance is on a reporting box: the list answers with a |
| + 4xx other than 429 (a console with no list for the app), or every targeted |
| + instance is on the list, the console read every box's report (the list |
| + carries no `rollout_incomplete`), and none of them has a report. A list cut |
| + short for the whole app neither causes nor blocks the fallback. A 5xx, a |
| + 429 or a network error is retried every 5 s until the deadline, and the |
| + timeout then says the instance list could not be read. So a flag-off |
| + deploy against a console whose list is failing times out instead of |
| + URL-watching. One deadline covers the list reads, the waits between them |
| + and the fallback. A box that refuses to arm is named in the timeout, and |
| + its instances are never URL-polled. A box that declined overlap runs |
| + today's stop-then-start paths, so its instances are URL-watched, as with |
| + the flag off. An instance counted as done goes back to pending if its box |
| + later reports a different build installed. The constants are |
| + `deployWatchListEvery` and `deployWatchPollEvery` |
| + (`deploywatch.go:140-142`); the fallback rule is `listAbsent` and |
| + `legacyOnly` (`deploywatch.go:107`, `deploywatch.go:413`). |
| - **A dash under SERVING means the build is not reported, for either of two |
| reasons.** No rollout report (flag off, or an older console), or a box that |
| set the flag and declined (`Unavailable`). `servingCell` is at |
| `instances_cli.go:450-455`. After Gate E, a dev instance still showing a |
| dash means the box did not arm. Look for the box's reason code and the |
| agent log line `REFUSING CARLOS_SWAP=overlap` |
| - (`internal/host/rollout.go:151`). |
| + (`internal/host/rollout.go:150`). |
| - **The migrations rule.** The new "Two builds at once" section of the public |
| CLI page (`docs/site/cli.md:330`, marked `TODO-COPY zdd.docs.migrations`) |
| and the release notes carry it: add in one release and remove in a later |
| @@ -276,9 +298,11 @@ checked here. Both must be live before Gate G. |
| Needs Paul's approval, naming the boxes: dev box `i-05498eada6f0afd23` (the |
| dev console box, eu-west-1) and dev box `i-046c7fe43620944bc` (eu-west-1), |
| -profile `kass`. Both ids come from the brief; only the first appears in |
| +profile `kass`. Both box ids come from the brief; only the first appears in |
| another repo document (`2026-10-06-console-release-defences-plan-4-d4.md`), |
| -and the second is not checked here. **Do not start this gate while the |
| +and the second is not checked here. The approval also names the probe's app |
| +actions on dev: creating the `zdd-probe` app, flipping its swap policy to |
| +`hold` and back, and four deploys of it. **Do not start this gate while the |
| "Before Gate E: owed" list has an open item that Paul has not waived.** |
| First the index's D1 guard: measure `systemctl daemon-reload` on a dev box |
| @@ -332,7 +356,10 @@ dash means the box did not arm (see the dash note above). |
| ## Gate F: the flagship probe box |
| -Needs Paul's approval, naming the box and its region. |
| +Needs Paul's approval, naming the box and its region. The approval also |
| +names the probe's app actions: flipping `bab/adopt-probe`'s swap policy to |
| +`hold` and back, and four deploys of it. **Do not start this gate while the |
| +"Before Gate E: owed" list has an open item that Paul has not waived.** |
| Find the probe's region with: |
| @@ -362,7 +389,9 @@ APP=adopt-probe ACCOUNT=bab CONSOLE=https://console.carloku.com HOST=probe1.bab. |
| ## Gate G: the rest of the flagship |
| -Needs Paul's approval, naming each box before it is done. |
| +Needs Paul's approval, naming each box before it is done. **Do not start |
| +this gate while the "Before Gate E: owed" list has an open item that Paul |
| +has not waived.** |
| One box at a time, with Gate E's flag command. After each box, do one |
| ordinary deploy of `adopt-probe` and run `carlos instances list --app |
| @@ -476,15 +505,14 @@ slowest is clearly above the baseline. If it is not, the probe cannot see a |
| pause and the overlap result means nothing. |
| The watch prints `swapped https://<host>` (draft word `swapped`, |
| -`deploywatch.go:459`) and `box <label>: ...` (`deploywatch.go:43`). With the |
| -flag off it prints `live https://<host>` instead (`deploywatch.go:454`), so a |
| +`deploywatch.go:569`) and `box <label>: ...` (`deploywatch.go:44`). With the |
| +flag off it prints `live https://<host>` instead (`deploywatch.go:564`), so a |
| `live` line during the overlap phase means that box is not armed. |
| -One caution on the control: the first `--swap hold` on an app creates |
| -`config.json` and causes one config delivery (the Q3 note above). If the |
| -hold phase's slowest request is high because of that restart rather than the |
| -zdd-4 swap, the control is contaminated. If the numbers look odd, rerun the |
| -hold phase once with the policy already set. |
| +On the control: the first `--swap hold` on an app creates `config.json`, and |
| +on boxes running this release that restarts nothing (the Q3 note above), so |
| +the hold phase's pause is the zdd-4 swap's. If the numbers look odd anyway, |
| +rerun the hold phase once with the policy already set. |
| ### 4. Clean up |
| @@ -496,7 +524,8 @@ On dev, run `carlos apps delete zdd-probe` once Paul is done with it. |
| ## Names checked against this branch |
| -Checked in the code on `zdd-6-surfaces` (HEAD `77d77b3a` plus this file): |
| +Checked in the code on `zdd-6-surfaces` (HEAD `77d77b3a` plus this file; |
| +the anchors final fix wave 1 moved were re-checked after it): |
| the verbs and flags `carlos system update|status`, `carlos apps set|create|delete`, |
| `carlos instances list|create|enable|retry`, `carlos deploy`, `carlos ship`, |
| `carlos promote`, `carlos promote --retry`, `carlos rollback`; the flag |