amadan
zdd-6 task 17: roll notes for the zero-downtime deploys (gates A to G, probe, owed list)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+515
−0
docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md+515 −0
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md| new file mode 100644 |
| index 000000000..595ccf46b |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| @@ -0,0 +1,515 @@ |
| +# Zero-downtime deploys: roll notes (plan 6, task 17) |
| + |
| +These are the notes for putting zero-downtime deploys into production, gate |
| +by gate. They are written, not run. Nothing in this file has been executed |
| +against a box, a bucket, a console or an AWS account, and every command |
| +below is quoted for Paul to run or approve. |
| + |
| +Spec: `2026-10-07-zero-downtime-deploys-design.md` (rev 17), section 15 for |
| +the order. The plans are `2026-10-07-zero-downtime-deploys-0-index.md` and |
| +plans 1 to 6, in `platform-design-zero-downtime-deploys`. |
| + |
| +## Ground rules |
| + |
| +- **Approval.** Every gate that touches dev or the flagship is production |
| + infrastructure (AGENTS.md, "Production infrastructure needs scoped |
| + approval"). Before running one, get Paul's explicit approval for that gate, |
| + naming the boxes. Approval for one gate is not approval for the next. |
| +- **AWS profiles.** Assumed-role profiles need the Bash sandbox off for that |
| + command. Dev is profile `kass`. The flagship is profile `keymail`. |
| +- **Order.** Boxes first, then the console's swap-policy settings, then the |
| + flag: dev, then `bab/adopt-probe` on the flagship, then the rest of the |
| + flagship. The deploy docs and release notes carry the migrations rule |
| + before the flag goes on across the fleet. Plan 5 keeps the flag off |
| + outside dev until this plan has merged. |
| +- **The flag.** `CARLOS_SWAP=overlap` in a box's `/etc/carlos/host.env`. Only |
| + the exact word `overlap` arms it. Anything else, or nothing, leaves the box |
| + on today's stop-then-start paths. |
| +- **Scratch.** The probe's scratch lives under `$TMPDIR`, never `/tmp`. |
| + |
| +## Before Gate E: owed |
| + |
| +This is the flag-on precondition. It is copied from amadan discussion #272 |
| +("Zero-downtime deploys: owed before CARLOS_SWAP can go on", opened by |
| +claude~paul@keymail.dev), as read on 2026-10-11, including its replies. |
| +Nothing on this list is closed. **The flag stays off on any box until every |
| +item is closed or Paul waives it in writing.** Gates A to D do not turn the |
| +flag on and can go ahead while this list is open. |
| + |
| +The discussion's opening text says zero-downtime deploys are merged |
| +dormant and `CARLOS_SWAP` must stay unset until everything below is closed. |
| + |
| +### Correctness, from Astra's plan 5 review (`CODEX-FINDINGS-ZDD-IMPL-5*.md`) |
| + |
| +- N5 to N7: gaps in the hold-ran marker. N7 is a disagreement over how it |
| + treats an existing failure record. |
| +- F5: failure-record durability across a failed write followed by a restart. |
| +- Boxes that declined to arm: worker restarts; refusals checked only at plan |
| + time; records read only at start. |
| +- The 15-item list in round 3, section 5. |
| + |
| +### Carried from plans 2 and 3 |
| + |
| +- With the flag off, a retained wake is proven by its leader's exit only; |
| + forked descendants are not checked (F6, pre-existing). |
| +- Leftover executable files are not swept from `/run`. |
| +- Claims-backed swaps are refused (P2-9). Main's claimed launch is closed, so |
| + a launch protocol and the strict-parent lifecycle are needed before claims |
| + boxes get swaps. |
| +- Strict hosts get no swaps until a claim path exists (P2-8). |
| + |
| +### Dev-box checks (need Paul's approval to run) |
| + |
| +- D1: `daemon-reload` cost at titogo-scale unit counts. Above about 1 s per |
| + swap means transient units instead of drop-ins. (Gate E below runs this |
| + measurement.) |
| +- The plan 2 kernel cgroup suite on arm64 with `User=carlos`. |
| +- Real-unit sidecar and control checks: the registry read under the control |
| + unit, and StopGen(kill) on a never-loaded unit. |
| +- A guard against legacy cgroup parents that still hold processes. |
| + |
| +### Surfaces (plan 6) |
| + |
| +- Copy review of all event and refusal wording. That includes the |
| + `WakePlan.Refuse` texts, "restart refused: rolling restarts are unavailable |
| + on this box", the refused-wake texts, and main's "sidecar could not be |
| + restarted" retry line, which has an em dash and now fires more often. |
| +- A member-safe wake reason for `ErrBuildNotPublished` (plan 4 R22). |
| +- `carlos sidecars ls` shows `ServingVersion`. |
| +- Roll notes: boxes first, then console policy, then dev, then |
| + `bab/adopt-probe` under load with zero failed requests, then the fleet. |
| + (This file.) |
| + |
| +### Reply: four claims-on gaps |
| + |
| +Plan 5 was merged dormant as `d905a67`, after merging main d6f66ef3 (#268 |
| +lease margin, region-HA). Astra's merge review |
| +(`CODEX-FINDINGS-ZDD-IMPL-5-MERGE.md`) said ACCEPT WITH FIXES for a dormant |
| +merge, with no flag-off defect. |
| + |
| +Main's `RoleLaunch.Root` is wired in production, so claims-on launches can |
| +execute. **Before the flag goes on, either decline arming on boxes with host |
| +claims on, or close all four:** |
| + |
| +1. Recovery selection and fallback cannot be honoured for claimed planned |
| + wakes. |
| +2. A claimed hold launch can lose its execution evidence and outcome fence. |
| +3. Retained claim processes are missing from the generations file. |
| +4. Successful claimed wakes cannot record their good build or serving |
| + version. |
| + |
| +I did not find a claims check in `swapArming` |
| +(`internal/host/rollout.go:145-158`), which is where the arming decision is |
| +made. I looked only there, so this is not proof that nothing declines |
| +elsewhere. Treat "decline arming on claims boxes" as unbuilt until someone |
| +shows the code. |
| + |
| +### Reply: wiring caution (settings plan 2, merged as `70f8119`) |
| + |
| +Held staging checks authorization inside `stageHeld`, so every entry point is |
| +covered, including main's claim-path `MaterializeHeldUnder`. |
| + |
| +Whoever wires zero-downtime plan 5 Task 9: when `movesProbeOwners.Release` |
| +is connected to `MaterializeHeldUnder`, it must pass the parsed |
| +`CARLOS_RELEASE_PUBKEY` and the agent's digest ratchet. A missing key or |
| +ratchet refuses every stage (fail closed). That is not a bug to "fix" by |
| +passing nil. |
| + |
| +### Where plan 6 stands against that list (for information) |
| + |
| +The discussion text predates plan 6 ("not started"). On this branch the |
| +`carlos sidecars ls` change is built (task 13), drafts for the refusal and |
| +wake wording exist behind `TODO-COPY` markers (task 14b), and this file is |
| +task 17. None of those closes its item: the copy items wait for Paul's |
| +review, and only Paul closes the discussion's items. |
| + |
| +## What plan 6 changed that the roll should check |
| + |
| +- **Copy is still pending.** No approved copy exists yet. Every new |
| + user-facing string is the plan's draft text with a `TODO-COPY zdd.<id>` |
| + marker beside it (Go: a `// TODO-COPY` comment; templates: |
| + `{{/* TODO-COPY ... */}}`; docs: `<!-- TODO-COPY ... -->`). The strings sit |
| + in `copy-review/` for review. Task 16's pin of the approved copy |
| + (`approved_copy_zdd_test.go`, reading `6-copy.json`) is deferred until Paul |
| + approves. **Do not turn the flag on until the approved copy is applied and |
| + pinned.** Where this file says "the approved line", it means the line after |
| + that review. Until then the draft is what prints. |
| +- **`carlos sidecars ls` changes with the flag off.** It reads |
| + `ServingVersion` and prints "unknown" for a sidecar that has never started |
| + anywhere. Plan 4 already changed what the registry `Version` means for a |
| + sidecar, so the old column would lie. Look at it on dev after Gate B and |
| + expect different output from the last release. The code is in |
| + `internal/console/apisidecars.go:121-130` and `sidecars_cli.go:368-378`. |
| +- **The swap-policy door is person-only.** `PUT |
| + /api/cli/apps/{account}/{app}/swap` is registered with the zero role set |
| + (`internal/console/authcli.go:858`, reasoning at |
| + `internal/console/apiswap.go:1-8`). A signed-in member can run |
| + `carlos apps set`. A CI token or any credential the app itself holds |
| + cannot. If CI needs to set the policy, that is a one-line role change and |
| + Paul's call. |
| +- **The first non-default swap policy creates `config.json` and causes one |
| + config delivery (plan Q3).** Setting the defaults (`overlap`, 1) on an app |
| + with no `config.json` writes nothing (`internal/appconfig/swap.go:47-56`). |
| + The first `hold` or concurrency above 1 creates the file, and boxes then |
| + see a config change once. After that, config delivery restarts an instance |
| + only when the rendered bytes change. In the probe this means the |
| + `--swap hold` control may restart `adopt-probe` once, a few seconds after |
| + the command and before the `hold` load phase has had its 20 s head start. |
| + Do not read that restart as part of the control's result. Q3 stays open with |
| + Paul. |
| +- **A host retry rides the unsigned restart-scope log (plan Q4).** |
| + `carlos instances retry` asks the box to retry through the restart-scope |
| + log. That log is not signed, and the box's `RetryHost` |
| + (`internal/rollout/manager.go:501-515`) trusts any request that names a host |
| + on that box. Only the app's record and the rollout report gate it, at the |
| + console, not at the box. Q4 stays open with Paul. |
| +- **`FailureBook` is not rendered anywhere.** The box publishes it |
| + (`internal/status/snapshot.go:268-275`) but no `carlos` surface or console |
| + page shows it. It is parked in #272. What `carlos` shows for a failed or |
| + held build comes from the per-host rollout report instead. |
| +- **The deploy watch polls the instance list every 5 s.** It does not poll |
| + every 2 s. It falls back to polling the URL (today's `watchVersion`) only |
| + after a complete list with no reports for the targeted instances, or after |
| + 3 failed first lists. A truncated list never triggers the fallback, and a |
| + box that refuses or declines is named, not URL-polled (declined boxes fall |
| + back to today's URL watch). The constants are `deployWatchListEvery`, |
| + `firstListTries` and `deployWatchPollEvery` |
| + (`deploywatch.go:85-88`). A deploy against a flaky console can therefore |
| + wait up to about 15 s longer before it falls back. |
| +- **A dash under SERVING means the build is not reported, for either of two |
| + reasons.** No rollout report (flag off, or an older console), or a box that |
| + set the flag and declined (`Unavailable`). `servingCell` is at |
| + `instances_cli.go:450-455`. After Gate E, a dev instance still showing a |
| + dash means the box did not arm. Look for the box's reason code and the |
| + agent log line `REFUSING CARLOS_SWAP=overlap` |
| + (`internal/host/rollout.go:151`). |
| +- **The migrations rule.** The new "Two builds at once" section of the public |
| + CLI page (`docs/site/cli.md:330`, marked `TODO-COPY zdd.docs.migrations`) |
| + and the release notes carry it: add in one release and remove in a later |
| + one, and keep the names of scheduled work the same across the two builds. |
| + `--swap hold` and `carlos apps set --swap hold` are the way out for a |
| + release that cannot follow it. |
| + |
| +## Gate A: release |
| + |
| +Needs Paul's approval, naming the release. This gate touches no box. |
| + |
| +Plans 1 to 6 are merged to amadan `main` and `main`'s CI is green. Cut a |
| +platform release from that commit, following README "Releasing the platform" |
| +(steps 1 to 9). Its release notes (step 5, copy-reviewed) lead with the |
| +migrations rule (spec section 9.5) and say the flag ships off. Call the |
| +release `vX.Y.Z` below. |
| + |
| +README's "Releasing the platform" section ends with a place for release |
| +specific operator notes. Add one for this release naming the flag-on |
| +preconditions above, so the person rolling it reads them. |
| + |
| +## Gate B: boxes, flag off |
| + |
| +Needs Paul's approval, naming the boxes: dev (`platform.dev.carlosframework.com`, |
| +channel `canary`) and the flagship (`console.carloku.com`, channel `stable`), |
| +which is `ie`, `us`, `uk` and `au`. |
| + |
| +```bash |
| +carlos system update --console https://platform.dev.carlosframework.com --channel canary --version vX.Y.Z --yes |
| +carlos system status --console https://platform.dev.carlosframework.com |
| +carlos system update --console https://console.carloku.com --channel stable --version vX.Y.Z --yes |
| +carlos system status --console https://console.carloku.com |
| +``` |
| + |
| +Expected: every box reports `vX.Y.Z`. Sydney is slow, so give it the full |
| +gate. With `CARLOS_SWAP` unset nothing changes: a `carlos deploy` still |
| +prints `live https://...` lines, and `carlos instances list` shows a dash |
| +under SERVING. |
| + |
| +## Gate C: consoles |
| + |
| +Needs Paul's approval, naming the consoles and the app. |
| + |
| +Roll each console after its boxes. The script is platform-infrastructure's |
| +`roll-console.sh`; it is not in this repo (not checked here). Or use direct |
| +mode: |
| + |
| +```bash |
| +carlos ship --app console --kind binary --version <sha> console-linux-arm64 |
| +carlos promote --app console <sha> canary/console |
| +``` |
| + |
| +with `CARLOS_DEPLOYMENT_BUCKET` set for both. Then check the door: |
| + |
| +```bash |
| +carlos apps set adopt-probe --account bab --swap overlap --console https://console.carloku.com |
| +``` |
| + |
| +Expected: the approved `zdd.apps-set.done` line for `adopt-probe`, overlap, |
| +1. Until the copy is approved the draft prints: |
| +`adopt-probe swaps in overlap mode, 1 at a time on each box`, then a muted |
| +note about backward-compatible migrations. Setting the default writes |
| +nothing (see the Q3 note above). |
| + |
| +A console that predates the door answers that the app is not in this account |
| +or that the console predates `carlos apps set`. Roll the console and run it |
| +again. In direct bucket mode (`CARLOS_DEPLOYMENT_BUCKET` or |
| +`CARLOS_DEPLOYMENT_DIR` set in the shell) `carlos apps set` refuses and says |
| +so, so run the door check from a shell that is logged in to the console and |
| +has neither variable set. |
| + |
| +## Gate D: docs and changelog |
| + |
| +Needs Paul's approval to publish. This gate touches no box. |
| + |
| +Run README step 10: `go run ./cmd/changelog` in a full checkout of `main` |
| +(after `git fetch origin --tags '+refs/amadan/closed/*:refs/amadan/closed/*'`, |
| +with `gh` and `amadan` signed in), map the release's bullets in |
| +`docs/site/changelog-links.json`, and land `docs/site/changelog.json` and the |
| +links file on a branch. Then run `node hack/sync-docs.mjs <checkout>` in |
| +`carlosframework/website` and ship the site the way its AGENTS.md describes. |
| +`hack/sync-docs.mjs` and that AGENTS.md live in the website repo (not checked |
| +here). |
| + |
| +Check that carlosframework.com/platform/changelog lists the release and the |
| +public CLI page shows "Two builds at once". The page's source is |
| +`docs/site/cli.md:330` here; where the live page sits in the website is not |
| +checked here. Both must be live before Gate G. |
| + |
| +## Gate E: dev, the daemon-reload check, then the flag |
| + |
| +Needs Paul's approval, naming the boxes: dev box `i-05498eada6f0afd23` (the |
| +dev console box, eu-west-1) and dev box `i-046c7fe43620944bc` (eu-west-1), |
| +profile `kass`. Both ids come from the brief; only the first appears in |
| +another repo document (`2026-10-06-console-release-defences-plan-4-d4.md`), |
| +and the second is not checked here. **Do not start this gate while the |
| +"Before Gate E: owed" list has an open item that Paul has not waived.** |
| + |
| +First the index's D1 guard: measure `systemctl daemon-reload` on a dev box |
| +carrying titogo-scale unit counts. |
| + |
| +```bash |
| +aws --profile kass --region eu-west-1 ssm send-command --instance-ids i-046c7fe43620944bc \ |
| + --document-name AWS-RunShellScript --query Command.CommandId --output text \ |
| + --parameters 'commands=["systemctl list-units --all --no-legend | wc -l","for i in 1 2 3; do s=$(date +%s.%N); systemctl daemon-reload; e=$(date +%s.%N); echo \"$e - $s\" | bc; done"]' |
| +aws --profile kass --region eu-west-1 ssm get-command-invocation --instance-id i-046c7fe43620944bc --command-id <the id printed> --query StandardOutputContent --output text |
| +``` |
| + |
| +Expected: a unit count, then three times. If any reload takes more than about |
| +1 s, stop: plan 2's D1 says nested generation units move to transient units in |
| +a follow-up before the flag goes on. Otherwise set the flag on both dev |
| +boxes: |
| + |
| +```bash |
| +aws --profile kass --region eu-west-1 ssm send-command \ |
| + --instance-ids i-05498eada6f0afd23 i-046c7fe43620944bc --document-name AWS-RunShellScript \ |
| + --parameters 'commands=["grep -q ^CARLOS_SWAP= /etc/carlos/host.env || echo CARLOS_SWAP=overlap >> /etc/carlos/host.env","systemctl restart carlos-edge"]' |
| +``` |
| + |
| +If `host.env` already has a `CARLOS_SWAP=` line, even an empty or different |
| +one, the command leaves it alone and the box does not arm. Read the file |
| +first. The unit is `carlos-edge` and it loads `/etc/carlos/host.env` |
| +(`internal/host/units/carlos-edge.service:18`). |
| + |
| +Restarting `carlos-edge` stops every instance on the box (spec section 10). |
| +Each wakes on its next request. |
| + |
| +Then run the probe below against a dev app. Create one with: |
| + |
| +```bash |
| +carlos apps create --app zdd-probe --console https://platform.dev.carlosframework.com |
| +``` |
| + |
| +then an operator's `carlos instances enable --app zdd-probe`, and: |
| + |
| +```bash |
| +carlos instances create --app zdd-probe --host <label>.<the account's platform domain> |
| +``` |
| + |
| +Use the host it prints as `HOST`. For this gate: `APP=zdd-probe`, |
| +`CONSOLE=https://platform.dev.carlosframework.com`, and `ACCOUNT` is the dev |
| +account (an empty `--account` uses your only account). |
| + |
| +Check the flag took before trusting the probe: `carlos instances list --app |
| +zdd-probe` should show a build under SERVING once the box has reported. A |
| +dash means the box did not arm (see the dash note above). |
| + |
| +## Gate F: the flagship probe box |
| + |
| +Needs Paul's approval, naming the box and its region. |
| + |
| +Find the probe's region with: |
| + |
| +```bash |
| +carlos instances list --app adopt-probe --account bab --console https://console.carloku.com |
| +``` |
| + |
| +(REGION column; the instance is `probe1.bab.oncarlos.com`.) The flagship |
| +boxes, profile `keymail`: |
| + |
| +| box | instance id | region | |
| +| --- | --- | --- | |
| +| `ie` | i-092c0c1eea75723cb | eu-west-1 | |
| +| `us` | i-0ed0641d8e476f1b8 | us-east-1 | |
| +| `uk` | i-03c310ceaf50a9bab | eu-west-2 | |
| +| `au` | i-022288023f5e7496d | ap-southeast-2 | |
| + |
| +`ie`'s id matches the flagship console box id in |
| +`2026-10-05-scanner-host-rollout.md`. The other three are not checked here. |
| + |
| +Run Gate E's flag command against that one box with `--profile keymail` and |
| +its region. Then run the probe with: |
| + |
| +```bash |
| +APP=adopt-probe ACCOUNT=bab CONSOLE=https://console.carloku.com HOST=probe1.bab.oncarlos.com |
| +``` |
| + |
| +## Gate G: the rest of the flagship |
| + |
| +Needs Paul's approval, naming each box before it is done. |
| + |
| +One box at a time, with Gate E's flag command. After each box, do one |
| +ordinary deploy of `adopt-probe` and run `carlos instances list --app |
| +adopt-probe --account bab`, which must show INSTALLED equal to SERVING. Tito |
| +is Tito's operator's call; do not touch Tito's boxes from here. |
| + |
| +## The probe |
| + |
| +Spec section 12: zero failed requests under continuous load, and no request |
| +held longer than a normal response. Run it at Gate E (dev) and Gate F |
| +(`adopt-probe`). |
| + |
| +### 1. Build a probe app that drains on SIGTERM, in scratch |
| + |
| +```bash |
| +mkdir -p "$TMPDIR/zdd-probe" && cd "$TMPDIR/zdd-probe" |
| +cat > main.go <<'EOF' |
| +package main |
| + |
| +import ( |
| + "context" |
| + "flag" |
| + "fmt" |
| + "net" |
| + "net/http" |
| + "os" |
| + "os/signal" |
| + "syscall" |
| + "time" |
| +) |
| + |
| +var version = "dev" |
| + |
| +func main() { |
| + sock := flag.String("socket", "", "unix socket to serve on") |
| + flag.Parse() |
| + _ = os.Remove(*sock) |
| + l, err := net.Listen("unix", *sock) |
| + if err != nil { |
| + fmt.Fprintln(os.Stderr, err) |
| + os.Exit(1) |
| + } |
| + mux := http.NewServeMux() |
| + mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { |
| + time.Sleep(50 * time.Millisecond) // keep requests in flight across the switch |
| + fmt.Fprintf(w, "ok %s\n", version) |
| + }) |
| + srv := &http.Server{Handler: mux} |
| + go func() { |
| + ch := make(chan os.Signal, 1) |
| + signal.Notify(ch, syscall.SIGTERM, syscall.SIGINT) |
| + <-ch |
| + ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) |
| + defer cancel() |
| + srv.Shutdown(ctx) |
| + }() |
| + if err := srv.Serve(l); err != nil && err != http.ErrServerClosed { |
| + fmt.Fprintln(os.Stderr, err) |
| + os.Exit(1) |
| + } |
| +} |
| +EOF |
| +go mod init zddprobe |
| +for v in zdd-1 zdd-2 zdd-3 zdd-4; do |
| + CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags "-X main.version=$v" -o "probe-$v" . |
| +done |
| +``` |
| + |
| +### 2. A load phase |
| + |
| +Four request loops for the given seconds, cache-busted, logging status and |
| +time: |
| + |
| +```bash |
| +load() { # usage: load <phase> <secs> |
| + mkdir -p "$TMPDIR/zdd-probe/$1" |
| + for i in 1 2 3 4; do |
| + ( end=$((SECONDS+$2)); n=0 |
| + while [ $SECONDS -lt $end ]; do n=$((n+1)) |
| + curl -s -o /dev/null -m 35 -w '%{http_code} %{time_total}\n' "https://$HOST/?zdd=$1-$i-$n" >> "$TMPDIR/zdd-probe/$1/load-$i.log" |
| + done ) & |
| + done |
| + wait |
| +} |
| +summary() { cat "$TMPDIR/zdd-probe/$1"/load-*.log | awk -v p="$1" '$1 !~ /^2/ {bad++} {if ($2>max) max=$2; n++} END {printf "%s: %d requests, %d failed, slowest %.2fs\n", p, n, bad+0, max}'; } |
| +``` |
| + |
| +### 3. Run it |
| + |
| +```bash |
| +cd "$TMPDIR/zdd-probe" |
| +carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-1 probe-zdd-1 |
| +load baseline 60; summary baseline |
| +( load overlap 240 ) & sleep 20 |
| +carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-2 probe-zdd-2 |
| +sleep 40 |
| +carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-3 probe-zdd-3 |
| +wait; summary overlap |
| +# Control: hold mode must show the pause, or the probe proves nothing. |
| +carlos apps set "$APP" --account "$ACCOUNT" --console "$CONSOLE" --swap hold |
| +( load hold 120 ) & sleep 20 |
| +carlos deploy --app "$APP" --account "$ACCOUNT" --console "$CONSOLE" --version zdd-4 probe-zdd-4 |
| +wait; summary hold |
| +carlos apps set "$APP" --account "$ACCOUNT" --console "$CONSOLE" --swap overlap |
| +``` |
| + |
| +**Pass.** `overlap` has 0 failed and its slowest is at most the larger of 1 s |
| +and twice the baseline's slowest. Each deploy's watch ends with the approved |
| +`swapped https://$HOST` line and a box line, exit 0. The `hold` control's |
| +slowest is clearly above the baseline. If it is not, the probe cannot see a |
| +pause and the overlap result means nothing. |
| + |
| +The watch prints `swapped https://<host>` (draft word `swapped`, |
| +`deploywatch.go:459`) and `box <label>: ...` (`deploywatch.go:43`). With the |
| +flag off it prints `live https://<host>` instead (`deploywatch.go:454`), so a |
| +`live` line during the overlap phase means that box is not armed. |
| + |
| +One caution on the control: the first `--swap hold` on an app creates |
| +`config.json` and causes one config delivery (the Q3 note above). If the |
| +hold phase's slowest request is high because of that restart rather than the |
| +zdd-4 swap, the control is contaminated. If the numbers look odd, rerun the |
| +hold phase once with the policy already set. |
| + |
| +### 4. Clean up |
| + |
| +```bash |
| +rm -rf "$TMPDIR/zdd-probe" |
| +``` |
| + |
| +On dev, run `carlos apps delete zdd-probe` once Paul is done with it. |
| + |
| +## Names checked against this branch |
| + |
| +Checked in the code on `zdd-6-surfaces` (HEAD `77d77b3a` plus this file): |
| +the verbs and flags `carlos system update|status`, `carlos apps set|create|delete`, |
| +`carlos instances list|create|enable|retry`, `carlos deploy`, `carlos ship`, |
| +`carlos promote`, `carlos promote --retry`, `carlos rollback`; the flag |
| +`CARLOS_SWAP` and its exact value `overlap`; the unit `carlos-edge` and |
| +`/etc/carlos/host.env`; the SERVING, INSTALLED and REGION columns; the |
| +`swapped`, `live` and `box` watch lines; `cmd/changelog`; |
| +`docs/site/changelog.json`, `docs/site/changelog-links.json` and the "Two |
| +builds at once" section. |
| + |
| +Not checked here (not in this repo, or not verifiable without running |
| +something): `roll-console.sh` and `update-platform-binary.sh` (platform-infrastructure); |
| +`hack/sync-docs.mjs` and the website's AGENTS.md (carlosframework/website); |
| +the live URLs; the AWS instance ids except `i-092c0c1eea75723cb` and |
| +`i-05498eada6f0afd23`; the SSM `AWS-RunShellScript` documents; that `bc` |
| +exists on the boxes; the dev account and the flagship's channel names; the |
| +behaviour of the running boxes. |