amadan
harness: Run 9 (B) rollback rehearsal PASS (23 s), Run 9 corrections R9-1..4, M1..M10
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+85
−0
harness/RUNLOG.md+85 −0
CI cancelled — superseded by a newer push — run details
CI log
=== gate === cd launchctx && go vet ./... && go test ./... -count=1 ok amadan.net/carlos/platform.git/launchctx 0.011s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] go vet ./... gofmt -l .: clean go test ./... -count=1 -timeout 30m amadan: job cancelled — a newer push superseded it
diff --git a/harness/RUNLOG.md b/harness/RUNLOG.md| index b2dea7040..f0d211c5e 100644 |
| --- a/harness/RUNLOG.md |
| +++ b/harness/RUNLOG.md |
| @@ -1813,3 +1813,88 @@ grace runs out (`internal/layout/pass.go:318-322`); an app with a 1-minute sched |
| `carlos status --layout`: `noor-marquetry (nested by default) moveapp.dev.harness.test flat` and the warning "runs |
| nested instances beside flat ... Nested isolation on this box is not in force until none of those remains". |
| +### B5. The rollback rehearsal on rollbackapp: abort after flip, before open: PASS (17:32:49-17:33:13Z) |
| + |
| +`blocks/r9-move.host.sh rollbackapp abort`. Before (woken on sophie): `bg` 2689 rows, `worker` 1351, instance key |
| +sha256 `eb5cb3433bc745c9`, files `instance.key` `2167c1dd...`, lease `sophie:230`. |
| + |
| +| step | at | took | result | |
| +|---|---|---|---| |
| +| start | 17:32:49.695Z | 2.1 s | as moveapp's; `instance exited (exit 0)` 17:32:50Z | |
| +| seal | 17:32:51.939Z | 6.1 s | `Copied and checked.`; data-ie lease `moved:mv_kai2mu6xs376umepnqhhvykfuu`; sophie's slice empty (0 processes) | |
| +| flip | 17:32:59.877Z | 6.1 s (try 1: inside the 2-minute window) | `Stopped in ie. Starting in uk.` / `uk holds everything ...`; dennis row `landing`, flat, env `165773df...` = sophie's, config.env 7 names, files `2167c1dd...` | |
| +| abort | 17:33:06.382Z | 4.1 s | `Moving rollbackapp.dev.harness.test back to ie.` / `Every box has put rollbackapp.dev.harness.test back in ie. Run carlos move offline finish ...` exit 3 (the CLI's "aborted" code); reports: sealer, both sources and the target `reverted` | |
| +| finish | 17:33:10.587Z | 0.1 s | **`rollbackapp.dev.harness.test is back in ie and serving. Nothing was lost.`** `Move mv_kai2mu6xs376umepnqhhvykfuu ... is aborted.` exit 3 | |
| +| first request on sophie | 17:33:12.342Z | 0.6 s | `/ready` 200 (0.44 s) at 17:33:12.979Z: **offline 23.3 s** from start (the console: `offline_s 20.9`) | |
| + |
| +Verified (17:33:32Z): sophie `/report` `bg` 1-2735 and `worker` 1-1375, contiguous and past the 2689/1351 read |
| +before start: **no write lost**; `platform_key_sha256 eb5cb343...` and the files unchanged; the store object |
| +`probe/a6f1b49d1c6b0487.txt` from the new boot. Sophie's row `ready` (nested, the local DB never purged); data-ie lease |
| +`sophie:230` generation 2 (the tombstone was lifted and the source took its own lease again); data-uk has no lease and |
| +no `moves/` object for the host (staging deleted); dennis has no row and nothing in `/data` for it, and parked what it |
| +had extracted under `/data/.offline-moves/rollbackapp.dev.harness.test/mv_kai2mu6xs376umepnqhhvykfuu/target/` |
| +(`<host>/instance.key`, `<host>.env`, `<host>.config.env`); the record is back in home (`region` absent), `move` null, |
| +`last_move {"outcome":"aborted","offline_s":20.9}`. |
| + |
| +## Run 9 corrections |
| + |
| +### To `claim-dev-cohort1-runbook` (the roll) and the README |
| + |
| +- **R9-1 (README "Checking dev", step 3).** On dev, leo (carlos-dev-ie-b) has had `CARLOS_POOLS=cohort1` since step |
| + 4.2 and no wakecheck row since 4.3, so the pinned request is answered by backhaul from the console box. Replace |
| + "A pass is a 200 whose `host=` is that box" with "A pass is a 200 whose `host=` is `carlos-dev-ie` (the request |
| + reaches carlos-dev-ie-b's edge and is backhauled; while leo is in pool cohort1 it holds no wakecheck row)". And |
| + step 4: wakecheck is `nested` on the console box only. |
| +- **R9-2 (G4).** Add sophie: "Sophie (canary, armed) takes v0.28.5 by the same seven steps with `--channel canary`; |
| + its console keeps running the protected v0.28.4 console (a router apply does not touch it)." Add under "Why not |
| + `system update`": "This holds for the v0.28.4 to v0.28.5 step only. Harness Run 9 showed a box on v0.28.5 taking the |
| + next release through its own `system update` with no hand step." |
| +- **R9-3 (G4 precondition).** Fill "Harness run ID": Run 9, `cohort1-20261020T161315Z` (the identity session) on |
| + harness-build; re-arm PASS on leo 16:19-16:23Z and sophie 16:30-16:33Z; normal road PASS 16:38-16:48Z. |
| +- **R9-4 (the console, NEW, blocking).** There is no block that moves sophie's guarded console to v0.28.5: `system |
| + claims install-console` refuses once a console is installed (defect 5). Until it is ruled, dev's console stays on |
| + v0.28.4, and the offline-move doors (only in the new console) do not exist on dev. E1 (move both slots aside, then |
| + install-console) worked in the harness; it is an experiment, not a ruled block. |
| +- **C4 holds:** the roster revision moved 9 -> 10 -> 11 for leo's roll, 11 -> 12 -> 13 for sophie's, 13 -> 15 for the |
| + normal road. |
| + |
| +### To the offline-move plan's "Morning runbook outline" (`2026-10-20-offline-operator-move.md` on main) |
| + |
| +- **M1 (P0.1/P0.2).** Add: "The console that serves the doors runs the release: `carlos move offline status` answers |
| + (an older console has no offline doors). On a guarded console box this needs R9-4's ruling first." |
| +- **M2 (P2.1, P1.2; finding 7).** Replace "`status` until sealed; `flip`; `status` until landed; `open`" with: "run |
| + `flip` within two minutes of the seal (and `open` within two minutes of the flip): the doors trust a box report for |
| + two minutes, and boxes rewrite theirs only on the moves pass that a door's serial bump starts, or every 5 to 6 |
| + minutes. If `flip` answers `<box> still runs a process for <host>` and `status` shows `Its report is not current`, |
| + that is a stale report, not a process: retry every 15 s until the next pass (at most 6 minutes). Only a current |
| + report with a process count is a real MV-O31." |
| +- **M3 (P0.7).** Replace "Make the first request yourself straight after `open`, then let the layout pass nest it |
| + inside the window" with: "Make the first request yourself straight after `open`. The row is flat. The layout pass |
| + nests it only when it is asleep, or stops it after a 1-hour grace (`layout: deferred until asleep -> <host> (awake |
| + (stopped for the move in 1h0m0s ...))`). An instance with a schedule shorter than the idle timeout never sleeps, so |
| + plan a second short stop about an hour after `open`, and confirm `carlos status --layout` shows it nested after |
| + that." (B4 below has the measured result.) |
| +- **M4 (P1.0).** In dev's current state (leo armed, claims off at the box, `CARLOS_POOLS=cohort1`), leo is not a |
| + blocker: it publishes `offline-move-1` (an armed guard does not withhold it), and it took part as a source box |
| + (held, fenced, reported, `gone` at finish) without holding the host. The gate applies only once leo's claims are on |
| + (after the window's 8.2). |
| +- **M5 (P1.1).** Before the first wake, check the box rendered `<host>.config.env` |
| + (`/data/instances/<host>/<host>.config.env` on a nested box). If it is missing, change any setting (`carlos env |
| + set`) to make a new bundle generation (finding 6). Otherwise the instance can run without its settings for up to an |
| + hour. |
| +- **M6 (P1.2/P2.3, defect 8).** After `finish`, on each source box: `ls -la /data/.instances/<host>/`. If a |
| + `<host>.env` is there (written by the source's scheduler after `open`), move it into |
| + `/data/.offline-moves/<host>/<id>/source/` and record it. Never delete it. |
| +- **M7 (Cleanup recipe).** A finished move does not delete the source's `instances/<host>/lock.json`: it rewrites it |
| + as `owner moved:<id>`, `expires_at 1970-01-01T00:00:00Z`. That expired object is the normal end state, not an |
| + abandoned move. The cleanup recipe is for a `moved:` owner whose `expires_at` is still in the future. |
| +- **M8 (P1.3, P2.4).** `abort` and the `finish` after it exit 3 (the CLI's "aborted" code); that is the expected |
| + result, not a failure. |
| +- **M9 (P0.2, as commands).** Per box: `grep -c '^CARLOS_HOST_CLAIM=.' /etc/carlos/host.env` = 0, no |
| + `CARLOS_MOVES_PHASE1`, `systemctl show carlos-edge -p User -p Delegate` = `User=carlos` and `Delegate=yes`, |
| + `systemctl is-active carlos-stage.path` = active (the UK box at least), and in each box's |
| + `control/edges/<box>.json`: `.capabilities` contains `offline-move-1` and every `.secrets_key_fp` is equal. |
| +- **M10 (P0.6 budget).** Measured: about 25 s of door and box work for a 4 MB nested instance (start 2 s, seal 6 s, |
| + flip 4-6 s, open 4 s, first request 6 s). Abort after flip: back serving in 23 s. Add the operator's gaps and up to |
| + 6 minutes per stale-report wait (M2). `finish` takes 4 to 5 minutes but the instance is serving by then. |
| + |