amadan
spec: region HA spec 1 rev 4.2, answering Astra round 5 (A5-1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 files changed,
+104
−9
docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md+22 −9docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-5.md+21 −0docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-6.md+61 −0
CI failed — run details
diff --git a/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md b/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md| index 20f70e5f0..4ddb77d47 100644 |
| --- a/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md |
| +++ b/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md |
| @@ -1,7 +1,8 @@ |
| # Region HA, spec 1: placing an instance on a box |
| -Design, 2026-10-04, **rev 4.1**. DRAFT. Nothing here is built. |
| +Design, 2026-10-04, **rev 4.2**. DRAFT. Nothing here is built. |
| +Rev 4.2 answers Astra round 5 (`CODEX-FINDINGS-REGION-HA-5.md`, A5-1). |
| Rev 4.1 answers Astra round 4 (`CODEX-FINDINGS-REGION-HA-4.md`, |
| **A4-n**). Rev 4 answers Astra round 3 (`CODEX-FINDINGS-REGION-HA-3.md`, **A3-n**, |
| plus the A2-5, A2-7 and A1-1 residuals). Rev 3 answered Astra round 2 (`CODEX-FINDINGS-REGION-HA-2.md`, cited |
| @@ -494,17 +495,21 @@ these in order, journaling each: |
| 1. **Stop every writer (A4-1).** Close admission, drain spawn attempts, |
| drain in-flight timer deliveries (moves §9.4), then stop the host's |
| process tree and wait for it to be empty: the per-host cgroup's |
| - `populated 0`, no member processes, and no holder of the database, |
| - WAL or SHM files (claim §6.1's process half). From here nothing can |
| - change the database, files or wake state. |
| + `populated 0` and no member processes. The database, WAL and SHM |
| + files may still be held, but **only** by the box's own replicator for |
| + this host, identified by the agent's pid and the replicator's own |
| + descriptors (A5-1). Any other holder fails the step. From here nothing |
| + but the replicator can touch the database, and nothing at all can |
| + change files or wake state. |
| 2. **Flush the database:** final checkpoint and `SyncAndWait`, then |
| record the **final TXID** `N`, the `MaxTXID` of the newest level-0 |
| file (moves rev 7.6 §5.1 step 1). |
| 3. **Publish every pending file and wake-state mutation**, whatever the |
| app's `files_durability`, and record the manifest's final `seq`. |
| -4. **Finish stop evidence:** close the replicators and the file |
| - publisher, and check claim §6.1's remaining conditions. Only now is |
| - the release complete. |
| +4. **Finish stop evidence:** close the replicator (`DB.Close`) and the |
| + file publisher, then run claim §6.1's **unrestricted** check: no |
| + holder at all of the database, WAL or SHM files in `/proc/*/fd`, and |
| + every other condition. Only now is the release complete. |
| Because step 1 empties the process tree first, the positions recorded in |
| steps 2 and 3 really are final. |
| @@ -757,7 +762,7 @@ those specs finalise a one-box-per-region assumption. |
| | # | Spec | Change | |
| |---|---|---| |
| | K1 | claim §5.1 step 4 | `exec` is the box `choose()` picks among the region's eligible boxes, not "the record region's box". | |
| -| K2 | claim §4, §8 | New op kind `relocating` (§5.8): same region, from `serving`, with its phase table, admission and rows per phase, the `installed` no-reinstall boundary, the phase-specific replacement table (termination and probe failure), the two-replacement bound and `stuck`, and the release sequence (stop every writer first, then journal the final TXID and manifest `seq`, then finish stop evidence; for the source and for a failed target). Recovery owner: the region placer. Excludes moves and placements, as `placing` does. | |
| +| K2 | claim §4, §8 | New op kind `relocating` (§5.8): same region, from `serving`, with its phase table, admission and rows per phase, the `installed` no-reinstall boundary, the phase-specific replacement table (termination and probe failure), the two-replacement bound and `stuck`, and the release sequence (stop every application writer first, allowing only the box's own replicator handles; then journal the final TXID and manifest `seq`; then close the replicator and run the unrestricted holder check; for the source and for a failed target). Recovery owner: the region placer. Excludes moves and placements, as `placing` does. | |
| | K3 | claim §7.2, CC5 | A **termination fence** (§5.9), bound to a verified box record, is a lost-box fence, written automatically by the region placer. `terminated` only; `stopped` stays operator-only. No `--readmit` for a terminated box; it still goes on `lost`. The operator path keeps §7.2's terminal-state rule: a CloudTrail request record never substitutes for it. | |
| | K4 | claim §8 | Region placers (boxes) need claim CAS rights for `relocating` and for removing a terminated box's obligations. | |
| | K5 | claim §9 | Adoption names "the record region's serving box". With several boxes that is ambiguous, so adoption must finish while the region has one box (§8 step 3). | |
| @@ -785,7 +790,9 @@ those specs finalise a one-box-per-region assumption. |
| - **Relocation:** every row of the replacement table, the |
| two-replacement bound, `probe-failed` without stop evidence, the |
| source's rows through every phase, and relocate → write → replicate → |
| - terminate → cold restore with the level-0 anchor. |
| + terminate → cold restore with the level-0 anchor; a release with a |
| + live embedded replicator (step 1 passes with only its descriptors, |
| + step 4 passes only after `DB.Close`, any other holder fails step 1). |
| - **Integration:** two agents against one fake region, one control |
| bucket: create lands on the chosen box only; the other box mints no |
| row; the edge backhauls to the executor; relocate by drain through |
| @@ -871,3 +878,9 @@ those specs finalise a one-box-per-region assumption. |
| | A4-1 P1: the release sequence flushes before stopping writers | accepted: stop and empty the process tree and timer deliveries first, then flush and journal, then finish stop evidence (§5.8), K2 | |
| | A4-2 P2: the conditional Deny blocks multipart preparation | accepted: `s3:ObjectCreationOperation: true` on every header Deny; multipart completion carries the header; tested (§6) | |
| | A4-3 P3: 29245f69 is not moves rev 7.6 | accepted: rev 7.6 named as an uncommitted draft, rev 7.5 at cdc98845 as the last committed; the plan pins the commit (header) | |
| + |
| +## 17. Astra round 5 dispositions |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A5-1 P1: step 1 waits for the replicator's handle, closed only in step 4 | accepted: step 1 allows only the box's own replicator descriptors; step 4 closes it and runs the unrestricted check; tested with a live replicator (§5.8), K2 | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-5.md b/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-5.md| new file mode 100644 |
| index 000000000..5613e260c |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-5.md |
| @@ -0,0 +1,21 @@ |
| +Reviewed rev 4.1 at `1cfa15bf`, against claim rev 6.3 and the supplied moves rev 7.6 draft. |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A4-1 — release ordering | **Partly resolved.** Application processes stop before flushing, but the new first step also requires the still-open replicator’s file descriptors to be gone. See A5-1. | |
| +| A4-2 — multipart preparation | **Resolved.** The object-creation qualifier exempts multipart preparation while enforcing conditional completion, matching [AWS’s documented policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes-enforce.html). | |
| +| A4-3 — dependency revision | **Resolved.** The header correctly identifies rev 7.6 as uncommitted and requires the plan to pin its commit. | |
| +| A3-2 — target durability barrier | **Partly resolved.** Final positions and minimum pinning requirements remain explicit, but A5-1 blocks the release sequence. | |
| +| A2-7 residual — failed-probe recovery | **Partly resolved.** Recovery preserves the required positions once release completes; A5-1 prevents normal completion. | |
| + |
| +**A5-1 — P1 — §5.8: the first release step waits for handles that close only in step 4.** [Location](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/region-ha/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md:494) |
| + |
| +**Problem:** Step 1 requires no remaining holder of the database, WAL or SHM files, citing claim §6.1. That claim’s check scans `/proc/*/fd`; it includes the platform’s embedded replicator. Litestream v0.5.14 explicitly opens a [long-running database descriptor](/home/paulca/go/pkg/mod/github.com/benbjohnson/litestream@v0.5.14/db.go:1000), which remains open until [DB.Close](/home/paulca/go/pkg/mod/github.com/benbjohnson/litestream@v0.5.14/db.go:799). This sequence keeps the replicator open for step 2’s `SyncAndWait` and closes it only in step 4. |
| + |
| +**Failing sequence:** A healthy, replicating source begins a drain. Admission closes, timer deliveries drain, and the application cgroup empties. The descriptor scan still finds Litestream’s database handle in the platform process. Step 1 cannot finish, so the final flush and replicator close are never reached. A failed-probe target encounters the same obstruction and becomes `stuck`. |
| + |
| +**Suggested fix:** In step 1, establish that application processes and other application writers have stopped, allowing only explicitly identified platform replication/flush handles to remain. Flush and journal the final positions, close those handles, then perform the unrestricted no-holder check as part of step 4’s final stop evidence. Carry this distinction into K2 and test release with an initialized embedded replicator. |
| + |
| +All code `file:line` citations were checked against `1cfa15bf`. No files were changed or tests run. |
| + |
| +**Verdict: not ready.** |
| \ No newline at end of file |
diff --git a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-6.md b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-6.md| new file mode 100644 |
| index 000000000..ee9ce3b51 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-6.md |
| @@ -0,0 +1,61 @@ |
| +# Review brief: region HA spec 1 (box placement), round 6 |
| + |
| +You are an independent second reader. Review |
| +`docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md` at |
| +the commit named in the prompt, in the repository at the working |
| +directory (Go; carlos platform). |
| + |
| +## Fixed: do not re-argue |
| + |
| +§0's rulings H1 to H6 are Paul's and are fixed, as are the standing |
| +rulings listed under them. Report only mechanisms that fail to honour |
| +them, are unsafe, or are missing. |
| + |
| +## What the spec builds on |
| + |
| +- Host lifecycle claim rev 6.3: file |
| + `/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md` |
| +- Move-any-app rev 7.6 (uncommitted draft in the moves worktree below; rev 7.5 is cdc98845): file |
| + `/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/specs/2026-10-02-move-any-app-design.md` |
| +- Nearest-region, on main: `docs/superpowers/specs/2026-10-01-nearest-region-default-design.md` |
| + and `internal/console/regioncandidates.go`. |
| + |
| +Those specs are still in review themselves. Where spec 1 relies on them, |
| +check it relies on what they actually say; §9 lists the changes it asks |
| +of them. |
| + |
| +## Focus |
| + |
| +1. **One writer.** Can any sequence (create, relocation, termination, |
| + a stale edges file, a stale warm copy, a placer that pauses, the |
| + console's box dying, a partition from the control bucket) leave two |
| + boxes running or writing one host, or a box serving a stale copy? |
| +2. **The termination fence (§5.9).** Is `terminated` from |
| + DescribeInstances sufficient evidence? What about eventual |
| + consistency of EC2 state, a box id that is a label not an instance |
| + id, a box in another account or region, a forged edges file. |
| +3. **Files durability (§5.6).** Restore ordering, tombstones, |
| + generation scoping, interaction with the moves archive, what a |
| + `snapshot` app loses. |
| +4. **Reconcile and the edges (§5.3, §5.4).** The slow sweep, the stamp |
| + rule, legacy files without stamps. |
| +5. **The K list (§9).** Is anything missing that the claim or moves |
| + specs would need to change for multi-box regions? |
| +6. **Code citations.** Verify every `file:line` cited against the tree. |
| + |
| +A mechanism named, bounded and assigned to spec 2, 3 or 4 (§12) is |
| +complete at this level; don't pull later specs' depth into this one. |
| + |
| +## Output |
| + |
| +Findings ranked P1 (unsafe or blocks build), P2 (wrong or missing), P3 |
| +(clarity). Each: section, the problem, a concrete failing sequence, and |
| +a suggested fix. End with a verdict: ready for plan, or not ready. |
| + |
| +## Round 6 |
| + |
| +This is rev 4.2, answering round 5 (`CODEX-FINDINGS-REGION-HA-5.md`). Give |
| +a disposition for A5-1 and the A4-1, A3-2 and A2-7 residuals, then |
| +report only new findings that rev 4.2's change to §5.8's release |
| +sequence introduces, or anything still unsafe. Don't re-raise anything a |
| +previous round marked resolved unless rev 4.2 broke it. |