amadan
spec: region HA spec 1 rev 4.1, answering Astra round 4 (A4-1 to A4-3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 files changed,
+145
−15
docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md+44 −15docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-4.md+39 −0docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-5.md+62 −0
diff --git a/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md b/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md| index 2ff01a52c..20f70e5f0 100644 |
| --- a/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md |
| +++ b/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md |
| @@ -1,8 +1,9 @@ |
| # Region HA, spec 1: placing an instance on a box |
| -Design, 2026-10-04, **rev 4**. DRAFT. Nothing here is built. |
| +Design, 2026-10-04, **rev 4.1**. DRAFT. Nothing here is built. |
| -Rev 4 answers Astra round 3 (`CODEX-FINDINGS-REGION-HA-3.md`, **A3-n**, |
| +Rev 4.1 answers Astra round 4 (`CODEX-FINDINGS-REGION-HA-4.md`, |
| +**A4-n**). Rev 4 answers Astra round 3 (`CODEX-FINDINGS-REGION-HA-3.md`, **A3-n**, |
| plus the A2-5, A2-7 and A1-1 residuals). Rev 3 answered Astra round 2 (`CODEX-FINDINGS-REGION-HA-2.md`, cited |
| **A2-n**, plus the round 1 findings it marked partly resolved). Rev 2 |
| answered round 1 (`CODEX-FINDINGS-REGION-HA-1.md`, **A1-n**). Every |
| @@ -25,9 +26,12 @@ It builds on two specs still in review, and asks them for changes (§9): |
| - the host lifecycle claim, rev 6.3 (`design-replica-ownership`, |
| 8faa89a4), cited as **claim §n**; |
| -- move-any-app, rev 7.6 (`design-moves-any-region`, 29245f69), cited as |
| - **moves §n**. Rev 7.6 adds the level-0 anchor that relocation's install |
| - relies on (A2-1). |
| +- move-any-app (`design-moves-any-region`), cited as **moves §n**. Rev |
| + 7.5 (cdc98845) is the last committed revision. **Rev 7.6**, which adds |
| + the level-0 anchor that relocation's install relies on (A2-1), is an |
| + uncommitted draft in the moves worktree as of 2026-10-04; its model |
| + half is committed at 29245f69. Citations of rev 7.6 are to that draft |
| + until it is committed, and the plan pins the commit (A4-3). |
| It also builds on nearest-region (on main since 3f4a9bfd), whose box |
| evidence in the edges file is how a console knows which boxes a region |
| @@ -487,13 +491,23 @@ replays from the claim alone (claim C2). |
| the source in `releasing` or a failed target in `probe-failed`, it does |
| these in order, journaling each: |
| -1. close admission and drain spawn attempts (claim §6.1); |
| -2. flush the database: final checkpoint and `SyncAndWait`, then record |
| - the **final TXID** `N`, the `MaxTXID` of the newest level-0 file |
| - (moves rev 7.6 §5.1 step 1); |
| -3. publish every pending file and wake-state mutation, **whatever the |
| - app's `files_durability`**, and record the manifest's final `seq`; |
| -4. take stop evidence (claim §6.1). |
| +1. **Stop every writer (A4-1).** Close admission, drain spawn attempts, |
| + drain in-flight timer deliveries (moves §9.4), then stop the host's |
| + process tree and wait for it to be empty: the per-host cgroup's |
| + `populated 0`, no member processes, and no holder of the database, |
| + WAL or SHM files (claim §6.1's process half). From here nothing can |
| + change the database, files or wake state. |
| +2. **Flush the database:** final checkpoint and `SyncAndWait`, then |
| + record the **final TXID** `N`, the `MaxTXID` of the newest level-0 |
| + file (moves rev 7.6 §5.1 step 1). |
| +3. **Publish every pending file and wake-state mutation**, whatever the |
| + app's `files_durability`, and record the manifest's final `seq`. |
| +4. **Finish stop evidence:** close the replicators and the file |
| + publisher, and check claim §6.1's remaining conditions. Only now is |
| + the release complete. |
| + |
| +Because step 1 empties the process tree first, the positions recorded in |
| +steps 2 and 3 really are final. |
| The journal entry is `{"released_by": "<box>", "generation": "<G>", |
| "final_txid": N, "manifest_seq": S}`, and `pinning` refuses to pin less |
| @@ -671,8 +685,15 @@ before alerting, and owns the loop that reads the queue and history. |
| S3 enforces both in policy (A3-3): the data bucket denies `PutObject` |
| on `…/objects/*` and `…/snapshots/*` without `s3:if-none-match: *`, and |
| on `…/manifest.json` without either `s3:if-match` or |
| - `s3:if-none-match: *`. The one writer, `hostfiles.Publish`, sends the |
| - headers and is tested against both refusals. The control bucket's |
| + `s3:if-none-match: *`. Each Deny carries |
| + `s3:ObjectCreationOperation: true`, so it applies to `PutObject`, |
| + `CopyObject` and `CompleteMultipartUpload`, never to |
| + `CreateMultipartUpload` or `UploadPart`, which can't carry the headers |
| + (A4-2). A large snapshot is uploaded in parts and completed with |
| + `If-None-Match: *`; the `control/boxes/` rules in §5.1 carry the same |
| + qualifier. The one writer, `hostfiles.Publish`, sends the |
| + headers and is tested against both refusals, and for a multipart |
| + snapshot that completes with the header and one refused without it. The control bucket's |
| `control/boxes/` rules are in §5.1. Each Deny is scoped to its prefix |
| only: a broad Deny on this bucket once broke every lease release. Plus |
| `s3:GetObjectVersion` on the files prefix, and the claim CAS rights the |
| @@ -736,7 +757,7 @@ those specs finalise a one-box-per-region assumption. |
| | # | Spec | Change | |
| |---|---|---| |
| | K1 | claim §5.1 step 4 | `exec` is the box `choose()` picks among the region's eligible boxes, not "the record region's box". | |
| -| K2 | claim §4, §8 | New op kind `relocating` (§5.8): same region, from `serving`, with its phase table, admission and rows per phase, the `installed` no-reinstall boundary, the phase-specific replacement table (termination and probe failure), the two-replacement bound and `stuck`, and the release sequence (final TXID and manifest `seq` journaled before stop evidence, for the source and for a failed target). Recovery owner: the region placer. Excludes moves and placements, as `placing` does. | |
| +| K2 | claim §4, §8 | New op kind `relocating` (§5.8): same region, from `serving`, with its phase table, admission and rows per phase, the `installed` no-reinstall boundary, the phase-specific replacement table (termination and probe failure), the two-replacement bound and `stuck`, and the release sequence (stop every writer first, then journal the final TXID and manifest `seq`, then finish stop evidence; for the source and for a failed target). Recovery owner: the region placer. Excludes moves and placements, as `placing` does. | |
| | K3 | claim §7.2, CC5 | A **termination fence** (§5.9), bound to a verified box record, is a lost-box fence, written automatically by the region placer. `terminated` only; `stopped` stays operator-only. No `--readmit` for a terminated box; it still goes on `lost`. The operator path keeps §7.2's terminal-state rule: a CloudTrail request record never substitutes for it. | |
| | K4 | claim §8 | Region placers (boxes) need claim CAS rights for `relocating` and for removing a terminated box's obligations. | |
| | K5 | claim §9 | Adoption names "the record region's serving box". With several boxes that is ambiguous, so adoption must finish while the region has one box (§8 step 3). | |
| @@ -842,3 +863,11 @@ those specs finalise a one-box-per-region assumption. |
| | A3-1 P1: a CloudTrail request is not compute fencing | accepted: claim §7.2's terminal-state rule unchanged; CloudTrail is a finding aid only; otherwise `awaiting-evidence` (§5.9), K3 | |
| | A3-3 (part): S3 can enforce conditional headers | accepted: `s3:if-none-match` and `s3:if-match` policy conditions, scoped per prefix (§6) | |
| | A3-4 P3: citations | accepted: `internal/host/fleetbind.go`, `edge.go:2432`, `homemap.go:538`/`:582`, `reconcile.go:335`/`:799` | |
| + |
| +## 16. Astra round 4 dispositions |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A4-1 P1: the release sequence flushes before stopping writers | accepted: stop and empty the process tree and timer deliveries first, then flush and journal, then finish stop evidence (§5.8), K2 | |
| +| A4-2 P2: the conditional Deny blocks multipart preparation | accepted: `s3:ObjectCreationOperation: true` on every header Deny; multipart completion carries the header; tested (§6) | |
| +| A4-3 P3: 29245f69 is not moves rev 7.6 | accepted: rev 7.6 named as an uncommitted draft, rev 7.5 at cdc98845 as the last committed; the plan pins the commit (header) | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-4.md b/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-4.md| new file mode 100644 |
| index 000000000..213472a65 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-4.md |
| @@ -0,0 +1,39 @@ |
| +| Finding | Disposition | |
| +|---|---| |
| +| A3-1 — CloudTrail fencing | **Resolved.** Only verified terminal state authorizes operator recovery; otherwise hosts remain `awaiting-evidence`. | |
| +| A3-2 — target durability barrier | **Partly resolved.** Final database and manifest positions are journaled, but the new sequence flushes before stopping existing writers. See A4-1. | |
| +| A3-3 — bind authentication and conditional policies | **Resolved.** Operator-only binds identify an immutable record version. Conditional headers are enforced in storage. A4-2 identifies a separate upload-policy gap. | |
| +| A3-4 — code citations | **Resolved.** The corrected paths and line anchors support their stated context. | |
| +| A2-5 residual — continuous publication | **Resolved.** Deadline publication and per-second polling replace the previously unbounded waits. | |
| +| A2-7 residual — failed-probe recovery | **Partly resolved.** Bounded recovery exists, but preserving the failed target’s final writes still requires A4-1. | |
| +| A1-1 residual — legacy identity | **Resolved.** The operator binding names the exact record version; replacement and deletion are denied. | |
| + |
| +Reviewed spec rev 4 at `ff92ca2e`. H1–H6 remain fixed. |
| + |
| +1. **A4-1 — P1 — §5.8: the release sequence flushes before stopping existing writers.** [Location](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/region-ha/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md:490) |
| + |
| + **Problem:** Closing admission and draining spawn attempts does not stop an already-running process. Claim §6.1 lists an empty process tree separately. Steps 2–3 therefore record “final” positions while the application can still mutate its database and files. |
| + |
| + **Failing sequence:** A publishes manifest sequence S in step 3. Its running app then writes a file before step 4 kills and drains the process tree. Stop evidence succeeds, and B installs the pinned manifest at S. The last write is lost during a live drain or failed-probe replacement. |
| + |
| + **Fix:** Close admission, drain existing application processes and timer deliveries, then checkpoint and flush all state. Journal the resulting positions, close remaining replication work, and establish final stop evidence. Carry that ordering into K2 and K8. |
| + |
| +2. **A4-2 — P2 — §6: the conditional-write Deny also blocks multipart upload preparation.** [Location](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/region-ha/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md:671) |
| + |
| + **Problem:** The specified Deny has no `s3:ObjectCreationOperation` qualification. Multipart initiation and part uploads require `s3:PutObject` authorization but cannot supply these conditional headers. AWS explicitly requires exempting those operations while enforcing the condition at completion. [AWS conditional-write enforcement](https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes-enforce.html). |
| + |
| + **Failing sequence:** A files snapshot exceeds the single-PUT limit. Its multipart initiation is denied, so no new recoverable snapshot can be published and a release flush cannot complete. The spec defines no size bound that excludes this case. [AWS upload limits](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html). |
| + |
| + **Fix:** Qualify the header Deny with `s3:ObjectCreationOperation: true`. Require the conditional header on `CompleteMultipartUpload`, and test successful multipart publication alongside refusal of unconditional completion. |
| + |
| +3. **A4-3 — P3 — dependency declaration: `29245f69` does not contain moves rev 7.6.** [Location](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/region-ha/docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md:28) |
| + |
| + **Problem:** At that commit, the moves document declares rev 7.5 and contains no level-0 anchor contract. The supplied worktree contains rev 7.6 as an uncommitted modification. |
| + |
| + **Failing sequence:** An implementer checks out the named dependency and follows §5.8’s references to rev 7.6 steps 2 and 7. Those referenced mechanisms are absent from that committed revision. |
| + |
| + **Fix:** Pin the committed revision containing rev 7.6, or explicitly identify the dependency as an uncommitted draft until that revision exists. This is a reproducibility correction; the HA spec’s explicit anchor requirement remains intact. |
| + |
| +All code `file:line` citations were checked against `ff92ca2e`. No files were changed or tests run. |
| + |
| +**Verdict: not ready.** |
| \ No newline at end of file |
diff --git a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-5.md b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-5.md| new file mode 100644 |
| index 000000000..10aedc273 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-REGION-HA-5.md |
| @@ -0,0 +1,62 @@ |
| +# Review brief: region HA spec 1 (box placement), round 5 |
| + |
| +You are an independent second reader. Review |
| +`docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md` at |
| +the commit named in the prompt, in the repository at the working |
| +directory (Go; carlos platform). |
| + |
| +## Fixed: do not re-argue |
| + |
| +§0's rulings H1 to H6 are Paul's and are fixed, as are the standing |
| +rulings listed under them. Report only mechanisms that fail to honour |
| +them, are unsafe, or are missing. |
| + |
| +## What the spec builds on |
| + |
| +- Host lifecycle claim rev 6.3: file |
| + `/home/paulca/amadan.net/carlos/platform/.claude/worktrees/replica-ownership/docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md` |
| +- Move-any-app rev 7.6 (uncommitted draft in this worktree; rev 7.5 is cdc98845): file |
| + `/home/paulca/amadan.net/carlos/platform-moves-design/docs/superpowers/specs/2026-10-02-move-any-app-design.md` |
| +- Nearest-region, on main: `docs/superpowers/specs/2026-10-01-nearest-region-default-design.md` |
| + and `internal/console/regioncandidates.go`. |
| + |
| +Those specs are still in review themselves. Where spec 1 relies on them, |
| +check it relies on what they actually say; §9 lists the changes it asks |
| +of them. |
| + |
| +## Focus |
| + |
| +1. **One writer.** Can any sequence (create, relocation, termination, |
| + a stale edges file, a stale warm copy, a placer that pauses, the |
| + console's box dying, a partition from the control bucket) leave two |
| + boxes running or writing one host, or a box serving a stale copy? |
| +2. **The termination fence (§5.9).** Is `terminated` from |
| + DescribeInstances sufficient evidence? What about eventual |
| + consistency of EC2 state, a box id that is a label not an instance |
| + id, a box in another account or region, a forged edges file. |
| +3. **Files durability (§5.6).** Restore ordering, tombstones, |
| + generation scoping, interaction with the moves archive, what a |
| + `snapshot` app loses. |
| +4. **Reconcile and the edges (§5.3, §5.4).** The slow sweep, the stamp |
| + rule, legacy files without stamps. |
| +5. **The K list (§9).** Is anything missing that the claim or moves |
| + specs would need to change for multi-box regions? |
| +6. **Code citations.** Verify every `file:line` cited against the tree. |
| + |
| +A mechanism named, bounded and assigned to spec 2, 3 or 4 (§12) is |
| +complete at this level; don't pull later specs' depth into this one. |
| + |
| +## Output |
| + |
| +Findings ranked P1 (unsafe or blocks build), P2 (wrong or missing), P3 |
| +(clarity). Each: section, the problem, a concrete failing sequence, and |
| +a suggested fix. End with a verdict: ready for plan, or not ready. |
| + |
| +## Round 5 |
| + |
| +This is rev 4.1, answering round 4 (`CODEX-FINDINGS-REGION-HA-4.md`). Give |
| +a disposition table for A4-1 to A4-3 and the A3-2 and A2-7 residuals. |
| +Then report only new findings that rev 4.1's changes introduce (the |
| +reordered release sequence in §5.8, the multipart qualifier in §6), or |
| +anything still unsafe. Don't re-raise anything a previous round marked |
| +resolved unless rev 4.1 broke it. |