amadan
region HA spec 2, rev 14: provider boundary corrected (H26: multi-box AWS-only until #299; spec 2's own AWS parts are adapters; s3compat for its bucket clients); round 13 findings committed unedited
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+78
−25
docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md+38 −25docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-SPEC2-13.md+40 −0
CI cancelled — superseded by a newer push — run details
diff --git a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md| index 7d684ee4b..36b5601be 100644 |
| --- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| +++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md |
| @@ -4,6 +4,8 @@ Design, 2026-10-16, **rev 13**. Approved by Paul on 2026-10-16, subject to this |
| **Rev 13** adds §25, the provider boundary, at Paul's request. Spec 2 must not make the platform depend on AWS: a deployment stays "one object-store bucket + N hosts running the `carlos` binary + DNS" (README), and its AWS-specific parts are adapters. |
| +**Rev 14** corrects §25 after Astra's round 13 (`CODEX-FINDINGS-REGION-HA-SPEC2-13.md`, **A13-n**). Multi-box regions are already AWS-only on main, because a box's identity is AWS's signed instance document. Paul ruled **H26** (2026-10-16): they stay AWS-only until a provider-neutral identity spec lands (#299). Spec 2 keeps its own new AWS parts as adapters, so that spec only has to replace identity. |
| + |
| **Rev 12** answers Astra's round 11 (`CODEX-FINDINGS-REGION-HA-SPEC2-11.md`, **A11-1**, P2, accepted). Rollback's exclusion is set on the region's group, and every box the group launches inherits it at enrolment, so a replacement can't reopen placement. §24. |
| **Rev 11** answers Astra's round 10, a full pass (`CODEX-FINDINGS-REGION-HA-SPEC2-10.md`, cited **A10-n**): three P2s and one P3, all accepted. Rollback now restores only the ingress and never destroys a group that has hosted an instance, so retiring a group is spec 4's drain too. Moves take their source from the claim, not from placement eligibility. §23 maps them. |
| @@ -68,6 +70,7 @@ from 2026-10-15: |
| | H23 | **The specs are recut.** Spec 2 covers two-box regions: this document. Spec 3 covers failover. Spec 4 shrinks to automatic scale-out and scale-in, rebalancing and latency steering. S1 §12's split is superseded where they differ. | |
| | H24 | **IE's current box becomes the platform box.** It's fixed, outside the group, and keeps the console, keymail, woodstar, the relay and the scanner grants. It takes no new customer instances. Its existing ones leave with spec 4's drain (H25). | |
| | H25 | **(2026-10-16) Spec 2 doesn't empty the old boxes.** Each region's old fixed box keeps the instances it has and takes no new ones. Those instances keep today's single-box behaviour until spec 4's drain moves them, or until spec 3's failover moves them if AWS terminates the box. Each region runs three boxes until then. | |
| +| H26 | **(2026-10-16) Multi-box regions stay AWS-only until a provider-neutral box identity exists** (#299). Box identity, enrolment, roster validation and placement eligibility rest on AWS's signed instance document on main today. Spec 2 doesn't make that worse: its own new AWS parts are adapters (§25). | |
| Design choices Paul approved in section review on the same day, recorded |
| here so they aren't reopened: |
| @@ -543,33 +546,43 @@ All three were answered in Astra's round 1 and folded in: |
| |---|---| |
| | A11-1 P2: rollback's exclusion doesn't survive box replacement | Accepted: a role on the region's group entry in the roster, inherited at enrolment in the same write, and applied to current members when changed (§7, §9) | |
| -## 25. The provider boundary (rev 13) |
| +## 25. The provider boundary (rev 14) |
| -Spec 2 is AWS-first because carloku.com runs on AWS. The platform isn't: a self-hosted deployment is one object-store bucket (S3-compatible, MinIO included), N hosts running `carlos`, and DNS. So every part of spec 2 is either **provider-neutral core** or an **AWS adapter** behind a small interface. The core never imports the AWS SDK. |
| +Spec 2 is AWS-first because carloku.com runs on AWS. The platform isn't: a self-hosted deployment is one object-store bucket (S3-compatible, MinIO included), N hosts running `carlos`, and DNS. |
| -**Core.** These work on any hosts, behind any load balancer or none: |
| +**What's true on main before spec 2** (A13-1, A13-2). Multi-box regions already need AWS. Under the claim, a box's identity is AWS's signed instance identity document: |
| +- enrolment requires an AWS region and verifies the AWS-signed evidence (`internal/hostactors/enroll.go:29`); |
| +- roster validation requires `Cloud.Provider == "aws"` (`internal/hostactors/roster.go:111`); |
| +- hosting re-verifies it (`internal/hostbox/identity.go:45`); |
| +- placement eligibility requires an AWS region and `Cloud.Complete()` (`internal/boxchoice/boxchoice.go:110`, `internal/boxid/verify.go:30`). |
| -| Piece | Without AWS | |
| -|---|---| |
| -| Placement policy, the `legacy`/`platform` box roles, the group role, pending creates (§7) | Unchanged. None of it calls a cloud API | |
| -| The health answer `/.carlos/lb-health` (§6) | Plain HTTP. Any load balancer (HAProxy, nginx, a hosting provider's) or a monitor can use it | |
| -| Forwarding by private address in a region, own public address between regions (§6) | Addresses come from an **address source**. Without AWS: `CARLOS_PRIVATE_IP` and `CARLOS_PUBLIC_IP` in `host.env`, or the primary interface's address when unset | |
| -| Joining and hosting modes (§5) | Generic. Enrolment uses the **membership confirmer** interface. Without AWS, there is none, and enrolment is operator-approved through the existing `hostactors.Enroll` path | |
| -| `host.env` (§5) | A file, as today. The SSM reader is one way of writing it | |
| -| The channel-release install and the guard (§5) | Unchanged. They read the bucket and the release key | |
| -| Single-box assumptions (§8) | Fixed in core, so they apply to every multi-box deployment | |
| - |
| -**AWS adapters.** They are loaded only when the deployment declares AWS, and they load the SDK through `internal/rootaws`, as every AWS client does since #267: |
| -- the **membership confirmer**: Auto Scaling `DescribeAutoScalingInstances` (H16, H17); |
| -- the **SSM and KMS parameter reader** that writes `host.env`; |
| -- the **IMDS address source**; |
| -- **termination evidence** (S1, spec 3); |
| -- the **`region-host` tofu module's group mode**: launch template, Auto Scaling group, NLB, alarms (§4). |
| - |
| -**What a deployment without AWS doesn't get.** |
| -- A sick box isn't replaced or enrolled automatically. An operator replaces it and approves its enrolment. |
| -- No automatic failover (spec 3). It triggers on a box's verified termination (H1), which has no equivalent without a provider's record. A provider adapter could supply one later, behind the same interface. |
| +A deployment without AWS runs one box per region with claim mode off, unaffected by any of this. **H26:** multi-box stays AWS-only until a provider-neutral identity spec (#299) replaces that identity. Operator approval alone can't stand in for it, because the claim re-verifies identity at every admission. |
| + |
| +**What spec 2 adds, and on which side it sits.** Spec 2 writes everything it adds so that #299 only has to replace identity: |
| + |
| +| Spec 2 adds | Side | Notes | |
| +|---|---|---| |
| +| Placement policy, box and group roles, pending creates (§7) | Core | Spec 2's new rules live in a function that takes candidates already filtered for identity. #299 then changes only the identity filter (`boxchoice.Eligible`'s AWS checks), not the policy | |
| +| The health answer `/.carlos/lb-health` (§6) | Core | Plain HTTP | |
| +| Forwarding by private address in a region, own public address between regions (§6) | Core, with an **address source** | The AWS adapter reads IMDS. The core fallback is `CARLOS_PRIVATE_IP` and `CARLOS_PUBLIC_IP` in `host.env`, or the primary interface's address | |
| +| Joining and hosting modes (§5) | Core, with a **membership confirmer** | The AWS adapter is the Auto Scaling describe (H16, H17). Joining itself names no provider | |
| +| `host.env` assembly from SSM and KMS (§5) | AWS adapter | `host.env` stays a file. SSM is one way of writing it | |
| +| The channel-release install and guard (§5) | Core | They read the bucket and the release key | |
| +| The `region-host` module's group mode (§4) | AWS adapter | Tofu, AWS by nature | |
| +| Termination evidence (S1, spec 3) | AWS adapter | | |
| + |
| +**Bucket access** (A13-3). Every bucket client spec 2 adds is built with `s3compat.Options()`, so `CARLOS_S3_ENDPOINT` applies, exactly as the existing clients do. One existing claim client doesn't: the generation-marker client (`internal/host/claimproduction_marker.go:21`). That's filed as #298 and fixed in claim code, separately from spec 2. |
| **Enforcement.** |
| -- An import test: the core packages spec 2 touches (`internal/boxchoice`, `internal/edge`, `internal/host` outside its adapter files, `internal/console`, `internal/hostactors`) import no `github.com/aws/...` package. Adapter files live in clearly named files or packages, for example `internal/host/awsaddr.go` and `internal/hostactors/awsmembership`. |
| -- A test runs the core path with no AWS configuration at all: two boxes with `CARLOS_PRIVATE_IP`/`CARLOS_PUBLIC_IP`, operator-approved enrolment, placement, forwarding and the health answer. |
| +- An import test: spec 2's new core files import no `github.com/aws/...` package. Adapters live in clearly named files or packages, for example `internal/host/awsaddr.go` and `internal/hostactors/awsmembership`. |
| +- A test that every `s3.NewFromConfig` call site in the packages spec 2 touches passes `s3compat.Options()`. |
| +- Unit tests for each core piece with the AWS adapters absent: the address source from `host.env`, the health answer, the placement policy over identity-filtered candidates, and joining with a fake confirmer. |
| +- A full multi-box lifecycle test without AWS waits for #299, because it needs a non-AWS identity. |
| + |
| +## 26. Astra round 13 dispositions (rev 14) |
| + |
| +| Finding | Disposition | |
| +|---|---| |
| +| A13-1 P1: operator approval doesn't remove the AWS identity requirement | Accepted as a fact about main. Under H26, multi-box stays AWS-only until #299, and §25 says so | |
| +| A13-2 P2: placement eligibility excludes non-AWS hosts | Accepted: spec 2's policy runs over identity-filtered candidates, so #299 changes only the filter. The import test doesn't claim to cover this (§25) | |
| +| A13-3 P2: the marker client ignores `CARLOS_S3_ENDPOINT` | Accepted: a bug on main, filed as #298 for the claim's code. Spec 2's own clients use `s3compat.Options()`, with a call-site test (§25) | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-SPEC2-13.md b/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-SPEC2-13.md| new file mode 100644 |
| index 000000000..b5441021e |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-REGION-HA-SPEC2-13.md |
| @@ -0,0 +1,40 @@ |
| +Three new findings: |
| + |
| +1. **P1 — Operator approval does not remove the AWS identity requirement.** |
| + [Spec:557](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:557) delegates non-AWS enrolment to existing `hostactors.Enroll`. That path still requires `AWSRegion` and verifies an AWS-signed identity ([enroll.go:29](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/hostactors/enroll.go:29)). Roster validation explicitly requires `Cloud.Provider == "aws"` ([roster.go:111](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/hostactors/roster.go:111)), and hosting re-verifies the AWS evidence ([identity.go:45](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/hostbox/identity.go:45)). |
| + |
| + **Failure scenario:** an operator approves a bare-metal host with MinIO and configured addresses, but it cannot enrol or enter hosting mode. The membership-confirmer interface does not cover this separate identity dependency. Specify a provider-neutral identity contract through enrolment and admission, preserving reference uniqueness and replacement isolation. |
| + |
| +2. **P2 — “Unchanged” placement still excludes non-AWS hosts.** |
| + [Spec:554](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:554) treats placement as provider-neutral because it calls no cloud API. However, [`boxchoice.Eligible`:110](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/boxchoice/boxchoice.go:110) requires an AWS region and `Cloud.Complete()`, which explicitly requires provider `aws` ([verify.go:30](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/boxid/verify.go:30)). |
| + |
| + **Failure scenario:** even after adding non-AWS enrolment, both healthy hosts remain ineligible and every create stays pending. Separate placement-region membership from AWS evidence. These checks already import no AWS SDK, so the proposed import test cannot enforce that distinction. |
| + |
| +3. **P2 — The no-AWS test omits a lifecycle write that still targets AWS.** |
| + [Spec:575](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md:575) requires enrolment, placement, forwarding and health checks, but no actual generation publication. The production marker client uses plain `s3.NewFromConfig`, without the `s3compat.Options()` that applies `CARLOS_S3_ENDPOINT` ([claimproduction_marker.go:21](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/host/claimproduction_marker.go:21)). |
| + |
| + **Failure scenario:** with MinIO configured through `CARLOS_S3_ENDPOINT`, ordinary bucket operations succeed, but generation publication sends its write to AWS and fails ([claimowners.go:586](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/design-region-ha-two-box/internal/host/claimowners.go:586)). Classify compatible-store transport explicitly and require the no-AWS test to exercise production admission, generation publication and relocation, with unexpected AWS/IMDS requests rejected. |
| + |
| +Reviewed platform `d2361b2a` and flagship `7c99ff8f`. No files changed; no builds, tests or infrastructure commands run. |
| + |
| +--- |
| + |
| +**FIRST PROMPT:** |
| +Review spec 2 round 13 at the pinned HEAD against flagship tofu, read-only. |
| + |
| +**WHAT:** |
| +Check the provider boundary and no-AWS fallback. |
| + |
| +**STATUS:** |
| +Review complete: one P1 and two P2 findings. |
| + |
| +**NEEDED:** |
| +Nothing. |
| + |
| +**RECOMMENDED:** |
| +Resolve the identity, placement and compatible-store gaps before planning. |
| + |
| +**SESSION:** |
| +Safe to close; nothing modified or started. |
| + |
| +**Verdict: not ready for a plan.** |
| \ No newline at end of file |