amadan

carlos / platform Public

plan: region HA 1b answers Astra round 2: claim passes obey the retry backoff, one discharge case per check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude 064366a9532ee50a18c49d32a47b3dfef0a90ef3 parent e17ce3c
2 files changed, +130 −9
  • docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md +74 −9
  • docs/superpowers/plans/CODEX-PLAN-REGION-HA-1B-BRIEF-3.md +56 −0

CI cancelled — superseded by a newer push — run details

diff --git a/docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md b/docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md
index 821c32955..e8a5e668b 100644
--- a/docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md
+++ b/docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md
@@ -1977,7 +1977,7 @@ Spec §5.9 under H19: regional workers record each enrolled box's block device m
- A `terminated` read is never recorded. Its mappings are empty (the dev test of 2026-10-06), and a record of it would list every volume as removed.
- "The first record taken at enrollment" is checked as: the earliest record for this enrollment must be observed within `EnrollmentWindow` (one hour, the refresh period) of the roster's enrollment time. A later first record cannot speak for a volume attached and removed before it.
- The newest record must be no more than `LastRecordWindow` (2 hours) older than `terminatedAt`, the fence's `observed_at` (controller ruling 1). A recorder that stopped long before the box died cannot speak for its last hours.
-- `Discharge` checks every stored record of this enrollment before using it (Astra plan R1-6): a version other than 1, a state that carries no mapping (`terminated`, or none), an empty mapping, or a volume with no id or device keeps the chores open. `List` returns whatever is stored, so a record `Write` would never produce is still met here.
+- `Discharge` checks every stored record of this enrollment before using it (Astra plan R1-6, R2-2): a version other than 1, a state that carries no mapping (`terminated`, or none), an empty mapping, or a volume with no id or no device keeps the chores open. The test has one stored case per check, each failing that check alone. `List` returns whatever is stored, so a record `Write` would never produce is still met here.
- `Discharge` recomputes removals from consecutive records as well as reading `removed_volumes`, so a record one worker failed to write cannot hide a removal.
Keys use a fixed-width UTC time with `-` in place of `:`, so they sort in observation order.
@@ -2222,6 +2222,10 @@ func TestDischargeRefusesStoredRecordsThatAreNotEvidence(t *testing.T) {
// After it, keeping vol-root, so nothing is removed.
"a volume with no id": {`{"v":1,` + after + `,"instance_state":"running","block_devices":[{"device":"/dev/xvda","volume_id":"vol-root","delete_on_termination":true,"status":"attached"},{"device":"/dev/xvdb","volume_id":"","delete_on_termination":true,"status":"attached"}],"removed_volumes":[]}`, 5 * time.Minute},
"a version 2 record": {`{"v":2,` + after + `,"instance_state":"running","block_devices":[{"device":"/dev/xvda","volume_id":"vol-root","delete_on_termination":true,"status":"attached"}],"removed_volumes":[]}`, 5 * time.Minute},
+ // Astra plan R2-2: one case per check, each failing only that check.
+ "a running read with no volume": {`{"v":1,` + before + `,"instance_state":"running","block_devices":[],"removed_volumes":[]}`, -5 * time.Minute},
+ "a terminated read with its root mapping": {`{"v":1,` + after + `,"instance_state":"terminated","block_devices":[{"device":"/dev/xvda","volume_id":"vol-root","delete_on_termination":true,"status":"attached"}],"removed_volumes":[]}`, 5 * time.Minute},
+ "a volume with no device": {`{"v":1,` + after + `,"instance_state":"running","block_devices":[{"device":"/dev/xvda","volume_id":"vol-root","delete_on_termination":true,"status":"attached"},{"device":"","volume_id":"vol-data","delete_on_termination":true,"status":"attached"}],"removed_volumes":[]}`, 5 * time.Minute},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
@@ -2664,7 +2668,7 @@ Spec §5.4: "The edges-file pass publishes a host as `home` only when this box i
- **The decision.** A claim whose `op.exec` is this box's reference: home, stamped with `holder.stamp`. A claim whose `op.exec` names another box, or none, or which has no holder: withheld. No claim: home, unstamped, exactly as before; §8's gate admits a second box only once every host has a claim.
- **Errors.** A read that fails keeps the last decision (claim C6: an error is never absence); a host never read publishes as before.
- **Cost.** Each read stands 10 minutes (H9), at most 256 reads a pass, never-attempted hosts first, then oldest attempt first. A failed read counts as an attempt, so a claim that keeps failing goes to the back of the queue and cannot starve the rest (Astra plan R1-4). About 288 thousand GETs a day per 2,000 hosts, under $0.12.
-- **Refresh.** The registry timer alone runs a pass every five to six minutes, which at 256 reads a pass would take 40 minutes to cover 2,000 hosts (Astra plan R1-5). So, with the flag on, the loop also runs a pass every 30 seconds while any claim read is due (`edgesLoopState`). Every claim is then re-read within about 14 minutes; the edges file is still written only when its body changes. With the flag off, `edgesLoopState` is exactly `edgesFileSchedule`.
+- **Refresh.** The registry timer alone runs a pass every five to six minutes, which at 256 reads a pass would take 40 minutes to cover 2,000 hosts (Astra plan R1-5). So, with the flag on, the loop also runs a pass every 30 seconds while any claim read is due (`edgesLoopState`). Every claim is then re-read within about 14 minutes; the edges file is still written only when its body changes. A claim pass is a publication like any other (Astra plan R2-1): it waits out the schedule's retry backoff after a failed write, and its own outcome goes through `edgesFileSchedule.done`, so a bucket that refuses every write is asked on the existing 2-second-to-5-minute ladder, not every 30 seconds. With the flag off, `edgesLoopState` is exactly `edgesFileSchedule`.
- **Aliases and catch-alls carry their primary's stamp** (Astra plan R1-2). An attached domain and a served `*.<parent>` catch-all are dialled through the primary's installation. Unstamped, a dead predecessor's alias would tie with the successor's at 0 and the edge (`dialBeats`) would keep the box that sorts first.
- **§18's open question.** A positive stamp is the mark: only relocatable hosts with a claim carry one, so the edge's report of two boxes at one positive stamp already concerns them alone. No separate marker field.
@@ -2684,7 +2688,7 @@ The withholding and stamping are applied before owners and aliases are computed,
const edgesClaimPassEvery = 30 * time.Second
type edgesClaims struct{ /* self, read, now, entries, hosts */ }
func (c *edgesClaims) backlog(now time.Time) bool
- type edgesLoopState struct{ sched *edgesFileSchedule; claims *edgesClaims; claimsNext time.Time; schedDue bool }
+ type edgesLoopState struct{ sched *edgesFileSchedule; claims *edgesClaims; claimsNext time.Time; schedDue, claimDue bool }
func (s *edgesLoopState) due(now time.Time, fp string, fpOK bool) bool
func (s *edgesLoopState) timed() bool
func (s *edgesLoopState) done(now time.Time, ok bool)
@@ -2982,6 +2986,51 @@ func TestEdgesLoopRefreshesEveryClaimWithinTheTTL(t *testing.T) {
t.Fatalf("a claim went %s without a read, want at most %s", worst, limit)
}
}
+
+// Astra plan R2-1: a claim pass is a publication. While the bucket refuses
+// every write, claim passes wait out the schedule's backoff and their own
+// failures move it; once writes succeed, the 30-second claim passes resume.
+func TestEdgesLoopClaimPassesObeyTheRetryBackoff(t *testing.T) {
+ book := &claimsBook{claims: map[string]*hostclaim.Claim{}}
+ var hosts []string
+ for i := 0; i < 4000; i++ { // enough that reads stay due throughout
+ h := fmt.Sprintf("h%04d.bab.test", i)
+ book.claims[h] = serving("annie", 1)
+ hosts = append(hosts, h)
+ }
+ start := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
+ clock := start
+ claims := newEdgesClaims("annie", book.read)
+ claims.now = func() time.Time { return clock }
+ state := &edgesLoopState{sched: newEdgesFileSchedule(start, func() time.Duration { return 5*time.Minute + 30*time.Second }), claims: claims}
+ failUntil := start.Add(20 * time.Minute)
+ var during, after []time.Time
+ for clock = start; clock.Before(failUntil.Add(8 * time.Minute)); clock = clock.Add(edgesFileWatchInterval) {
+ if !state.due(clock, "fp", true) {
+ continue
+ }
+ claims.decide(context.Background(), hosts)
+ ok := !clock.Before(failUntil)
+ if ok {
+ after = append(after, clock)
+ } else {
+ during = append(during, clock)
+ }
+ state.done(clock, ok)
+ }
+ // Unthrottled, a claim pass every 30 seconds would be 40 attempts in
+ // 20 minutes. The backoff (2 s doubling to 5 min) plus the timer allows
+ // about a dozen.
+ t.Logf("%d attempts while failing, %d after", len(during), len(after))
+ if len(during) > 16 {
+ t.Fatalf("%d publication attempts while every write failed, want at most 16: %v", len(during), during)
+ }
+ // Recovery waits out the backoff (at most five minutes), then the
+ // 30-second claim passes resume.
+ if len(after) < 3 || after[len(after)-1].Sub(after[len(after)-2]) != edgesClaimPassEvery {
+ t.Fatalf("passes after writes recovered = %v, want the 30-second claim passes back within eight minutes", after)
+ }
+}
```
And create `internal/edge/homemap_alias_stamp_test.go`, which pins the edge's half: with stamped aliases, the successor's alias wins over a dead box's that sorts first. It passes on main already (`dialBeats` compares stamps after the alias rule); it is here so the reason for stamping aliases is tested where the decision is made.
@@ -3208,27 +3257,34 @@ func applyEdgesClaims(ctx context.Context, claims *edgesClaims, routes []registr
// edgesLoopState is edgesFileLoop's per-tick decision: the registry and
// timer schedule, plus, with region HA on, an extra pass every
-// edgesClaimPassEvery while claim reads are due.
+// edgesClaimPassEvery while claim reads are due. A claim pass is a
+// publication like any other: it waits out the schedule's retry backoff
+// after a failed write, and its own outcome moves that backoff, so a bucket
+// that refuses every write is not asked every 30 seconds (Astra plan R2-1).
type edgesLoopState struct {
sched *edgesFileSchedule
claims *edgesClaims
claimsNext time.Time
schedDue bool
+ claimDue bool
}
// due reports whether to run a pass on this tick.
func (s *edgesLoopState) due(now time.Time, fp string, fpOK bool) bool {
s.schedDue = s.sched.due(now, fp, fpOK)
- claimPass := s.claims != nil && !now.Before(s.claimsNext) && s.claims.backlog(now)
- return s.schedDue || claimPass
+ s.claimDue = s.claims != nil && !now.Before(s.claimsNext) && !now.Before(s.sched.retryAt) && s.claims.backlog(now)
+ return s.schedDue || s.claimDue
}
// timed reports whether the pass due just allowed is the timer's.
func (s *edgesLoopState) timed() bool { return s.schedDue && s.sched.timed }
-// done records the pass's outcome.
+// done records the pass's outcome. Every pass, claim passes included, goes
+// through the schedule: a failure advances its backoff, and a success
+// clears it and records the fingerprint the pass published. A claim pass
+// is never the timer's (due cleared timed), so the timer does not move.
func (s *edgesLoopState) done(now time.Time, ok bool) {
- if s.schedDue {
+ if s.schedDue || s.claimDue {
s.sched.done(now, ok)
}
if s.claims != nil {
@@ -3426,7 +3482,7 @@ Run:
go test ./internal/host/ -run 'EdgesFile|EdgesClaims|EdgesLoop' -count=1 -v
go test ./internal/edge/ -run 'TestHomeMap' -count=1
```
-Expected: PASS: the eight new host tests (including `TestEdgesFileStampsAliasesAndCatchAllsWithTheirPrimary`, `TestEdgesClaimsAFailingClaimDoesNotStarveTheRest` and `TestEdgesLoopRefreshesEveryClaimWithinTheTTL`), every existing edges-file test, and the home map tests. The `internal/edge` run of `TestHomeMap` alone needs no socket.
+Expected: PASS: the nine new host tests (including `TestEdgesFileStampsAliasesAndCatchAllsWithTheirPrimary`, `TestEdgesClaimsAFailingClaimDoesNotStarveTheRest`, `TestEdgesLoopRefreshesEveryClaimWithinTheTTL` and `TestEdgesLoopClaimPassesObeyTheRetryBackoff`), every existing edges-file test, and the home map tests. The `internal/edge` run of `TestHomeMap` alone needs no socket.
- [ ] **Step 6: Gate and commit**
@@ -4230,3 +4286,12 @@ Findings are in `CODEX-PLAN-REGION-HA-1B-R1.md` (reviewed at 856767d8). All seve
| 5 | P2 Task 6: 256 reads a pass at the timer's 5 to 6 minutes cannot refresh 2,000 hosts in 10 minutes | fixed: `edgesLoopState` adds a pass every 30 seconds while claim reads are due (flag on only); `TestEdgesLoopRefreshesEveryClaimWithinTheTTL` simulates the loop's schedule for three hours with 2,000 hosts and requires every claim re-read within 15 minutes (killed with the extra passes off) |
| 6 | P2 Task 5: incomplete, non-recordable or unknown-version records can authorize discharge | fixed: `Discharge` validates every stored record of the enrollment (version 1, a state that carries mappings, a non-empty mapping, every volume with an id and device); anything else keeps the chores open. `TestDischargeRefusesStoredRecordsThatAreNotEvidence` writes each bad record through the store and lists it back, placed so that no removal masks it; each case killed by removing its check |
| 7 | P2 Task 8: the policy test checks resources only | fixed: each statement is compared whole (effect, principal, actions, condition, resources); five mutations (Deny to Allow, a dropped delete action, the wrong condition key, a narrower principal, a wider prefix) and main's policy each fail it |
+
+## Astra plan round 2: dispositions
+
+Findings are in `CODEX-PLAN-REGION-HA-1B-R2.md` (reviewed at 161728ab). Round 2 marked R1-1 to R1-4 and R1-7 resolved and R1-5 and R1-6 partly resolved, and raised two findings, both checked and accepted. Tasks 5 and 6 were dry-run again from the plan's text, with every task applied in order to a clean copy of bf813540, built, vetted, cross-built and tested; each check below was removed in turn and its test watched to fail.
+
+| # | Finding | Disposition |
+|---|---|---|
+| R2-1 | P2 Task 6: claim passes bypass the publication retry backoff | fixed: a claim pass is gated on the schedule's `retryAt`, and its outcome goes through `edgesFileSchedule.done` (a failure advances the backoff, a success clears it; the timer does not move). `TestEdgesLoopClaimPassesObeyTheRetryBackoff` runs the loop's schedule against a bucket that fails for 20 minutes then recovers: 12 attempts while failing (40 unthrottled), and the 30-second passes back after recovery. It fails with the `retryAt` gate removed, with claim outcomes not recorded, and with claim passes off |
+| R2-2 | P2 Task 5: three validation checks survive removal | fixed: three more stored cases, each failing exactly one check: `running` with no volume (empty mapping), `terminated` with its root mapping (state), a volume with an id but no device (device). Removing each of the five checks (state, empty mapping, volume id, device, version) fails its own case |
diff --git a/docs/superpowers/plans/CODEX-PLAN-REGION-HA-1B-BRIEF-3.md b/docs/superpowers/plans/CODEX-PLAN-REGION-HA-1B-BRIEF-3.md
new file mode 100644
index 000000000..33d2ed145
--- /dev/null
+++ b/docs/superpowers/plans/CODEX-PLAN-REGION-HA-1B-BRIEF-3.md
@@ -0,0 +1,56 @@
+# Review brief: region HA plan 1b, round 3
+
+You are an independent second reader. Review the implementation plan
+`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md` at the
+commit named in the prompt, against the tree at the working directory
+(Go, module github.com/carlosframework/platform) and the approved spec
+`docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`
+(rev 5.4; rulings H1–H19 in §0). Your round 2 findings are in
+`CODEX-PLAN-REGION-HA-1B-R2.md`; the author's answers are the table
+"Astra plan round 2: dispositions" at the end of the plan.
+
+## Fixed: do not re-argue
+
+- The spec's rulings H1 to H19, and Paul's approval of rev 5.4.
+- The scope rule: plan 1b builds only what needs none of the spec's
+ L1–L20 claim/moves changes; everything under "Waits for the claim" is
+ out of scope.
+- The controller rulings in the plan's header (six, unchanged).
+- Everything stays dormant behind CARLOS_REGION_HA and changes nothing on
+ a one-box region.
+- Round 1 findings you marked resolved in round 2 (R1-1 to R1-4, R1-7),
+ and round 1 and 2 findings the dispositions mark fixed, unless the fix
+ itself is wrong; say which, and why.
+
+## What changed since round 2
+
+- Task 6, `edgesLoopState` (in `internal/host/edgesclaims.go`): a claim
+ pass now waits for the schedule's `retryAt`, and every pass, claim
+ passes included, records its outcome through `edgesFileSchedule.done`.
+ New test `TestEdgesLoopClaimPassesObeyTheRetryBackoff`.
+- Task 5, `TestDischargeRefusesStoredRecordsThatAreNotEvidence`: three new
+ stored cases, one per check (empty mapping, terminated state with a
+ mapping, a volume with no device).
+
+Nothing else changed.
+
+## What to check
+
+1. The round 2 table first: are R2-1 and R2-2 (and so R1-5 and R1-6)
+ resolved, partly resolved (say what is left), or not resolved?
+2. Then only new findings, in the two changed places or anything they
+ break. In particular: can recording a claim pass's success through
+ `edgesFileSchedule.done` advance the change-detection baseline past a
+ registry change that pass did not publish? Does the backoff gate
+ delay a timer pass or a change pass that would not have been delayed
+ before, with the flag off or on?
+3. As before: the code compiles, the tests pass as written, and each new
+ test fails with its mechanism removed.
+
+## Output
+
+Findings ranked P1 (would not compile, a test fails, or unsafe), P2
+(wrong or missing), P3 (clarity). Each: task and step, the problem, the
+evidence (file:line, or the plan's own line), and a concrete fix. Start
+with the round 2 table, then new findings only. End with a verdict:
+ready to build, or not ready.