plan: claim Plan 9 rev 3 (B8 record_seq fencing with an ABA regression)
Co-Authored-By: Codex <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+1278
−135
docs/superpowers/plans/2026-10-08-host-lifecycle-claim-9-activation-readiness.md+1278 −135
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-08-host-lifecycle-claim-9-activation-readiness.md b/docs/superpowers/plans/2026-10-08-host-lifecycle-claim-9-activation-readiness.md| index 4c8c5789c..72c6be666 100644 |
| --- a/docs/superpowers/plans/2026-10-08-host-lifecycle-claim-9-activation-readiness.md |
| +++ b/docs/superpowers/plans/2026-10-08-host-lifecycle-claim-9-activation-readiness.md |
| @@ -1,6 +1,8 @@ |
| # Host lifecycle claim, Plan 9: activation readiness |
| -Implementation plan, 2026-10-08, rev 2. Base: `origin/main` at **4f26f15cb41ecfdd9214b49393c228399f34490b**. The remote main ref was checked without fetching or writing git state. Spec: `docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`, rev **6.30**. Scope: amadan discussion **#250**, including its Plan 5, Plan 6b and integration additions. Plans 1 through 8b, 6b, and Plan 5 Tasks 1 through 9 are already on this base; do not replay them. |
| +Implementation plan, 2026-10-08, rev 3. Base: `origin/main` at **4f26f15cb41ecfdd9214b49393c228399f34490b**. The remote main ref was checked without fetching or writing git state. Spec: `docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`, rev **6.30**. Scope: amadan discussion **#250**, including its Plan 5, Plan 6b and integration additions. Plans 1 through 8b, 6b, and Plan 5 Tasks 1 through 9 are already on this base; do not replay them. |
| + |
| +**Rev 3 (2026-10-08).** Applied the B8 ruling: every claim-on record write advances `record_seq`; the resolver proves not-applied from that sequence or its own conditional sequence fence. B8 now includes complete code, passing fence and retry acceptance tests, and the ordinary-rewrite ABA regression. `record_seq` joins C1's release-floor inventory. The six Paul decisions remain open with their text unchanged, and all Group D signatures and executable blocks are unchanged. Base remains the latest remote `origin/main`, `4f26f15`. Verification is recorded at the end. |
| **Rev 2 (2026-10-08).** Rebased the executable blocks on `4f26f15`, which includes moves Plan 2. Round 1's two P2 findings are addressed: `bytes` enters `systemapply.go` in B2, and A1 preserves the first definite remote-floor refusal against a later offline error, with mixed-error regressions. B8 adds positive move-start recovery and records the failed fencing proof that still blocks not-applied settlement. Group D adds the agreed moves C1 integration contracts and D1-D5 tasks; its exported signatures appear first so the moves session can implement against them. The six Paul decisions from rev 1 remain open, with their options and recommendations unchanged. |
| @@ -18,7 +20,7 @@ The task groups retain the discussion's order. Every task states the authority o |
| Use a clean, later implementation checkout based on the pinned main or a reviewed descendant. If main moved, refresh the exact patches and repeat this document's replay checks; do not force a failed hunk. This authoring session performs no git writes. Implementation must later use the repository's reviewed amadan branch workflow, all required gates, and authority review before landing. |
| -Save the following as `$TMPDIR/claim9-apply.py` in the later implementation checkout. It applies one task and phase, without staging or committing. `B4` and `B7` contain only decision-gated tests; B8 also contains explicitly blocked fence-acceptance tests; do not apply them to a landing branch before Paul's ruling and the follow-on implementation. Apply test blocks first, run the task's focused command and record the expected failure, then apply implementation blocks and repeat. Patches use the exact pinned base or the preceding task's result; a failed patch means refresh the plan, not accept fuzz. |
| +Save the following as `$TMPDIR/claim9-apply.py` in the later implementation checkout. It applies one task and phase, without staging or committing. `B4` and `B7` contain only decision-gated tests; do not apply them to a landing branch before Paul's ruling and the follow-on implementation. B8's rev 3 sequence design is decided and its complete acceptance suite must pass. Apply test blocks first, run the task's focused command and record the expected failure, then apply implementation blocks and repeat. Patches use the exact pinned base or the preceding task's result; a failed patch means refresh the plan, not accept fuzz. |
| ```python |
| import pathlib |
| @@ -4634,33 +4636,38 @@ func TestPlan9JournalReplacementRemainsUntouched(t *testing.T) { |
| } |
| ``` |
| -### Task B8: resolve an uncertain move start (ResolveMoveStart, rev 2 addendum) |
| +### Task B8: resolve an uncertain move start (ResolveMoveStart, rev 3 record_seq ruling) |
| + |
| +Invariant: only the exact issuing handle may settle its move-start effect, through the effect runner's settlement path. An exact journal entry proves complete. With that entry absent, not-applied requires a current `record_seq` greater than the base sequence of every attempted Put. Neither age, a negative read nor a changed content-hash ETag proves that result. |
| + |
| +**Rev 3 ruling and complete implementation.** The rev 2 identical-body and whitespace fences were unsafe: a later ordinary serialization could reproduce the original bytes and ETag, admitting the delayed If-Match Put (ABA). Every claim-on record Put now advances a body counter owned by the conditional record store. A legacy record without `record_seq` has sequence zero; the first claim-on write emits one. Every subsequent write preserves the caller's read sequence and the store advances it by exactly one. Callers cannot remove, lower or choose the counter. A malformed counter or exhausted `uint64` refuses before the raw write. The fence preserves all other fields and conditionally advances the same counter. Its new sequence makes the failed Put's base bytes permanently unavailable to later claim-on rewrites. |
| -Invariant: the exact issuing handle may settle its move-start effect only through the effect runner's settlement path. An exact journal entry proves complete. Not-applied requires proof that every delayed record Put is fenced; neither age nor a negative read grants that proof. |
| +**Audit: every move-start attempt is conditional.** `StartClaimMoveRecord` calls `setLifecycleRecordMove`, then `provision.SetRecordMove`/`updateRecord`. Each attempt uses the record ETag it read, including the single retry after a CAS conflict. The mutation boundary and `conditionalRecordStore.putLifecycle` require and forward that nonempty ETag to the raw Put. The recorder associates every read ETag with its exact integer sequence and captures the base of every attempted Put before forwarding it. Resolution uses the highest captured base, not merely the first or last ETag. An unknown or empty If-Match value refuses. The production call path has no unconditional record write; it must retain the mutation boundary and conditional record store. |
| -**Audit: every write is conditional.** `StartClaimMoveRecord` calls `setLifecycleRecordMove`, then `provision.SetRecordMove`/`updateRecord`. Each attempt passes the record ETag it read, including the single retry after a CAS conflict. The mutation boundary and `conditionalRecordStore.putLifecycle` require and forward that same nonempty ETag to the raw Put. The production call path has no unconditional write. The recorder below captures every candidate If-Match ETag on the exact issuing handle. Production must keep using the mutation boundary or conditional record store, because that boundary calls `MarkAppMutationIssued`. |
| +**Writer and parser inventory.** `ConfigureClaimRuntime` installs the conditional record store only with `HostClaim` enabled and gives that data handle to the console and claim owners. All three provisioning-record arms of `conditionalRecordStore.Put` are covered: lifecycle writes, issued record/control-object effects, and ordinary updates. `PutFrom` buffers record bodies and delegates to that same Put. `putSequencedRecord` checks the current ETag again, validates the caller's preserved sequence, advances it and issues the original conditional Put. Create-only record effects start at one. Non-record keys pass through unchanged. The resolver's narrowly scoped raw fence uses the same sequence helper and the exact current ETag; it grants no generic raw-write or settlement API. |
| -**Rev 2 blocker: the requested fencing proof does not hold on main's stores.** `DirStore.Put` hashes the bytes; the S3 store also does not promise an ETag that changes on every write. An identical-body CAS can retain the failed Put's ETag and fences nothing. An unrelated write or a whitespace-only CAS can change the ETag temporarily, but an ordinary `provision.SetRecordRegion` with the same region serializes the original record again. The old ETag then returns and the delayed Put succeeds. Admission serialization does not help after the resolver settles and the door releases the claim. This is the ABA failure: bytes A become B, then become A again. |
| +`provision.Record` and `provision.SidecarRecord` retain the additive `RecordSeq uint64` field with `omitempty`. This covers instance `DecodeRecord`/Get/list/reconcile, sidecar Get/list/reconcile, hostops `ReadRecord`, ordinary typed updates and lifecycle serialization. Claim holder identity and incarnation do not include the counter. The sequence helper retains unknown JSON fields and parses integers without a float conversion; duplicate, aliased, null, fractional, negative and overflowing counter encodings refuse. Lifecycle callers receive the counter actually written. `Group C/C1` includes `record_seq` in the reader and writer floor inventory. Record deletion is not a sequence reset operation: the resolver refuses an absent or replaced incarnation, and existing retirement/recreation authority still requires a new incarnation before creating another record. An unresolved move registration continues to block retirement and recreation. |
| -The proposed newline fence was prototyped and rejected by an Opus authority review. The requested tests passed against that candidate, but `TestPlan9B8RefusesUnprovenContentFence` failed with `delayed Put survived acknowledged fence and ordinary rewrite: <nil>`. This is a correctness blocker, not a residual risk to accept. It applies both to the unchanged-ETag fence path and to the addendum's changed-ETag shortcut. The plan does not settle by signature age, a grace period or negative read-back. |
| +**Box writers and versioned writes.** The box moves pass captures `hostwrite.Config.Enabled` independently of the moves switch. With the claim flag on, it sets `provision.WithRecordSequence` on the pass context; all ordinary, phase-1, cut and landing record updates inherit it. `updateRecord` preserves the callback's base counter, increments before each conditional Put and returns that actual value; a CAS retry rereads and advances the new base. Console writes do not set this context because their conditional store owns advancement. `protectedFieldsEqual` prevents typed callbacks and whole-record updates from altering the counter. These changes add no Group D signature. |
| -The complete blocks below expose `ResolveMoveStart(context.Context, *StartHandle, hostclaim.Observed, string) (Outcome, error)` and the runner's shared settlement path. They resolve a landed Put whose response was lost to `OutcomeComplete`; wrong handles, wrong move IDs and read errors refuse; a record without the matching journal returns `OutcomeUncertain` with `ErrMoveStartFenceUnproven` and keeps its effect uncertain. An already completed pre-issue refusal can still report its recorded `OutcomeNotApplied`; no new uncertain effect is settled not-applied by these blocks. This is a safe partial implementation, not completion of B8. Do not land B8 as a delivered retry feature or remove the moves door's skip until the negative-resolution acceptance tests pass with a durable fence. |
| +The claim-on `versionedBoundary` refuses record `CopyVersion` destinations and `PutIfAbsent`, and refuses generic record `DeleteVersion`: any of these could restore older bytes without advancing the counter. Source reads retain their existing authorization. Only `DeletePurgeSweep` can supply the private retired-record deletion scope, under its existing exclusive purge authority. It rechecks current-record absence before every deletion and orders record data versions before delete markers, including replay of older journals. Both the sweep and boundary refuse a marker deletion while any data version remains, so a sweep cannot temporarily resurrect an old sequence. Tests exercise versioned calls with a real issued app permit, late live records, and an interleaved data/marker history whose record must stay absent throughout deletion. |
| -A complete solution needs a record revision or nonce that changes the conditional-write identity permanently and is preserved by every record writer, including ordinary updates, lifecycle updates, retries and raw-write guards. It must also bind the exact current move and incarnation. Inventory all record parsers and writers, certify their release floor, and test a normal write after settlement followed by the delayed Put. The identical-body addendum does not define that field or the writer-wide migration; those changes are not supplied by these blocks. The six existing Paul decisions remain exactly as framed; this is newly observed failed evidence for B8's proposed mechanism. |
| +**Resolver outcomes and races.** Read the same key and verify its holder/incarnation before interpreting evidence. The exact move ID and incarnation in its journal return complete. A matching terminal `LastMove` on that verified record incarnation also proves complete if a box finished before resolution. Otherwise, a sequence above every captured base returns not-applied without a fencing write. An equal sequence requires a CAS that advances it, after the creation-freeze check and a fresh matching claim ETag; an acknowledged fence permits not-applied. If the delayed Put wins that race, reread and return complete from its journal. Retry only CAS conflicts, at most three fence attempts. A lower sequence, missing capture, unreadable/replaced record, failed fence or exhausted retries returns uncertain with an error and leaves the effect held. A lost fence response is uncertain even if the fence landed; the same handle can later prove it from the newer sequence. A changed ETag at the same sequence still requires a real sequence fence. |
| -**Runner and handle scope.** `EffectRunner.saveSettlement` contains Issue's existing settle CAS, with the same owner check, uncancelled bounded save, retry and already-decided result. The new `resolveUncertain` hook is private and accepts only this registration's `move-start` effect in `uncertain` state. `ResolveMoveStart` is the minimal exported hook. A public generic settle method would let a caller assert a result without supplying its proof. The issuing handle records the move ID, incarnation, key and candidate Put ETags; another handle with the same registration tuple cannot resolve. Losing that in-memory handle leaves the effect uncertain. There is no new persisted field or floor change in these partial blocks. |
| +**Runner and handle scope.** `EffectRunner.saveSettlement` is Issue's existing settle CAS, with the same owner check, bounded uncancelled save, retry and already-decided result. Its private `resolveUncertain` hook accepts only this registration's uncertain `move-start` effect. `ResolveMoveStart(context.Context, *StartHandle, hostclaim.Observed, string) (Outcome, error)` remains the minimal exported hook. Another handle with the same registration tuple cannot resolve; losing the issuing handle leaves uncertainty. The slot contains an atomic pointer, so do not copy `StartHandle` by value. B3's added fields and B8's slot apply together. B4's eventual runner design must retain one settlement implementation. |
| -**Moves integration.** Reproduce `TestMoveDoorHostClaimRetryAfterAFailedRecordStart` from moves-phase1-2 `26d9e4d1` in hostops. Its retry-and-drain requirement and the other fence-dependent cases are retained below as `TestPlan9B8Acceptance*`; they compile and remain red until a durable fence is specified and implemented. Call the resolver before releasing the claim. On complete, use the recorded result. On uncertain, keep the registration and claim held and promise no retry. Only a proved not-applied result permits the existing grant-bound compensation and registration drain. Main's moves door is unchanged by this plan. |
| +**Moves integration and rollout.** The hostops acceptance tests reproduce moves-phase1-2 `26d9e4d1`'s `TestMoveDoorHostClaimRetryAfterAFailedRecordStart`: resolve before releasing the claim, use the exact journal on complete, and compensate and drain only on proved not-applied. Then a new registration/move ID can retry. Complete reports application of the record-start effect; the move's success, abort or rollback remains in its journal. On uncertain, retain the registration and claim and promise no retry. All `TestPlan9B8Acceptance*` cases and `TestPlan9B8RefusesUnprovenContentFence` must now pass; the ABA case performs a real ordinary `SetRecordRegion` through the claim-on record store after grant-bound release, restores the original business fields, and proves the delayed Put still conflicts. Wiring the moves door and removing its skip remains that session's integration step; this task supplies its completed resolver contract. |
| -**Flag-off and task overlap.** The move-start entry is used by the claim-mode moves door. Issue's shared settlement refactor preserves its original outcomes; ordinary flag-off paths do not acquire a resolver. B3 adds fields after `StartHandle.epoch`, while B8 adds its slot after `w`; both patches apply together. B4's eventual runner changes must retain one settlement implementation. The slot contains an atomic pointer, so do not copy StartHandle by value. |
| +Deploy the sequence-aware readers and every claim-on writer before enabling the resolver or claim mode, under Group C's existing approvals. Do not mix older writers that can strip the field into that rollout. Flag-off records at legacy zero retain main's exact bytes and emit no `record_seq`; the sequence store is never installed on that path. Populated records must remain readable without losing their counter. The six Paul decisions remain open and unchanged. No Group D interface or signature changes. |
| -Tests first: the new API names initially fail to compile. With the partial implementation, the safety and positive-resolution tests pass; the acceptance command must remain red and must not be presented as a green B8 gate. |
| +Tests first: apply the new test files before the implementation and record the missing API/field compile failure. Rev 2's four fence-dependent acceptance tests also fail against its partial resolver. After applying the complete blocks, run these commands serially. There are no excluded B8 acceptance cases. |
| ```bash |
| -nice -n 10 go test -p 1 ./internal/hostops -run '^TestPlan9B8(PutLanded|ReadError|Refuses|SettleHook|Identical)' -count=1 |
| -nice -n 10 go test -p 1 ./internal/hostops -run '^TestPlan9B8Acceptance' -count=1 |
| -nice -n 10 go test -p 1 ./internal/hostops -run 'ClaimMove|StartClaimMoveRecord|MoveRecovery|Effect|AppMutation|RecoverMutation|Restore|Mail|Registration' -count=1 |
| -nice -n 10 go vet ./internal/hostops |
| +nice -n 10 go test -p 1 ./internal/hostops ./internal/provision ./internal/host -run '^TestPlan9B8' -count=1 |
| +nice -n 10 go test -p 1 ./internal/hostops ./internal/provision -count=1 |
| +nice -n 10 go test -p 1 ./internal/console -run 'ClaimAuthority|MoveDoorHostClaimFailedRecordStart' -count=1 |
| +nice -n 10 go test -p 1 ./internal/host -run 'MovesPass|Phase1|Cut|Landing' -count=1 |
| +nice -n 10 go vet ./internal/hostops ./internal/provision ./internal/host |
| ``` |
| #### New file: `internal/hostops/plan9_b8_test.go` |
| @@ -4845,8 +4852,12 @@ func TestPlan9B8AcceptancePutLostBeforeLandingIsFencedNotApplied(t *testing.T) { |
| if tag == r.tag { |
| t.Fatal("the fence left the ETag the start Put matched on") |
| } |
| - if !bytes.Equal(body, append(append([]byte(nil), r.before...), '\n')) { |
| - t.Fatalf("the fence changed the record value:\n%s", body) |
| + before, _, _ := recordSequence(r.before) |
| + after, seq, parseErr := recordSequence(body) |
| + delete(before, "record_seq") |
| + delete(after, "record_seq") |
| + if parseErr != nil || seq != 1 || !maps.EqualFunc(before, after, func(a, b json.RawMessage) bool { return bytes.Equal(a, b) }) { |
| + t.Fatalf("fence did not preserve the record and advance sequence: %s %v", body, parseErr) |
| } |
| if rec, err := provision.DecodeRecord(body); err != nil || rec.Move != nil { |
| t.Fatalf("fenced record: %+v %v", rec.Move, err) |
| @@ -4925,6 +4936,7 @@ func TestPlan9B8AcceptanceUnrelatedWriteIsAlreadyAFence(t *testing.T) { |
| t.Fatal(err) |
| } |
| rec.Channel = "beta" |
| + rec.RecordSeq = 1 // An unrelated claim-on writer advanced the counter. |
| body, err := json.Marshal(rec) |
| if err != nil { |
| t.Fatal(err) |
| @@ -4965,8 +4977,8 @@ func TestPlan9B8ReadErrorIsUncertain(t *testing.T) { |
| t.Fatalf("effect = %+v, want uncertain", e) |
| } |
| r.control.failGet = false |
| - if got, err = ResolveMoveStart(ctx, r.h, r.moving, moveStartID); !errors.Is(err, ErrMoveStartFenceUnproven) || got != OutcomeUncertain { |
| - t.Fatalf("unproved negative after read recovery: %q, %v", got, err) |
| + if got, err = ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err != nil || got != OutcomeNotApplied { |
| + t.Fatalf("resolution after read recovery: %q, %v", got, err) |
| } |
| } |
| @@ -5007,8 +5019,8 @@ func TestPlan9B8RefusesAnotherHandleOrMove(t *testing.T) { |
| if e := r.effect(t); e.State != "uncertain" { |
| t.Fatalf("a refused resolution settled the effect: %+v", e) |
| } |
| - if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); !errors.Is(err, ErrMoveStartFenceUnproven) || got != OutcomeUncertain { |
| - t.Fatalf("issuing handle still needs a proved fence: %q, %v", got, err) |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err != nil || got != OutcomeNotApplied { |
| + t.Fatalf("issuing handle resolution: %q, %v", got, err) |
| } |
| } |
| @@ -5059,100 +5071,884 @@ func TestPlan9B8AcceptanceNotAppliedDrainsAndARetryStarts(t *testing.T) { |
| if err := hostclaim.RequireMoveOp(moveStartID, hostclaim.KindMoving, hostclaim.PhaseMoving)(c); err != nil { |
| return err |
| } |
| - return hostclaim.ReleaseMove(moveStartID, "ie-box")(c) |
| - } |
| - if _, err := hostclaim.Update(r.h.Lifecycle(ctx), r.data, r.moving, releaseMove); err != nil { |
| - t.Fatalf("the door's claim release: %v", err) |
| - } |
| - if err := r.w.SettleRegistration(ctx, "bab", "app", "canonical", moveStartID, r.now.Add(RegistrationTTL+2*RegistrationSkew)); err != nil { |
| - t.Fatalf("the failed attempt's registration does not drain: %v", err) |
| + return hostclaim.ReleaseMove(moveStartID, "ie-box")(c) |
| + } |
| + if _, err := hostclaim.Update(r.h.Lifecycle(ctx), r.data, r.moving, releaseMove); err != nil { |
| + t.Fatalf("the door's claim release: %v", err) |
| + } |
| + if err := r.w.SettleRegistration(ctx, "bab", "app", "canonical", moveStartID, r.now.Add(RegistrationTTL+2*RegistrationSkew)); err != nil { |
| + t.Fatalf("the failed attempt's registration does not drain: %v", err) |
| + } |
| + a, err := ReadAdmission(ctx, r.raw.Store, "bab", "app", "canonical") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, ok := a.Value.Registrations[moveStartID]; ok { |
| + t.Fatal("the failed attempt's registration is still held") |
| + } |
| + retry := "mv_" + strings.Repeat("b", 26) |
| + o := r.read(t) |
| + h, err := r.authority.RegisterMove(ctx, o, "canonical", retry) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + moving, err := BeginClaimMove(ctx, r.data, r.authority.Directory, h, o, retry, "operator", "uk-box", "uk", r.now) |
| + if err != nil { |
| + t.Fatalf("retry start CAS: %v", err) |
| + } |
| + start := r.start |
| + start.ID = retry |
| + rec, err := StartClaimMoveRecord(ctx, r.data, h, moving, start) |
| + if err != nil || rec.Move == nil || rec.Move.ID != retry || rec.Move.State != provision.MovePreparing { |
| + t.Fatalf("retry record start = %+v, %v", rec, err) |
| + } |
| + if a, err = ReadAdmission(ctx, r.raw.Store, "bab", "app", "canonical"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if e := a.Value.Effects[retry]; e.State != "complete" || e.ResultReference != retry { |
| + t.Fatalf("retry effect = %+v", e) |
| + } |
| + if e := a.Value.Effects[moveStartID]; e.State != "complete" || e.ResultReference != "not-applied" { |
| + t.Fatalf("failed attempt's effect = %+v", e) |
| + } |
| +} |
| + |
| +// Why the fence rewrites the value with different bytes: the DirStore's |
| +// ETag, like a single-part S3 ETag, is a content hash, so a byte-identical |
| +// rewrite keeps the ETag a late Put matched on. |
| +func TestPlan9B8IdenticalBytesKeepAContentETag(t *testing.T) { |
| + ctx := context.Background() |
| + s := release.NewDirStore(t.TempDir()) |
| + key := provision.RecordKey("bab", "app", "shop.example.com") |
| + tag, err := s.Put(ctx, key, []byte(`{"schema":1}`), "") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + same, err := s.Put(ctx, key, []byte(`{"schema":1}`), tag) |
| + if err != nil || same != tag { |
| + t.Fatalf("identical rewrite: %q, %v (was %q)", same, err, tag) |
| + } |
| + moved, err := s.Put(ctx, key, []byte("{\"schema\":1}\n"), same) |
| + if err != nil || moved == same { |
| + t.Fatalf("newline rewrite: %q, %v", moved, err) |
| + } |
| +} |
| + |
| +// A changed ETag without a sequence increment is not proof. The resolver |
| +// must perform its own sequence CAS before acknowledging not-applied. |
| +func TestPlan9B8RefusesUnprovenContentFence(t *testing.T) { |
| + ctx := context.Background() |
| + r := newResolveFixture(t, false) |
| + if _, err := r.raw.Store.Put(ctx, r.raw.recordKey, append(r.before, '\n'), r.tag); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err != nil || got != OutcomeNotApplied || r.control.fences != 1 { |
| + t.Fatalf("content-only change was accepted without a sequence fence: %q %v fences=%d", got, err, r.control.fences) |
| + } |
| + r.ordinaryRewrite(t) |
| + if err := r.latePut(); !errors.Is(err, release.ErrCASConflict) { |
| + t.Fatalf("late Put: %v", err) |
| + } |
| +} |
| + |
| +// ordinaryRewrite releases the claim with the existing grant, then exercises |
| +// the real ordinary record writer and claim-on conditional store. Its |
| +// attempted semantic value is exactly the original record's value. |
| +func (r *resolveFixture) ordinaryRewrite(t *testing.T) { |
| + t.Helper() |
| + ctx := context.Background() |
| + if _, err := hostclaim.Update(r.h.Lifecycle(ctx), r.data, r.moving, hostclaim.ReleaseMove(moveStartID, "ie-box")); err != nil { |
| + t.Fatal(err) |
| + } |
| + records, err := NewConditionalRecordStore(r.raw, r.pub) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err = provision.SetRecordRegion(ctx, records, "bab", "app", r.host, "ie"); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| + |
| +func TestPlan9B8AcceptanceOrdinaryRewriteCannotReopenFenceABA(t *testing.T) { |
| + ctx := context.Background() |
| + r := newResolveFixture(t, false) |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err != nil || got != OutcomeNotApplied { |
| + t.Fatalf("resolve: %q %v", got, err) |
| + } |
| + r.ordinaryRewrite(t) |
| + body, tag := r.record(t) |
| + fields, seq, err := recordSequence(body) |
| + original, _, _ := recordSequence(r.before) |
| + delete(fields, "record_seq") |
| + delete(original, "record_seq") |
| + if err != nil || seq != 2 || tag == r.tag || !maps.EqualFunc(fields, original, func(a, b json.RawMessage) bool { return bytes.Equal(a, b) }) { |
| + t.Fatalf("ordinary rewrite restored conditional identity: seq=%d tag=%q body=%s err=%v", seq, tag, body, err) |
| + } |
| + if err := r.latePut(); !errors.Is(err, release.ErrCASConflict) { |
| + t.Fatalf("delayed Put survived fence and ordinary rewrite: %v", err) |
| + } |
| +} |
| +``` |
| + |
| +#### New file: `internal/hostops/plan9_b8_sequence_test.go` |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "fmt" |
| + "strings" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/provision" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func TestPlan9B8SequenceEveryRecordWriter(t *testing.T) { |
| + ctx := context.Background() |
| + f := newMoveStartFixture(t, nil) |
| + records, err := NewConditionalRecordStore(f.raw, f.pub) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + key := f.raw.recordKey |
| + // Ordinary typed writer: old absent sequence becomes one and then two. |
| + for want := uint64(1); want <= 2; want++ { |
| + if err := provision.SetRecordRegion(ctx, records, "bab", "app", f.host, "ie"); err != nil { |
| + t.Fatal(err) |
| + } |
| + body, _, _ := records.Get(ctx, key) |
| + rec, err := provision.DecodeRecord(body) |
| + if err != nil || rec.RecordSeq != want { |
| + t.Fatalf("ordinary: %+v %v", rec, err) |
| + } |
| + } |
| + // Both effect branches also sequence their writes, including PutFrom. |
| + for _, resource := range []string{key, "control-object:" + key} { |
| + body, tag, _ := records.Get(ctx, key) |
| + _, seq, _ := recordSequence(body) |
| + permitted := context.WithValue(ctx, effectPermitKey{}, effectPermit{Resource: resource}) |
| + if _, err := release.PutObjectFrom(permitted, records, key, bytes.NewReader(body), int64(len(body)), tag); err != nil { |
| + t.Fatal(err) |
| + } |
| + next, _, _ := records.Get(ctx, key) |
| + _, got, _ := recordSequence(next) |
| + if got != seq+1 { |
| + t.Fatalf("effect %s: seq %d -> %d", resource, seq, got) |
| + } |
| + } |
| + // Lifecycle updates keep the same counter after the first journal Put. |
| + r := newResolveFixture(t, true) |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err != nil || got != OutcomeComplete { |
| + t.Fatalf("resolve: %q %v", got, err) |
| + } |
| + lifecycle, err := NewConditionalRecordStore(r.raw, r.pub) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if rec, err := setLifecycleRecordMove(ctx, lifecycle, r.moving, func(rec *provision.Record) error { return nil }); err != nil || rec.RecordSeq != 2 { |
| + t.Fatalf("returned lifecycle record: %+v %v", rec, err) |
| + } |
| + body, _ := r.record(t) |
| + _, seq, err := recordSequence(body) |
| + if err != nil || seq != 2 { |
| + t.Fatalf("lifecycle seq=%d err=%v", seq, err) |
| + } |
| + // Sidecar create and typed decode/rewrite preserve the counter too. |
| + side := provision.SidecarRecord{Schema: 1, Account: "bab", App: "app", Name: "worker", Host: provision.SidecarHost("worker", "app", "bab"), CreatedAt: f.now} |
| + key = provision.SidecarRecordKey("bab", "app", "worker") |
| + permitted := context.WithValue(ctx, effectPermitKey{}, effectPermit{Resource: key}) |
| + body, _ = json.Marshal(side) |
| + tag, err := records.Put(permitted, key, body, "") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + body, _, _ = records.Get(ctx, key) |
| + if err := json.Unmarshal(body, &side); err != nil || side.RecordSeq != 1 { |
| + t.Fatalf("sidecar: %+v %v", side, err) |
| + } |
| + body, _ = json.Marshal(side) |
| + if _, err := records.Put(permitted, key, body, tag); err != nil { |
| + t.Fatal(err) |
| + } |
| + body, _, _ = records.Get(ctx, key) |
| + _, seq, err = recordSequence(body) |
| + if err != nil || seq != 2 { |
| + t.Fatalf("sidecar seq=%d %v", seq, err) |
| + } |
| +} |
| + |
| +func TestPlan9B8SequenceCannotBeRemovedLoweredOrChosen(t *testing.T) { |
| + for _, mode := range []string{"ordinary", "effect", "control-object", "stream"} { |
| + for _, counter := range []string{"", `,"record_seq":0`, `,"record_seq":1`, `,"record_seq":3`, `,"record_seq":null`, `,"record_seq":-1`, `,"record_seq":1.5`, `,"record_seq":18446744073709551616`, `,"Record_Seq":2`, `,"record_seq":2,"record_seq":2`} { |
| + t.Run(mode+counter, func(t *testing.T) { |
| + f := newMoveStartFixture(t, nil) |
| + ctx := context.Background() |
| + body, tag, _ := f.raw.Get(ctx, f.raw.recordKey) |
| + var fields map[string]json.RawMessage |
| + _ = json.Unmarshal(body, &fields) |
| + fields["record_seq"] = json.RawMessage(`2`) |
| + before, _ := json.Marshal(fields) |
| + tag, err := f.raw.Put(ctx, f.raw.recordKey, before, tag) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + delete(fields, "record_seq") |
| + body, _ = json.Marshal(fields) |
| + candidate := append(append([]byte(nil), body[:len(body)-1]...), []byte(counter+"}")...) |
| + records, err := NewConditionalRecordStore(f.raw, f.pub) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + switch mode { |
| + case "effect", "stream": |
| + ctx = context.WithValue(ctx, effectPermitKey{}, effectPermit{Resource: f.raw.recordKey}) |
| + case "control-object": |
| + ctx = context.WithValue(ctx, effectPermitKey{}, effectPermit{Resource: "control-object:" + f.raw.recordKey}) |
| + } |
| + if mode == "stream" { |
| + _, err = release.PutObjectFrom(ctx, records, f.raw.recordKey, bytes.NewReader(candidate), int64(len(candidate)), tag) |
| + } else { |
| + _, err = records.Put(ctx, f.raw.recordKey, candidate, tag) |
| + } |
| + if !errors.Is(err, ErrIdentity) { |
| + t.Fatalf("counter accepted: %s err=%v", candidate, err) |
| + } |
| + after, afterTag, _ := f.raw.Get(ctx, f.raw.recordKey) |
| + if !bytes.Equal(before, after) || afterTag != tag { |
| + t.Fatal("refusal changed record") |
| + } |
| + }) |
| + } |
| + } |
| +} |
| + |
| +func TestPlan9B8SequencePrecisionOverflowAndUnknownFields(t *testing.T) { |
| + const high = uint64(1<<53) + 1 |
| + before := []byte(fmt.Sprintf(`{"record_seq":%d,"future":{"value":9007199254740993}}`, high)) |
| + body, err := advanceRecordSequence(before, before) |
| + fields, seq, parseErr := recordSequence(body) |
| + if err != nil || parseErr != nil || seq != high+1 || string(fields["future"]) != `{"value":9007199254740993}` { |
| + t.Fatalf("precision: %s %v %v", body, err, parseErr) |
| + } |
| + max := []byte(`{"record_seq":18446744073709551615}`) |
| + if _, err := advanceRecordSequence(max, max); !errors.Is(err, ErrPending) { |
| + t.Fatalf("overflow: %v", err) |
| + } |
| +} |
| + |
| +func TestPlan9B8FlagOffBytesAndNonRecordBytesUnchanged(t *testing.T) { |
| + f := newMoveStartFixture(t, nil) |
| + ctx := context.Background() |
| + before, tag, _ := f.raw.Get(ctx, f.raw.recordKey) |
| + off, err := NewMutationBoundary(false, f.raw, nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := provision.SetRecordRegion(ctx, off, "bab", "app", f.host, "ie"); err != nil { |
| + t.Fatal(err) |
| + } |
| + after, got, _ := f.raw.Get(ctx, f.raw.recordKey) |
| + if !bytes.Equal(before, after) || got != tag || bytes.Contains(after, []byte(`"record_seq"`)) { |
| + t.Fatalf("flag-off changed: %s", after) |
| + } |
| + records, err := NewConditionalRecordStore(f.raw, f.pub) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + key := "apps/bab/app/metadata.json" |
| + body := []byte("{\"value\": 1}\n") |
| + if _, err := records.Put(ctx, key, body, ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + after, _, _ = records.Get(ctx, key) |
| + if !bytes.Equal(body, after) { |
| + t.Fatal("non-record bytes changed") |
| + } |
| + side, _ := json.Marshal(provision.SidecarRecord{Name: "worker"}) |
| + if strings.Contains(string(side), "record_seq") { |
| + t.Fatalf("legacy sidecar: %s", side) |
| + } |
| +} |
| + |
| +func TestPlan9B8ResolverRefusesMissingReplacedOrMalformedRecord(t *testing.T) { |
| + for _, mode := range []string{"missing", "incarnation", "null-sequence", "lower-sequence", "uncaptured", "exhausted"} { |
| + t.Run(mode, func(t *testing.T) { |
| + ctx := context.Background() |
| + r := newResolveFixture(t, false) |
| + body, tag := r.record(t) |
| + fields, _, _ := recordSequence(body) |
| + switch mode { |
| + case "missing": |
| + if err := r.raw.Store.Delete(ctx, r.raw.recordKey); err != nil { |
| + t.Fatal(err) |
| + } |
| + case "incarnation": |
| + fields["created_at"] = json.RawMessage(`"2026-10-08T00:00:00Z"`) |
| + case "null-sequence": |
| + fields["record_seq"] = json.RawMessage(`null`) |
| + case "lower-sequence": |
| + r.h.moveStart.puts.Load().bases = []uint64{1} |
| + case "uncaptured": |
| + r.h.moveStart.puts.Load().bases = nil |
| + case "exhausted": |
| + fields["record_seq"] = json.RawMessage(`18446744073709551615`) |
| + r.h.moveStart.puts.Load().bases = []uint64{^uint64(0)} |
| + } |
| + if mode == "incarnation" || mode == "null-sequence" || mode == "exhausted" { |
| + next, _ := json.Marshal(fields) |
| + if _, err := r.raw.Store.Put(ctx, r.raw.recordKey, next, tag); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err == nil || got != OutcomeUncertain { |
| + t.Fatalf("refusal: %q %v", got, err) |
| + } |
| + if r.effect(t).State != "uncertain" || r.control.fences != 0 { |
| + t.Fatal("unproven resolution wrote or settled") |
| + } |
| + }) |
| + } |
| +} |
| + |
| +type sequenceFenceFailure struct { |
| + release.Store |
| + key string |
| + land bool |
| + conflicts bool |
| + calls int |
| +} |
| + |
| +func (s *sequenceFenceFailure) Put(ctx context.Context, key string, body []byte, tag string) (string, error) { |
| + if key != s.key { |
| + return s.Store.Put(ctx, key, body, tag) |
| + } |
| + s.calls++ |
| + if s.conflicts { |
| + return "", release.ErrCASConflict |
| + } |
| + if s.land { |
| + if _, err := s.Store.Put(ctx, key, body, tag); err != nil { |
| + return "", err |
| + } |
| + } |
| + return "", errors.New("fence answer lost") |
| +} |
| +func TestPlan9B8FenceFailureRemainsUncertainAndCanRecover(t *testing.T) { |
| + for _, land := range []bool{false, true} { |
| + t.Run(fmt.Sprint(land), func(t *testing.T) { |
| + ctx := context.Background() |
| + r := newResolveFixture(t, false) |
| + failure := &sequenceFenceFailure{Store: r.control, key: r.raw.recordKey, land: land} |
| + r.h.w.raw = failure |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err == nil || got != OutcomeUncertain { |
| + t.Fatalf("lost fence: %q %v", got, err) |
| + } |
| + if r.effect(t).State != "uncertain" { |
| + t.Fatal("lost fence settled") |
| + } |
| + r.h.w.raw = r.control |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err != nil || got != OutcomeNotApplied { |
| + t.Fatalf("retry: %q %v", got, err) |
| + } |
| + if err := r.latePut(); !errors.Is(err, release.ErrCASConflict) { |
| + t.Fatalf("late Put: %v", err) |
| + } |
| + }) |
| + } |
| + r := newResolveFixture(t, false) |
| + failure := &sequenceFenceFailure{Store: r.control, key: r.raw.recordKey, conflicts: true} |
| + r.h.w.raw = failure |
| + if got, err := ResolveMoveStart(context.Background(), r.h, r.moving, moveStartID); err == nil || got != OutcomeUncertain || failure.calls != 3 { |
| + t.Fatalf("bounded conflicts: %q %v calls=%d", got, err, failure.calls) |
| + } |
| + if r.effect(t).State != "uncertain" { |
| + t.Fatal("conflicts settled") |
| + } |
| +} |
| + |
| +func TestPlan9B8CapturesEveryCASAttemptBase(t *testing.T) { |
| + ctx := context.Background() |
| + r := newResolveFixture(t, false) |
| + p := r.h.moveStart.puts.Load() |
| + // Model updateRecord's retry reading another revision before its own Put. |
| + body, tag := r.record(t) |
| + next, err := advanceRecordSequence(body, body) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := r.raw.Store.Put(ctx, r.raw.recordKey, next, tag); err != nil { |
| + t.Fatal(err) |
| + } |
| + failed := &lostRecordPut{Store: r.raw, recordKey: r.raw.recordKey, armed: true} |
| + recorder := moveStartRecorder{s: failed, puts: p} |
| + current, etag, err := recorder.Get(ctx, r.raw.recordKey) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := recorder.Put(ctx, r.raw.recordKey, current, etag); err == nil { |
| + t.Fatal("lost retry succeeded") |
| + } |
| + if high, ok := p.highestBase(); !ok || high != 1 || len(p.bases) != 2 { |
| + t.Fatalf("bases: %v", p.bases) |
| + } |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err != nil || got != OutcomeNotApplied || r.control.fences != 1 { |
| + t.Fatalf("retry fence: %q %v fences=%d", got, err, r.control.fences) |
| + } |
| + if _, err := r.raw.Store.Put(ctx, r.raw.recordKey, failed.body, failed.etag); !errors.Is(err, release.ErrCASConflict) { |
| + t.Fatalf("late retry: %v", err) |
| + } |
| +} |
| + |
| +func TestPlan9B8TerminalJournalStillProvesComplete(t *testing.T) { |
| + for _, outcome := range []string{"done", "rolled_back", "aborted", "different-id"} { |
| + t.Run(outcome, func(t *testing.T) { |
| + ctx := context.Background() |
| + r := newResolveFixture(t, outcome != "different-id") |
| + id := moveStartID |
| + want := OutcomeComplete |
| + if outcome == "different-id" { |
| + id = "mv_" + strings.Repeat("b", 26) |
| + want = OutcomeNotApplied |
| + } |
| + // Complete describes the record-start effect, including a move that |
| + // subsequently aborted or rolled back. The journal retains its outcome. |
| + _, err := provision.SetRecordMove(provision.WithRecordSequence(ctx), r.raw.Store, "bab", "app", r.host, func(rec *provision.Record) error { |
| + rec.LastMove = &provision.MoveOutcome{ID: id, Outcome: outcome, At: r.now} |
| + rec.Move = nil |
| + return nil |
| + }) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err != nil || got != want { |
| + t.Fatalf("terminal evidence: %q %v want %q", got, err, want) |
| + } |
| + result := moveStartID |
| + if want == OutcomeNotApplied { |
| + result = "not-applied" |
| + } |
| + if r.control.fences != 0 || r.effect(t).ResultReference != result { |
| + t.Fatal("wrong terminal resolution") |
| + } |
| + }) |
| + } |
| +} |
| +``` |
| + |
| +#### New file: `internal/hostops/plan9_b8_versions_test.go` |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| + "github.com/carlosframework/platform/internal/lease" |
| + "github.com/carlosframework/platform/internal/lease/leasetest" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func TestPlan9B8VersionedRecordWritesCannotReopenSequence(t *testing.T) { |
| + r := newResolveFixture(t, false) |
| + ctx := context.Background() |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); err != nil || got != OutcomeNotApplied { |
| + t.Fatalf("resolve: %q %v", got, err) |
| + } |
| + versions := leasetest.NewVersionedBucket() |
| + versions.AddBucket("control") |
| + old := versions.Put("control", r.raw.recordKey, r.before) |
| + guarded, err := NewVersionedBoundary(true, versions, r.raw, r.w, "control", nil, nil) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + issued, settle, err := r.h.Runner(r.w.now).Issue(ctx, Effect{ID: "versions", Resource: r.raw.recordKey, Action: "test-versions"}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer func() { _ = settle("", ErrNotApplied) }() |
| + if _, err = guarded.PutIfAbsent(issued, "control", r.raw.recordKey, r.before); !errors.Is(err, ErrContract) { |
| + t.Fatalf("PutIfAbsent: %v", err) |
| + } |
| + if _, _, err = guarded.CopyVersion(issued, "control", r.raw.recordKey, old, "control", r.raw.recordKey, false); !errors.Is(err, ErrContract) { |
| + t.Fatalf("CopyVersion: %v", err) |
| + } |
| + if err = guarded.DeleteVersion(issued, "control", r.raw.recordKey, old); !errors.Is(err, ErrContract) { |
| + t.Fatalf("DeleteVersion: %v", err) |
| + } |
| + retained, err := versions.GetVersion(ctx, "control", r.raw.recordKey, old) |
| + if err != nil || string(retained) != string(r.before) { |
| + t.Fatalf("old version changed: %v", err) |
| + } |
| + // Even the private sweep scope cannot delete a live record's version. |
| + sweep := context.WithValue(issued, retiredRecordSweepKey{}, r.raw.recordKey) |
| + if err = guarded.DeleteVersion(sweep, "control", r.raw.recordKey, old); !errors.Is(err, ErrPending) { |
| + t.Fatalf("live sweep: %v", err) |
| + } |
| +} |
| + |
| +func TestPlan9B8PurgeSweepRechecksRetiredRecord(t *testing.T) { |
| + ctx := context.Background() |
| + control := release.NewDirStore(t.TempDir()) |
| + versions := leasetest.NewVersionedBucket() |
| + versions.AddBucket("control") |
| + key := "apps/bab/demo/instances/shop.example.com.json" |
| + old := versions.Put("control", key, []byte(`{"record_seq":1}`)) |
| + journal := PurgeJournal{V: 1, Purge: Purge{ID: "p", Phase: "retired"}, AppID: "old", Account: "bab", App: "demo", Hosts: map[string]PurgeHost{}} |
| + w := seedPurge(t, ctx, control, time.Now(), journal) |
| + if err := PreparePurgeSweep(ctx, control, w, versions, "control", "bab", "demo", "p", "old"); err != nil { |
| + t.Fatal(err) |
| + } |
| + tag, err := control.Put(ctx, key, []byte(`{"record_seq":2}`), "") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := DeletePurgeSweep(ctx, control, w, versions, "control", "bab", "demo", "p", "old"); !errors.Is(err, ErrPending) { |
| + t.Fatalf("late live record: %v", err) |
| + } |
| + if _, err := versions.GetVersion(ctx, "control", key, old); err != nil { |
| + t.Fatalf("live version removed: %v", err) |
| + } |
| + if err := release.DeleteIfMatch(ctx, control, key, tag); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := DeletePurgeSweep(ctx, control, w, versions, "control", "bab", "demo", "p", "old"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := versions.GetVersion(ctx, "control", key, old); !errors.Is(err, lease.ErrMirrorNotFound) { |
| + t.Fatalf("retired version not swept: %v", err) |
| + } |
| +} |
| + |
| +func TestPlan9B8FenceRefusesChangedClaim(t *testing.T) { |
| + r := newResolveFixture(t, false) |
| + ctx := context.Background() |
| + if _, err := hostclaim.Update(r.h.Lifecycle(ctx), r.data, r.moving, hostclaim.ReleaseMove(moveStartID, "ie-box")); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID); got != OutcomeUncertain || !errors.Is(err, hostclaim.ErrConflict) { |
| + t.Fatalf("stale fence: %q %v", got, err) |
| + } |
| + if r.control.fences != 0 || r.effect(t).State != "uncertain" { |
| + t.Fatal("stale observation fenced or settled") |
| + } |
| +} |
| + |
| +type sweepWithoutResurrection struct { |
| + *leasetest.VersionedBucket |
| + t *testing.T |
| + key string |
| +} |
| + |
| +func (v sweepWithoutResurrection) DeleteVersion(ctx context.Context, bucket, key, id string) error { |
| + if err := v.VersionedBucket.DeleteVersion(ctx, bucket, key, id); err != nil { |
| + return err |
| + } |
| + if _, live := v.Current(bucket, v.key); live { |
| + v.t.Fatal("sweep resurrected a record body") |
| + } |
| + return nil |
| +} |
| +func TestPlan9B8PurgeRemovesRecordMarkersLast(t *testing.T) { |
| + ctx := context.Background() |
| + control := release.NewDirStore(t.TempDir()) |
| + versions := leasetest.NewVersionedBucket() |
| + versions.AddBucket("control") |
| + key := "apps/bab/demo/instances/shop.example.com.json" |
| + versions.Put("control", key, []byte(`{"record_seq":1}`)) |
| + versions.BareDelete("control", key) |
| + versions.Put("control", key, []byte(`{"record_seq":2}`)) |
| + marker := versions.BareDelete("control", key) |
| + if err := retiredRecordVersion(ctx, control, versions, "control", key, marker); !errors.Is(err, ErrPending) { |
| + t.Fatalf("marker could expose old body: %v", err) |
| + } |
| + journal := PurgeJournal{V: 1, Purge: Purge{ID: "p", Phase: "retired"}, AppID: "old", Account: "bab", App: "demo", Hosts: map[string]PurgeHost{}} |
| + w := seedPurge(t, ctx, control, time.Now(), journal) |
| + guarded := sweepWithoutResurrection{versions, t, key} |
| + if err := PreparePurgeSweep(ctx, control, w, guarded, "control", "bab", "demo", "p", "old"); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := DeletePurgeSweep(ctx, control, w, guarded, "control", "bab", "demo", "p", "old"); err != nil { |
| + t.Fatal(err) |
| + } |
| + all, err := versions.ListVersions(ctx, "control", key) |
| + if err != nil || len(all) != 0 { |
| + t.Fatalf("retained versions: %+v %v", all, err) |
| + } |
| +} |
| +``` |
| + |
| +#### New file: `internal/provision/plan9_b8_sequence_test.go` |
| + |
| +```go |
| +package provision |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +func TestPlan9B8BoxRecordSequenceAndLegacyBytes(t *testing.T) { |
| + s := release.NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + r := acmeRecord("bdm") |
| + if err := CreateRecord(ctx, s, r); err != nil { |
| + t.Fatal(err) |
| + } |
| + before, tag, err := s.Get(ctx, RecordKey(r.Account, r.App, r.Host)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := SetRecordRegion(ctx, s, r.Account, r.App, r.Host, r.Region); err != nil { |
| + t.Fatal(err) |
| + } |
| + unchanged, same, err := s.Get(ctx, RecordKey(r.Account, r.App, r.Host)) |
| + if err != nil || string(before) != string(unchanged) || tag != same { |
| + t.Fatalf("legacy bytes changed: %v", err) |
| + } |
| + for want := uint64(1); want <= 2; want++ { |
| + got, err := SetRecordMove(WithRecordSequence(ctx), s, r.Account, r.App, r.Host, func(*Record) error { return nil }) |
| + if err != nil || got.RecordSeq != want { |
| + t.Fatalf("sequence: %+v %v", got, err) |
| + } |
| + } |
| + if _, err := SetRecordMove(WithRecordSequence(ctx), s, r.Account, r.App, r.Host, func(r *Record) error { r.RecordSeq = 0; return nil }); !errors.Is(err, ErrMaintenanceProjection) { |
| + t.Fatalf("reset: %v", err) |
| + } |
| + current, etag, err := GetRecordETag(ctx, s, r.Account, r.App, r.Host) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := s.Put(ctx, RecordKey(r.Account, r.App, r.Host), before, tag); !errors.Is(err, release.ErrCASConflict) { |
| + t.Fatalf("old Put: %v", err) |
| + } |
| + current.RecordSeq = 0 |
| + if err := UpdateRecord(ctx, s, *current, etag); !errors.Is(err, ErrMaintenanceProjection) { |
| + t.Fatalf("whole record removed sequence: %v", err) |
| + } |
| +} |
| +``` |
| + |
| +#### New file: `internal/host/plan9_b8_sequence_test.go` |
| + |
| +```go |
| +package host |
| + |
| +import ( |
| + "context" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/provision" |
| +) |
| + |
| +func TestPlan9B8MovesPassUsesCapturedClaimFlag(t *testing.T) { |
| + for _, enabled := range []bool{false, true} { |
| + h := newMoveHarness(t) |
| + host := "a.go.example.com" |
| + h.record(t, host, provision.MoveRequest("uk", time.Now()), provision.MoveAbort()) |
| + h.cfg.claimRecords = enabled |
| + if err := movesPass(context.Background(), h.s, h.worker()); err != nil { |
| + t.Fatal(err) |
| + } |
| + rec := h.read(t, host) |
| + want := uint64(0) |
| + if enabled { |
| + want = 1 |
| + } |
| + if rec.Move != nil || rec.RecordSeq != want { |
| + t.Fatalf("claim=%v record=%+v", enabled, rec) |
| + } |
| + } |
| +} |
| +``` |
| + |
| +#### New file: `internal/hostops/record_sequence.go` |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "bytes" |
| + "context" |
| + "encoding/json" |
| + "errors" |
| + "io" |
| + "os" |
| + "strconv" |
| + "strings" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// recordSequence preserves unknown fields and parses the counter without a |
| +// float64 round trip. Missing means legacy zero; null, aliases, duplicates, |
| +// fractions, negatives and overflow refuse at the claim-on write boundary. |
| +func recordSequence(body []byte) (map[string]json.RawMessage, uint64, error) { |
| + d := json.NewDecoder(bytes.NewReader(body)) |
| + if first, err := d.Token(); err != nil || first != json.Delim('{') { |
| + return nil, 0, ErrIdentity |
| + } |
| + fields := map[string]json.RawMessage{} |
| + var seq uint64 |
| + for d.More() { |
| + token, err := d.Token() |
| + if err != nil { |
| + return nil, 0, ErrIdentity |
| + } |
| + name, ok := token.(string) |
| + if !ok { |
| + return nil, 0, ErrIdentity |
| + } |
| + if _, exists := fields[name]; exists { |
| + return nil, 0, ErrIdentity |
| + } |
| + var value json.RawMessage |
| + if d.Decode(&value) != nil { |
| + return nil, 0, ErrIdentity |
| + } |
| + if strings.EqualFold(name, "record_seq") { |
| + if name != "record_seq" { |
| + return nil, 0, ErrIdentity |
| + } |
| + seq, err = strconv.ParseUint(string(value), 10, 64) |
| + if err != nil { |
| + return nil, 0, ErrIdentity |
| + } |
| + } |
| + fields[name] = value |
| } |
| - a, err := ReadAdmission(ctx, r.raw.Store, "bab", "app", "canonical") |
| - if err != nil { |
| - t.Fatal(err) |
| + if end, err := d.Token(); err != nil || end != json.Delim('}') { |
| + return nil, 0, ErrIdentity |
| } |
| - if _, ok := a.Value.Registrations[moveStartID]; ok { |
| - t.Fatal("the failed attempt's registration is still held") |
| + if _, err := d.Token(); err != io.EOF { |
| + return nil, 0, ErrIdentity |
| } |
| - retry := "mv_" + strings.Repeat("b", 26) |
| - o := r.read(t) |
| - h, err := r.authority.RegisterMove(ctx, o, "canonical", retry) |
| + return fields, seq, nil |
| +} |
| + |
| +// advanceRecordSequence alone owns the counter. Callers must preserve the |
| +// read value, including on retries; they cannot reset it or choose a future |
| +// revision. Exhaustion refuses instead of wrapping to a reusable body. |
| +func advanceRecordSequence(before, body []byte) ([]byte, error) { |
| + _, base, err := recordSequence(before) |
| if err != nil { |
| - t.Fatal(err) |
| + return nil, err |
| } |
| - moving, err := BeginClaimMove(ctx, r.data, r.authority.Directory, h, o, retry, "operator", "uk-box", "uk", r.now) |
| + fields, candidate, err := recordSequence(body) |
| if err != nil { |
| - t.Fatalf("retry start CAS: %v", err) |
| - } |
| - start := r.start |
| - start.ID = retry |
| - rec, err := StartClaimMoveRecord(ctx, r.data, h, moving, start) |
| - if err != nil || rec.Move == nil || rec.Move.ID != retry || rec.Move.State != provision.MovePreparing { |
| - t.Fatalf("retry record start = %+v, %v", rec, err) |
| + return nil, err |
| } |
| - if a, err = ReadAdmission(ctx, r.raw.Store, "bab", "app", "canonical"); err != nil { |
| - t.Fatal(err) |
| - } |
| - if e := a.Value.Effects[retry]; e.State != "complete" || e.ResultReference != retry { |
| - t.Fatalf("retry effect = %+v", e) |
| + if candidate != base { |
| + return nil, ErrIdentity |
| } |
| - if e := a.Value.Effects[moveStartID]; e.State != "complete" || e.ResultReference != "not-applied" { |
| - t.Fatalf("failed attempt's effect = %+v", e) |
| + if base == ^uint64(0) { |
| + return nil, ErrPending |
| } |
| + fields["record_seq"] = json.RawMessage(strconv.FormatUint(base+1, 10)) |
| + return json.Marshal(fields) |
| } |
| -// Why the fence rewrites the value with different bytes: the DirStore's |
| -// ETag, like a single-part S3 ETag, is a content hash, so a byte-identical |
| -// rewrite keeps the ETag a late Put matched on. |
| -func TestPlan9B8IdenticalBytesKeepAContentETag(t *testing.T) { |
| - ctx := context.Background() |
| - s := release.NewDirStore(t.TempDir()) |
| - key := provision.RecordKey("bab", "app", "shop.example.com") |
| - tag, err := s.Put(ctx, key, []byte(`{"schema":1}`), "") |
| - if err != nil { |
| - t.Fatal(err) |
| +// putSequencedRecord is shared by every conditional-record Put arm, including |
| +// issued record/control-object effects and PutFrom's buffered path. The |
| +// constructor is claim-on only. Non-record and flag-off writes never use it. |
| +func putSequencedRecord(ctx context.Context, raw release.Store, key string, body []byte, etag string) (string, error) { |
| + before, seen, err := raw.Get(ctx, key) |
| + if errors.Is(err, os.ErrNotExist) { |
| + if etag != "" { |
| + return "", release.ErrCASConflict |
| + } |
| + before, seen = []byte(`{}`), "" |
| + } else if err != nil { |
| + return "", err |
| } |
| - same, err := s.Put(ctx, key, []byte(`{"schema":1}`), tag) |
| - if err != nil || same != tag { |
| - t.Fatalf("identical rewrite: %q, %v (was %q)", same, err, tag) |
| + if seen != etag { |
| + return "", release.ErrCASConflict |
| } |
| - moved, err := s.Put(ctx, key, []byte("{\"schema\":1}\n"), same) |
| - if err != nil || moved == same { |
| - t.Fatalf("newline rewrite: %q, %v", moved, err) |
| + next, err := advanceRecordSequence(before, body) |
| + if err != nil { |
| + return "", err |
| } |
| + MarkAppMutationIssued(ctx) |
| + return raw.Put(ctx, key, next, etag) |
| } |
| +``` |
| -// A content-hash change is not a durable fence: ordinary record serialization |
| -// can restore the old bytes after settlement has released serialization. |
| -// The resolver must retain uncertainty unless it has a fence that survives |
| -// those later writes. This exercises the same record encoder as that writer. |
| -func TestPlan9B8RefusesUnprovenContentFence(t *testing.T) { |
| - ctx := context.Background() |
| - r := newResolveFixture(t, false) |
| - got, err := ResolveMoveStart(ctx, r.h, r.moving, moveStartID) |
| - if got == OutcomeUncertain && err != nil { |
| - if e := r.effect(t); e.State != "uncertain" { |
| - t.Fatalf("unproven fence settled: %+v", e) |
| +#### New file: `internal/hostops/retired_record_versions.go` |
| + |
| +```go |
| +package hostops |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "os" |
| + |
| + "github.com/carlosframework/platform/internal/lease" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// A delete marker must stay until every data version below it is gone. |
| +// Removing it first could resurrect a failed Put's base bytes. The caller |
| +// must also hold the existing exclusive purge authority against recreation. |
| +func retiredRecordVersion(ctx context.Context, control release.Store, versions lease.VersionedStore, bucket, key, id string) error { |
| + if _, _, err := control.Get(ctx, key); !errors.Is(err, os.ErrNotExist) { |
| + if err != nil { |
| + return err |
| } |
| - return |
| - } |
| - if got != OutcomeNotApplied || err != nil { |
| - t.Fatalf("unexpected resolution: %q, %v", got, err) |
| + return ErrPending |
| } |
| - if err := provision.SetRecordRegion(ctx, r.raw.Store, "bab", "app", r.host, "ie"); err != nil { |
| - t.Fatal(err) |
| + all, err := versions.ListVersions(ctx, bucket, key) |
| + if err != nil { |
| + return err |
| } |
| - if _, tag := r.record(t); tag != r.tag { |
| - t.Fatalf("ordinary encoder did not restore original tag: %q != %q", tag, r.tag) |
| + marker, data := false, false |
| + for _, v := range all { |
| + if v.Key != key { |
| + continue |
| + } |
| + if v.VersionID == id { |
| + marker = v.DeleteMarker |
| + } |
| + if !v.DeleteMarker { |
| + data = true |
| + } |
| } |
| - if err := r.latePut(); !errors.Is(err, release.ErrCASConflict) { |
| - t.Fatalf("delayed Put survived acknowledged fence and ordinary rewrite: %v", err) |
| + if marker && data { |
| + return ErrPending |
| } |
| + return nil |
| } |
| ``` |
| +#### New file: `internal/provision/record_sequence.go` |
| -Implementation. |
| +```go |
| +package provision |
| + |
| +import "context" |
| + |
| +type recordSequenceKey struct{} |
| + |
| +// WithRecordSequence selects sequence advancement for box-side move writers |
| +// using the raw conditional store. The host sets it from its captured claim |
| +// flag. Console writers instead advance at their conditional-record boundary; |
| +// they must not set this context. This supplies no mutation authority. |
| +func WithRecordSequence(ctx context.Context) context.Context { |
| + return context.WithValue(ctx, recordSequenceKey{}, true) |
| +} |
| +``` |
| #### New file: `internal/hostops/move_start_resolve.go` |
| @@ -5162,7 +5958,6 @@ package hostops |
| import ( |
| "context" |
| "errors" |
| - "slices" |
| "sync" |
| "sync/atomic" |
| @@ -5179,7 +5974,8 @@ const moveStartAction = "move-start" |
| type Outcome string |
| const ( |
| - // OutcomeComplete: the record carries this move's journal entry. |
| + // OutcomeComplete: the record-start effect applied. The journal retains |
| + // the move's own outcome, including a later abort or rollback. |
| OutcomeComplete Outcome = "complete" |
| // OutcomeNotApplied: it does not, and no record Put the start issued |
| // can still land. |
| @@ -5189,33 +5985,30 @@ const ( |
| OutcomeUncertain Outcome = "uncertain" |
| ) |
| -// ErrMoveStartFenceUnproven retains the effect until a durable fence is |
| -// available. Content-hash ETags may return to an earlier value. |
| +// ErrMoveStartFenceUnproven retains uncertainty when capture is missing, |
| +// the sequence regressed or repeated CAS conflicts prevented a proof. |
| var ErrMoveStartFenceUnproven = errors.New("hostops: move-start fence is unproven") |
| -// moveStartPuts is every If-Match ETag StartClaimMoveRecord handed the |
| -// record store for one move start. It lives only on the StartHandle whose |
| -// effect issued those Puts: knowing the registration's owner tuple is not |
| -// enough to resolve its effect, and a process that lost its handle leaves |
| -// the effect uncertain. |
| +// moveStartPuts binds every attempted conditional Put to its read sequence. |
| +// Only the issuing handle holds it. Missing capture cannot prove a fence. |
| type moveStartPuts struct { |
| id, incarnation, key string |
| + holder hostclaim.Holder |
| mu sync.Mutex |
| - etags []string |
| + reads map[string]uint64 |
| + bases []uint64 |
| } |
| -func (p *moveStartPuts) add(etag string) { |
| +func (p *moveStartPuts) highestBase() (uint64, bool) { |
| p.mu.Lock() |
| defer p.mu.Unlock() |
| - if !slices.Contains(p.etags, etag) { |
| - p.etags = append(p.etags, etag) |
| + var high uint64 |
| + for _, seq := range p.bases { |
| + if seq > high { |
| + high = seq |
| + } |
| } |
| -} |
| - |
| -func (p *moveStartPuts) matched(etag string) bool { |
| - p.mu.Lock() |
| - defer p.mu.Unlock() |
| - return slices.Contains(p.etags, etag) |
| + return high, len(p.bases) > 0 |
| } |
| // moveStartSlot is a StartHandle's move-start Puts. StartClaimMoveRecord |
| @@ -5236,12 +6029,36 @@ type moveStartRecorder struct { |
| var _ release.Store = moveStartRecorder{} |
| func (r moveStartRecorder) Get(ctx context.Context, key string) ([]byte, string, error) { |
| - return r.s.Get(ctx, key) |
| + body, etag, err := r.s.Get(ctx, key) |
| + if err == nil && key == r.puts.key { |
| + _, seq, parseErr := recordSequence(body) |
| + if parseErr != nil { |
| + return nil, "", parseErr |
| + } |
| + if etag == "" { |
| + return nil, "", ErrIdentity |
| + } |
| + r.puts.mu.Lock() |
| + if r.puts.reads == nil { |
| + r.puts.reads = map[string]uint64{} |
| + } |
| + r.puts.reads[etag] = seq |
| + r.puts.mu.Unlock() |
| + } |
| + return body, etag, err |
| } |
| func (r moveStartRecorder) Put(ctx context.Context, key string, body []byte, etag string) (string, error) { |
| if key == r.puts.key { |
| - r.puts.add(etag) |
| + r.puts.mu.Lock() |
| + seq, ok := r.puts.reads[etag] |
| + if ok && etag != "" { |
| + r.puts.bases = append(r.puts.bases, seq) |
| + } |
| + r.puts.mu.Unlock() |
| + if !ok || etag == "" { |
| + return "", ErrIdentity |
| + } |
| } |
| return r.s.Put(ctx, key, body, etag) |
| } |
| @@ -5259,9 +6076,9 @@ func (r moveStartRecorder) Delete(ctx context.Context, key string) error { |
| } |
| // ResolveMoveStart resolves the exact issuing handle's uncertain move-start |
| -// effect when the record carries that move ID and incarnation. It reuses the |
| -// runner's settlement path. A missing journal stays uncertain: main's |
| -// content-hash ETags cannot prove a permanent fence against a delayed Put. |
| +// effect from its exact journal or a newer record sequence that fences every |
| +// attempted Put. Only the runner records settlement; read errors and failed |
| +// fence writes retain uncertainty. |
| // Previously settled results are returned without issuing any new write. |
| func ResolveMoveStart(ctx context.Context, h *StartHandle, o hostclaim.Observed, moveID string) (Outcome, error) { |
| if h == nil || h.w.ready() != nil || moveID == "" || moveID != h.id || o.Absent || o.Claim == nil || o.Claim.Holder == nil { |
| @@ -5293,7 +6110,7 @@ func ResolveMoveStart(ctx context.Context, h *StartHandle, o hostclaim.Observed, |
| // Prepared or issued: the issuer has not joined and settled its call. |
| return OutcomeUncertain, ErrPending |
| } |
| - result, err := decideMoveStart(ctx, h.w, puts) |
| + result, err := decideMoveStart(ctx, h.w, puts, o) |
| if err != nil { |
| return OutcomeUncertain, err |
| } |
| @@ -5324,25 +6141,172 @@ func settledMoveStart(e Effect, moveID string) (Outcome, error) { |
| return OutcomeUncertain, ErrIdentity |
| } |
| -// decideMoveStart accepts positive evidence only. Neither an absent record, |
| -// a changed content-hash ETag nor a whitespace rewrite proves that a delayed |
| -// conditional Put can never land after another record writer runs. |
| -func decideMoveStart(ctx context.Context, w *ControlWriter, puts *moveStartPuts) (string, error) { |
| - body, _, err := w.raw.Get(ctx, puts.key) |
| - if err != nil { |
| - return "", err |
| - } |
| - rec, err := provision.DecodeRecord(body) |
| - if err != nil { |
| - return "", err |
| +// decideMoveStart never uses ETag inequality as proof. A lost fence answer |
| +// stays uncertain; a later call may prove that fence from its sequence. |
| +func decideMoveStart(ctx context.Context, w *ControlWriter, puts *moveStartPuts, o hostclaim.Observed) (string, error) { |
| + base, captured := puts.highestBase() |
| + if !captured { |
| + return "", ErrMoveStartFenceUnproven |
| } |
| - if rec.Move != nil && rec.Move.ID == puts.id && rec.Move.Incarnation == puts.incarnation { |
| - return puts.id, nil |
| + for attempt := 0; attempt < 3; attempt++ { |
| + current, err := ReadRecord(ctx, w.raw, puts.key) |
| + if err != nil { |
| + return "", err |
| + } |
| + if !matches(current, &puts.holder) { |
| + return "", ErrIdentity |
| + } |
| + _, seq, err := recordSequence(current.Body) |
| + if err != nil { |
| + return "", err |
| + } |
| + rec, err := provision.DecodeRecord(current.Body) |
| + if err != nil { |
| + return "", err |
| + } |
| + if rec.Move != nil && rec.Move.ID == puts.id && rec.Move.Incarnation == puts.incarnation { |
| + return puts.id, nil |
| + } |
| + // A box may finish the move while the console's start result is |
| + // uncertain. The terminal journal on this verified incarnation is |
| + // positive evidence too, even after Move has been cleared. |
| + if rec.LastMove != nil && rec.LastMove.ID == puts.id { |
| + return puts.id, nil |
| + } |
| + if seq > base { |
| + return "not-applied", nil |
| + } |
| + // A lower counter is a violated migration/write invariant, not authority |
| + // to repair it and settle. Equal requires a new conditional sequence. |
| + if seq < base { |
| + return "", ErrMoveStartFenceUnproven |
| + } |
| + if err := CheckCreateFreeze(ctx, w.raw, w.pub); err != nil { |
| + return "", err |
| + } |
| + fresh, err := hostclaim.Read(ctx, w.raw, o.Host) |
| + if err != nil { |
| + return "", err |
| + } |
| + if fresh.Absent || fresh.ETag != o.ETag { |
| + return "", hostclaim.ErrConflict |
| + } |
| + body, err := advanceRecordSequence(current.Body, current.Body) |
| + if err != nil { |
| + return "", err |
| + } |
| + if _, err = w.raw.Put(ctx, puts.key, body, current.ETag); err == nil { |
| + return "not-applied", nil |
| + } else if !errors.Is(err, release.ErrCASConflict) { |
| + return "", err |
| + } |
| + // The delayed Put may have won: reread its exact journal before deciding. |
| } |
| return "", ErrMoveStartFenceUnproven |
| } |
| ``` |
| +#### Patch: `internal/provision/provision.go` |
| + |
| +```diff |
| +--- a/internal/provision/provision.go |
| ++++ b/internal/provision/provision.go |
| +@@ -36,6 +36,8 @@ |
| + // Account vs the key's segments); GetRecord and ListRecords enforce that, a |
| + // record that lies about identity is never guessed at (design §3.2). |
| + type Record struct { |
| ++ // RecordSeq is advanced by the claim-on record store; absent is legacy zero. |
| ++ RecordSeq uint64 `json:"record_seq,omitempty"` |
| + Acquisition string `json:"acq,omitempty"` |
| + // Maintenance is preserved by ordinary writers; only a future guarded |
| + // maintenance transition may install, replace, or clear it. |
| +@@ -909,6 +911,12 @@ |
| + if !protectedFieldsEqual(before, rec) { |
| + return nil, ErrMaintenanceProjection |
| + } |
| ++ if enabled, _ := ctx.Value(recordSequenceKey{}).(bool); enabled { |
| ++ if rec.RecordSeq == ^uint64(0) { |
| ++ return nil, ErrMaintenanceProjection |
| ++ } |
| ++ rec.RecordSeq++ |
| ++ } |
| + body, err := json.Marshal(rec) |
| + if err != nil { |
| + return nil, err |
| +``` |
| + |
| +#### Patch: `internal/provision/sidecar.go` |
| + |
| +```diff |
| +--- a/internal/provision/sidecar.go |
| ++++ b/internal/provision/sidecar.go |
| +@@ -69,6 +69,8 @@ |
| + // identity is never guessed at, the same discipline GetRecord applies to |
| + // instances. |
| + type SidecarRecord struct { |
| ++ // RecordSeq is advanced by the claim-on record store; absent is legacy zero. |
| ++ RecordSeq uint64 `json:"record_seq,omitempty"` |
| + Acquisition string `json:"acq,omitempty"` |
| + Schema int `json:"schema"` // 1 |
| + Name string `json:"name"` |
| +``` |
| + |
| +#### Patch: `internal/hostops/record_writes.go` |
| + |
| +```diff |
| +--- a/internal/hostops/record_writes.go |
| ++++ b/internal/hostops/record_writes.go |
| +@@ -91,7 +91,7 @@ |
| + return s.putLifecycle(ctx, key, body, etag, permit) |
| + } |
| + if permit, ok := ctx.Value(effectPermitKey{}).(effectPermit); ok && (permit.Resource == key || permit.Resource == "control-object:"+key) { |
| +- return s.raw.Put(ctx, key, body, etag) |
| ++ return putSequencedRecord(ctx, s.raw, key, body, etag) |
| + } |
| + // Not-found is never turned into create outside the record effect. |
| + if etag == "" { |
| +@@ -131,7 +131,7 @@ |
| + if obs.Absent || obs.Claim.Op.Kind != hostclaim.KindServing || !matches(current, obs.Claim.Holder) { |
| + return "", ErrPending |
| + } |
| +- return s.raw.Put(ctx, key, body, etag) |
| ++ return putSequencedRecord(ctx, s.raw, key, body, etag) |
| + } |
| + |
| + func (s conditionalRecordStore) Delete(ctx context.Context, key string) error { |
| +``` |
| + |
| +#### Patch: `internal/hostops/lifecycle_record.go` |
| + |
| +```diff |
| +--- a/internal/hostops/lifecycle_record.go |
| ++++ b/internal/hostops/lifecycle_record.go |
| +@@ -21,7 +21,13 @@ |
| + return nil, ErrIdentity |
| + } |
| + h := o.Claim.Holder |
| +- return provision.SetRecordMove(context.WithValue(ctx, lifecycleRecordKey{}, lifecycleRecordPermit{o}), s, h.Account, h.App, o.Host, mut) |
| ++ rec, err := provision.SetRecordMove(context.WithValue(ctx, lifecycleRecordKey{}, lifecycleRecordPermit{o}), s, h.Account, h.App, o.Host, mut) |
| ++ if err == nil { |
| ++ // The claim-on store advances the serialized counter after the |
| ++ // mutation callback. Return that same counter to lifecycle callers. |
| ++ rec.RecordSeq++ |
| ++ } |
| ++ return rec, err |
| + } |
| + |
| + func (s conditionalRecordStore) putLifecycle(ctx context.Context, key string, body []byte, etag string, p lifecycleRecordPermit) (string, error) { |
| +@@ -131,7 +137,7 @@ |
| + } |
| + // Accepted work drains under closed admission; expiry does not revoke it. |
| + MarkAppMutationIssued(ctx) |
| +- return s.raw.Put(ctx, key, body, etag) |
| ++ return putSequencedRecord(ctx, s.raw, key, body, etag) |
| + } |
| + |
| + func sameMoveJournal(a, b []byte) bool { |
| +``` |
| + |
| #### Patch: `internal/hostops/effect_runner.go` |
| ```diff |
| @@ -5449,7 +6413,7 @@ func decideMoveStart(ctx context.Context, w *ControlWriter, puts *moveStartPuts) |
| + _, err := runRecordEffect(ctx, h, Effect{ID: p.ID, Resource: c.Holder.Key, Action: moveStartAction, Inputs: map[string]string{"move_id": p.ID, "incarnation": p.Incarnation, "to": p.To}}, func(ctx context.Context) (string, error) { |
| + // The handle keeps every If-Match ETag this start's record Puts |
| + // carry, so ResolveMoveStart can fence one whose answer was lost. |
| -+ puts := &moveStartPuts{id: p.ID, incarnation: p.Incarnation, key: c.Holder.Key} |
| ++ puts := &moveStartPuts{id: p.ID, incarnation: p.Incarnation, key: c.Holder.Key, holder: *c.Holder} |
| + h.moveStart.puts.Store(puts) |
| + rec, callErr = setLifecycleRecordMove(ctx, moveStartRecorder{s: s, puts: puts}, moving, provision.MoveStartTransition(p)) |
| if callErr != nil { |
| @@ -5472,6 +6436,170 @@ func decideMoveStart(ctx context.Context, w *ControlWriter, puts *moveStartPuts) |
| epoch uint64 |
| ``` |
| +#### Patch: `internal/provision/maintenance.go` |
| + |
| +```diff |
| +--- a/internal/provision/maintenance.go |
| ++++ b/internal/provision/maintenance.go |
| +@@ -79,7 +79,7 @@ |
| + // Equality preserves the projection, not permission to change a fenced record. |
| + // Durable guard and lifecycle checks belong to the maintenance integration. |
| + func protectedFieldsEqual(before, after Record) bool { |
| +- if before.Account != after.Account || before.App != after.App || before.Host != after.Host || !before.CreatedAt.Equal(after.CreatedAt) { |
| ++ if before.RecordSeq != after.RecordSeq || before.Account != after.Account || before.App != after.App || before.Host != after.Host || !before.CreatedAt.Equal(after.CreatedAt) { |
| + return false |
| + } |
| + if len(before.Maintenance) == 0 || len(after.Maintenance) == 0 { |
| +``` |
| + |
| +#### Patch: `internal/host/movespass.go` |
| + |
| +```diff |
| +--- a/internal/host/movespass.go |
| ++++ b/internal/host/movespass.go |
| +@@ -228,9 +228,10 @@ |
| + defer rep.Close(ctx) |
| + return rep.SyncAndWait(ctx) |
| + }, |
| +- Now: time.Now, |
| +- ReadyDelta: readyDelta, |
| +- Enabled: moves.Enabled(os.Getenv), |
| ++ Now: time.Now, |
| ++ ReadyDelta: readyDelta, |
| ++ Enabled: moves.Enabled(os.Getenv), |
| ++ claimRecords: writes.Enabled, |
| + } |
| + w := &movesWorker{mirrors: map[string]lease.MirrorStore{}} |
| + if cfg.Enabled { |
| +@@ -392,6 +393,7 @@ |
| + |
| + // moveCfg carries what the worker needs, injectable for tests. |
| + type moveCfg struct { |
| ++ claimRecords bool // Captured CARLOS_HOST_CLAIM, independent of the moves switch. |
| + Region string // this box's placement code, "" included (the home) |
| + OwnBucket string // CARLOS_HIBERNATE_BUCKET |
| + OwnBucketRegion string // CARLOS_HIBERNATE_REGION |
| +@@ -583,6 +585,9 @@ |
| + w.pass.Lock() |
| + defer w.pass.Unlock() |
| + cfg := w.cfg |
| ++ if cfg.claimRecords { |
| ++ ctx = provision.WithRecordSequence(ctx) |
| ++ } |
| + recs, err := provision.ListAllRecords(ctx, s) |
| + if err != nil { |
| + return fmt.Errorf("moves: list records: %w", err) |
| +``` |
| + |
| +#### Patch: `internal/hostops/versioned_boundary.go` |
| + |
| +```diff |
| +--- a/internal/hostops/versioned_boundary.go |
| ++++ b/internal/hostops/versioned_boundary.go |
| +@@ -2,7 +2,9 @@ |
| + |
| + import ( |
| + "context" |
| ++ "errors" |
| + "io" |
| ++ "os" |
| + "strings" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| +@@ -74,6 +76,20 @@ |
| + func (v *versionedBoundary) classify(ctx context.Context, bucket string, t MutationTarget) error { |
| + switch { |
| + case bucket == v.bucket: |
| ++ if provisioningKey(t.Key) && t.Method != "CopyVersion:source" { |
| ++ // Version replacement/deletion can resurrect a lower counter. |
| ++ // Only the typed retired-record sweep may remove old versions. |
| ++ key, _ := ctx.Value(retiredRecordSweepKey{}).(string) |
| ++ if t.Method != "DeleteVersion" || key != t.Key { |
| ++ return ErrContract |
| ++ } |
| ++ if _, _, err := v.control.Get(ctx, t.Key); !errors.Is(err, os.ErrNotExist) { |
| ++ if err != nil { |
| ++ return err |
| ++ } |
| ++ return ErrPending |
| ++ } |
| ++ } |
| + if err := structuralRefusal(t.Key); err != nil { |
| + return err |
| + } |
| +@@ -141,6 +157,11 @@ |
| + if err := v.authorize(ctx, "DeleteVersion", bucket, key, v.versionBody(ctx, bucket, key, versionID), nil); err != nil { |
| + return err |
| + } |
| ++ if bucket == v.bucket && provisioningKey(key) { |
| ++ if err := retiredRecordVersion(ctx, v.control, v.raw, bucket, key, versionID); err != nil { |
| ++ return err |
| ++ } |
| ++ } |
| + MarkAppMutationIssued(ctx) |
| + return v.raw.DeleteVersion(ctx, bucket, key, versionID) |
| + } |
| +``` |
| + |
| +#### Patch: `internal/hostops/purge_journal.go` |
| + |
| +```diff |
| +--- a/internal/hostops/purge_journal.go |
| ++++ b/internal/hostops/purge_journal.go |
| +@@ -5,6 +5,7 @@ |
| + "encoding/json" |
| + "errors" |
| + "os" |
| ++ "sort" |
| + "strings" |
| + |
| + "github.com/carlosframework/platform/internal/hostclaim" |
| +@@ -142,6 +143,9 @@ |
| + _, err = w.SavePurgeJournal(ctx, j, tag) |
| + return err |
| + } |
| ++ |
| ++type retiredRecordSweepKey struct{} |
| ++ |
| + func DeletePurgeSweep(ctx context.Context, s release.Store, w *ControlWriter, v lease.VersionedStore, bucket, account, app, id, appID string) error { |
| + if v == nil { |
| + return ErrContract |
| +@@ -156,7 +160,14 @@ |
| + if j.Cleanup.Sweep.Deleted { |
| + return nil |
| + } |
| +- for _, o := range j.Cleanup.Sweep.Versions { |
| ++ batch := append([]lease.ObjectVersion(nil), j.Cleanup.Sweep.Versions...) |
| ++ // Replay old journals safely too: retire record data before markers. |
| ++ sort.SliceStable(batch, func(i, k int) bool { |
| ++ a := provisioningKey(batch[i].Key) && batch[i].DeleteMarker |
| ++ b := provisioningKey(batch[k].Key) && batch[k].DeleteMarker |
| ++ return !a && b |
| ++ }) |
| ++ for _, o := range batch { |
| + fresh, _, err := ReadPurgeJournal(ctx, s, account, app, id, appID) |
| + if err != nil { |
| + return err |
| +@@ -167,7 +178,16 @@ |
| + if o.VersionID == "" || o.VersionID == "null" || purgeExcluded(j, o.Key) { |
| + return ErrIdentity |
| + } |
| +- if err = v.DeleteVersion(ctx, bucket, o.Key, o.VersionID); err != nil { |
| ++ deleteCtx := ctx |
| ++ if provisioningKey(o.Key) { |
| ++ // Recheck after preparation, before every version deletion. The |
| ++ // existing exclusive purge authority excludes recreation. |
| ++ if err := retiredRecordVersion(ctx, s, v, bucket, o.Key, o.VersionID); err != nil { |
| ++ return err |
| ++ } |
| ++ deleteCtx = context.WithValue(ctx, retiredRecordSweepKey{}, o.Key) |
| ++ } |
| ++ if err = v.DeleteVersion(deleteCtx, bucket, o.Key, o.VersionID); err != nil { |
| + return err |
| + } |
| + } |
| +``` |
| + |
| ## Group C: rollout ordering and witnesses |
| @@ -5488,13 +6616,14 @@ The mandatory release inventory is: |
| | `region`, `claim_etag`, `observed_boot_id` | `hostops.StopReceipt` plus claim regional bindings | Root, console and regional readers verify identical receipt bytes and reject a changed value. | |
| | `park_target`, `evidence_refs` | `hostclaim.Op` and retirement evidence | Populated claims round-trip, including every keyed evidence reference. | |
| | `acq` | `hostclaim.Holder`, records and ownership backfill | Strict read of the first acquired holder before #232 can write it. | |
| +| `record_seq` | B8: `provision.Record`, `provision.SidecarRecord`, every claim-on conditional record writer and the move-start resolver | Read legacy absent/zero and populated counters on every participating binary; ordinary/lifecycle/effect/stream writes advance without losing precision, and an ordinary rewrite cannot reopen the failed Put's fence. | |
| Also inventory `Box.CensusPublicKey`, the release identity fields and every new field approved for Task B4. The first-review M5 remains open as a deployment ordering requirement: `acq` is on main, but its presence in the type does not prove that every deployed reader understands it. Do not remove the field to make old readers pass. All named fields must be in the release certification, even when their test fixture normally omits them. |
| Check the exact candidate source and run the reader tests serially: |
| ```bash |
| -rg -n 'json:"(discovery_bound|discovery_boxes|request_limit_ns|region|claim_etag|observed_boot_id|park_target|evidence_refs|acq)' internal/hostclaim internal/hostops internal/hostactors |
| +rg -n 'json:"(discovery_bound|discovery_boxes|request_limit_ns|region|claim_etag|observed_boot_id|park_target|evidence_refs|acq|record_seq)' internal/hostclaim internal/hostops internal/hostactors internal/provision |
| rg -n 'DisallowUnknownFields|VerifyStop|CensusPublicKey' internal/hostclaim internal/hostops internal/hostactors internal/hostbox |
| nice -n 10 go test -p 1 ./internal/hostclaim ./internal/hostactors ./internal/hostops ./internal/hostbox -count=1 |
| ``` |
| @@ -11020,7 +12149,7 @@ No new marker or claim write is unconditional. The Group D production bindings a |
| ## Self-review and verification |
| -The deliverable changes only this plan in the design worktree. There are no git writes or live operations. Rev 1 exported `76f728a910ff9c78856620801e712650b962f2cd` and reviewed its authority-bearing work with explicitly selected Opus delegates. Rev 2 uses a new disposable export of `4f26f15cb41ecfdd9214b49393c228399f34490b`, confirmed by read-only `git ls-remote`. The verification recorded below distinguishes the two revisions. |
| +The deliverable changes only this plan in the design worktree. There are no git writes or live operations. Rev 1 exported `76f728a910ff9c78856620801e712650b962f2cd` and reviewed its authority-bearing work with explicitly selected Opus delegates. Rev 2 uses a new disposable export of `4f26f15cb41ecfdd9214b49393c228399f34490b`, confirmed by read-only `git ls-remote`. The verification recorded below distinguishes all three revisions. |
| Review tightened two candidate exceptions: a currently fenced/quarantined writer cannot recover authority by claiming an earlier observation timestamp, and a fresh report from another process cannot be overwritten merely because this process started later. Both exceptions were removed before the combined green run. Review also rejected the journal connector that still handed SQLite an attacker-replaceable pathname. B7 records its actual failing witness and the unruled architecture; it is not in the decided implementation. |
| @@ -11041,7 +12170,7 @@ Coverage: |
| | R1 / 5-M4 | C1, every named strict field and exact binary/receipt certification. | |
| | M-4 / M-7 | C2/C3, missing evidence producers remain refusal; journal-owned staging retained. | |
| | First-review M7 | C3, one-box-region guard contract and production witness. | |
| -| Move-start recovery | B8, positive resolution through the exact runner; not-applied settlement blocked by the content-ETag ABA witness. | |
| +| Move-start recovery | B8 rev 3, complete resolution through the exact runner; monotonic `record_seq` fences every failed Put, with passing acceptance, drain/retry and ordinary-rewrite ABA regressions. | |
| | Moves C1 H3 / H2 / H12 / H15 / H16 | D1-D5, guarded phase progression, attempts, readiness, retirement and owner dispatch; production dependencies G-1 through G-6 remain explicit. | |
| | I-3 / I-5 / M1-1 | Already fixed on main; no duplicate implementation. Keep wrapped-not-found, bounded binary reads and locked/torn marker recovery regression coverage. | |
| @@ -11057,10 +12186,24 @@ A final fresh export of `4f26f15` replayed every block with `patch --fuzz=0`. Al |
| Rev 2 authority review used explicitly selected Opus reviewers. Corrections included the source proof before admission, retention of legacy owners with moves disabled, exact attempt capture, required C8 materialization, fenced and joined publication after a real root, unused-lease cleanup on admission failure, session-specific wake stops without consuming retirement chores, and replay of a failed inventory CAS after the old marker was parked. Focused regressions passed for these corrections. The final bounded reviews reported no remaining P1/P2 in the corrected paths. This does not close the production dependencies G-1 through G-6 or substitute fixture kernels for C3's privileged witnesses. |
| -Rev 2 negative evidence remains explicit. B4 still fails because authority reads hold the SQL connection and a transaction checks four times; B7 still modifies the substituted journal database. B8's identical-body fence retains the ETag, and its rejected whitespace variant permits the old ETag to return after an ordinary record rewrite; the delayed Put then succeeds. The four fence-dependent B8 acceptance tests remain red, while positive resolution and conservative refusal tests pass. No not-applied recovery feature, production wake, live IAM property, privileged launch/stop witness, arming or activation is claimed complete. All authoring overlays and scratch are removed after saving this revision. |
| +Historical rev 2 negative evidence (B8 is superseded by rev 3 below). B4 still fails because authority reads hold the SQL connection and a transaction checks four times; B7 still modifies the substituted journal database. B8's identical-body fence retains the ETag, and its rejected whitespace variant permits the old ETag to return after an ordinary record rewrite; the delayed Put then succeeds. The four fence-dependent B8 acceptance tests remained red in rev 2, while positive resolution and conservative refusal tests passed. That revision did not complete not-applied recovery. Rev 3 supersedes that B8 limitation; production wake, live IAM properties, privileged launch/stop witnesses, arming and activation remain unproven. Authoring overlays and scratch are removed after saving each revision. |
| Before landing the later implementation, refresh against current amadan main, make every resolved task's tests pass, complete the chosen design tasks and their exact code, run the repository's build/vet/test/gofmt and cross-build gates, and obtain authority review on the final branch. Push and land only through the reviewed amadan branch workflow. Local compilation and inert landing are not approval to arm or activate. |
| Rev 1 final replay extracted every new file and patch from that revision into a fresh main export, applied with `patch --fuzz=0`, and compared all resulting bytes with the reviewed prototype. `go build -p 1 ./...` passed and `go test -p 1 ./... -run '^$'` compiled all 93 package entries. Exported trees have no usable VCS metadata, so those authoring commands used `GOFLAGS=-buildvcs=false`; the initial build without it refused VCS stamping, not source compilation. The final extracted-tree focused run passed all new decided Go tests plus reset-door, root-signer, census-forgery and instance/sidecar list regressions. `gofmt -l .` was empty and module `tofu fmt -check` passed. The two final pending-list error changes were re-extracted, byte-compared and tested after the broad compile. No full runtime suite, live deployment or privileged isolation witness is claimed. |
| Decisions still required before completion of activation readiness: B4's transaction completion/recovery authority; B7's journal service or pinned VFS; B3's remote stop protocol and concrete mode-scoped owner adapters; A3's distinct box-writer credentials; A1's pending-controller crash recovery; and C2's publication/arming order. All C2 rollout milestones then require separate explicit Paul approvals. The plan is reviewable now; arming and activation are not authorized by it. |
| + |
| +### Rev 3 verification (2026-10-08) |
| + |
| +The remote main ref was checked again without fetching and remained `4f26f15cb41ecfdd9214b49393c228399f34490b`. Rev 3 was authored and tested in disposable exports under `/home/paulca/.cache/tmp/claude-1001/`. Every task's blocks replayed with `patch --fuzz=0`; all 104 resulting files matched the tested prototype byte-for-byte. The final terminal-journal additions were re-extracted and compared too. The six complete decision sections (A1, A3, B3, B4, B7 and C2) and the entire Group D section were compared against rev 2 and remained byte-identical. No Group D signature changed. |
| + |
| +Negative evidence came first: all four original fence-dependent B8 acceptance tests failed against rev 2's partial resolver with `ErrMoveStartFenceUnproven`. The revised test files, applied before B8's implementation to a fresh base plus the preceding tasks, failed on the expected missing APIs and `RecordSeq` fields. The complete rev 3 B8 suite then passed, with no skipped or excluded acceptance cases. Coverage includes lost and landed Puts, fence races and lost fence responses, multiple captured Put bases, exact-handle refusal, registration drain and retry, unrelated writes, an ordinary-rewrite ABA attempt, box-side writes, streamed/effect/ordinary record updates, counter removal/reset/precision/overflow, versioned-write refusal, marker-last purge and legacy flag-off bytes. Terminal journal tests distinguish application of the start effect from the move's eventual success, abort or rollback, and refuse to use another move ID as positive evidence. |
| + |
| +All executed Go checks were serial under `nice -n 10`, with tests using `-p 1`. The default build cache refused writes; checks used the brief's shared fallback `/home/paulca/.cache/tmp/claude-1001/gocache` and exported-tree `GOFLAGS=-buildvcs=false`. The full hostops and provision packages passed. Focused host moves/phase-1/cut/landing tests passed (41.372 seconds), as did the new box-sequence test, console authority/failed-start tests, and all Group D tests across hostclaim, hostbox, host, activator, hostguard and park. Vet passed for hostops, provision and host. `go build -p 1 ./...` passed, `go test -p 1 ./... -run '^$'` compiled all 93 package entries, and `gofmt -l .` was empty. The final terminal-journal cases were followed by another passing full hostops run. Rev 3 did not rerun the full runtime suite, cross-builds or privileged/live witnesses. |
| + |
| +Opus performed a writer audit and bounded authority reviews. The audit found the box-side record writers and the versioned handle as additional ways to bypass the sequence. Rev 3 covers both, preserves delete markers until old data versions are gone, and checks the current claim and creation freeze before a fence write. The final review reported no confirmed P1/P2. Its source assumptions were checked: an empty Store Put condition is create-only, `updateRecord` always performs its Put before returning success, and record streaming delegates to the conditional Put. The lifecycle helper remains console-only; box writers use the separate captured-flag context. Restore on this base does not recreate instance/sidecar records from archived record bodies; any future such path must retain the new-incarnation and sequence rules. Uncaptured attempts, an incomplete purge version listing or a changed claim retain refusal rather than manufacture evidence. |
| + |
| +One broader console probe remains red independently of B8: `TestMoveDoorHostClaimRefusals/target-fenced` returns 503 where the fixture expects 409. A separate clean `4f26f15` export with only rev 2's A2 applied reproduced the same failure, before any B8 code. That fixture was not changed by this scoped revision, and the passing focused console command does not claim to cover it. B4 and B7's decision tests remain intentionally red; the six Paul decisions and Group D's production dependencies G-1 through G-6 remain open. Rev 3 completes B8's planned implementation, not activation readiness as a whole. |
| + |
| +Only this plan is saved in the requested worktree. There are no git writes or live operations. All authoring overlays, review inputs, logs and scratch are removed at hand-back; the shared Go cache remains shared. |