amadan
plan: self-serve moves 1, rev 3 (Astra plan review 2: F1 to F12)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+162
−125
docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md+162 −125
CI cancelled — superseded by a newer push — run details
CI log
=== gate === .amadan/test-gate.sh === launchctx (1 s) ok amadan.net/carlos/platform.git/launchctx 0.011s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] === vet (5 s) === gofmt (0 s) gofmt -l .: clean === rest (147 s) ? github.com/carlosframework/platform/cmd/changelog [no test files] ok github.com/carlosframework/platform/cmd/console 4.764s ok github.com/carlosframework/platform/cmd/moveprobe 11.537s ok github.com/carlosframework/platform/cmd/zddprobe 0.702s ok github.com/carlosframework/platform/internal/activator 58.142s ok github.com/carlosframework/platform/internal/agentsock 0.017s ok github.com/carlosframework/platform/internal/alerting 0.005s ok github.com/carlosframework/platform/internal/analytics 0.020s ok github.com/carlosframework/platform/internal/appconfig 2.687s ok github.com/carlosframework/platform/internal/apperr 0.074s ok github.com/carlosframework/platform/internal/applog 0.134s ok github.com/carlosframework/platform/internal/awsinv 0.005s ok github.com/carlosframework/platform/internal/boxchoice 0.007s ok github.com/carlosframework/platform/internal/boxid 0.468s ? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files] ? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files] ok github.com/carlosframework/platform/internal/boxvolumes 0.008s ok github.com/carlosframework/platform/internal/caps 0.013s ok github.com/carlosframework/platform/internal/certwarm 0.002s ok github.com/carlosframework/platform/internal/cgroup 0.697s ok github.com/carlosframework/platform/internal/changelog 0.341s ok github.com/carlosframework/platform/internal/claimactivation 0.819s ok github.com/carlosframework/platform/internal/clamd 0.054s ? github.com/carlosframework/platform/internal/cliflag [no test files] ? github.com/carlosframework/platform/internal/controlname [no test files] ? github.com/carlosframework/platform/internal/controlsock [no test files] ok github.com/carlosframework/platform/internal/dbdoor 6.518s ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.010s ok github.com/carlosframework/platform/internal/dbverify 0.004s ok github.com/carlosframework/platform/internal/dnsrec 0.005s ok github.com/carlosframework/platform/internal/docsite 0.006s ok github.com/carlosframework/platform/internal/domains 2.714s ok github.com/carlosframework/platform/internal/doorkey 0.006s ok github.com/carlosframework/platform/internal/economics 0.003s ok github.com/carlosframework/platform/internal/edge 44.868s ? github.com/carlosframework/platform/internal/edgeproof [no test files] ok github.com/carlosframework/platform/internal/edgesfile 0.007s ok github.com/carlosframework/platform/internal/fleetapi 0.132s ok github.com/carlosframework/platform/internal/gate 1.704s ok github.com/carlosframework/platform/internal/genselect 0.014s ok github.com/carlosframework/platform/internal/geo 0.019s ok github.com/carlosframework/platform/internal/geoip 0.029s ok github.com/carlosframework/platform/internal/historyreadback 0.151s ok github.com/carlosframework/platform/internal/host 135.799s ok github.com/carlosframework/platform/internal/hostactors 6.020s ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.007s ok github.com/carlosframework/platform/internal/hostbox 2.989s ok github.com/carlosframework/platform/internal/hostclaim 0.737s ok github.com/carlosframework/platform/internal/hostenv 0.006s ok github.com/carlosframework/platform/internal/hostevidence 0.009s ok github.com/carlosframework/platform/internal/hostexec 5.298s ok github.com/carlosframework/platform/internal/hostfence 0.060s ok github.com/carlosframework/platform/internal/hostfiles 10.277s ok github.com/carlosframework/platform/internal/hostfloor 0.027s ok github.com/carlosframework/platform/internal/hostguard 1.064s ok github.com/carlosframework/platform/internal/hostops 2.038s ? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files] ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.005s ok github.com/carlosframework/platform/internal/hostquarantine 0.002s ok github.com/carlosframework/platform/internal/hostrelocate 0.646s ok github.com/carlosframework/platform/internal/hostresolve 0.206s ok github.com/carlosframework/platform/internal/hostwrite 0.006s ok github.com/carlosframework/platform/internal/infraview 0.003s ok github.com/carlosframework/platform/internal/instancectl 5.224s ok github.com/carlosframework/platform/internal/instanceexec 2.753s ok github.com/carlosframework/platform/internal/instancefs 0.017s ok github.com/carlosframework/platform/internal/instancekey 0.005s ok github.com/carlosframework/platform/internal/instanceupdates 0.008s ok github.com/carlosframework/platform/internal/layout 1.876s ok github.com/carlosframework/platform/internal/lbhealth 0.008s ok github.com/carlosframework/platform/internal/lease 10.165s ok github.com/carlosframework/platform/internal/lease/leasetest 0.503s ok github.com/carlosframework/platform/internal/ledger 0.022s ok github.com/carlosframework/platform/internal/mail 0.279s ok github.com/carlosframework/platform/internal/maintenance 0.097s ok github.com/carlosframework/platform/internal/maintenanceexec 5.548s ok github.com/carlosframework/platform/internal/maintenanceguard 0.138s ok github.com/carlosframework/platform/internal/metrics 0.006s ok github.com/carlosframework/platform/internal/moves 21.261s ok github.com/carlosframework/platform/internal/moves/archive 1.095s ok github.com/carlosframework/platform/internal/objstore 0.282s ok github.com/carlosframework/platform/internal/ops 0.239s ok github.com/carlosframework/platform/internal/park 0.271s ok github.com/carlosframework/platform/internal/plans 0.007s ok github.com/carlosframework/platform/internal/privatefile 0.004s ok github.com/carlosframework/platform/internal/provision 13.606s ok github.com/carlosframework/platform/internal/publicrelease 0.009s ? github.com/carlosframework/platform/internal/regionname [no test files] ok github.com/carlosframework/platform/internal/registry 5.469s ok github.com/carlosframework/platform/internal/relay 7.234s ok github.com/carlosframework/platform/internal/release 25.062s ok github.com/carlosframework/platform/internal/rollout 3.796s ok github.com/carlosframework/platform/internal/rolloutwords 0.003s ok github.com/carlosframework/platform/internal/rootaws 0.007s ok github.com/carlosframework/platform/internal/rootparticipation 0.059s ok github.com/carlosframework/platform/internal/rootstate 0.029s ok github.com/carlosframework/platform/internal/runs 0.521s ok github.com/carlosframework/platform/internal/s3compat 0.009s ok github.com/carlosframework/platform/internal/safefile 0.005s ok github.com/carlosframework/platform/internal/schedule 2.571s ok github.com/carlosframework/platform/internal/seeds 0.011s ok github.com/carlosframework/platform/internal/sim/hostlife 10.305s ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.199s ok github.com/carlosframework/platform/internal/status 0.083s ok github.com/carlosframework/platform/internal/steering 0.007s ok github.com/carlosframework/platform/internal/stopevidence 0.315s ok github.com/carlosframework/platform/internal/svcrole 0.002s ok github.com/carlosframework/platform/internal/telemetry 0.004s ok github.com/carlosframework/platform/internal/termfence 0.006s ok github.com/carlosframework/platform/internal/tunnelclient 0.004s ok github.com/carlosframework/platform/internal/unixdial 0.003s ok github.com/carlosframework/platform/internal/updatewords 0.004s ok github.com/carlosframework/platform/internal/usage 0.008s ok github.com/carlosframework/platform/internal/version 0.001s ok github.com/carlosframework/platform/internal/vet 1.648s === internal_console@0 (86 s) ok github.com/carlosframework/platform/internal/console 84.404s === internal_console@1 (37 s) ok github.com/carlosframework/platform/internal/console 35.204s === internal_console@2 (42 s) ok github.com/carlosframework/platform/internal/console 40.238s === internal_console@3 (36 s) ok github.com/carlosframework/platform/internal/console 35.004s === internal_console@4 (36 s) ok github.com/carlosframework/platform/internal/console 34.452s === internal_console@5 (34 s) ok github.com/carlosframework/platform/internal/console 32.683s === internal_console@6 (36 s) ok github.com/carlosframework/platform/internal/console 34.659s === internal_console@7 (42 s) ok github.com/carlosframework/platform/internal/console 40.548s === root@0 (31 s) ok github.com/carlosframework/platform 29.899s === root@1 (43 s) ok github.com/carlosframework/platform 41.849s === root@2 (40 s) ok github.com/carlosframework/platform 38.592s === cross === --- shard 0 of 5 --- cross-compiling darwin/arm64 amadan: job cancelled — a newer push superseded it --- shard 1 of 5 --- cross-compiling darwin/amd64 amadan: job cancelled — a newer push superseded it --- shard 2 of 5 --- cross-compiling linux/arm64 amadan: job cancelled — a newer push superseded it --- shard 3 of 5 --- cross-compiling windows/amd64 amadan: job cancelled — a newer push superseded it --- shard 4 of 5 --- cross-compiling windows/arm64 amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md| index d90ae4fee..ffbc9ae90 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| @@ -2,240 +2,277 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 2, 2026-10-22. Rev 1 was reviewed by Astra (`after236/selfserve-plan1-review.md`, NOT READY, 16 findings); rev 2 answers every one. The largest changes: the protocol is scoped to claims-off deployments (claims-on records keep their own lifecycle), the migration marker is fenced inside the one record-write primitive, the create checks the record before allocating, a legacy record is migrated before it is retired, the pair never moves on an ordinary upsert, and the alias precedence rule moves to plan 3 with the holding entries it orders. |
| +Rev 3, 2026-10-22. Rev 1 and rev 2 were reviewed by Astra (`after236/selfserve-plan1-review.md`, 16 findings; `after236/selfserve-plan1-review2.md`, 12 findings F1 to F12). Rev 3 answers F1 to F12: the slot carries its owning record key; every claims-off record removal (the import rule's ordinary delete, seed finishing, the detach and rollback callers) goes through one retirement function that classifies the record and the slot together before it writes; the create re-checks hostname ownership under the exclusion and reports a pending allocation as its own result so a caller never releases a claim over one; the consumers are named by function with concrete judged readers and a re-read on a lifetime mismatch; the row's pair moves only through guarded setters that name the lifetime they expect; the fold compares only entries of one lifetime; and the fixtures are corrected. |
| **Goal:** Give every claims-off instance record a durable lifetime (the host incarnation slot) and every served row a serving order `(incarnation, serving_seq)` that boxes publish and edges honour, with crash-safe create, retirement and migration, so a later plan's holding pages and moves can never be out-ordered by a stale publication. |
| -**Architecture:** A per-hostname slot object in the deployment bucket (`control/host-incarnations/<host>.json`), created once and changed only by conditional writes, is the authority for a record's lifetime; the console allocates, retires and migrates through persisted intents in SQLite under a per-hostname exclusion and recovers them at start. Records gain `incarnation`, `incarnation_migrating`, `move_seq` and a tombstone form; registry rows gain the pair; the edges file publishes it with each alias's target; the home map orders comparable entries by it before the existing claim stamp. Retirement replaces the record with a tombstone before retiring the slot. Lazy migration marks a record by CAS before binding a slot to it, and the write primitive refuses every other writer while the marker stands. |
| +**Architecture:** A per-hostname slot object in the deployment bucket (`control/host-incarnations/<host>.json`), created once and changed only by conditional writes, is the authority for a record's lifetime and names the record key that owns it; the console allocates, retires and migrates through persisted intents in SQLite under a per-hostname exclusion and recovers them at start. Records gain `incarnation`, `incarnation_migrating`, `move_seq` and a tombstone form; registry rows gain the pair; the edges file publishes it with each alias's target; the home map orders entries of one lifetime by it before the existing claim stamp. Retirement replaces the record with a tombstone before retiring the slot. Lazy migration marks a record by CAS before binding a slot to it, and the write primitive refuses every other writer while the marker stands. |
| -**Tech Stack:** Go. `internal/provision` (records, sequence, reconcile, offline transitions), `internal/registry`, `internal/console` (doors, store, store gate, passes), `internal/host` (edges file, offline hold and release), `internal/edge` (home map). Tests use `release.NewDirStore`, real registries, the console's doors in-process, and the existing two-box offline witness `TestOfflineMoveE2E` (`internal/host/movesoffline_e2e_test.go`). |
| +**Tech Stack:** Go. `internal/provision` (records, sequence, reconcile, offline transitions), `internal/registry`, `internal/console` (doors, store, store gate, passes), `internal/host` (edges file, offline hold and release, the agent's config), `internal/edge` (home map). Tests use `release.NewDirStore`, real registries, the console's doors in-process, and the existing two-box offline witness `TestOfflineMoveE2E` (`internal/host/movesoffline_e2e_test.go`). |
| **Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md`, §6.4 ("Ordering by a per-host sequence", "Lifetimes: the host incarnation slot", "Aliases", "Freshness") and §11. Plans 2 (door, token, job, reservations, reports), 3 (holding entries, probe mode, the alias rule) and 4 (cleanup) build on this one. |
| ## Global Constraints |
| -- **Scope: claims-off.** The slot protocol, migration pass, tombstone retirement and validity checks run only where `CARLOS_HOST_CLAIM` is off on the console (`!s.cfg.HostClaim`) and apply only to records that carry `incarnation` or `incarnation_migrating`. Claims-on records are created by `finishClaimCreate` and retired by the host-claim operations; this plan does not touch them, and a claims-on console's record consumers see no new field. Self-serve moves refuse on a claims-on console (MV-O20), so nothing here is needed there. |
| -- Every claims-off consumer treats a record that fails `RecordLifetimeValid` (Task 1) as absent: reconcile, the console's record reads, the boxes' publication. "Absent" for reconcile means removed from the live set before minting, so an existing row is retired as "record gone". |
| -- The slot is never deleted and never advanced to abandon a create (spec §6.4). |
| -- One per-hostname exclusion for create, retirement, migration and recovery in the console process (Task 3's `hostnameMutex`), and a persisted pending operation for a hostname refuses any new create or delete of that hostname (409 "another change to this instance is still settling. try again in a minute") until recovery resolves it; unreadable evidence leaves it pending. |
| -- Every record write stays on `provision.WriteRecord` with the exact bytes read and the sequence discipline of `record_sequence.go`; no new raw `Put` of a record key. |
| +- **Scope: claims-off, every record.** On the console the protocol runs only when `!s.cfg.HostClaim`; on a box only when `!hostwrite.Enabled(getenv)` (`CARLOS_HOST_CLAIM` unset). Where it runs it applies to **every** instance record, fieldless ones included (a fieldless record beside a slot is stale, Task 1). Claims-on records are created by `finishClaimCreate` and retired by the host-claim operations; this plan does not touch them, their consumers read no slot (the slot cache is nil there, pinned by `TestClaimsOnConsumersReadNoSlot`), and their rows keep the claim stamp order. Self-serve moves refuse on a claims-on console (MV-O20). |
| +- Every claims-off consumer treats a record whose lifetime verdict is `stale` (Task 1) as absent, and one whose verdict is `unreadable` as a record it cannot judge this pass: reconcile protects the row, a door answers 502, a listing reports the host unreadable. |
| +- The slot is never deleted and never advanced to abandon a create (spec §6.4). It never moves backwards: every slot write names the exact prior state it replaces. |
| +- One per-hostname exclusion (`withHostname`, Task 3) around create, retirement, migration and their recovery in the console process. A persisted pending operation for a hostname refuses any create or retirement of that hostname by another operation (409 "another change to this instance is still settling. try again in a minute") until recovery resolves it; unreadable evidence leaves it pending and keeps refusing. |
| +- Every record write stays on `provision.WriteRecord` with the exact bytes read and the sequence discipline of `record_sequence.go`; no new raw `Put` of a record key. Every record removal on a claims-off console is `retireRecordWithLifetime` (Task 4); `provision.DeleteRecord` gains no new caller and loses its claims-off instance callers. |
| - `release.DeleteIfMatch` never falls back to a plain delete; absence counts as success. |
| - No console write to a bucket outside the deployment bucket; no root-side change. |
| -- Census: every changed unit in `internal/host` or `internal/provision` takes a row in `internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv`, loaded last by a new block at the end of `TestPlan14RootReadCensus` in `internal/provision/plan14_root_reads_test.go`, `rev3=PROPOSED-self-serve-moves-1`, with the note `docs/superpowers/notes/census-proposal-self-serve-moves-1.md`; `TestPlan9B8RecordWriteCensus` (the record-write census) is updated for the new record writers (tombstone, marker, migration); both tests green. |
| -- The new SQLite tables are classified in `claimSQLTables` (`internal/console/store_gate.go`) so a claims-on store still opens (its boot census refuses an unlisted table). |
| -- Copy: the two new sentences are copy rows pending Paul's review: "the instance changed. read it again and try the delete once more" and "another change to this instance is still settling. try again in a minute". No tenant names: `shop.example.com`, account `bab`, app `shop`. |
| +- Census: every changed unit in `internal/host` or `internal/provision` takes a row in `internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv`, loaded last by a new block at the end of `TestPlan14RootReadCensus` in `internal/provision/plan14_root_reads_test.go`, `rev3=PROPOSED-self-serve-moves-1`, with the `@fields` line naming the new record fields and the note `docs/superpowers/notes/census-proposal-self-serve-moves-1.md`. `TestPlan9B8RecordWriteCensus` takes its inventory from a new `plan14SelfServeMoves1WriteCensusInventory()` that wraps `plan14ClaimRosterBindingWriteCensusInventory()` and lists each added, changed and removed record-write site of this plan (the tombstone, the marker, the migration's two writes, the lifetime create, the transitions whose digests change, the removed `DeleteIfMatch` sites in `deleteRead` and `finishSeedOnce`). Both tests green. |
| +- The new SQLite tables are classified in `claimSQLTables` (`internal/console/store_gate.go`) so a claims-on store still opens. |
| +- Copy: two new sentences are copy rows pending Paul's review: "the instance changed. read it again and try the delete once more" and "another change to this instance is still settling. try again in a minute". Existing sentences are reused where the plan says so. No tenant names: `shop.example.com`, account `bab`, app `shop`. |
| - Boxes roll before the console (the roll order rule); the migration pass is behind `CARLOS_LIFETIMES_MIGRATE=1`, default off, until every box of a deployment runs this code. |
| - Local tests: only the touched packages with `-run`; CI runs the full suite. Go needs the sandbox off; never set `GOCACHE`. |
| ## Review Focus |
| -1. A console crash between the slot write and the record write, a second console start before the first process's write has landed, then the late write: the replay and the late write must both be harmless (Task 3, `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`). |
| -2. A box-side offline transition issued against the pre-marker ETag during migration: its CAS must fail and its retry succeed after the field is set (Task 5, `TestAnIssuedWriteLosesToTheMarkerAndRetries`). |
| -3. A legacy record with no incarnation beside a retired slot (an old console's late create): every consumer treats it as absent and the next create replaces it (Task 6 `TestReconcileRemovesAFieldlessRecordBesideAnUnrelatedSlot`, Task 3 `TestCreateReplacesStaleRecord`). |
| -4. A reconstructed registry after a finished move, and after an abort from landed: reconcile installs the record's pair with the minted row (Task 7, `TestReconcileInstallsTheRecordsPairOnAMintedRow`). |
| -5. An existing row kept across a recreation or a migration: it must take the new lifetime's pair without inventing a higher sequence (Task 7, `TestReconcileInstallsANewLifetimeOnAnExistingRow`). |
| +1. A console crash between the slot write and the record write, a second console start, then the late write: the replay and the late write must both be harmless (Task 3, `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`), and a delete arriving during that recovery is refused as settling (`TestDeleteDuringCreateRecoveryIsRefused`). |
| +2. A box-side offline transition issued against the pre-marker ETag during migration: its CAS fails and its retry succeeds after the field is set (Task 5, `TestAnIssuedWriteLosesToTheMarkerAndRetries`); a consumer whose record read straddles the marker must not retire the host (Task 6, `TestReconcileRereadsAcrossAMigration`). |
| +3. A legacy fieldless record beside a retired slot (an old console's late create): the strict listings, the console doors, reconcile and the offline keep view all treat it as absent, and the next create replaces it (Task 6 `TestEveryClaimsOffConsumerIgnoresAFieldlessRecordBesideASlot`, Task 3 `TestCreateReplacesStaleRecord`). |
| +4. A reconstructed registry row after a finished move and after an abort from landed: reconcile installs the record's pair on the minted row, `(1, 3)` in both cases (Task 7, `TestReconcileInstallsTheRecordsPairOnAMintedRow`). |
| +5. An existing row across a recreation and a migration, held and unheld: a held row never takes a new lifetime's pair, an unheld row of the old lifetime takes the new one once, and a setter from the old lifetime cannot move the successor's row (Task 7, `TestExistingRowsAcrossLifetimes`). |
| -(The alias repointed during a move is plan 3's Review Focus: it needs holding entries.) |
| +(The alias repointed during a move and the holding entry's fold are plan 3's Review Focus.) |
| --- |
| -### Task 1: Record fields, the slot type, the tombstone form and the validity rule |
| +### Task 1: Record fields, the slot type, the tombstone form and the lifetime verdict |
| **Files:** |
| -- Modify: `internal/provision/provision.go` (the `Record` struct after `Gate`; `GetRecordETag`, `GetRecord`) |
| +- Modify: `internal/provision/provision.go` (the `Record` struct after `Gate`; `GetRecordETag`) |
| - Create: `internal/provision/lifetime.go` |
| - Test: `internal/provision/lifetime_test.go`; extend `TestRecordWireShapeIsAdditive` in `internal/provision/provision_test.go` |
| **Interfaces:** |
| - Produces: |
| - - `Record.Incarnation uint64 \`json:"incarnation,omitempty"\``, `Record.IncarnationMigrating string \`json:"incarnation_migrating,omitempty"\``, `Record.MoveSeq uint64 \`json:"move_seq,omitempty"\``, `Record.Retired bool \`json:"retired,omitempty"\``. |
| - - `type HostIncarnation struct { Incarnation uint64 \`json:"incarnation"\`; State string \`json:"state"\`; Intent string \`json:"intent,omitempty"\`; RecordETag string \`json:"record_etag,omitempty"\` }`; `const SlotLive = "live"`, `SlotRetired = "retired"`. `RecordETag` is set only by a migration slot: the marked record's ETag it is bound to. |
| - - `func HostIncarnationKey(host string) string` → `control/host-incarnations/<host>.json`. |
| - - `func ReadHostIncarnation(ctx context.Context, s release.Store, host string) (*HostIncarnation, string, error)`: nil, "", nil when absent (the store's not-found). |
| - - `func WriteHostIncarnation(ctx context.Context, s release.Store, host string, slot HostIncarnation, ifMatch string) (string, error)`: `ifMatch == ""` is create-only (`Store.Put` with "" sends `If-None-Match: *` on S3 and refuses an existing file on `DirStore`), otherwise `If-Match`. |
| - - `func RecordLifetimeValid(rec *Record, recordETag string, slot *HostIncarnation) bool`. |
| - - `func TombstoneBody(before []byte) ([]byte, error)`: decodes `before`, keeps `schema`, `host`, `account`, `app`, `incarnation` and `record_seq`, sets `retired: true`, drops every other field; the caller writes it with `WriteRecord(ctx, s, key, before, body, etag)` so the sequence discipline holds (`record_seq` equal to `before`'s, then advanced by `WriteRecord`). |
| - - `func GetRecordRaw(ctx context.Context, s release.Store, acct, app, host string) (*Record, []byte, string, error)`: the decoded record (tombstones included, path check applied), its exact bytes and ETag. |
| - - `GetRecordETag` and `GetRecord` return the store's not-found error for a tombstone (`Retired`), so every existing caller sees a retired record as absent; `GetRecordRaw` is the only reader that sees it. |
| - |
| -The validity rule, exactly: `rec.Retired` → false. `rec.Incarnation == 0 && rec.IncarnationMigrating == ""`: slot nil → true; slot non-nil → false. `rec.IncarnationMigrating != ""` (fieldless with the marker): slot nil → true; slot `{1, live, "migration " + marker, RecordETag == recordETag}` → true; otherwise false. `rec.Incarnation > 0`: slot non-nil, `slot.State == live`, `slot.Incarnation == rec.Incarnation` → true; otherwise false. |
| - |
| -- [ ] **Step 1: Write the failing tests** in `lifetime_test.go`: `TestRecordLifetimeValid` (a table with one case per line of the rule above, including a marked record whose ETag differs from the slot's `RecordETag` → false); `TestHostIncarnationWrites` against `release.NewDirStore(t.TempDir())` (create-only once, second create-only fails with `release.ErrCASConflict`, `If-Match` with the returned ETag succeeds, a stale ETag fails, absent reads nil); `TestTombstoneBodyKeepsTheSequence` (a record body with `record_seq: 7` and many fields → the tombstone has exactly the six fields, `record_seq 7`, `retired true`; `WriteRecord` of it against the before bytes advances to 8); `TestGetRecordHidesATombstone` (after writing a tombstone, `GetRecord` returns the not-found error and `GetRecordRaw` returns `Retired: true`). Extend `TestRecordWireShapeIsAdditive` for the four fields. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/provision -run 'RecordLifetimeValid|HostIncarnationWrites|TombstoneBody|GetRecordHidesATombstone|RecordWireShapeIsAdditive' -count=1`. |
| -- [ ] **Step 3: Implement** the fields, `lifetime.go`, `GetRecordRaw`, and the tombstone check in `GetRecordETag`. |
| + - `Record.Incarnation uint64 \`json:"incarnation,omitempty"\``, `Record.IncarnationMigrating string \`json:"incarnation_migrating,omitempty"\``, `Record.MoveSeq uint64 \`json:"move_seq,omitempty"\``, `Record.Retired bool \`json:"retired,omitempty"\``. None of the four joins `protectedFieldsEqual` (`maintenance.go:81`): ordinary updates do not touch them, the migration's writes do, and the fence (Task 5) stops everyone else while a marker stands. |
| + - `type HostIncarnation struct { Incarnation uint64 \`json:"incarnation"\`; State string \`json:"state"\`; Account string \`json:"account"\`; App string \`json:"app"\`; Intent string \`json:"intent,omitempty"\`; RecordETag string \`json:"record_etag,omitempty"\` }`; `const SlotLive = "live"`, `SlotRetired = "retired"`. `Account`/`App` are the record key the lifetime belongs to (the sqid and app, as `RecordKey` takes them); `RecordETag` is set only by a migration slot. |
| + - `func HostIncarnationKey(host string) string` → `control/host-incarnations/<host>.json`; "" for a host failing `ValidHostShape`. |
| + - `func ReadHostIncarnation(ctx context.Context, s release.Store, host string) (*HostIncarnation, string, error)`: nil, "", nil when absent. |
| + - `func WriteHostIncarnation(ctx context.Context, s release.Store, host string, slot HostIncarnation, ifMatch string) (string, error)`: `ifMatch == ""` is create-only (`Store.Put` with "" is `If-None-Match: *` on S3 and refuses an existing file on `DirStore`), otherwise `If-Match`. |
| + - `type LifetimeVerdict int`; `const (LifetimeValid LifetimeVerdict = iota; LifetimeStale; LifetimeUnreadable)`. |
| + - `func LifetimeOf(rec *Record, recordETag string, slot *HostIncarnation) LifetimeVerdict` (pure; never `LifetimeUnreadable`, which only a failed slot read produces, Task 6). |
| + - `func TombstoneBody(before []byte) ([]byte, error)`: decodes `before` as JSON fields, keeps exactly `schema`, `host`, `account`, `app`, `incarnation` and `record_seq` (the last two only when present in `before`), adds `retired: true`, drops everything else. The caller writes it with `WriteRecord(ctx, s, key, before, body, etag)`, so `record_seq` advances by one when `before` carries it. |
| + - `func GetRecordRaw(ctx context.Context, s release.Store, acct, app, host string) (*Record, []byte, string, error)`: the decoded record (tombstones included), its exact bytes and ETag; the path check of `GetRecordETag` applies (a body naming another key is an error). |
| + - `GetRecordETag` (and so `GetRecord`) return `os.ErrNotExist` for a tombstone; `GetRecordRaw` is the only reader that sees one. |
| + |
| +The verdict, exactly, with `owned := slot != nil && slot.Account == rec.Account && slot.App == rec.App`: |
| +- `rec.Retired` → stale. |
| +- `rec.Incarnation == 0 && rec.IncarnationMigrating == ""` (fieldless): slot nil → valid; slot non-nil → stale. |
| +- `rec.IncarnationMigrating != ""`: slot nil → valid; slot `{1, live, owned, Intent == "migration " + marker, RecordETag == recordETag}` → valid; otherwise stale. |
| +- `rec.Incarnation > 0`: slot non-nil, `owned`, `State == live`, `slot.Incarnation == rec.Incarnation` → valid; otherwise stale. |
| + |
| +- [ ] **Step 1: Write the failing tests** in `lifetime_test.go`: `TestLifetimeOf` (one case per line above, plus a slot owned by another app → stale, and a marked record whose ETag differs from the slot's → stale); `TestHostIncarnationWrites` against `release.NewDirStore(t.TempDir())` (create-only once; a second create-only fails `release.ErrCASConflict`; `If-Match` with the returned ETag succeeds; a stale ETag fails; absent reads nil); `TestTombstoneBodyKeepsTheSequence` (a body with `record_seq: 7` and many fields → exactly the seven keys, `record_seq 7`, `retired true`; `WriteRecord` of it against `before` lands `record_seq 8`; a body without `record_seq` → six keys and no sequence); `TestGetRecordHidesATombstone`. Extend `TestRecordWireShapeIsAdditive` for the four fields. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/provision -run 'LifetimeOf|HostIncarnationWrites|TombstoneBody|GetRecordHidesATombstone|RecordWireShapeIsAdditive' -count=1`. |
| +- [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/provision -run 'Record' -count=1`. |
| -- [ ] **Step 5: Commit** `provision: record lifetime fields, the host incarnation slot, the tombstone form and the validity rule`. |
| +- [ ] **Step 5: Commit** `provision: record lifetime fields, the owned host incarnation slot, the tombstone form and the lifetime verdict`. |
| ### Task 2: Console tables for pending lifetimes, classified for the store gate |
| **Files:** |
| - Modify: `internal/console/store.go` (`migrate`, after the `account_plans` table), `internal/console/store_gate.go` (`claimSQLTables`) |
| - Create: `internal/console/lifetimes_store.go` |
| -- Test: `internal/console/lifetimes_store_test.go`; extend the existing claims-on store-open test beside `store_gate.go`'s tests so the three tables are classified |
| +- Test: `internal/console/lifetimes_store_test.go`; a new case in the store-gate tests |
| **Interfaces:** |
| -- Tables, all keyed by the record (`account` is the sqid, as the mail tables use): |
| - - `host_allocations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, state TEXT, body BLOB, predecessor_etag TEXT, created_at INTEGER)` |
| +- Tables (`account` is the sqid): |
| + - `host_allocations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, state TEXT, body BLOB, predecessor_etag TEXT, stale_etag TEXT, created_at INTEGER)` |
| - `record_retirements (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, record_etag TEXT, slot_etag TEXT, state TEXT, created_at INTEGER)` |
| - - `record_migrations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, record_etag TEXT, state TEXT, created_at INTEGER)` |
| + - `record_migrations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, marked_etag TEXT, state TEXT, created_at INTEGER)` |
| - Classification in `claimSQLTables`: `{Sqid: "account", App: "app"}` for all three. |
| - States: `pending`, `committed`, `superseded`, `void`, `refused`. |
| -- On `*Store`: `BeginAllocation(a Allocation) error`, `SetAllocationState(opID, state string) error`, `PendingAllocations() ([]Allocation, error)`, `LastIncarnation(host string) (uint64, error)` (the highest `incarnation` over every state but `void` for the host; 0 when none); `BeginRetirement(r Retirement) error`, `SetRetirementState(opID, state string) error`, `PendingRetirements() ([]Retirement, error)`; `BeginMigration(m Migration) error`, `SetMigrationState(opID, state string) error`, `PendingMigrations() ([]Migration, error)`; `PendingForHost(host string) (int, error)` (pending rows across the three tables). |
| -- `type Allocation struct { OpID, Account, App, Host string; Incarnation uint64; State string; Body []byte; PredecessorETag string }`, `type Retirement struct { OpID, Account, App, Host string; Incarnation uint64; RecordETag, SlotETag, State string }`, `type Migration struct { OpID, Account, App, Host, RecordETag, State string }`. |
| +- On `*Store`: `BeginAllocation(a Allocation) error`, `SetAllocationState(opID, state string) error`, `PendingAllocations() ([]Allocation, error)`, `LastIncarnation(host string) (uint64, error)` (the highest `incarnation` over every state but `void`; 0 when none); `BeginRetirement(r Retirement) error`, `SetRetirementState(opID, state string) error`, `PendingRetirements() ([]Retirement, error)`; `BeginMigration(m Migration) error`, `SetMigrationMarkedETag(opID, etag string) error`, `SetMigrationState(opID, state string) error`, `PendingMigrations() ([]Migration, error)`; `PendingForHost(host, exceptOpID string) (int, error)` (pending rows across the three tables, not counting `exceptOpID`). |
| +- `type Allocation struct { OpID, Account, App, Host string; Incarnation uint64; State string; Body []byte; PredecessorETag, StaleETag string }`, `type Retirement struct { OpID, Account, App, Host string; Incarnation uint64; RecordETag, SlotETag, State string }`, `type Migration struct { OpID, Account, App, Host, MarkedETag, State string }`. |
| -- [ ] **Step 1: Write the failing tests**: `TestLifetimeTablesRoundTrip` (begin, list pending, set state, list again, for all three); `TestLastIncarnationSkipsVoid` (pending 2 and void 3 → 2); `TestPendingForHostCountsAllThree`; and in the store-gate tests, `TestClaimStoreOpensWithLifetimeTables` (a claims-on gated open succeeds with the three tables present). |
| +- [ ] **Step 1: Write the failing tests**: `TestLifetimeTablesRoundTrip` (begin, list pending, set state and the marked ETag, list again, for all three); `TestLastIncarnationSkipsVoid` (pending 2 and void 3 → 2); `TestPendingForHostCountsAllThreeExceptOwn`; in the store-gate tests `TestClaimStoreOpensWithLifetimeTables`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement** with the store's `CREATE TABLE IF NOT EXISTS` pattern and the classification entries. |
| -- [ ] **Step 4: Run to verify they pass**: `go test ./internal/console -run 'LifetimeTables|LastIncarnation|PendingForHost|ClaimStoreOpensWithLifetimeTables' -count=1`. |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'LifetimeTables|LastIncarnation|PendingForHost|ClaimStoreOpensWithLifetimeTables' -count=1`. |
| - [ ] **Step 5: Commit** `console: tables for pending allocations, retirements and migrations, classified for the gate`. |
| ### Task 3: The create protocol and its recovery |
| **Files:** |
| -- Create: `internal/console/lifetimes.go` (the exclusion and the shared helpers), `internal/console/lifetimes_create.go`, `internal/console/lifetimes_recovery.go` |
| -- Modify: `internal/console/apiinstances.go` (the claims-off branch at the create door, the `provision.CreateRecord` call near line 1664), `internal/console/apiinstanceseed.go` (line 341), `internal/console/apiinstanceimport.go` (line 172), `internal/console/domainattach.go` (the first-route create near line 237), `internal/console/claim_workers.go` (register `"StartLifetimeRecoveryPass"`), `cmd/console/main.go` (start it beside the other passes) |
| +- Create: `internal/console/lifetimes.go` (the exclusion, admission and shared classification), `internal/console/lifetimes_create.go`, `internal/console/lifetimes_recovery.go` |
| +- Modify: `internal/console/apiinstances.go` (the claims-off branch at line 1664: `recordWriteErr = provision.CreateRecord(...)` becomes the lifetime create; the sibling undo at line 1684 stays on `retireInstanceRecord`, which Task 4 routes), `internal/console/apiinstanceseed.go` (line 341), `internal/console/apiinstanceimport.go` (line 172), `internal/console/domainattach.go` (the first-route create near line 237 and the two `rollbackAttach` calls that follow it), `internal/console/claim_workers.go` (register `"StartLifetimeRecoveryPass"`), `cmd/console/main.go` (start it beside the other passes) |
| - Test: `internal/console/lifetimes_create_test.go` |
| **Interfaces:** |
| -- Consumes: Task 1 and Task 2. |
| +- Consumes: Tasks 1 and 2. |
| - Produces: |
| - - `func (s *Server) lifetimesOn() bool` (`!s.cfg.HostClaim`). |
| - - `func (s *Server) hostnameMutex(host string) *sync.Mutex` (a `sync.Map` of mutexes). |
| - - `func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record) error`: the one function every claims-off creation path calls; returns `release.ErrCASConflict` for a live existing record (the doors keep their existing 409 mapping), `errLifetimeSettling` (mapped to 409 with the settling sentence) while a pending operation exists for the host. |
| - - `func (s *Server) recoverLifetimes(ctx context.Context) error` and `func (s *Server) StartLifetimeRecoveryPass(ctx context.Context)` (once at start, then every 60 s). |
| - |
| -The protocol, under `hostnameMutex(host)`: |
| -1. `PendingForHost(host) > 0` → `errLifetimeSettling`. |
| -2. Read the record (`GetRecordRaw`) and the slot. A record present and `RecordLifetimeValid` → `release.ErrCASConflict` (the existing conflict; the slot is not touched). A record present and not valid (a tombstone, or a lifetime behind the slot) is stale: remembered with its ETag for step 6. |
| -3. N = max(`LastIncarnation(host)`, slot.Incarnation) + 1; `rec.Incarnation = N`; `body = json.Marshal(rec)`; `BeginAllocation({OpID: new id, Account, App, Host, N, pending, body, predecessor ETag or "" when the slot is absent})`. |
| -4. `WriteHostIncarnation({N, live, "create " + OpID}, predecessor)`. |
| -5. If a stale record was found, `release.DeleteIfMatch` its ETag (a conflict here means the key changed: reread and resolve by the recovery branches). |
| -6. `WriteRecord(ctx, s, key, []byte("{}"), body, "")` (create-only). |
| -7. `SetAllocationState(committed)`. |
| - |
| -Recovery (`recoverLifetimes`), per pending allocation, under the hostname mutex, reading the slot and the record raw: slot absent or at the predecessor ETag → step 4 onward; slot `{N, live, "create " + OpID}` and record absent → step 6 onward; slot at N and the record at N → committed; slot at N and a stale record under the key → step 5 onward; slot past N or at N with another intent → `superseded` (an operator moved it by hand); any read error → leave pending. Then the retirements (Task 4) and migrations (Task 5). |
| + - `func (s *Server) lifetimesOn() bool` (`!s.cfg.HostClaim && s.cfg.Releases != nil`). |
| + - `func (s *Server) withHostname(host string, fn func() error) error` (a `sync.Map` of `*sync.Mutex`; not reentrant; every exported helper below takes it, every `...Locked` helper assumes it). |
| + - `func (s *Server) admit(host, ownOpID string) error`: `errLifetimeSettling` when `PendingForHost(host, ownOpID) > 0`. |
| + - `func (s *Server) classifyLocked(ctx context.Context, acct, app, host string) (lifetimeClass, error)` where `type lifetimeClass struct { Rec *provision.Record; Raw []byte; RecordETag string; Slot *provision.HostIncarnation; SlotETag string; Verdict provision.LifetimeVerdict }` (`Verdict` is `LifetimeUnreadable` and the error non-nil when either read fails for a reason other than absence). |
| + - `func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record) (*provision.Record, error)`: the one function every claims-off creation path calls; returns the record as written (incarnation set, `RecordSeq` set). Errors: `release.ErrCASConflict` (a valid record under this key; the doors keep their 409), `errHostHeld` (a live slot owned by another record key, or `hostHolder` reports another holder; the doors answer the existing `hostHeldSentence`), `errLifetimeSettling` (409, the settling sentence), `errLifetimePending` (the allocation is persisted and the record write's outcome is unknown: recovery completes it; the door answers its existing "could not create" sentence and **does not** undo anything). |
| + - `func (s *Server) recoverLifetimes(ctx context.Context) error`, `func (s *Server) StartLifetimeRecoveryPass(ctx context.Context)` (returns at once unless `lifetimesOn()`; once at start, then every 60 s). |
| + |
| +The protocol, inside `withHostname(host)`: |
| +1. `admit(host, "")`. |
| +2. `classifyLocked`. Unreadable → the read error (502 at the doors). Valid record under this key → `ErrCASConflict`; the slot is untouched. A live slot owned by another key → `errHostHeld`. Then `s.hostHolder(ctx, acct, host, app)` again, under the lock (the pre-lock call at `apiinstances.go:1164` stays): a holder → `errHostHeld`. A stale record under this key (a tombstone, or a lifetime behind the slot) is remembered: `StaleETag`. |
| +3. `N = max(LastIncarnation(host), slot.Incarnation) + 1`; `rec.Incarnation = N`; `body = json.Marshal(rec)`; `BeginAllocation({OpID, Account, App, Host, N, pending, body, PredecessorETag: slot's ETag or "", StaleETag})`. |
| +4. `WriteHostIncarnation({N, live, rec.Account, rec.App, "create " + OpID}, PredecessorETag)`. A CAS conflict here (the slot moved after step 2) → `superseded`, and the caller gets `errLifetimeSettling` (the next request re-classifies). |
| +5. If `StaleETag != ""`: `DeleteIfMatch` the record on it; a conflict means the key changed → leave `pending` and return `errLifetimePending` (recovery re-classifies). |
| +6. `WriteRecord(ctx, s, key, []byte("{}"), body, "")`. Success → 7. `ErrCASConflict` → re-read raw: the body at N under this key → 7 (a replay landed); otherwise `pending` and `errLifetimePending`. Any other error → `pending` and `errLifetimePending`. |
| +7. `SetAllocationState(committed)`; return the written record. |
| + |
| +Recovery (`recoverLifetimes`), per pending allocation, inside `withHostname` with `admit(host, OpID)` (its own operation does not refuse itself), after `classifyLocked`, the first matching branch: |
| +- unreadable → leave pending; |
| +- slot at an incarnation past N, or at N with an intent other than `"create " + OpID` → `superseded` (an operator or a later lifetime moved it); |
| +- slot `{N, live, this key, "create " + OpID}` and a record at N under this key → `committed`; |
| +- slot as above and a record under the key that is not at N → step 5 with the current ETag, then 6; |
| +- slot as above and no record → step 6; |
| +- slot absent, or at `PredecessorETag` → step 4; |
| +- anything else → `superseded`. |
| +Then the retirements (Task 4) and the migrations (Task 5), in that order. |
| + |
| +Door mappings: the create door (`apiinstances.go`): `errHostHeld` → 409 `hostHeldSentence`; `errLifetimeSettling` → 409 settling; `errLifetimePending` → 502 "could not create the instance — try again" (the existing sentence), no undo. Seed (`apiinstanceseed.go:341`) and import (`apiinstanceimport.go:172`): the same three mappings onto their existing failure sentences; on `errLifetimePending` the door writes nothing further, and the record recovery completes carries the seed state the body had, which the existing seed sweep then owns (a begun seed with no upload times out as today). Attach (`domainattach.go`): `ErrCASConflict` keeps today's adoption branch; `errLifetimePending` → the ambiguous branch, but `rollbackAttach(ctx, acct, app, host, false, false)`: the claim is **kept** (it is this app's; the record recovery completes is this app's first-route record under its own claim, the case the door already adopts on retry), and the answer is the existing "could not declare the route for this domain — nothing was attached, try again"; `errLifetimeSettling` and `errHostHeld` → 409 with their sentences and `rollbackAttach(..., false, created)` as today's refusal branch does. |
| - [ ] **Step 1: Write the failing tests** against an in-process console with `release.NewDirStore` and `HostClaim` off: |
| - - `TestCreateAllocatesSlotThenRecord`, `TestCreateAfterRetirementGetsNextIncarnation` (seeded slot `{1, retired}` → `{2, live}`), `TestCreateReplacesStaleRecord` (a fieldless record beside `{1, retired}` is deleted conditionally and incarnation 2 written), `TestCreateOfALiveInstanceIsTheExistingConflict` (record at 1 valid → `ErrCASConflict`, slot unchanged, no allocation row), `TestCreateRefusesWhileAnOperationIsPending` (a pending allocation row for the host → `errLifetimeSettling`). |
| - - `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`: a pending allocation with the slot written and no record; recovery writes the record and commits; an injected late create-only write of the same body then fails with `ErrCASConflict` and the record's ETag is unchanged. |
| - - `TestRecoveryWritesTheSlotWhenMissing`, `TestRecoveryContinuesFromAPredecessorSlot` (slot still at the predecessor's ETag), `TestRecoveryLeavesALandedCreateServing` (record ETag unchanged, state committed), `TestRecoveryRemovesAStaleRecordThenWrites`, `TestRecoverySupersedesWhenSlotMovedPast`, `TestRecoveryLeavesPendingWhenUnreadable` (a store whose `Get` of the slot fails), `TestRecoverySurvivesASecondCrash` (recovery run twice with a crash injected after step 4 on the first run). |
| - - `TestRecoveryUsesOnlyPersistedState`: close and reopen the SQLite store between the crash and the recovery; no request-local state. |
| + - `TestCreateAllocatesSlotThenRecord` (slot `{1, live, bab, shop, "create <id>"}`, record `Incarnation 1`, the returned record has it), `TestCreateAfterRetirementGetsNextIncarnation` (`{1, retired}` → `{2, live}`), `TestCreateReplacesStaleRecord` (a fieldless record beside `{1, retired}` is deleted on its ETag; incarnation 2 written), `TestCreateOfALiveInstanceIsTheExistingConflict` (slot unchanged, no allocation row), `TestCreateRefusesWhileAnOperationIsPending`, `TestCreateRefusesALiveSlotOfAnotherApp` (slot `{1, live, bab, other}` → `errHostHeld`, slot unchanged), `TestTwoAppsCreatingOneHostnameSerialise` (two goroutines, apps `shop` and `other`, one host: exactly one slot write, one record, the loser gets `errHostHeld` or `ErrCASConflict`, never a second lifetime). |
| + - `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`: a pending allocation with the slot written and no record; recovery writes the record and commits; an injected late create-only write of the same body fails `ErrCASConflict` and the record's ETag is unchanged. |
| + - `TestDeleteDuringCreateRecoveryIsRefused` (a DELETE while the allocation is pending → 409 settling, nothing written). |
| + - `TestRecoveryWritesTheSlotWhenMissing`, `TestRecoveryContinuesFromAPredecessorSlot`, `TestRecoveryLeavesALandedCreateServing`, `TestRecoveryRemovesAStaleRecordThenWrites`, `TestRecoverySupersedesWhenSlotMovedPast`, `TestRecoveryLeavesPendingWhenUnreadable`, `TestRecoverySurvivesASecondCrash` (a crash injected after step 4 on the first recovery run), `TestRecoveryUsesOnlyPersistedState` (the SQLite store is closed and reopened between the crash and the recovery). |
| + - `TestAttachKeepsItsClaimOverAPendingAllocation` (failure injected after the slot write in the attach's first-route create: the claim stands, the answer is the ambiguous sentence, recovery completes the record, a second attach adopts it). |
| + - `TestSeedCreatePendingIsOwnedByTheSweep` (the same injection at the seed door: recovery completes the record with its seed state; the seed sweep's existing timeout path then rolls it back). |
| - [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement**; route the four claims-off creation paths through `createRecordWithLifetime` (the claims-on branches are untouched). |
| -- [ ] **Step 4: Run** `go test ./internal/console -run 'Create|Lifetime|Recovery|DomainAttach|Import|Seed' -count=1`. |
| -- [ ] **Step 5: Commit** `console: claims-off creates allocate a host incarnation through a persisted intent, with recovery`. |
| +- [ ] **Step 3: Implement**; route the four claims-off creation paths through `createRecordWithLifetime`. |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'Create|Lifetime|Recovery|Attach|Import|Seed' -count=1`. |
| +- [ ] **Step 5: Commit** `console: claims-off creates allocate an owned host incarnation through a persisted intent, with recovery`. |
| -### Task 4: Retirement through a tombstone |
| +### Task 4: Retirement through a tombstone, from every claims-off removal |
| **Files:** |
| - Create: `internal/console/lifetimes_retire.go` |
| -- Modify: `internal/console/hostclaimops.go` (the claims-off branch of `retireHostRecord`, line ~23: call `retireRecordWithLifetime` for the record, then the rest of `provision.RetireRecord`'s effects with the record already gone), `internal/provision/retirement.go` (`RetireOptions` gains `RecordRetired bool` so `RetireRecord` skips its own record deletion when the console already retired it), `internal/console/apiinstances.go` (`handleAPIInstanceDelete`: map `errRetireRefused` to 409 with its sentence, `errLifetimeSettling` to 409, a `void` outcome to the existing already-absent 200), `internal/console/lifetimes_recovery.go` (the retirement branches) |
| +- Modify: `internal/console/hostclaimops.go` (`retireHostRecord`, claims-off branch: `ref.Family == "instance"` → `retireRecordWithLifetime(ctx, ref.Account, ref.App, ref.Host, "")`; sidecars and everything else unchanged on `provision.RetireRecord`), `internal/console/apiinstanceimport.go` (`deleteRead`: `release.DeleteIfMatch(...)` becomes `retireRecordWithLifetime(ctx, acct.Sqid, app, host, etag)`; `ErrCASConflict` keeps the `again` re-decision; `errLifetimeSettling` → 409 settling; `errRetireRefused` → 409 its sentence), `internal/console/seedfinish.go` (`finishSeedOnce`'s final `DeleteIfMatch` becomes `retireRecordWithLifetime(ctx, sqid, app, host, etag)` with the ETag `deleteSeedReplica` returned; the seed resource teardown before it is unchanged; `ErrCASConflict` keeps `finishSeed`'s one retry), `internal/console/apiinstances.go` (`handleAPIInstanceDelete`: map `errRetireRefused` and `errLifetimeSettling` to 409 on the `retireInstanceRecord` branch), `internal/console/lifetimes_recovery.go` (the retirement branches) |
| - Test: `internal/console/lifetimes_retire_test.go` |
| **Interfaces:** |
| -- Consumes: Tasks 1 to 3; `release.DeleteIfMatch`; Task 5's `migrateHostLifetime` for a legacy record (so build Task 5's function first or stub it behind the same signature). |
| -- Produces: `func (s *Server) retireRecordWithLifetime(ctx context.Context, acct, app, host string) error`; `var errRetireRefused`. |
| - |
| -The protocol, under the hostname mutex, pending-op check as in Task 3: |
| -- A record with no `Incarnation` and no slot (legacy): run `migrateHostLifetime` first, under the same exclusion; if it cannot complete (unreadable), refuse with `errLifetimeSettling`. Never a plain delete. |
| -- Read the record raw (bytes B, ETag E) and the slot (ETag S); `BeginRetirement({OpID, Account, App, Host, N, E, S, pending})`. |
| -- (1) `WriteRecord(ctx, s, key, B, TombstoneBody(B), E)`. On `ErrCASConflict`: reread; a non-tombstone record at N → `refused` (`errRetireRefused`); another incarnation, or slot past N → `void`. |
| -- (2) `WriteHostIncarnation({N, retired, "retire " + OpID}, S)`. |
| -- (3) `DeleteIfMatch` the tombstone on its ETag. |
| -- `committed`. |
| - |
| -Recovery branches, from the persisted E and S: record at E → (1); tombstone for N present and slot at S → (2); tombstone present and slot `{N, retired}` → (3); record absent and slot retired → committed; slot `{N, retired}` and a non-tombstone record at N (any ETag) → committed, and the record is deleted conditionally on its ETag (stale under a retired slot); slot live at S and a non-tombstone record at N with an ETag other than E → `refused`; another incarnation or slot past N with no tombstone → `void`; unreadable → pending. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestRetireWritesTombstoneThenRetiresSlotThenRemovesTombstone` (with a hook after (1): `GetRecord` not found, `GetRecordRaw` retired, `record_seq` advanced by one); `TestRetireRefusesWhenTheRecordChangedInItsLifetime` (a `SetRecordRegion` between the read and (1) → the door answers 409 with the sentence; record kept; state refused); `TestRetireVoidsAgainstASuccessor`; `TestLateCreateFailsOnTheTombstoneAndIsStaleAfterIt`; `TestLegacyRecordIsMigratedThenRetired` (no plain delete; the slot ends `{1, retired}`); `TestDelayedLegacyDeleteCannotEraseASuccessor` (a plain `Store.Delete` issued against the old key is not part of the protocol; assert the door never calls `Store.Delete` for a record key, through a counting store wrapper); `TestRetirementRecoveryBranches` (one subtest per branch above, each asserting the final slot, key and state, with a second crash injected during recovery in two of them). |
| +- Consumes: Tasks 1 to 3; Task 5's `migrateHostLifetimeLocked` (build Task 5's function first or stub it behind the same signature). |
| +- Produces: `func (s *Server) retireRecordWithLifetime(ctx context.Context, acct, app, host, expectETag string) error`; `var errRetireRefused`. A nil return means the key is absent and the slot, if any, retired or someone else's: the callers' "deleted" and "already gone" answers. |
| + |
| +Inside `withHostname(host)`: `admit(host, "")`; `classifyLocked`; when `expectETag != ""` and the record's ETag differs → `release.ErrCASConflict` (the caller re-decides, as `deleteRead` does today). Then the first matching case: |
| +- (a) record absent, slot absent → nil. |
| +- (b) record absent, slot `{N, live}` owned by this key → an **orphan slot**: `BeginRetirement({..., N, RecordETag: "", SlotETag})`, then step (2) below, `committed`. (A crashed create would be a pending allocation and refused at `admit`; what remains is an old console's plain delete, and retiring the slot makes its late record stale.) |
| +- (c) record absent, slot owned by another key, or retired → nil. |
| +- (d) record present, verdict stale: a tombstone or a fieldless/behind record → `DeleteIfMatch` on its ETag (a conflict → `errLifetimeSettling`); the slot is untouched; nil. |
| +- (e) record present, fieldless, slot absent (legacy) → `migrateHostLifetimeLocked` (Task 5); if it does not reach `committed` → `errLifetimeSettling`; then re-classify and continue at (f). |
| +- (f) record valid at N under slot `{N, live, this key}` at ETag S → `BeginRetirement({OpID, Account, App, Host, N, E, S, pending})`, then: |
| + - (1) `WriteRecord(ctx, s, key, raw, TombstoneBody(raw), E)`. `ErrCASConflict` → re-classify: a non-tombstone record at N → `refused` (`errRetireRefused`); a tombstone at N → continue at (2); anything else → `void`, nil. |
| + - (2) `WriteHostIncarnation({N, retired, Account, App, "retire " + OpID}, S)`. `ErrCASConflict` → re-read the slot: `{N, retired}` → continue; otherwise `void` (a successor exists; the tombstone is left for (d) of a later call or for recovery). |
| + - (3) `DeleteIfMatch` the tombstone on its ETag (absence is success). |
| + - `committed`. |
| + |
| +Recovery, per pending retirement, inside `withHostname` with `admit(host, OpID)`, after `classifyLocked`, the first matching branch: unreadable → pending; slot past N, or at N and live with another intent and a non-tombstone record → `void`; record at `RecordETag` (unchanged) and slot at `SlotETag` → (1); tombstone at N and slot `{N, live}` at `SlotETag` → (2); tombstone at N and slot `{N, retired}` → (3); record absent and slot `{N, retired}` → `committed`; `RecordETag == ""` (an orphan slot retirement) and slot `{N, live}` at `SlotETag` → (2); slot `{N, live}` at `SlotETag` and a non-tombstone record at N whose ETag is not `RecordETag` → `refused`; anything else → `void`. |
| + |
| +- [ ] **Step 1: Write the failing tests** through the real doors (`DELETE /api/cli/.../instances/{host}` and the seed sweep) and the function: |
| + - `TestDeleteDoorRetiresThroughTheTombstone` (the import rule's ordinary path: after a hook at (1) the key holds a tombstone, `GetRecord` says not found, `record_seq` advanced by one; at the end the slot is `{1, retired}` and the key absent; the door answers "deleted"); `TestSlotRetiresBeforeTheKeyIsAbsent` (a hook between (2) and (3): the slot is already retired while the tombstone stands, and a create-only write of a record at incarnation 1 is stale under it). |
| + - `TestSeedFinishRetiresThroughTheTombstone`. |
| + - `TestRetireRefusesWhenTheRecordChangedInItsLifetime` (a `SetRecordRegion` between the read and (1) → 409 with the sentence; record kept; state refused); `TestRetireVoidsAgainstASuccessor` (a stale record at 1 beside a live slot `{2}`: case (d), the slot stays `{2, live}`; and a retirement begun at `{1, live}` whose slot moves to `{2}` before (2) → void, slot untouched); `TestRetireNeverMovesTheSlotBackwards` (every case of a record at 1 beside slots `{2, live}` and `{2, retired}`: the slot's ETag is unchanged afterwards). |
| + - `TestOrphanSlotIsRetired` (record absent, slot `{1, live}`, no pending allocation → slot `{1, retired}`; a late fieldless create-only write then lands stale). |
| + - `TestLegacyRecordIsMigratedThenRetired` (no plain delete; the slot ends `{1, retired}`). |
| + - `TestDelayedDeleteCannotEraseASuccessor`: block the conditional tombstone write (1) of a retirement with a hook; let recovery run after a restart (it finds the record at `RecordETag` and completes (1)–(3)); create the host again (`{2, live}`, record at 2); release the blocked write: it fails `ErrCASConflict`; the successor's record and slot are intact. |
| + - `TestRetirementRecoveryBranches` (one subtest per branch above; a second crash injected during recovery in two of them). |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/console -run 'Retire|Tombstone|Lifetime|InstanceDelete' -count=1` and `go test ./internal/provision -run 'Retire' -count=1`. |
| -- [ ] **Step 5: Commit** `console: claims-off retirement replaces the record with a tombstone before retiring the slot`. |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'Retire|Tombstone|Lifetime|InstanceDelete|SeedFinish|DeleteDoor' -count=1` and `go test ./internal/provision -run 'Retire' -count=1`. |
| +- [ ] **Step 5: Commit** `console: every claims-off record removal retires through a tombstone, classifying the record and its slot together`. |
| ### Task 5: Lazy migration with a marker fenced in the write primitive |
| **Files:** |
| -- Modify: `internal/provision/record_sequence.go` (`WriteRecord`: after decoding `before`, if its `incarnation_migrating` is non-empty and the context does not carry `WithMigration(ctx, thatID)`, return `ErrRecordMigrating`; a create-only write is unaffected), `internal/provision/provision.go` (`updateRecordSeeding` and `SetRecordRegion`: treat `ErrRecordMigrating` as a retryable conflict with backoff, at most 30 s, then return it), `internal/console/claim_workers.go` (register `"StartIncarnationMigrationPass"`), `cmd/console/main.go` |
| -- Create: `internal/console/lifetimes_migrate.go`, `internal/provision/lifetime_migration.go` (`WithMigration`, `ErrRecordMigrating`) |
| +- Modify: `internal/provision/record_sequence.go` (`WriteRecord`: after `RecordSequence(before)`, if `before`'s `incarnation_migrating` is non-empty and `ctx` does not carry `WithMigration(ctx, thatID)`, return `ErrRecordMigrating`; a create-only write, `before == {}`, is unaffected), `internal/provision/provision.go` (`updateRecordSeeding`: when `update()` returns an error that `Is` `ErrRecordMigrating`, retry every second up to `recordMigrationWait` (30 s; a test hook shortens it), re-reading each time, then return it; `claimRawUpdateRecord`: returns `ErrRecordMigrating` as it comes, which `Is` `release.ErrCASConflict`, so every hostops writer re-reads as it does for a lost race), `internal/console/claim_workers.go` (register `"StartIncarnationMigrationPass"`), `cmd/console/main.go` |
| +- Create: `internal/provision/lifetime_migration.go` (`ErrRecordMigrating`, `WithMigration`), `internal/console/lifetimes_migrate.go` |
| - Test: `internal/provision/record_sequence_test.go`, `internal/console/lifetimes_migrate_test.go` |
| **Interfaces:** |
| -- Produces: `var provision.ErrRecordMigrating = errors.New("provision: the record is being migrated; try again")`; `func provision.WithMigration(ctx context.Context, id string) context.Context`; `func (s *Server) migrateHostLifetime(ctx context.Context, acct, app, host string) error`; `func (s *Server) StartIncarnationMigrationPass(ctx context.Context)` (behind `CARLOS_LIFETIMES_MIGRATE=1`, every 5 minutes, at most 50 records per pass, oldest first, only records with no `Incarnation` and no marker whose slot is absent). |
| +- Produces: `var ErrRecordMigrating = fmt.Errorf("%w: the record is being migrated; read it again", release.ErrCASConflict)`; `func WithMigration(ctx context.Context, id string) context.Context`; `func (s *Server) migrateHostLifetimeLocked(ctx context.Context, acct, app, host string) (state string, err error)` (assumes `withHostname`; returns the final row state); `func (s *Server) StartIncarnationMigrationPass(ctx context.Context)` (behind `CARLOS_LIFETIMES_MIGRATE=1` and `lifetimesOn()`, every 5 minutes, at most 50 records per pass, oldest `created_at` first, each inside `withHostname` and `admit`, only fieldless unmarked records whose slot is absent). |
| + |
| +The protocol (inside the exclusion): `admit(host, "")`; `BeginMigration({OpID, Account, App, Host, "", pending})`; (1) `classifyLocked`; the record must be fieldless, unmarked, slot absent (else `superseded`); `WriteRecord(WithMigration(ctx, OpID), s, key, raw, raw with IncarnationMigrating = OpID, E)`; on `ErrCASConflict` re-read and retry from the current record, at most 5 times, then leave pending; `SetMigrationMarkedETag(OpID, M)` with the returned ETag; (2) `WriteHostIncarnation({1, live, Account, App, "migration " + OpID, RecordETag: M}, "")` create-only, or `If-Match` when a slot `{1, live, this key, "migration " + OpID}` already exists (an earlier attempt's); (3) `WriteRecord(WithMigration(ctx, OpID), ..., Incarnation = 1, IncarnationMigrating = "", against M)`; `committed`. The slot's `RecordETag` stays M afterwards; the verdict of the finished record does not read it. |
| -The protocol, under the hostname mutex: `BeginMigration({OpID, Account, App, Host, "", pending})`; (1) read raw (B, E); `WriteRecord(WithMigration(ctx, OpID), s, key, B, B with IncarnationMigrating = OpID, E)`; on `ErrCASConflict` reread and retry from the current record (bounded to 5 attempts, then leave pending); persist the marked record's ETag M in the row; (2) `WriteHostIncarnation({1, live, "migration " + OpID, RecordETag: M}, "")` create-only, or `If-Match` when a slot at 1 with this intent exists from an earlier attempt; (3) `WriteRecord(WithMigration(ctx, OpID), ..., Incarnation = 1, IncarnationMigrating = "", against M)`; `committed`. Recovery: record unmarked and fieldless, slot absent → (1); record marked with this OpID, slot absent → (2); record marked, slot bound to M → (3); record carries `Incarnation 1` → committed; record gone, another lifetime, or a slot not of this migration → `superseded`; unreadable → pending. A fieldless record beside an unrelated existing slot is never migrated: it is stale (Task 1's rule) and the create door removes it. |
| +Recovery, per pending migration, inside the exclusion with `admit(host, OpID)`: unreadable → pending; record carries `Incarnation 1` and slot `{1, live, this key, "migration " + OpID}` → `committed`; record marked with OpID, slot bound to the record's current ETag → (3); record marked with OpID and `MarkedETag == ""` → `SetMigrationMarkedETag(OpID, current ETag)` then (2); record marked with OpID, slot absent → (2); record fieldless and unmarked, slot absent → (1); record gone, another lifetime, a marker of another op, or a slot not of this migration → `superseded`. |
| - [ ] **Step 1: Write the failing tests**: |
| - - In `record_sequence_test.go`: `TestWriteRecordRefusesAMarkedRecordWithoutTheMigrationContext`, `TestWriteRecordAllowsTheMigrationsOwnWrite`, `TestCreateOnlyWriteIgnoresTheMarker`. |
| - - `TestMigrationMarksThenSlotsThenSets` (final: slot `{1, live, "migration <id>", RecordETag M}`, record `Incarnation 1`, no marker, state committed). |
| - - `TestMigrationRestartsFromAnUpdateThatWon` (an update between the read and (1); the marker CAS fails; the pass rereads and completes; the update's field survives). |
| - - `TestAnIssuedWriteLosesToTheMarkerAndRetries` (a box-shaped `SetRecordMove` goroutine blocked on a hook with the pre-marker ETag: its `WriteRecord` fails `ErrCASConflict`, its reread sees the marker and gets `ErrRecordMigrating`, it waits, and succeeds after the migration's (3)). |
| - - `TestWriterGivesUpAfterTheBound` (marker never cleared → `ErrRecordMigrating` after the bound, shortened by a test hook). |
| - - `TestMigrationRecoveryBranches` (one subtest per branch). |
| + - `record_sequence_test.go`: `TestWriteRecordRefusesAMarkedRecordWithoutTheMigrationContext` (the error `Is` both `ErrRecordMigrating` and `release.ErrCASConflict`), `TestWriteRecordAllowsTheMigrationsOwnWrite`, `TestCreateOnlyWriteIgnoresTheMarker`, `TestClaimRawUpdateRecordSurfacesTheMarker`. |
| + - `TestMigrationMarksThenSlotsThenSets` (final: slot `{1, live, bab, shop, "migration <id>", RecordETag M}`, record `Incarnation 1`, no marker, committed). |
| + - `TestMigrationRestartsFromAnUpdateThatWon` (an update between the read and (1); the marker CAS fails; the pass re-reads and completes; the update's field survives). |
| + - `TestAnIssuedWriteLosesToTheMarkerAndRetries` (a box-shaped `SetRecordMove` goroutine blocked on a hook with the pre-marker ETag: its `WriteRecord` fails `ErrCASConflict`, its re-read gets `ErrRecordMigrating`, it waits, and succeeds after (3)). |
| + - `TestWriterGivesUpAfterTheBound` (the marker never clears → `ErrRecordMigrating` after the shortened bound). |
| + - `TestMigrationRecoveryBranches` (one subtest per branch, including a restart after the marker write and before `MarkedETag` is persisted). |
| - `TestMigrationPassIsOffWithoutTheFlag`, `TestMigrationSkipsAFieldlessRecordBesideAnUnrelatedSlot`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/provision -run 'WriteRecord|Migrat|Record' -count=1` and `go test ./internal/console -run 'Migrat' -count=1`. |
| +- [ ] **Step 4: Run** `go test ./internal/provision -run 'WriteRecord|Migrat|Record|ClaimRawUpdate' -count=1` and `go test ./internal/console -run 'Migrat' -count=1`. |
| - [ ] **Step 5: Commit** `lifetimes: lazy migration marks the record by CAS, fenced in the write primitive`. |
| -### Task 6: Consumers check the lifetime |
| +### Task 6: Consumers judge the lifetime |
| **Files:** |
| -- Create: `internal/provision/lifetime_cache.go` |
| -- Modify: `internal/provision/reconcile.go` (where the pass lists and decodes records into its live set, before the minting loop: drop invalid records there, so the removal loop retires their rows as "record gone"), `internal/provision/provision.go` (`ListRecords`: unchanged; add `func ListRecordsValid(ctx, s, cache *SlotCache, ...)` used by reconcile and by the box's moves worker's record listing in `internal/host/movespass.go`), `internal/console/apiinstances.go` (the door's record read helper that every instance door uses after `instanceGate`: a record that fails validity reads as not found), `internal/host/edgesfile.go` (a row whose record is invalid is not published: the writer already has the records in hand for owners; add the check there), the agent's `provision.ReconcileConfig{...}` construction in `internal/host` (set `Slots: provision.NewSlotCache()`) |
| -- Test: `internal/provision/reconcile_lifetime_test.go`, `internal/console/apiinstances_lifetime_test.go`, `internal/host/edgesfile_lifetime_test.go` |
| +- Create: `internal/provision/lifetime_cache.go`, `internal/console/lifetimes_read.go` |
| +- Modify: `internal/provision/reconcile.go` (the decode loop at lines 855–941: `getEach` and `fetched` gain `etag`; after the path check and before `ValidHostShape`, the lifetime judgement; `ListAllRecordsStrict` gains a judged sibling and `readAllRecords` takes the cache), `internal/provision/provision.go` (`GetRecordETag`'s judged sibling; `ListRecordsStrict`, `ListRecordsComplete`, `ListRecords` gain judged siblings), `internal/host/agent.go` (`provisionTickConfig`: `Slots: provision.NewSlotCache()` when `!hostwrite.Enabled(getenv)`, nil otherwise), `internal/host/movespass.go` (line 1079) and `internal/host/movesoffline.go` (`refreshOfflineView`): the judged listing with the box's cache, `internal/console/*.go` (every `provision.GetRecord`, `GetRecordETag`, `ListRecords*` and `ListAllRecords*` call on instance records in the files the inventory below names goes through the `lifetimes_read.go` helpers) |
| +- Test: `internal/provision/reconcile_lifetime_test.go`, `internal/provision/lifetime_cache_test.go`, `internal/console/lifetimes_read_test.go`, `internal/host/lifetime_gating_test.go` |
| **Interfaces:** |
| -- Produces: `type SlotCache struct{...}`; `func NewSlotCache() *SlotCache`; `func (c *SlotCache) Valid(ctx context.Context, s release.Store, rec *Record, recordETag string) (bool, error)` (reads the slot for every record on a claims-off deployment, fieldless ones included, cached per host by the slot's ETag with a conditional read; a read error returns `false, err` and the caller treats the record as unreadable for this pass, not as absent); `ReconcileConfig.Slots *SlotCache` (required in production; a nil cache makes reconcile log once and treat every record as valid, for tests that predate this, and `TestProductionReconcileConfigHasASlotCache` pins that the agent sets it). |
| +- Produces: |
| + - `type SlotCache struct{...}`; `func NewSlotCache() *SlotCache`; `func (c *SlotCache) Judge(ctx context.Context, s release.Store, rec *Record, recordETag string) (LifetimeVerdict, error)`: reads the slot (one `Get`; cached per host with the slot's ETag for 5 s; absence cached too; safe for concurrent use; a failed read → `LifetimeUnreadable, err`); then `LifetimeOf`. On a `stale` verdict where the slot's `Intent` begins `"migration "`, or the record is fieldless, it re-reads the record once (`GetRecordRaw`) and judges the fresh pair; still stale → stale. A nil `*SlotCache` judges everything valid (claims-on). |
| + - `func ReadRecordJudged(ctx context.Context, s release.Store, c *SlotCache, acct, app, host string) (*Record, string, error)`: `GetRecordRaw` then `Judge`; a tombstone or stale → `os.ErrNotExist`; unreadable → the slot read error. |
| + - `func ListRecordsStrictJudged(ctx context.Context, s release.Store, c *SlotCache, acct, app string, max int) (recs []Record, unreadable map[string]bool, err error)`, `ListRecordsCompleteJudged(ctx, s, c, acct, app, max) (recs []Record, truncated bool, skipped []string, err error)`, `ListRecordsJudged(ctx, s, c, acct, app, max) (recs []Record, truncated bool, err error)`, `ListAllRecordsStrictJudged(ctx context.Context, s release.Store, c *SlotCache) (recs []Record, unreadable map[string]bool, err error)`: each is its sibling with stale records dropped and an unreadable slot reported as that host unreadable (strict: the listing error, as a failed record read is today; complete: `skipped` names the host; the tolerant one: `complete = false`). |
| + - `ReconcileConfig.Slots *SlotCache`. In the decode loop: verdict stale → `continue` (not in `recs`, not protected: the removal phase retires the row as "record gone"); unreadable → `protect[host] = true` and `readFailed = true`, as a failed read is today. |
| + - Console: `func (s *Server) readRecord(ctx, acct, app, host) (*Record, string, error)` and `listRecords`, `listRecordsStrict`, `listRecordsComplete`, `listAllRecordsStrict` wrappers passing `s.slots` (`*provision.SlotCache`, nil when `s.cfg.HostClaim`). Inventory to route (every instance-record read in these files, by `grep -n 'provision.GetRecord\|provision.GetRecordETag\|provision.ListRecords\|provision.ListAllRecords'`): `apiinstancemove.go`, `apiinstances.go`, `seedfinish.go`, `channelmove.go`, `apiruns.go`, `apiinstancemove_offline.go`, `apiinstanceimport.go` (`readImportRecord`, `readDeletedRecord`), `updatescore.go`, `movesrecovery.go`, `domainattach.go`, `appsettings.go`, `apirunsfollow.go`, `apiretry.go`, `apidb.go`, `apex.go` (`hostHolder`), `apidomains.go`, `apiinstanceseed.go`, `apilegacyroutes.go`, `relayops.go`, `seedsweep.go`, `server.go`, `regiondrift.go`, `runsweep.go`, `seedmove.go`. A test pins the inventory: `TestNoConsoleReadBypassesTheJudgedReaders` walks `internal/console` (non-test) and fails on a direct `provision.GetRecord`, `GetRecordETag`, `ListRecords*` or `ListAllRecords*` call outside `lifetimes_read.go`, `lifetimes*.go` and the claims-on files it lists by name. |
| -- [ ] **Step 1: Write the failing tests**: `TestReconcileRemovesARecordBehindItsSlot` (record at 1, slot `{2, live}`: no mint; an existing row is retired as "record gone"); `TestReconcileRemovesATombstone`; `TestReconcileRemovesAFieldlessRecordBesideAnUnrelatedSlot` (slot `{1, retired}`); `TestReconcileKeepsALegacyRecordWithNoSlot` and `TestReconcileKeepsAMarkedRecordUnderItsMigrationSlot`; `TestReconcileLeavesARecordAloneWhenTheSlotIsUnreadable` (no mint, no retire); `TestInstanceDoorsAnswer404ForAStaleRecord`; `TestEdgesFileOmitsAStaleRecordsRow`; `TestProductionReconcileConfigHasASlotCache`. |
| +- [ ] **Step 1: Write the failing tests**: `TestSlotCacheJudgesAndCaches` (two judgements, one slot read; absence cached; a changed slot seen after the window); `TestSlotCacheRereadsAcrossAMigration` (record read before the marker, slot read after it → valid, one re-read); `TestNilSlotCacheJudgesEverythingValid`; `TestReconcileRemovesARecordBehindItsSlot` (record at 1, slot `{2, live}`: no mint; an existing row retired as "record gone"); `TestReconcileRemovesATombstone` (the tombstone has no channel: it leaves before the channel check, so the row is retired, not protected); `TestReconcileKeepsALegacyRecordWithNoSlot`; `TestReconcileKeepsAMarkedRecordUnderItsMigrationSlot`; `TestReconcileProtectsWhenTheSlotIsUnreadable`; `TestReconcileRereadsAcrossAMigration` (hooks: the record body read before the marker, the slot after; the row survives); `TestEveryClaimsOffConsumerIgnoresAFieldlessRecordBesideASlot` (slot `{1, retired}`, fieldless record: `ReadRecordJudged` not found; the four judged listings omit it; the console GET answers 404; reconcile mints nothing; the offline keep view after `refreshOfflineView` does not keep a move it carries); `TestClaimsOnConsumersReadNoSlot` (`HostClaim` on: a counting store sees no `control/host-incarnations/` read from the doors, reconcile or the box listings); `TestProvisionTickConfigGatesTheSlotCache` (`CARLOS_HOST_CLAIM` set → `Slots == nil`; unset → non-nil); `TestNoConsoleReadBypassesTheJudgedReaders`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** the touched packages' targeted tests, then `go test ./internal/provision -run 'TestPlan14RootReadCensus|TestPlan9B8RecordWriteCensus' -count=1` and add the overlay rows the census names. |
| -- [ ] **Step 5: Commit** `lifetimes: reconcile, the doors and the edges file treat a record behind its slot as absent`. |
| +- [ ] **Step 4: Run** the touched packages' targeted tests, then the two census tests and add the rows they name. |
| +- [ ] **Step 5: Commit** `lifetimes: reconcile, the box listings and every console read judge a record's lifetime; a mismatch is re-read before it is believed`. |
| ### Task 7: `move_seq` on the record and the pair on rows |
| **Files:** |
| -- Modify: `internal/provision/offline_move.go` (`OfflineStart` and `OfflineStartStaged` (the hold), `OfflineOpenTransition` (the target serves), `OfflineAbort` (the source reopens) and `OfflineFinish` each set `rec.MoveSeq = rec.MoveSeq + 1` (1 when 0) and `rec.Move.Seq` (nil-safe) to that value; `OfflineStage`, `OfflineRestage`, `OfflineSeal`, `OfflineSealedBy`, `OfflineFlip`, `OfflineLandedBy`, `OfflineFinishingTransition` leave both alone), `internal/provision/provision.go` (`MoveState.Seq uint64 \`json:"seq,omitempty"\``), `internal/provision/offline_move_test.go` (`TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo`: `move_seq` joins the allowlist, the preservation assertions stay), `internal/registry/registry.go` (columns `incarnation INTEGER NOT NULL DEFAULT 0`, `serving_seq INTEGER NOT NULL DEFAULT 0` in the `ALTER TABLE` list; `Route.Incarnation uint64`, `Route.ServingSeq uint64`; `routeCols`, `routeColCount` (+2), `scanRoute`, `routeArgs`, `normalizeRoute`; `Upsert`'s conflict clause **always keeps the existing row's pair** (`incarnation=routes.incarnation, serving_seq=routes.serving_seq`), so an ordinary upsert never moves it; the pair changes only through the three setters below; `remote_dispatch.go` gains the new methods), `internal/provision/reconcile.go` (`RouteFor` copies `rec.Incarnation` and `rec.MoveSeq` into a minted row; after the mint or for an existing provisioned row, the pass calls `InstallLifetime` when the row's incarnation differs from the record's, or the row is `(0, 0)` and the record has a pair; it never raises `serving_seq` within the same incarnation), `internal/host/movesoffline.go` (`offlineHold`'s result gains the pair: `func(rec provision.Record) (status, op string, pair registry.Pair, ok bool)`, with `pair = {rec.Incarnation, rec.Move.Seq}`; `holdExisting` and `mintHeld` in reconcile take the pair and call `SetHoldIfSeq` / write it on the mint), `internal/host/movesoffline_land.go` and `internal/host/movesoffline.go` (`moveCfg.ReleaseHold` becomes `func(host, op string, pair registry.Pair) (bool, error)`, backed by `ReleaseHoldSeq`, and the open and abort paths pass the record's `Move.Seq` at that transition with its incarnation) |
| +- Modify: `internal/provision/offline_move.go` (`OfflineStart`, `OfflineStartStaged`, `OfflineOpenTransition`, `OfflineFinish` each set `rec.MoveSeq = rec.MoveSeq + 1` and `rec.Move.Seq` (nil-safe) to that value; `OfflineAbort` does so only from `OfflineStopping`, `OfflineSealing`, `OfflineSealed`, `OfflineLanding` or `OfflineLanded` (the source reopens), not from `OfflineStaging` (nothing was held); `OfflineStage`, `OfflineRestage`, `OfflineSeal`, `OfflineSealedBy`, `OfflineFlip`, `OfflineLandedBy`, `OfflineFinishingTransition` leave both alone), `internal/provision/provision.go` (`MoveState.Seq uint64 \`json:"seq,omitempty"\``), `internal/provision/offline_move_test.go` (`TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo`: `move_seq` joins the allowlist and nothing else changes), `internal/registry/registry.go` (columns `incarnation INTEGER NOT NULL DEFAULT 0`, `serving_seq INTEGER NOT NULL DEFAULT 0` in the `ALTER TABLE` list; `Route.Incarnation uint64`, `Route.ServingSeq uint64`; `routeCols`, `routeColCount` (+2), `scanRoute`, `routeArgs`, `normalizeRoute`; `Upsert`'s conflict clause gains `incarnation=routes.incarnation, serving_seq=routes.serving_seq` (an ordinary upsert never moves the pair; the claim-ownership `WHERE` predicate is unchanged); `Fingerprint`'s column list gains both; `remote_dispatch.go` gains the new methods), `internal/provision/reconcile.go` (`RouteFor` copies `rec.Incarnation` and `rec.MoveSeq` into a minted row, so `mintHeld`'s mint carries them too; `ReconcileConfig.PairOf func(rec Record) registry.Pair` (nil → the zero pair, which the setters write as today's behaviour for Phase 1); `mintHeld` and `holdExisting` pass `cfg.PairOf(rec)` to the mint and to `SetHoldIfSeq`; after the converge of an existing provisioned row of this pass's app and account, `installLifetime(reg, existing, rec)`: when `existing.HoldOp == ""` and either (`existing.Incarnation == 0 && rec.Incarnation > 0`) or (`0 < existing.Incarnation < rec.Incarnation`), call `InstallLifetimeIf(host, Pair{existing.Incarnation, existing.ServingSeq}, Pair{rec.Incarnation, rec.MoveSeq})`; a held row, or one at the record's incarnation, is never touched here, and a held row of an older lifetime is logged once per pass), `internal/host/movesoffline.go` (`withOfflineHolds` sets `cfg.PairOf = func(rec) Pair{rec.Incarnation, rec.Move.Seq}` (nil-safe); `moveCfg` gains `ReleaseHoldSeq func(host, op string, pair registry.Pair) (bool, error)`, backed by a new `releaseHeldSeq(reg, written)` in `movesoffline.go`; the open path at `movesoffline.go:1077` and the abort path at `movesoffline_land.go:165` call it with `{rec.Incarnation, rec.Move.Seq}`; Phase 1's `releaseHeld` and `moveCfg.ReleaseHold` are untouched) |
| - Test: `internal/provision/offline_move_test.go`, `internal/registry/registry_test.go`, `internal/provision/reconcile_lifetime_test.go`, `internal/host/movesoffline_e2e_test.go` |
| **Interfaces:** |
| -- Produces: `type registry.Pair struct { Incarnation, ServingSeq uint64 }`; `func (r *Registry) SetHoldIfSeq(host, status, op, expectOp string, pair Pair) (bool, error)` (the `SetHoldIf` predicate, plus the pair in the same statement); `func (r *Registry) ReleaseHoldSeq(host, op string, pair Pair) (bool, error)`; `func (r *Registry) InstallLifetime(host string, pair Pair) error` (the pair alone, used by reconcile for a new lifetime on an existing row); `Route.Incarnation`, `Route.ServingSeq`; `MoveState.Seq`; `Record.MoveSeq` advanced as above. |
| +- Produces: `type registry.Pair struct { Incarnation, ServingSeq uint64 }`; `func (r *Registry) SetHoldIfSeq(host, status, op, expectOp string, pair Pair) (bool, error)` (`UPDATE routes SET status=?, hold_op=?, serving_seq=? WHERE host=? AND hold_op=? AND incarnation=?` with `pair.Incarnation` in the predicate, so a setter of another lifetime changes nothing; a zero `pair.Incarnation` matches only a row at 0, today's rows); `func (r *Registry) ReleaseHoldSeq(host, op string, pair Pair) (bool, error)` (`SET status='ready', hold_op='', serving_seq=? WHERE host=? AND hold_op=? AND incarnation=?`); `func (r *Registry) InstallLifetimeIf(host string, expect, next Pair) (bool, error)` (`SET incarnation=?, serving_seq=? WHERE host=? AND incarnation=? AND serving_seq=? AND hold_op=''`); `Route.Incarnation`, `Route.ServingSeq`; `MoveState.Seq`; `Record.MoveSeq`; `ReconcileConfig.PairOf`; `moveCfg.ReleaseHoldSeq`. |
| - [ ] **Step 1: Write the failing tests**: |
| - - `TestOfflineTransitionsAdvanceMoveSeq`: direct path start 1, open 2, finish 3; staged path stage 0, start-staged 1, open 2, finish 3; abort after start → 2 (reopening), finish → 3; stage, restage, seal, flip, landed, finishing leave it; a record at `MoveSeq 5` starts at 6; `Move.Seq` equals the record's value at each advancing transition. |
| + - `TestOfflineTransitionsAdvanceMoveSeq`: direct path start 1, open 2, finish 3; staged path stage 0, start-staged 1, open 2, finish 3; abort after start 2, finish 3; abort from staging leaves 0 and finish then leaves 0; a second abort call on a reverting record is the existing refusal and changes nothing; stage, restage, seal, flip, landed, finishing leave it; a record at `MoveSeq 5` starts at 6; `Move.Seq` equals the record's value at each advancing transition. |
| - `TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo` passes with `move_seq` in its allowlist. |
| - - `TestRegistryPairColumns` (upsert a row with the pair, read back; an ordinary upsert of the same host with another pair keeps the first); `TestSetHoldIfSeqWritesStatusOpAndPairTogether` (and refuses when `expectOp` differs); `TestReleaseHoldSeqWritesThePair`; `TestInstallLifetimeReplacesBothComponents` (`(1, 9)` → `(2, 0)`). |
| - - `TestRouteForCopiesThePair`; `TestReconcileInstallsTheRecordsPairOnAMintedRow` (after a finished move at `move_seq 3`, delete the row, reconcile → `(1, 3)`); `TestReconcileInstallsANewLifetimeOnAnExistingRow` (row `(1, 9)`, record `{2, 0}` → `(2, 0)`); `TestReconcileNeverRaisesTheSequenceWithinALifetime` (row `(1, 1)`, record `{1, 3}` with a move in flight → the row stays until the move's own transition writes it). |
| - - In `TestOfflineMoveE2E`: the records are created through the lifetime (slot `{1, live}`, `Incarnation 1`) and the validity checks are on; after `start` the source row is held with `(1, 1)`; after `open` the target row is ready with `(1, 2)`; a second run aborting from landed leaves the source ready with `(1, 2)` and the target without a row. |
| + - `TestRegistryPairColumns` (upsert with the pair; an ordinary upsert of another pair keeps the first; `Fingerprint` changes when only the pair changes); `TestSetHoldIfSeqNamesItsLifetime` (a setter at incarnation 1 against a row at 2 changes nothing; at 2 it writes status, op and seq together); `TestReleaseHoldSeqNamesItsLifetime`; `TestInstallLifetimeIfIsGuarded` (`(1, 9)` → `(2, 0)` with the right expectation; a wrong expectation or a held row → unchanged). |
| + - `TestRouteForCopiesThePair`; `TestReconcileInstallsTheRecordsPairOnAMintedRow` (after a finished move at `move_seq 3`, delete the row, reconcile → `(1, 3)`; after an abort from landed and finish, likewise `(1, 3)`); `TestExistingRowsAcrossLifetimes` (subtests: an unheld row `(1, 9)` with record `{2, 0}` → `(2, 0)`; an unheld row `(0, 0)` with a migrated record `{1, 2}` → `(1, 2)` once and never again while the record is at 1 (a later record at `{1, 5}` with the row unheld at `(1, 2)` stays `(1, 2)`); a held row `(1, 1)` by an old op with record `{2, 0}` → unchanged and logged; a held row `(0, 0)` with a migrated record `{1, 0}` → unchanged; a stale `ReleaseHoldSeq` for incarnation 1 against the successor's row at 2 → no change). |
| + - In `TestOfflineMoveE2E`: the records are created through the lifetime (slot `{1, live}`, `Incarnation 1`) with judging on; after `start` the source row is held with `(1, 1)`; after `open` the target row is ready with `(1, 2)`; a second trace aborting from landed leaves the source ready with `(1, 2)` and no target row. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/registry -run 'Pair|Hold|Lifetime' -count=1`, `go test ./internal/provision -run 'MoveSeq|RouteFor|Offline|Reconcile' -count=1`, `go test ./internal/host -run 'OfflineMoveE2E|Offline' -count=1`, then both census tests (the record-write census gains the transitions' changed digests; the root-read census the `internal/host` units). |
| -- [ ] **Step 5: Commit** `moves: the offline hold, open, abort and finish advance move_seq, and rows carry (incarnation, serving_seq)`. |
| +- [ ] **Step 4: Run** `go test ./internal/registry -run 'Pair|Hold|Lifetime|Fingerprint' -count=1`, `go test ./internal/provision -run 'MoveSeq|RouteFor|Offline|Reconcile|Lifetime' -count=1`, `go test ./internal/host -run 'OfflineMoveE2E|Offline' -count=1`, then both census tests. |
| +- [ ] **Step 5: Commit** `moves: the offline hold, open, abort and finish advance move_seq; rows carry (incarnation, serving_seq) and change it only by setters that name the lifetime`. |
| -### Task 8: Publish the pair and order comparable entries by it |
| +### Task 8: Publish the pair and order one lifetime's entries by it |
| **Files:** |
| -- Modify: `internal/host/edgesfile.go` (`edgeHost` gains `Incarnation uint64 \`json:"incarnation,omitempty"\``, `ServingSeq uint64 \`json:"serving_seq,omitempty"\``, `TargetHost string \`json:"target_host,omitempty"\``; a route publishes its row's pair; an explicit alias entry publishes its primary's pair and `TargetHost: a.Host`; a catch-all's derived alias entry (the `catchallServed` branch near line 624) publishes `TargetHost: wd.Host` and the primary's pair), `internal/edge/homemap.go` (`edgesHostView` and `homeEntry` gain `incarnation`, `servingSeq`, `targetHost`; `foldHomes` copies all three with the dial winner; `dialBeats` becomes: `!in.dial` → false; `!prev.dial` → true; `prev.alias != in.alias` → the real route wins as today; then higher `(incarnation, servingSeq)` wins; equal → `in.stamp > prev.stamp` as today) |
| +- Modify: `internal/host/edgesfile.go` (`edgeHost` gains `Incarnation uint64 \`json:"incarnation,omitempty"\``, `ServingSeq uint64 \`json:"serving_seq,omitempty"\``, `TargetHost string \`json:"target_host,omitempty"\``; a served route publishes its row's pair; the explicit alias entry at line 546 publishes `hosts[a.Host]`'s pair and `TargetHost: a.Host`; the wildcard dial entry at line 632, when `catchallServed[wd.Host]` names a `primary`, publishes `hosts[primary]`'s pair and `TargetHost: primary`; a certificate-only wildcard entry publishes none), `internal/edge/homemap.go` (`edgesHostView` and `homeEntry` gain `incarnation`, `servingSeq`, `targetHost`; `foldHomes` copies all three with the dial winner; `dialBeats` becomes: `!in.dial` → false; `!prev.dial` → true; `prev.alias != in.alias` → the real route wins (today); both real → higher `(incarnation, servingSeq)` wins, equal → `in.stamp > prev.stamp` (today); both alias with `prev.targetHost != "" && prev.targetHost == in.targetHost` → the same pair rule; both alias otherwise → `in.stamp > prev.stamp` (today, unchanged)) |
| - Test: `internal/host/edgesfile_test.go`, `internal/edge/homemap_test.go` |
| **Interfaces:** |
| -- Produces: the three fields on the wire and in the fold; the precedence order above. The different-target alias rule is plan 3's (it needs holding entries); plan 1 only publishes `target_host`. |
| +- Produces: the three fields on the wire and in the fold; the comparable rule above. The different-target alias rule (a ready route of another target over a hold) is plan 3's; plan 1 publishes `target_host` and never orders aliases of different or unknown targets by the pair. |
| -- [ ] **Step 1: Write the failing tests**: `TestEdgesFilePublishesThePairAndTargetHost` (a row `(2, 3)`; its explicit alias; a derived catch-all alias); `TestFoldPrefersTheHigherPair` (`(1, 2)` over `(1, 1)`; `(2, 0)` over `(1, 9)`; `(0, 0)` loses to any pair; equal pairs → the stamp rule); `TestFoldKeepsRealRouteOverAHigherPairAlias`; `TestFoldNeverDialsACertOnlyEntryForItsPair`; `TestFoldCopiesTheWinnersPairAndTarget` (three candidates, the final winner's fields intact); `TestFoldLegacyPublicationsUnchanged` (existing fixtures without the fields fold as before; `TestHomeMapPicksTheSameHomeOnADuplicateClaim` still passes). |
| +- [ ] **Step 1: Write the failing tests**: `TestEdgesFilePublishesThePairAndTargetHost` (a row `(2, 3)`; its explicit alias; a wildcard dial entry whose parent name differs from the primary; a certificate-only wildcard entry publishes no target and no pair); `TestFoldPrefersTheHigherPairOfOneLifetime` (`(1, 2)` over `(1, 1)`; `(2, 0)` over `(1, 9)`; `(0, 0)` loses to any pair; equal → the stamp rule); `TestFoldKeepsRealRouteOverAHigherPairAlias`; `TestFoldComparesAliasesOnlyForOneTarget` (aliases to A at `(5, 9)` and to B at `(1, 0)`: today's stamp rule decides, not the pair; aliases to A at `(1, 2)` and A at `(1, 1)`: the pair decides; an alias with no target never compares by pair); `TestFoldNeverDialsACertOnlyEntryForItsPair`; `TestFoldCopiesTheWinnersPairAndTarget`; `TestFoldLegacyPublicationsUnchanged` (fixtures without the fields fold as before; `TestHomeMapPicksTheSameHomeOnADuplicateClaim` still passes). |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/host -run 'EdgesFile' -count=1`, `go test ./internal/edge -run 'Fold|HomeMap' -count=1`, then the census tests (the edges writer's changed unit). |
| -- [ ] **Step 5: Commit** `edges: publish (incarnation, serving_seq) and target_host; the home map orders comparable entries by the pair`. |
| +- [ ] **Step 4: Run** `go test ./internal/host -run 'EdgesFile' -count=1`, `go test ./internal/edge -run 'Fold|HomeMap' -count=1`, then the census tests. |
| +- [ ] **Step 5: Commit** `edges: publish (incarnation, serving_seq) and target_host; the home map orders one lifetime's entries by the pair`. |
| ### Task 9: Lifetimes end to end |
| **Files:** |
| - Test: `internal/host/movesoffline_lifetime_test.go` (extending `TestOfflineMoveE2E`'s fixtures), `internal/console/lifetimes_e2e_test.go` |
| -These are §11's witnesses plan 1 owns; the ones that need holding publications are plan 3's and are not claimed here. |
| +These are §11's witnesses plan 1 owns: record and row transitions and the ordering of ready publications. Every witness that needs a holding publication (a hold winning a fold, a current hold beating a previous lifetime's route, a stale ready alias losing to its own hold, holding entries at the edge) is plan 3's and is not claimed here. |
| -- [ ] **Step 1: Write the tests**: `TestLifetimeAcrossAMoveAndAnAbort` (create `(1, 0)`; start → source held `(1, 1)` and a fold with the hold's publication and a retained `(1, 0)` route prefers the hold's publisher for the dial; open → target `(1, 2)` wins; abort from landed → source `(1, 2)` wins over a retained target publication at `(1, 1)`); `TestReconstructedRowCarriesTheRecordsPair` (after a finished move at `move_seq 3`, delete the target's row, reconcile → `(1, 3)`, beats a retained `(1, 1)`); `TestReconstructedSourceAfterAbortWins` (abort from landed, finish, delete the source row, reconcile → `(1, 2)`, beats the target's retained `(1, 1)`); `TestRecreatedHostOutranksItsPreviousLifetime` (retire, create → `(2, 0)` beats a retained `(1, 9)`); `TestLateLegacyCreateIsInert` (slot `{1, retired}`; a fieldless record written by the test; reconcile mints nothing, the door 404s, the next create replaces it); `TestDelayedCreateReplayMoveAndRetirementInterleave` (pending create replayed; a move completes; retirement crashes between the tombstone and the slot; the late original create fails on the tombstone; recovery retires the slot and removes the tombstone). |
| +- [ ] **Step 1: Write the tests**: `TestLifetimeAcrossAForwardMove` (create `(1, 0)`; start → source row held `(1, 1)`; open → target row ready `(1, 2)`; finish → `(1, 3)`; a fold of the target's ready publication at `(1, 3)` against a retained source publication at `(1, 0)` prefers the target); `TestLifetimeAcrossAnAbort` (a second trace: start, flip to landed, abort → source reopens at `(1, 2)`, finish → `(1, 3)`; the source's publication at `(1, 3)` beats a retained target publication at `(1, 1)`); `TestReconstructedRowCarriesTheRecordsPair` (after a finished move at `move_seq 3`, delete the target's row, reconcile → `(1, 3)`, beats a retained `(1, 1)`); `TestReconstructedSourceAfterAbortWins` (abort from landed, finish, delete the source row, reconcile → `(1, 3)`, beats the target's retained `(1, 1)`); `TestRecreatedHostOutranksItsPreviousLifetime` (retire, create → `(2, 0)` beats a retained `(1, 9)`); `TestLateLegacyCreateIsInert` (slot `{1, retired}`; a fieldless record written by the test; reconcile mints nothing, the door 404s, the strict listings omit it, the next create replaces it); `TestDelayedCreateReplayMoveAndRetirementInterleave` (pending create replayed; a move completes; retirement crashes between the tombstone and the slot; the late original create fails on the tombstone; recovery retires the slot and removes the tombstone); `TestLandedCreateThenSecondRecoveryCrash` (a create whose record landed and serves, a crash before `committed`, recovery marks it committed with the record's ETag unchanged, including a second crash during that recovery). |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| - [ ] **Step 3: Run the census tests and both cross-builds** (`GOOS=darwin go build ./internal/host/`, `GOOS=windows go vet ./internal/host/`); write the census note. |
| - [ ] **Step 4: Commit** `lifetimes: end-to-end witnesses across a move, an abort, a reconstruction, a recreation and an interleaved retirement`; push. |
| @@ -244,7 +281,7 @@ These are §11's witnesses plan 1 owns; the ones that need holding publications |
| ## Self-review notes |
| -- §6.4 coverage: slot and tombstone form (T1, T4), create and recovery (T3), retirement and recovery (T4), migration, its fence and recovery (T5), validity in every claims-off consumer (T1, T6), `move_seq` and the row pair with the reconcile installation rule (T7), publication and the fold's comparable-entry order (T8), witnesses (T9). Deferred by name: holding entries and their five-minute validity, the probe mode and the alias different-target rule (plan 3); the door, token, job, reservations and reports (plan 2); cleanup (plan 4). |
| -- Names used across tasks: `RecordLifetimeValid(rec, etag, slot)`, `ReadHostIncarnation`, `WriteHostIncarnation`, `HostIncarnation{Incarnation, State, Intent, RecordETag}`, `TombstoneBody`, `GetRecordRaw`, `createRecordWithLifetime`, `retireRecordWithLifetime`, `migrateHostLifetime`, `errLifetimeSettling`, `errRetireRefused`, `ErrRecordMigrating`, `WithMigration`, `SlotCache.Valid`, `ListRecordsValid`, `registry.Pair`, `SetHoldIfSeq`, `ReleaseHoldSeq`, `InstallLifetime`, `Route.Incarnation`, `Route.ServingSeq`, `MoveState.Seq`, `edgeHost.Incarnation/ServingSeq/TargetHost`. |
| -- Review Focus 1 to 5 are pinned by T3, T5, T6+T3, T7 (two tests) and T7. |
| -- Store primitives, as verified in rev 1's review: `Store.Put(..., "")` is create-only on both backends; `WriteRecord` with `before={}` and `etag=""` is the create; `DeleteIfMatch` treats absence as success and refuses unsupported backends. |
| +- §6.4 coverage: slot, owner key and tombstone form (T1, T4), create and recovery (T3), every claims-off removal and its recovery (T4), migration, its fence and recovery (T5), the verdict in every claims-off consumer with the re-read (T1, T6), `move_seq` and the row pair with guarded setters and the installation rule (T7), publication and the one-lifetime fold order (T8), witnesses (T9). Deferred by name: holding entries and their five-minute validity, the probe mode, the different-target alias rule and every holding-dependent witness (plan 3); the door, token, job, reservations and reports (plan 2); cleanup (plan 4). |
| +- Names used across tasks: `LifetimeOf`, `LifetimeVerdict` (`LifetimeValid`, `LifetimeStale`, `LifetimeUnreadable`), `ReadHostIncarnation`, `WriteHostIncarnation`, `HostIncarnation{Incarnation, State, Account, App, Intent, RecordETag}`, `TombstoneBody`, `GetRecordRaw`, `withHostname`, `admit`, `classifyLocked`, `createRecordWithLifetime`, `retireRecordWithLifetime`, `migrateHostLifetimeLocked`, `errLifetimeSettling`, `errLifetimePending`, `errHostHeld`, `errRetireRefused`, `ErrRecordMigrating`, `WithMigration`, `SlotCache.Judge`, `ReadRecordJudged`, `List*Judged`, `ReconcileConfig.Slots`, `ReconcileConfig.PairOf`, `registry.Pair`, `SetHoldIfSeq`, `ReleaseHoldSeq`, `InstallLifetimeIf`, `moveCfg.ReleaseHoldSeq`, `Route.Incarnation`, `Route.ServingSeq`, `MoveState.Seq`, `edgeHost.Incarnation/ServingSeq/TargetHost`. |
| +- Review Focus 1 to 5 are pinned by T3 (two tests), T5+T6, T6+T3, T7, T7. |
| +- Store primitives, as verified in the reviews: `Store.Put(..., "")` is create-only on both backends; `WriteRecord` with `before={}` and `etag=""` is the create and needs the real CAS base otherwise; `DeleteIfMatch` treats absence as success and has no unconditional fallback. |