amadan
plan: self-serve moves 1, rev 2 (Astra plan review 1: 16 findings)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+154
−141
docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md+154 −141
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (2 s)
ok amadan.net/carlos/platform.git/launchctx 0.011s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (6 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (145 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.834s
ok github.com/carlosframework/platform/cmd/moveprobe 11.530s
ok github.com/carlosframework/platform/cmd/zddprobe 0.703s
ok github.com/carlosframework/platform/internal/activator 58.835s
ok github.com/carlosframework/platform/internal/agentsock 0.016s
ok github.com/carlosframework/platform/internal/alerting 0.004s
ok github.com/carlosframework/platform/internal/analytics 0.019s
ok github.com/carlosframework/platform/internal/appconfig 2.798s
ok github.com/carlosframework/platform/internal/apperr 0.074s
ok github.com/carlosframework/platform/internal/applog 0.128s
ok github.com/carlosframework/platform/internal/awsinv 0.005s
ok github.com/carlosframework/platform/internal/boxchoice 0.004s
ok github.com/carlosframework/platform/internal/boxid 1.256s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.011s
ok github.com/carlosframework/platform/internal/caps 0.018s
ok github.com/carlosframework/platform/internal/certwarm 0.003s
ok github.com/carlosframework/platform/internal/cgroup 0.702s
ok github.com/carlosframework/platform/internal/changelog 0.342s
ok github.com/carlosframework/platform/internal/claimactivation 0.887s
ok github.com/carlosframework/platform/internal/clamd 0.054s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.157s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.009s
ok github.com/carlosframework/platform/internal/dbverify 0.004s
ok github.com/carlosframework/platform/internal/dnsrec 0.002s
ok github.com/carlosframework/platform/internal/docsite 0.014s
ok github.com/carlosframework/platform/internal/domains 2.785s
ok github.com/carlosframework/platform/internal/doorkey 0.008s
ok github.com/carlosframework/platform/internal/economics 0.005s
ok github.com/carlosframework/platform/internal/edge 45.028s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.007s
ok github.com/carlosframework/platform/internal/fleetapi 0.130s
ok github.com/carlosframework/platform/internal/gate 1.900s
ok github.com/carlosframework/platform/internal/genselect 0.015s
ok github.com/carlosframework/platform/internal/geo 0.015s
ok github.com/carlosframework/platform/internal/geoip 0.030s
ok github.com/carlosframework/platform/internal/historyreadback 0.153s
ok github.com/carlosframework/platform/internal/host 134.093s
ok github.com/carlosframework/platform/internal/hostactors 6.054s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.004s
ok github.com/carlosframework/platform/internal/hostbox 3.186s
ok github.com/carlosframework/platform/internal/hostclaim 0.825s
ok github.com/carlosframework/platform/internal/hostenv 0.008s
ok github.com/carlosframework/platform/internal/hostevidence 0.005s
ok github.com/carlosframework/platform/internal/hostexec 5.458s
ok github.com/carlosframework/platform/internal/hostfence 0.052s
ok github.com/carlosframework/platform/internal/hostfiles 10.263s
ok github.com/carlosframework/platform/internal/hostfloor 0.021s
ok github.com/carlosframework/platform/internal/hostguard 1.053s
ok github.com/carlosframework/platform/internal/hostops 2.127s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.002s
ok github.com/carlosframework/platform/internal/hostquarantine 0.003s
ok github.com/carlosframework/platform/internal/hostrelocate 0.675s
ok github.com/carlosframework/platform/internal/hostresolve 0.180s
ok github.com/carlosframework/platform/internal/hostwrite 0.003s
ok github.com/carlosframework/platform/internal/infraview 0.003s
ok github.com/carlosframework/platform/internal/instancectl 5.252s
ok github.com/carlosframework/platform/internal/instanceexec 2.779s
ok github.com/carlosframework/platform/internal/instancefs 0.007s
ok github.com/carlosframework/platform/internal/instancekey 0.005s
ok github.com/carlosframework/platform/internal/instanceupdates 0.011s
ok github.com/carlosframework/platform/internal/layout 1.813s
ok github.com/carlosframework/platform/internal/lbhealth 0.007s
ok github.com/carlosframework/platform/internal/lease 10.146s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.607s
ok github.com/carlosframework/platform/internal/ledger 0.017s
ok github.com/carlosframework/platform/internal/mail 0.276s
ok github.com/carlosframework/platform/internal/maintenance 0.095s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.546s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.092s
ok github.com/carlosframework/platform/internal/metrics 0.011s
ok github.com/carlosframework/platform/internal/moves 21.329s
ok github.com/carlosframework/platform/internal/moves/archive 1.128s
ok github.com/carlosframework/platform/internal/objstore 0.250s
ok github.com/carlosframework/platform/internal/ops 0.250s
ok github.com/carlosframework/platform/internal/park 0.275s
ok github.com/carlosframework/platform/internal/plans 0.010s
ok github.com/carlosframework/platform/internal/privatefile 0.006s
ok github.com/carlosframework/platform/internal/provision 13.681s
ok github.com/carlosframework/platform/internal/publicrelease 0.008s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.565s
ok github.com/carlosframework/platform/internal/relay 7.230s
ok github.com/carlosframework/platform/internal/release 24.384s
ok github.com/carlosframework/platform/internal/rollout 3.778s
ok github.com/carlosframework/platform/internal/rolloutwords 0.009s
ok github.com/carlosframework/platform/internal/rootaws 0.013s
ok github.com/carlosframework/platform/internal/rootparticipation 0.071s
ok github.com/carlosframework/platform/internal/rootstate 0.027s
ok github.com/carlosframework/platform/internal/runs 0.499s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.007s
ok github.com/carlosframework/platform/internal/schedule 2.653s
ok github.com/carlosframework/platform/internal/seeds 0.010s
ok github.com/carlosframework/platform/internal/sim/hostlife 10.056s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.079s
ok github.com/carlosframework/platform/internal/status 0.080s
ok github.com/carlosframework/platform/internal/steering 0.007s
ok github.com/carlosframework/platform/internal/stopevidence 0.221s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.010s
ok github.com/carlosframework/platform/internal/termfence 0.007s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.002s
ok github.com/carlosframework/platform/internal/updatewords 0.005s
ok github.com/carlosframework/platform/internal/usage 0.006s
ok github.com/carlosframework/platform/internal/version 0.001s
ok github.com/carlosframework/platform/internal/vet 1.652s
=== internal_console@0 (81 s)
ok github.com/carlosframework/platform/internal/console 79.694s
=== internal_console@1 (36 s)
ok github.com/carlosframework/platform/internal/console 34.842s
=== internal_console@2 (41 s)
ok github.com/carlosframework/platform/internal/console 39.300s
=== internal_console@3 (36 s)
ok github.com/carlosframework/platform/internal/console 34.636s
=== internal_console@4 (36 s)
ok github.com/carlosframework/platform/internal/console 34.699s
=== internal_console@5 (34 s)
ok github.com/carlosframework/platform/internal/console 32.494s
=== internal_console@6 (36 s)
ok github.com/carlosframework/platform/internal/console 34.270s
=== internal_console@7 (42 s)
ok github.com/carlosframework/platform/internal/console 40.959s
=== root@0 (31 s)
ok github.com/carlosframework/platform 29.729s
=== root@1 (43 s)
ok github.com/carlosframework/platform 41.872s
=== root@2 (40 s)
ok github.com/carlosframework/platform 38.506s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.021s
=== RUN TestT4StorageProbeMinio
2026/10/22 00:03:17 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0B11ED44EED9B, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.00s)
PASS
ok github.com/carlosframework/platform/internal/host 0.020s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md| index cab648472..d90ae4fee 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| @@ -2,236 +2,249 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -**Goal:** Give every instance record a durable lifetime (the host incarnation slot) and every served row a serving order `(incarnation, serving_seq)` that boxes publish and edges honour, with crash-safe create, retirement and migration, so a later plan's holding pages and moves can never be out-ordered by a stale publication. |
| +Rev 2, 2026-10-22. Rev 1 was reviewed by Astra (`after236/selfserve-plan1-review.md`, NOT READY, 16 findings); rev 2 answers every one. The largest changes: the protocol is scoped to claims-off deployments (claims-on records keep their own lifecycle), the migration marker is fenced inside the one record-write primitive, the create checks the record before allocating, a legacy record is migrated before it is retired, the pair never moves on an ordinary upsert, and the alias precedence rule moves to plan 3 with the holding entries it orders. |
| -**Architecture:** A per-hostname slot object in the deployment bucket (`control/host-incarnations/<host>.json`), created once and changed only by conditional writes, is the authority for a record's lifetime; the console allocates and retires it through persisted intents in SQLite and recovers them at start. Records gain `incarnation` and `move_seq`; registry rows gain the pair; the edges file publishes it; the home map orders by it before the existing claim stamp. Retirement replaces the record with a tombstone before retiring the slot. Lazy migration marks a record by CAS before binding a slot to it. |
| +**Goal:** Give every claims-off instance record a durable lifetime (the host incarnation slot) and every served row a serving order `(incarnation, serving_seq)` that boxes publish and edges honour, with crash-safe create, retirement and migration, so a later plan's holding pages and moves can never be out-ordered by a stale publication. |
| -**Tech Stack:** Go. `internal/provision` (records, reconcile), `internal/registry`, `internal/console` (doors, store, passes), `internal/host` (edges file, offline move steps), `internal/edge` (home map). Tests use `release.DirStore`, real registries, the console's doors in-process, and the existing offline two-box witness. |
| +**Architecture:** A per-hostname slot object in the deployment bucket (`control/host-incarnations/<host>.json`), created once and changed only by conditional writes, is the authority for a record's lifetime; the console allocates, retires and migrates through persisted intents in SQLite under a per-hostname exclusion and recovers them at start. Records gain `incarnation`, `incarnation_migrating`, `move_seq` and a tombstone form; registry rows gain the pair; the edges file publishes it with each alias's target; the home map orders comparable entries by it before the existing claim stamp. Retirement replaces the record with a tombstone before retiring the slot. Lazy migration marks a record by CAS before binding a slot to it, and the write primitive refuses every other writer while the marker stands. |
| -**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md`, §6.4 ("Ordering by a per-host sequence", "Lifetimes: the host incarnation slot", "Aliases", "Freshness") and §11. Plans 2 to 4 build on this one. |
| +**Tech Stack:** Go. `internal/provision` (records, sequence, reconcile, offline transitions), `internal/registry`, `internal/console` (doors, store, store gate, passes), `internal/host` (edges file, offline hold and release), `internal/edge` (home map). Tests use `release.NewDirStore`, real registries, the console's doors in-process, and the existing two-box offline witness `TestOfflineMoveE2E` (`internal/host/movesoffline_e2e_test.go`). |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md`, §6.4 ("Ordering by a per-host sequence", "Lifetimes: the host incarnation slot", "Aliases", "Freshness") and §11. Plans 2 (door, token, job, reservations, reports), 3 (holding entries, probe mode, the alias rule) and 4 (cleanup) build on this one. |
| ## Global Constraints |
| -- Every record consumer treats a record that fails `RecordLifetimeValid` (Task 1) as absent: reconcile, the console doors, the boxes' publication. |
| -- The slot is never deleted and never advanced to abandon a create; a `pending` allocation is replayed or superseded, never dropped (spec §6.4). |
| -- No console write to a bucket outside the deployment bucket; no root-side change in this plan. |
| -- Every new root-state read or changed unit in `internal/host` or `internal/provision` takes a row in a new Plan 14 census overlay `plan14-root-reads-self-serve-moves-1.tsv`, loaded last, `rev3=PROPOSED-self-serve-moves-1`, with a note `docs/superpowers/notes/census-proposal-self-serve-moves-1.md`; run `TestPlan14RootReadCensus` and `TestPlan9B8RecordWriteCensus`. |
| -- Every user-facing sentence is a copy row, pending Paul's review; the one in this plan is the delete door's "the instance changed. read it again and try the delete once more". |
| -- No tenant names in code, tests or copy: `shop.example.com`, account `bab`, app `shop`. |
| -- Boxes roll before the console (the roll order rule) so every box understands the migration marker before any migration runs; the migration pass therefore checks `CARLOS_LIFETIMES_MIGRATE=1` on the console, default off, until the fleet is rolled. |
| +- **Scope: claims-off.** The slot protocol, migration pass, tombstone retirement and validity checks run only where `CARLOS_HOST_CLAIM` is off on the console (`!s.cfg.HostClaim`) and apply only to records that carry `incarnation` or `incarnation_migrating`. Claims-on records are created by `finishClaimCreate` and retired by the host-claim operations; this plan does not touch them, and a claims-on console's record consumers see no new field. Self-serve moves refuse on a claims-on console (MV-O20), so nothing here is needed there. |
| +- Every claims-off consumer treats a record that fails `RecordLifetimeValid` (Task 1) as absent: reconcile, the console's record reads, the boxes' publication. "Absent" for reconcile means removed from the live set before minting, so an existing row is retired as "record gone". |
| +- The slot is never deleted and never advanced to abandon a create (spec §6.4). |
| +- One per-hostname exclusion for create, retirement, migration and recovery in the console process (Task 3's `hostnameMutex`), and a persisted pending operation for a hostname refuses any new create or delete of that hostname (409 "another change to this instance is still settling. try again in a minute") until recovery resolves it; unreadable evidence leaves it pending. |
| +- Every record write stays on `provision.WriteRecord` with the exact bytes read and the sequence discipline of `record_sequence.go`; no new raw `Put` of a record key. |
| +- `release.DeleteIfMatch` never falls back to a plain delete; absence counts as success. |
| +- No console write to a bucket outside the deployment bucket; no root-side change. |
| +- Census: every changed unit in `internal/host` or `internal/provision` takes a row in `internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv`, loaded last by a new block at the end of `TestPlan14RootReadCensus` in `internal/provision/plan14_root_reads_test.go`, `rev3=PROPOSED-self-serve-moves-1`, with the note `docs/superpowers/notes/census-proposal-self-serve-moves-1.md`; `TestPlan9B8RecordWriteCensus` (the record-write census) is updated for the new record writers (tombstone, marker, migration); both tests green. |
| +- The new SQLite tables are classified in `claimSQLTables` (`internal/console/store_gate.go`) so a claims-on store still opens (its boot census refuses an unlisted table). |
| +- Copy: the two new sentences are copy rows pending Paul's review: "the instance changed. read it again and try the delete once more" and "another change to this instance is still settling. try again in a minute". No tenant names: `shop.example.com`, account `bab`, app `shop`. |
| +- Boxes roll before the console (the roll order rule); the migration pass is behind `CARLOS_LIFETIMES_MIGRATE=1`, default off, until every box of a deployment runs this code. |
| - Local tests: only the touched packages with `-run`; CI runs the full suite. Go needs the sandbox off; never set `GOCACHE`. |
| ## Review Focus |
| -1. A console crash between the slot write and the record write, with a second console start before the first process's write has landed: the replay and the late write must both be harmless (Task 3's recovery test with an injected late write). |
| -2. A record updated by a box-side offline transition during migration: the marker must make the box's CAS fail and its retry succeed after the field is set (Task 5's concurrent-writer test). |
| -3. An alias repointed to another instance while the first is held: the different-target rule must choose the new target whatever the pairs (Task 8's alias test). |
| -4. A legacy record with no incarnation beside a retired slot (an old console's late create): every consumer must treat it as absent and the next create must replace it (Task 6's validity test and Task 3's stale-record branch). |
| -5. A reconstructed registry after a finished move: reconcile must write the record's pair with the minted row so the row outranks a stale holding publication (Task 7's reconciliation test). |
| +1. A console crash between the slot write and the record write, a second console start before the first process's write has landed, then the late write: the replay and the late write must both be harmless (Task 3, `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`). |
| +2. A box-side offline transition issued against the pre-marker ETag during migration: its CAS must fail and its retry succeed after the field is set (Task 5, `TestAnIssuedWriteLosesToTheMarkerAndRetries`). |
| +3. A legacy record with no incarnation beside a retired slot (an old console's late create): every consumer treats it as absent and the next create replaces it (Task 6 `TestReconcileRemovesAFieldlessRecordBesideAnUnrelatedSlot`, Task 3 `TestCreateReplacesStaleRecord`). |
| +4. A reconstructed registry after a finished move, and after an abort from landed: reconcile installs the record's pair with the minted row (Task 7, `TestReconcileInstallsTheRecordsPairOnAMintedRow`). |
| +5. An existing row kept across a recreation or a migration: it must take the new lifetime's pair without inventing a higher sequence (Task 7, `TestReconcileInstallsANewLifetimeOnAnExistingRow`). |
| + |
| +(The alias repointed during a move is plan 3's Review Focus: it needs holding entries.) |
| --- |
| -### Task 1: Record fields, the slot type and the validity rule |
| +### Task 1: Record fields, the slot type, the tombstone form and the validity rule |
| **Files:** |
| -- Modify: `internal/provision/provision.go` (the `Record` struct, after `Gate`) |
| +- Modify: `internal/provision/provision.go` (the `Record` struct after `Gate`; `GetRecordETag`, `GetRecord`) |
| - Create: `internal/provision/lifetime.go` |
| -- Test: `internal/provision/lifetime_test.go`, `internal/provision/provision_test.go` (the additivity test) |
| +- Test: `internal/provision/lifetime_test.go`; extend `TestRecordWireShapeIsAdditive` in `internal/provision/provision_test.go` |
| **Interfaces:** |
| - Produces: |
| - `Record.Incarnation uint64 \`json:"incarnation,omitempty"\``, `Record.IncarnationMigrating string \`json:"incarnation_migrating,omitempty"\``, `Record.MoveSeq uint64 \`json:"move_seq,omitempty"\``, `Record.Retired bool \`json:"retired,omitempty"\``. |
| - - `type HostIncarnation struct { Incarnation uint64; State string; Intent string }` with `json` names `incarnation`, `state`, `intent`; `const SlotLive = "live"`, `SlotRetired = "retired"`. |
| - - `func HostIncarnationKey(host string) string` returning `control/host-incarnations/<host>.json`. |
| - - `func ReadHostIncarnation(ctx, s release.Store, host string) (*HostIncarnation, string, error)`: nil, "", nil when absent. |
| - - `func WriteHostIncarnation(ctx, s release.Store, host string, slot HostIncarnation, ifMatch string) (string, error)`: `ifMatch == ""` is create-only (`If-None-Match: *`), otherwise `If-Match`. |
| - - `func RecordLifetimeValid(rec *Record, slot *HostIncarnation) bool`. |
| - |
| -- [ ] **Step 1: Write the failing tests** in `lifetime_test.go`: |
| - - `TestRecordLifetimeValid`: a table over (record, slot) with the spec's rules: tombstone → false; fieldless and no slot → true; fieldless with marker `m1` and slot `{1, live, "migration m1"}` → true; fieldless with marker and slot `{1, live, "create c9"}` → false; fieldless, no marker, slot `{1, live, ...}` → false; fieldless and slot `{1, retired}` → false; `Incarnation 2` and slot `{2, live}` → true; `Incarnation 1` and slot `{2, live}` → false; `Incarnation 2` and slot `{2, retired}` → false; `Incarnation 2` and no slot → false. |
| - - `TestHostIncarnationWrites`: against `release.NewDirStore(t.TempDir())`: create-only succeeds once and fails the second time with `release.ErrCASConflict`; `If-Match` with the returned ETag succeeds; with a stale ETag fails; `ReadHostIncarnation` on an absent key returns nil, "", nil. |
| - - In `provision_test.go`, extend `TestRecordWireShapeIsAdditive` so a record without the four new fields round-trips byte-identical and one with them keeps them. |
| -- [ ] **Step 2: Run them to verify they fail**: `go test ./internal/provision -run 'RecordLifetimeValid|HostIncarnationWrites|RecordWireShapeIsAdditive' -count=1` fails with undefined symbols. |
| -- [ ] **Step 3: Implement** the fields and `lifetime.go` as named above; `WriteHostIncarnation` marshals with `json.Marshal` and calls `s.Put(ctx, key, body, ifMatch)`; `ReadHostIncarnation` maps the store's not-found to nil. |
| -- [ ] **Step 4: Run the tests to verify they pass.** |
| -- [ ] **Step 5: Commit** `provision: record lifetime fields, the host incarnation slot and the validity rule`. |
| - |
| -### Task 2: Console tables for pending lifetimes |
| + - `type HostIncarnation struct { Incarnation uint64 \`json:"incarnation"\`; State string \`json:"state"\`; Intent string \`json:"intent,omitempty"\`; RecordETag string \`json:"record_etag,omitempty"\` }`; `const SlotLive = "live"`, `SlotRetired = "retired"`. `RecordETag` is set only by a migration slot: the marked record's ETag it is bound to. |
| + - `func HostIncarnationKey(host string) string` → `control/host-incarnations/<host>.json`. |
| + - `func ReadHostIncarnation(ctx context.Context, s release.Store, host string) (*HostIncarnation, string, error)`: nil, "", nil when absent (the store's not-found). |
| + - `func WriteHostIncarnation(ctx context.Context, s release.Store, host string, slot HostIncarnation, ifMatch string) (string, error)`: `ifMatch == ""` is create-only (`Store.Put` with "" sends `If-None-Match: *` on S3 and refuses an existing file on `DirStore`), otherwise `If-Match`. |
| + - `func RecordLifetimeValid(rec *Record, recordETag string, slot *HostIncarnation) bool`. |
| + - `func TombstoneBody(before []byte) ([]byte, error)`: decodes `before`, keeps `schema`, `host`, `account`, `app`, `incarnation` and `record_seq`, sets `retired: true`, drops every other field; the caller writes it with `WriteRecord(ctx, s, key, before, body, etag)` so the sequence discipline holds (`record_seq` equal to `before`'s, then advanced by `WriteRecord`). |
| + - `func GetRecordRaw(ctx context.Context, s release.Store, acct, app, host string) (*Record, []byte, string, error)`: the decoded record (tombstones included, path check applied), its exact bytes and ETag. |
| + - `GetRecordETag` and `GetRecord` return the store's not-found error for a tombstone (`Retired`), so every existing caller sees a retired record as absent; `GetRecordRaw` is the only reader that sees it. |
| + |
| +The validity rule, exactly: `rec.Retired` → false. `rec.Incarnation == 0 && rec.IncarnationMigrating == ""`: slot nil → true; slot non-nil → false. `rec.IncarnationMigrating != ""` (fieldless with the marker): slot nil → true; slot `{1, live, "migration " + marker, RecordETag == recordETag}` → true; otherwise false. `rec.Incarnation > 0`: slot non-nil, `slot.State == live`, `slot.Incarnation == rec.Incarnation` → true; otherwise false. |
| + |
| +- [ ] **Step 1: Write the failing tests** in `lifetime_test.go`: `TestRecordLifetimeValid` (a table with one case per line of the rule above, including a marked record whose ETag differs from the slot's `RecordETag` → false); `TestHostIncarnationWrites` against `release.NewDirStore(t.TempDir())` (create-only once, second create-only fails with `release.ErrCASConflict`, `If-Match` with the returned ETag succeeds, a stale ETag fails, absent reads nil); `TestTombstoneBodyKeepsTheSequence` (a record body with `record_seq: 7` and many fields → the tombstone has exactly the six fields, `record_seq 7`, `retired true`; `WriteRecord` of it against the before bytes advances to 8); `TestGetRecordHidesATombstone` (after writing a tombstone, `GetRecord` returns the not-found error and `GetRecordRaw` returns `Retired: true`). Extend `TestRecordWireShapeIsAdditive` for the four fields. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/provision -run 'RecordLifetimeValid|HostIncarnationWrites|TombstoneBody|GetRecordHidesATombstone|RecordWireShapeIsAdditive' -count=1`. |
| +- [ ] **Step 3: Implement** the fields, `lifetime.go`, `GetRecordRaw`, and the tombstone check in `GetRecordETag`. |
| +- [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/provision -run 'Record' -count=1`. |
| +- [ ] **Step 5: Commit** `provision: record lifetime fields, the host incarnation slot, the tombstone form and the validity rule`. |
| + |
| +### Task 2: Console tables for pending lifetimes, classified for the store gate |
| **Files:** |
| -- Modify: `internal/console/store.go` (`migrate`, after the `account_plans` table) |
| +- Modify: `internal/console/store.go` (`migrate`, after the `account_plans` table), `internal/console/store_gate.go` (`claimSQLTables`) |
| - Create: `internal/console/lifetimes_store.go` |
| -- Test: `internal/console/lifetimes_store_test.go` |
| +- Test: `internal/console/lifetimes_store_test.go`; extend the existing claims-on store-open test beside `store_gate.go`'s tests so the three tables are classified |
| **Interfaces:** |
| -- Produces, on `*Store`: |
| - - tables `host_allocations (host TEXT, incarnation INTEGER, create_id TEXT PRIMARY KEY, state TEXT, body BLOB, predecessor_etag TEXT, created_at INTEGER)`, `record_retirements (host TEXT, incarnation INTEGER, retire_id TEXT PRIMARY KEY, record_etag TEXT, slot_etag TEXT, state TEXT, created_at INTEGER)`, `record_migrations (host TEXT PRIMARY KEY, migration_id TEXT, state TEXT, created_at INTEGER)`. |
| - - `func (s *Store) BeginAllocation(host string, incarnation uint64, createID string, body []byte, predecessorETag string) error`, `func (s *Store) SetAllocationState(createID, state string) error`, `func (s *Store) PendingAllocations() ([]Allocation, error)`, `func (s *Store) LastIncarnation(host string) (uint64, error)` (the highest committed or pending for the host, 0 when none). |
| - - `type Allocation struct { Host string; Incarnation uint64; CreateID string; Body []byte; PredecessorETag string; State string }`. |
| - - `func (s *Store) BeginRetirement(host string, incarnation uint64, retireID, recordETag, slotETag string) error`, `func (s *Store) SetRetirementState(retireID, state string) error`, `func (s *Store) PendingRetirements() ([]Retirement, error)`; `type Retirement struct { Host string; Incarnation uint64; RetireID, RecordETag, SlotETag, State string }`. |
| - - `func (s *Store) BeginMigration(host, migrationID string) error`, `func (s *Store) SetMigrationState(host, state string) error`, `func (s *Store) PendingMigrations() ([]Migration, error)`; `type Migration struct { Host, MigrationID, State string }`. |
| - - States: `pending`, `committed`, `superseded`, `void`, `refused`. |
| -- [ ] **Step 1: Write the failing tests**: `TestLifetimeTablesRoundTrip` (begin, list pending, set state, list again) for all three; `TestLastIncarnationCountsPending` (a pending allocation at 2 makes `LastIncarnation` 2). |
| +- Tables, all keyed by the record (`account` is the sqid, as the mail tables use): |
| + - `host_allocations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, state TEXT, body BLOB, predecessor_etag TEXT, created_at INTEGER)` |
| + - `record_retirements (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, record_etag TEXT, slot_etag TEXT, state TEXT, created_at INTEGER)` |
| + - `record_migrations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, record_etag TEXT, state TEXT, created_at INTEGER)` |
| + - Classification in `claimSQLTables`: `{Sqid: "account", App: "app"}` for all three. |
| +- States: `pending`, `committed`, `superseded`, `void`, `refused`. |
| +- On `*Store`: `BeginAllocation(a Allocation) error`, `SetAllocationState(opID, state string) error`, `PendingAllocations() ([]Allocation, error)`, `LastIncarnation(host string) (uint64, error)` (the highest `incarnation` over every state but `void` for the host; 0 when none); `BeginRetirement(r Retirement) error`, `SetRetirementState(opID, state string) error`, `PendingRetirements() ([]Retirement, error)`; `BeginMigration(m Migration) error`, `SetMigrationState(opID, state string) error`, `PendingMigrations() ([]Migration, error)`; `PendingForHost(host string) (int, error)` (pending rows across the three tables). |
| +- `type Allocation struct { OpID, Account, App, Host string; Incarnation uint64; State string; Body []byte; PredecessorETag string }`, `type Retirement struct { OpID, Account, App, Host string; Incarnation uint64; RecordETag, SlotETag, State string }`, `type Migration struct { OpID, Account, App, Host, RecordETag, State string }`. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestLifetimeTablesRoundTrip` (begin, list pending, set state, list again, for all three); `TestLastIncarnationSkipsVoid` (pending 2 and void 3 → 2); `TestPendingForHostCountsAllThree`; and in the store-gate tests, `TestClaimStoreOpensWithLifetimeTables` (a claims-on gated open succeeds with the three tables present). |
| - [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement** with the store's existing `migrate` pattern (`CREATE TABLE IF NOT EXISTS`) and plain SQL. |
| -- [ ] **Step 4: Run to verify they pass**: `go test ./internal/console -run 'LifetimeTables|LastIncarnation' -count=1`. |
| -- [ ] **Step 5: Commit** `console: tables for pending allocations, retirements and migrations`. |
| +- [ ] **Step 3: Implement** with the store's `CREATE TABLE IF NOT EXISTS` pattern and the classification entries. |
| +- [ ] **Step 4: Run to verify they pass**: `go test ./internal/console -run 'LifetimeTables|LastIncarnation|PendingForHost|ClaimStoreOpensWithLifetimeTables' -count=1`. |
| +- [ ] **Step 5: Commit** `console: tables for pending allocations, retirements and migrations, classified for the gate`. |
| ### Task 3: The create protocol and its recovery |
| **Files:** |
| -- Create: `internal/console/lifetimes_create.go`, `internal/console/lifetimes_recovery.go` |
| -- Modify: `internal/console/apiinstances.go` (the create door around line 1664, `provision.CreateRecord`), `internal/console/apiinstanceseed.go:341`, `internal/console/apiinstanceimport.go:172`, `cmd/console/main.go` (start the recovery pass beside the other `RunClaimWorker` passes) |
| +- Create: `internal/console/lifetimes.go` (the exclusion and the shared helpers), `internal/console/lifetimes_create.go`, `internal/console/lifetimes_recovery.go` |
| +- Modify: `internal/console/apiinstances.go` (the claims-off branch at the create door, the `provision.CreateRecord` call near line 1664), `internal/console/apiinstanceseed.go` (line 341), `internal/console/apiinstanceimport.go` (line 172), `internal/console/domainattach.go` (the first-route create near line 237), `internal/console/claim_workers.go` (register `"StartLifetimeRecoveryPass"`), `cmd/console/main.go` (start it beside the other passes) |
| - Test: `internal/console/lifetimes_create_test.go` |
| **Interfaces:** |
| -- Consumes: Task 1's slot helpers and fields; Task 2's store methods; `provision.WriteRecord` (create-only when the ETag is "") and `release.DeleteIfMatch`. |
| -- Produces: `func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record) error`, the one function every creation path calls; `func (s *Server) hostnameMutex(host string) *sync.Mutex`; `func (s *Server) recoverLifetimes(ctx context.Context) error` and `StartLifetimeRecoveryPass` (runs `recoverLifetimes` once at start and then every 60 s). |
| - |
| -The protocol, in `createRecordWithLifetime`: lock the hostname; read the slot (ETag P or absent); N = `LastIncarnation(host) + 1` (and at least slot.Incarnation + 1); `BeginAllocation(host, N, createID, body, P)`; `WriteHostIncarnation({N, live, "create " + createID}, P)` (create-only when absent); set `rec.Incarnation = N`; create-only `WriteRecord`; if the key holds a stale record (a tombstone, or a record whose lifetime is not valid against the new slot) delete it with `DeleteIfMatch` on its ETag and write again; `SetAllocationState(committed)`. Recovery, per pending allocation, exactly the branches of spec §6.4: slot absent or at the predecessor → write the slot; slot `{N, live, this create}` and record absent → write the record; slot at N and record at N → committed; slot at N and stale record → conditional delete then write; slot past N → superseded; unreadable → leave pending. |
| - |
| -- [ ] **Step 1: Write the failing tests** against an in-process console with `release.NewDirStore`: |
| - - `TestCreateAllocatesSlotThenRecord`: after a create, the slot is `{1, live, "create <id>"}`, the record carries `Incarnation 1`, the allocation is `committed`. |
| - - `TestCreateAfterRetirementGetsNextIncarnation`: slot `{1, retired}` seeded; a create yields `{2, live}` and a record at 2. |
| - - `TestCreateReplacesStaleRecord`: a fieldless record under the key beside slot `{1, retired}`; a create deletes it conditionally and writes incarnation 2. |
| - - `TestRecoveryReplaysCreateAtItsIncarnation`: a pending allocation at 1 with the slot written and no record; recovery writes the record at 1 and commits; then an injected "late original write" (create-only, same body) fails with `release.ErrCASConflict` and changes nothing. |
| - - `TestRecoveryWritesTheSlotWhenMissing`: pending allocation, slot absent; recovery writes the slot then the record. |
| - - `TestRecoveryLeavesALandedCreateServing`: slot and record at 1, allocation still pending; recovery marks it committed and rewrites nothing (the record ETag is unchanged). |
| - - `TestRecoverySupersedesWhenSlotMovedPast`: slot at 2, pending allocation at 1 → superseded; no record write. |
| - - `TestRecoveryLeavesPendingWhenUnreadable`: a store whose `Get` of the slot fails → state stays pending, no writes. |
| - - `TestConcurrentCreatesOfOneHostnameSerialise`: two creates in parallel; one succeeds at 1, the other fails the create-only record write with the existing 409 and allocates nothing committed. |
| +- Consumes: Task 1 and Task 2. |
| +- Produces: |
| + - `func (s *Server) lifetimesOn() bool` (`!s.cfg.HostClaim`). |
| + - `func (s *Server) hostnameMutex(host string) *sync.Mutex` (a `sync.Map` of mutexes). |
| + - `func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record) error`: the one function every claims-off creation path calls; returns `release.ErrCASConflict` for a live existing record (the doors keep their existing 409 mapping), `errLifetimeSettling` (mapped to 409 with the settling sentence) while a pending operation exists for the host. |
| + - `func (s *Server) recoverLifetimes(ctx context.Context) error` and `func (s *Server) StartLifetimeRecoveryPass(ctx context.Context)` (once at start, then every 60 s). |
| + |
| +The protocol, under `hostnameMutex(host)`: |
| +1. `PendingForHost(host) > 0` → `errLifetimeSettling`. |
| +2. Read the record (`GetRecordRaw`) and the slot. A record present and `RecordLifetimeValid` → `release.ErrCASConflict` (the existing conflict; the slot is not touched). A record present and not valid (a tombstone, or a lifetime behind the slot) is stale: remembered with its ETag for step 6. |
| +3. N = max(`LastIncarnation(host)`, slot.Incarnation) + 1; `rec.Incarnation = N`; `body = json.Marshal(rec)`; `BeginAllocation({OpID: new id, Account, App, Host, N, pending, body, predecessor ETag or "" when the slot is absent})`. |
| +4. `WriteHostIncarnation({N, live, "create " + OpID}, predecessor)`. |
| +5. If a stale record was found, `release.DeleteIfMatch` its ETag (a conflict here means the key changed: reread and resolve by the recovery branches). |
| +6. `WriteRecord(ctx, s, key, []byte("{}"), body, "")` (create-only). |
| +7. `SetAllocationState(committed)`. |
| + |
| +Recovery (`recoverLifetimes`), per pending allocation, under the hostname mutex, reading the slot and the record raw: slot absent or at the predecessor ETag → step 4 onward; slot `{N, live, "create " + OpID}` and record absent → step 6 onward; slot at N and the record at N → committed; slot at N and a stale record under the key → step 5 onward; slot past N or at N with another intent → `superseded` (an operator moved it by hand); any read error → leave pending. Then the retirements (Task 4) and migrations (Task 5). |
| + |
| +- [ ] **Step 1: Write the failing tests** against an in-process console with `release.NewDirStore` and `HostClaim` off: |
| + - `TestCreateAllocatesSlotThenRecord`, `TestCreateAfterRetirementGetsNextIncarnation` (seeded slot `{1, retired}` → `{2, live}`), `TestCreateReplacesStaleRecord` (a fieldless record beside `{1, retired}` is deleted conditionally and incarnation 2 written), `TestCreateOfALiveInstanceIsTheExistingConflict` (record at 1 valid → `ErrCASConflict`, slot unchanged, no allocation row), `TestCreateRefusesWhileAnOperationIsPending` (a pending allocation row for the host → `errLifetimeSettling`). |
| + - `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`: a pending allocation with the slot written and no record; recovery writes the record and commits; an injected late create-only write of the same body then fails with `ErrCASConflict` and the record's ETag is unchanged. |
| + - `TestRecoveryWritesTheSlotWhenMissing`, `TestRecoveryContinuesFromAPredecessorSlot` (slot still at the predecessor's ETag), `TestRecoveryLeavesALandedCreateServing` (record ETag unchanged, state committed), `TestRecoveryRemovesAStaleRecordThenWrites`, `TestRecoverySupersedesWhenSlotMovedPast`, `TestRecoveryLeavesPendingWhenUnreadable` (a store whose `Get` of the slot fails), `TestRecoverySurvivesASecondCrash` (recovery run twice with a crash injected after step 4 on the first run). |
| + - `TestRecoveryUsesOnlyPersistedState`: close and reopen the SQLite store between the crash and the recovery; no request-local state. |
| - [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement** `lifetimes_create.go` and `lifetimes_recovery.go`; route the three creation paths through `createRecordWithLifetime`; start the pass in `cmd/console/main.go`. |
| -- [ ] **Step 4: Run the tests and the existing create-door tests** (`go test ./internal/console -run 'Create|Lifetime|Recovery' -count=1`). |
| -- [ ] **Step 5: Commit** `console: creates allocate a host incarnation through a persisted intent, with recovery`. |
| +- [ ] **Step 3: Implement**; route the four claims-off creation paths through `createRecordWithLifetime` (the claims-on branches are untouched). |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'Create|Lifetime|Recovery|DomainAttach|Import|Seed' -count=1`. |
| +- [ ] **Step 5: Commit** `console: claims-off creates allocate a host incarnation through a persisted intent, with recovery`. |
| ### Task 4: Retirement through a tombstone |
| **Files:** |
| - Create: `internal/console/lifetimes_retire.go` |
| -- Modify: `internal/console/apiinstances.go` (`retireInstanceRecord`, called from `handleAPIInstanceDelete`), `internal/provision/provision.go` (`GetRecord`, `GetRecordETag`: a tombstone reads as not found for every caller except the lifetime code, which uses a new `GetRecordRaw`), `internal/console/lifetimes_recovery.go` (the retirement branches) |
| +- Modify: `internal/console/hostclaimops.go` (the claims-off branch of `retireHostRecord`, line ~23: call `retireRecordWithLifetime` for the record, then the rest of `provision.RetireRecord`'s effects with the record already gone), `internal/provision/retirement.go` (`RetireOptions` gains `RecordRetired bool` so `RetireRecord` skips its own record deletion when the console already retired it), `internal/console/apiinstances.go` (`handleAPIInstanceDelete`: map `errRetireRefused` to 409 with its sentence, `errLifetimeSettling` to 409, a `void` outcome to the existing already-absent 200), `internal/console/lifetimes_recovery.go` (the retirement branches) |
| - Test: `internal/console/lifetimes_retire_test.go` |
| **Interfaces:** |
| -- Consumes: Task 2's retirement rows; Task 1's slot helpers; `release.DeleteIfMatch`; `provision.WriteRecord` with an `If-Match` ETag. |
| -- Produces: `func (s *Server) retireRecordWithLifetime(ctx context.Context, acct, app, host string) error` replacing the plain delete inside `retireInstanceRecord`; `func provision.GetRecordRaw(ctx, s, acct, app, host) (*Record, string, error)` (tombstones included); `func provision.Tombstone(host string, incarnation uint64) Record` (`Host`, `Incarnation`, `Retired: true`, nothing else). |
| +- Consumes: Tasks 1 to 3; `release.DeleteIfMatch`; Task 5's `migrateHostLifetime` for a legacy record (so build Task 5's function first or stub it behind the same signature). |
| +- Produces: `func (s *Server) retireRecordWithLifetime(ctx context.Context, acct, app, host string) error`; `var errRetireRefused`. |
| -The protocol: read the record and the slot (ETags E, S); `BeginRetirement(host, N, id, E, S)`; (1) `WriteRecord` the tombstone with `If-Match: E`; on a CAS failure reread: a record at N that is not a tombstone → `refused` with the copy sentence; another incarnation or slot past N → `void`; (2) `WriteHostIncarnation({N, retired, "retire " + id}, S)`; (3) `DeleteIfMatch` the tombstone on its ETag; `committed`. A record with no `Incarnation` (legacy, no slot) keeps today's plain delete, after writing a slot `{1, retired}` create-only so the hostname's next create starts at 2. Recovery branches exactly as spec §6.4: record at E → step 1; tombstone present and slot at S → step 2; tombstone present and slot retired → step 3; record absent and slot retired → committed; slot retired and a non-tombstone record at N → committed after a conditional delete of that record; slot live and a non-tombstone record at N with another ETag → `refused`; another incarnation or slot past N with no tombstone → `void`; unreadable → pending. |
| +The protocol, under the hostname mutex, pending-op check as in Task 3: |
| +- A record with no `Incarnation` and no slot (legacy): run `migrateHostLifetime` first, under the same exclusion; if it cannot complete (unreadable), refuse with `errLifetimeSettling`. Never a plain delete. |
| +- Read the record raw (bytes B, ETag E) and the slot (ETag S); `BeginRetirement({OpID, Account, App, Host, N, E, S, pending})`. |
| +- (1) `WriteRecord(ctx, s, key, B, TombstoneBody(B), E)`. On `ErrCASConflict`: reread; a non-tombstone record at N → `refused` (`errRetireRefused`); another incarnation, or slot past N → `void`. |
| +- (2) `WriteHostIncarnation({N, retired, "retire " + OpID}, S)`. |
| +- (3) `DeleteIfMatch` the tombstone on its ETag. |
| +- `committed`. |
| -- [ ] **Step 1: Write the failing tests**: |
| - - `TestRetireWritesTombstoneThenRetiresSlotThenRemovesTombstone`: after the door, the record key is absent, the slot is `{1, retired}`, the retirement is `committed`; a probe `GetRecord` mid-way (after step 1, with a hook) returns not found while `GetRecordRaw` returns the tombstone. |
| - - `TestRetireRefusesWhenTheRecordChangedInItsLifetime`: the record's ETag moves between read and step 1 (a `SetRecordRegion` in between); the door answers 409 with the sentence; the record is kept; the retirement is `refused`. |
| - - `TestRetireVoidsAgainstASuccessor`: slot at 2 when the door read 1 → `void`, nothing written. |
| - - `TestLateCreateFailsOnTheTombstone`: with the tombstone in place, a create-only write fails; after step 3 with the slot retired, a late record at 1 is written by the test and `RecordLifetimeValid` is false for it. |
| - - `TestRetirementRecoveryBranches`: one subtest per branch above, each asserting the final slot, key state and retirement state. |
| - - `TestLegacyRecordRetiresWithASlot`: a fieldless record with no slot; the door deletes it and leaves `{1, retired}`. |
| +Recovery branches, from the persisted E and S: record at E → (1); tombstone for N present and slot at S → (2); tombstone present and slot `{N, retired}` → (3); record absent and slot retired → committed; slot `{N, retired}` and a non-tombstone record at N (any ETag) → committed, and the record is deleted conditionally on its ETag (stale under a retired slot); slot live at S and a non-tombstone record at N with an ETag other than E → `refused`; another incarnation or slot past N with no tombstone → `void`; unreadable → pending. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestRetireWritesTombstoneThenRetiresSlotThenRemovesTombstone` (with a hook after (1): `GetRecord` not found, `GetRecordRaw` retired, `record_seq` advanced by one); `TestRetireRefusesWhenTheRecordChangedInItsLifetime` (a `SetRecordRegion` between the read and (1) → the door answers 409 with the sentence; record kept; state refused); `TestRetireVoidsAgainstASuccessor`; `TestLateCreateFailsOnTheTombstoneAndIsStaleAfterIt`; `TestLegacyRecordIsMigratedThenRetired` (no plain delete; the slot ends `{1, retired}`); `TestDelayedLegacyDeleteCannotEraseASuccessor` (a plain `Store.Delete` issued against the old key is not part of the protocol; assert the door never calls `Store.Delete` for a record key, through a counting store wrapper); `TestRetirementRecoveryBranches` (one subtest per branch above, each asserting the final slot, key and state, with a second crash injected during recovery in two of them). |
| - [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement.** `GetRecord`/`GetRecordETag` return the store's not-found error for a tombstone. |
| -- [ ] **Step 4: Run** `go test ./internal/console -run 'Retire|Tombstone|Lifetime' -count=1` and `go test ./internal/provision -run 'Record' -count=1`. |
| -- [ ] **Step 5: Commit** `console: retirement replaces the record with a tombstone before retiring the slot`. |
| +- [ ] **Step 3: Implement.** |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'Retire|Tombstone|Lifetime|InstanceDelete' -count=1` and `go test ./internal/provision -run 'Retire' -count=1`. |
| +- [ ] **Step 5: Commit** `console: claims-off retirement replaces the record with a tombstone before retiring the slot`. |
| -### Task 5: Lazy migration with a marker |
| +### Task 5: Lazy migration with a marker fenced in the write primitive |
| **Files:** |
| -- Create: `internal/console/lifetimes_migrate.go` |
| -- Modify: `internal/provision/provision.go` (`updateRecordSeeding`: a record carrying `IncarnationMigrating` makes the mutation wait and retry, bounded to 30 s, then returns `ErrRecordMigrating`), `cmd/console/main.go` (start `StartIncarnationMigrationPass` behind `CARLOS_LIFETIMES_MIGRATE=1`) |
| -- Test: `internal/console/lifetimes_migrate_test.go`, `internal/provision/provision_test.go` |
| +- Modify: `internal/provision/record_sequence.go` (`WriteRecord`: after decoding `before`, if its `incarnation_migrating` is non-empty and the context does not carry `WithMigration(ctx, thatID)`, return `ErrRecordMigrating`; a create-only write is unaffected), `internal/provision/provision.go` (`updateRecordSeeding` and `SetRecordRegion`: treat `ErrRecordMigrating` as a retryable conflict with backoff, at most 30 s, then return it), `internal/console/claim_workers.go` (register `"StartIncarnationMigrationPass"`), `cmd/console/main.go` |
| +- Create: `internal/console/lifetimes_migrate.go`, `internal/provision/lifetime_migration.go` (`WithMigration`, `ErrRecordMigrating`) |
| +- Test: `internal/provision/record_sequence_test.go`, `internal/console/lifetimes_migrate_test.go` |
| **Interfaces:** |
| -- Consumes: Task 2's migration rows; Task 1's helpers. |
| -- Produces: `func (s *Server) migrateHostLifetime(ctx context.Context, acct, app, host string) error`; `StartIncarnationMigrationPass` (every 5 minutes, up to 50 hosts per pass, oldest records first, only records with no `Incarnation`); `var provision.ErrRecordMigrating = errors.New("provision: the record is being migrated; try again")`. |
| +- Produces: `var provision.ErrRecordMigrating = errors.New("provision: the record is being migrated; try again")`; `func provision.WithMigration(ctx context.Context, id string) context.Context`; `func (s *Server) migrateHostLifetime(ctx context.Context, acct, app, host string) error`; `func (s *Server) StartIncarnationMigrationPass(ctx context.Context)` (behind `CARLOS_LIFETIMES_MIGRATE=1`, every 5 minutes, at most 50 records per pass, oldest first, only records with no `Incarnation` and no marker whose slot is absent). |
| -The protocol: `BeginMigration(host, id)`; (1) CAS `IncarnationMigrating = id` against the ETag read; on conflict reread and retry from the current record (a legitimate update won); (2) `WriteHostIncarnation({1, live, "migration " + id}, "")` create-only, or `If-Match` when a slot at 1 with this intent exists from an earlier attempt; (3) CAS `Incarnation = 1`, `IncarnationMigrating = ""`; `committed`. Recovery branches as spec §6.4. |
| +The protocol, under the hostname mutex: `BeginMigration({OpID, Account, App, Host, "", pending})`; (1) read raw (B, E); `WriteRecord(WithMigration(ctx, OpID), s, key, B, B with IncarnationMigrating = OpID, E)`; on `ErrCASConflict` reread and retry from the current record (bounded to 5 attempts, then leave pending); persist the marked record's ETag M in the row; (2) `WriteHostIncarnation({1, live, "migration " + OpID, RecordETag: M}, "")` create-only, or `If-Match` when a slot at 1 with this intent exists from an earlier attempt; (3) `WriteRecord(WithMigration(ctx, OpID), ..., Incarnation = 1, IncarnationMigrating = "", against M)`; `committed`. Recovery: record unmarked and fieldless, slot absent → (1); record marked with this OpID, slot absent → (2); record marked, slot bound to M → (3); record carries `Incarnation 1` → committed; record gone, another lifetime, or a slot not of this migration → `superseded`; unreadable → pending. A fieldless record beside an unrelated existing slot is never migrated: it is stale (Task 1's rule) and the create door removes it. |
| - [ ] **Step 1: Write the failing tests**: |
| - - `TestMigrationMarksThenSlotsThenSets`: a fieldless record; after the pass the slot is `{1, live, "migration <id>"}`, the record has `Incarnation 1` and no marker. |
| - - `TestMigrationRestartsFromAnUpdateThatWon`: an update changes the record between the read and step 1; the marker CAS fails; the pass rereads and completes on the updated record; the update's field survives. |
| - - `TestWriterWaitsOnTheMarker`: with the marker set and no slot yet, `updateRecordSeeding` blocks until a goroutine completes the migration, then applies; with the marker never cleared it returns `ErrRecordMigrating` after 30 s (use a short bound via a test hook). |
| - - `TestMigrationRecoveryBranches`: marked and no slot → slot then set; marked and slot bound → set; `Incarnation 1` present → committed; record gone → superseded. |
| - - `TestMigrationPassIsOffWithoutTheFlag`. |
| + - In `record_sequence_test.go`: `TestWriteRecordRefusesAMarkedRecordWithoutTheMigrationContext`, `TestWriteRecordAllowsTheMigrationsOwnWrite`, `TestCreateOnlyWriteIgnoresTheMarker`. |
| + - `TestMigrationMarksThenSlotsThenSets` (final: slot `{1, live, "migration <id>", RecordETag M}`, record `Incarnation 1`, no marker, state committed). |
| + - `TestMigrationRestartsFromAnUpdateThatWon` (an update between the read and (1); the marker CAS fails; the pass rereads and completes; the update's field survives). |
| + - `TestAnIssuedWriteLosesToTheMarkerAndRetries` (a box-shaped `SetRecordMove` goroutine blocked on a hook with the pre-marker ETag: its `WriteRecord` fails `ErrCASConflict`, its reread sees the marker and gets `ErrRecordMigrating`, it waits, and succeeds after the migration's (3)). |
| + - `TestWriterGivesUpAfterTheBound` (marker never cleared → `ErrRecordMigrating` after the bound, shortened by a test hook). |
| + - `TestMigrationRecoveryBranches` (one subtest per branch). |
| + - `TestMigrationPassIsOffWithoutTheFlag`, `TestMigrationSkipsAFieldlessRecordBesideAnUnrelatedSlot`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/console -run 'Migration|Migrat' -count=1` and `go test ./internal/provision -run 'Migrating|updateRecord' -count=1`. |
| -- [ ] **Step 5: Commit** `console: lazy incarnation migration marks the record by CAS before binding its slot`. |
| +- [ ] **Step 4: Run** `go test ./internal/provision -run 'WriteRecord|Migrat|Record' -count=1` and `go test ./internal/console -run 'Migrat' -count=1`. |
| +- [ ] **Step 5: Commit** `lifetimes: lazy migration marks the record by CAS, fenced in the write primitive`. |
| ### Task 6: Consumers check the lifetime |
| **Files:** |
| -- Modify: `internal/provision/reconcile.go` (the ready-row minting loop near line 1258, `holdExisting`, `mintHeld`), `internal/console/apiinstances.go` (`instanceGate`'s record read), `internal/host/edgesfile.go` (the publication loop: a row whose record fails validity is not published) |
| -- Create: `internal/provision/lifetime_cache.go` (`type SlotCache` keyed by host with the slot's ETag, refreshed per reconcile pass) |
| -- Test: `internal/provision/reconcile_lifetime_test.go`, `internal/console/apiinstances_lifetime_test.go` |
| +- Create: `internal/provision/lifetime_cache.go` |
| +- Modify: `internal/provision/reconcile.go` (where the pass lists and decodes records into its live set, before the minting loop: drop invalid records there, so the removal loop retires their rows as "record gone"), `internal/provision/provision.go` (`ListRecords`: unchanged; add `func ListRecordsValid(ctx, s, cache *SlotCache, ...)` used by reconcile and by the box's moves worker's record listing in `internal/host/movespass.go`), `internal/console/apiinstances.go` (the door's record read helper that every instance door uses after `instanceGate`: a record that fails validity reads as not found), `internal/host/edgesfile.go` (a row whose record is invalid is not published: the writer already has the records in hand for owners; add the check there), the agent's `provision.ReconcileConfig{...}` construction in `internal/host` (set `Slots: provision.NewSlotCache()`) |
| +- Test: `internal/provision/reconcile_lifetime_test.go`, `internal/console/apiinstances_lifetime_test.go`, `internal/host/edgesfile_lifetime_test.go` |
| **Interfaces:** |
| -- Consumes: Task 1's `RecordLifetimeValid`, `ReadHostIncarnation`. |
| -- Produces: `func (c *SlotCache) Valid(ctx context.Context, s release.Store, rec *Record) (bool, error)`; `ReconcileConfig.Slots *SlotCache` (nil keeps today's behaviour for tests that predate this; production sets it). |
| +- Produces: `type SlotCache struct{...}`; `func NewSlotCache() *SlotCache`; `func (c *SlotCache) Valid(ctx context.Context, s release.Store, rec *Record, recordETag string) (bool, error)` (reads the slot for every record on a claims-off deployment, fieldless ones included, cached per host by the slot's ETag with a conditional read; a read error returns `false, err` and the caller treats the record as unreadable for this pass, not as absent); `ReconcileConfig.Slots *SlotCache` (required in production; a nil cache makes reconcile log once and treat every record as valid, for tests that predate this, and `TestProductionReconcileConfigHasASlotCache` pins that the agent sets it). |
| -- [ ] **Step 1: Write the failing tests**: |
| - - `TestReconcileSkipsARecordBehindItsSlot`: a record at 1 with slot `{2, live}` mints no row and retires an existing row as "record gone". |
| - - `TestReconcileSkipsATombstone`. |
| - - `TestReconcileKeepsALegacyRecordWithNoSlot` and `...AMarkedRecordUnderItsMigrationSlot`. |
| - - `TestInstanceGateAnswers404ForAStaleRecord` (the console door). |
| - - `TestEdgesFileOmitsAStaleRecordsRow`. |
| +- [ ] **Step 1: Write the failing tests**: `TestReconcileRemovesARecordBehindItsSlot` (record at 1, slot `{2, live}`: no mint; an existing row is retired as "record gone"); `TestReconcileRemovesATombstone`; `TestReconcileRemovesAFieldlessRecordBesideAnUnrelatedSlot` (slot `{1, retired}`); `TestReconcileKeepsALegacyRecordWithNoSlot` and `TestReconcileKeepsAMarkedRecordUnderItsMigrationSlot`; `TestReconcileLeavesARecordAloneWhenTheSlotIsUnreadable` (no mint, no retire); `TestInstanceDoorsAnswer404ForAStaleRecord`; `TestEdgesFileOmitsAStaleRecordsRow`; `TestProductionReconcileConfigHasASlotCache`. |
| - [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement**, reading the slot only for records that carry `Incarnation` or a marker (legacy records cost no read). |
| -- [ ] **Step 4: Run** the touched packages' targeted tests, then `go test ./internal/provision -run 'TestPlan14RootReadCensus|TestPlan9B8RecordWriteCensus' -count=1`; add the census overlay rows the test names (the slot read inside reconcile is a bucket read, not a root read; expect envelope re-pins for `internal/host`). |
| +- [ ] **Step 3: Implement.** |
| +- [ ] **Step 4: Run** the touched packages' targeted tests, then `go test ./internal/provision -run 'TestPlan14RootReadCensus|TestPlan9B8RecordWriteCensus' -count=1` and add the overlay rows the census names. |
| - [ ] **Step 5: Commit** `lifetimes: reconcile, the doors and the edges file treat a record behind its slot as absent`. |
| ### Task 7: `move_seq` on the record and the pair on rows |
| **Files:** |
| -- Modify: `internal/provision/offline_move.go` (`offlineBegin`, `OfflineFlip`, `offlineOpenMut`'s transition, `OfflineAbort`'s reopening, `OfflineFinish`: each increments `rec.MoveSeq`, initialising to 1 when 0; the move state gains `Seq uint64 \`json:"seq,omitempty"\`` carrying the sequence of the transition that produced it), `internal/registry/registry.go` (columns `incarnation INTEGER NOT NULL DEFAULT 0`, `serving_seq INTEGER NOT NULL DEFAULT 0`; `Route.Incarnation uint64`, `Route.ServingSeq uint64`; `Upsert` writes them when non-zero; `SetHold` and `ReleaseHold` gain variants `SetHoldSeq(host, status, op string, incarnation, seq uint64)` and `ReleaseHoldSeq(host, op string, incarnation, seq uint64)` that write the pair in the same statement), `internal/provision/reconcile.go` (`RouteFor` copies `rec.Incarnation` and `rec.MoveSeq` into the minted row), `internal/host/movesoffline.go` (the hold reconcile hook passes the move's `Seq`), `internal/host/movesoffline_land.go` (the target's held mint and its open carry `open`'s seq), the abort path's reopening (the sealer's `ReleaseHoldSeq` with the abort's seq) |
| -- Test: `internal/provision/offline_move_test.go`, `internal/registry/registry_test.go`, `internal/host/movesoffline_test.go` (the two-box witness) |
| +- Modify: `internal/provision/offline_move.go` (`OfflineStart` and `OfflineStartStaged` (the hold), `OfflineOpenTransition` (the target serves), `OfflineAbort` (the source reopens) and `OfflineFinish` each set `rec.MoveSeq = rec.MoveSeq + 1` (1 when 0) and `rec.Move.Seq` (nil-safe) to that value; `OfflineStage`, `OfflineRestage`, `OfflineSeal`, `OfflineSealedBy`, `OfflineFlip`, `OfflineLandedBy`, `OfflineFinishingTransition` leave both alone), `internal/provision/provision.go` (`MoveState.Seq uint64 \`json:"seq,omitempty"\``), `internal/provision/offline_move_test.go` (`TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo`: `move_seq` joins the allowlist, the preservation assertions stay), `internal/registry/registry.go` (columns `incarnation INTEGER NOT NULL DEFAULT 0`, `serving_seq INTEGER NOT NULL DEFAULT 0` in the `ALTER TABLE` list; `Route.Incarnation uint64`, `Route.ServingSeq uint64`; `routeCols`, `routeColCount` (+2), `scanRoute`, `routeArgs`, `normalizeRoute`; `Upsert`'s conflict clause **always keeps the existing row's pair** (`incarnation=routes.incarnation, serving_seq=routes.serving_seq`), so an ordinary upsert never moves it; the pair changes only through the three setters below; `remote_dispatch.go` gains the new methods), `internal/provision/reconcile.go` (`RouteFor` copies `rec.Incarnation` and `rec.MoveSeq` into a minted row; after the mint or for an existing provisioned row, the pass calls `InstallLifetime` when the row's incarnation differs from the record's, or the row is `(0, 0)` and the record has a pair; it never raises `serving_seq` within the same incarnation), `internal/host/movesoffline.go` (`offlineHold`'s result gains the pair: `func(rec provision.Record) (status, op string, pair registry.Pair, ok bool)`, with `pair = {rec.Incarnation, rec.Move.Seq}`; `holdExisting` and `mintHeld` in reconcile take the pair and call `SetHoldIfSeq` / write it on the mint), `internal/host/movesoffline_land.go` and `internal/host/movesoffline.go` (`moveCfg.ReleaseHold` becomes `func(host, op string, pair registry.Pair) (bool, error)`, backed by `ReleaseHoldSeq`, and the open and abort paths pass the record's `Move.Seq` at that transition with its incarnation) |
| +- Test: `internal/provision/offline_move_test.go`, `internal/registry/registry_test.go`, `internal/provision/reconcile_lifetime_test.go`, `internal/host/movesoffline_e2e_test.go` |
| **Interfaces:** |
| -- Produces: the record's `MoveSeq` and `Move.Seq`; `registry.Route.Incarnation`, `registry.Route.ServingSeq`; `SetHoldSeq`, `ReleaseHoldSeq`. |
| +- Produces: `type registry.Pair struct { Incarnation, ServingSeq uint64 }`; `func (r *Registry) SetHoldIfSeq(host, status, op, expectOp string, pair Pair) (bool, error)` (the `SetHoldIf` predicate, plus the pair in the same statement); `func (r *Registry) ReleaseHoldSeq(host, op string, pair Pair) (bool, error)`; `func (r *Registry) InstallLifetime(host string, pair Pair) error` (the pair alone, used by reconcile for a new lifetime on an existing row); `Route.Incarnation`, `Route.ServingSeq`; `MoveState.Seq`; `Record.MoveSeq` advanced as above. |
| - [ ] **Step 1: Write the failing tests**: |
| - - `TestOfflineTransitionsAdvanceMoveSeq`: start → 1, open → 2, finish → 3 on one record; abort after start → 2 (the reopening) and finish → 3; a record with `MoveSeq 5` starts at 6. |
| - - `TestRegistryPairColumns`: upsert with the pair, read back; `SetHoldSeq` writes status, op and pair in one statement; `ReleaseHoldSeq` the same. |
| - - `TestRouteForCopiesThePair` (a record `{Incarnation 2, MoveSeq 3}` mints a row `(2, 3)`; a record with neither mints `(0, 0)`). |
| - - In the two-box witness: after `start` the source row is held with `(1, 1)`; after `open` the target row is ready with `(1, 2)`; after an abort from landed the source row is ready with `(1, 2)` and the target has no row. |
| + - `TestOfflineTransitionsAdvanceMoveSeq`: direct path start 1, open 2, finish 3; staged path stage 0, start-staged 1, open 2, finish 3; abort after start → 2 (reopening), finish → 3; stage, restage, seal, flip, landed, finishing leave it; a record at `MoveSeq 5` starts at 6; `Move.Seq` equals the record's value at each advancing transition. |
| + - `TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo` passes with `move_seq` in its allowlist. |
| + - `TestRegistryPairColumns` (upsert a row with the pair, read back; an ordinary upsert of the same host with another pair keeps the first); `TestSetHoldIfSeqWritesStatusOpAndPairTogether` (and refuses when `expectOp` differs); `TestReleaseHoldSeqWritesThePair`; `TestInstallLifetimeReplacesBothComponents` (`(1, 9)` → `(2, 0)`). |
| + - `TestRouteForCopiesThePair`; `TestReconcileInstallsTheRecordsPairOnAMintedRow` (after a finished move at `move_seq 3`, delete the row, reconcile → `(1, 3)`); `TestReconcileInstallsANewLifetimeOnAnExistingRow` (row `(1, 9)`, record `{2, 0}` → `(2, 0)`); `TestReconcileNeverRaisesTheSequenceWithinALifetime` (row `(1, 1)`, record `{1, 3}` with a move in flight → the row stays until the move's own transition writes it). |
| + - In `TestOfflineMoveE2E`: the records are created through the lifetime (slot `{1, live}`, `Incarnation 1`) and the validity checks are on; after `start` the source row is held with `(1, 1)`; after `open` the target row is ready with `(1, 2)`; a second run aborting from landed leaves the source ready with `(1, 2)` and the target without a row. |
| - [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement.** The `Upsert` conflict clause keeps the pair when the incoming value is 0 (`CASE WHEN excluded.serving_seq = 0 THEN routes.serving_seq ELSE excluded.serving_seq END`, and the same for `incarnation`). |
| -- [ ] **Step 4: Run** `go test ./internal/registry -run 'Pair|Hold' -count=1`, `go test ./internal/provision -run 'MoveSeq|RouteFor|Offline' -count=1`, `go test ./internal/host -run 'Offline' -count=1`, then the two census tests; add rows for the changed `internal/host` units (`offlineHold`, the landing's mint, the abort's release) at their existing classes. |
| -- [ ] **Step 5: Commit** `moves: the offline transitions advance move_seq and rows carry (incarnation, serving_seq)`. |
| +- [ ] **Step 3: Implement.** |
| +- [ ] **Step 4: Run** `go test ./internal/registry -run 'Pair|Hold|Lifetime' -count=1`, `go test ./internal/provision -run 'MoveSeq|RouteFor|Offline|Reconcile' -count=1`, `go test ./internal/host -run 'OfflineMoveE2E|Offline' -count=1`, then both census tests (the record-write census gains the transitions' changed digests; the root-read census the `internal/host` units). |
| +- [ ] **Step 5: Commit** `moves: the offline hold, open, abort and finish advance move_seq, and rows carry (incarnation, serving_seq)`. |
| -### Task 8: Publish the pair and order by it |
| +### Task 8: Publish the pair and order comparable entries by it |
| **Files:** |
| -- Modify: `internal/host/edgesfile.go` (`edgeHost` gains `Incarnation uint64 \`json:"incarnation,omitempty"\``, `ServingSeq uint64 \`json:"serving_seq,omitempty"\``, `TargetHost string \`json:"target_host,omitempty"\``; routes publish the row's pair; an alias entry publishes its primary's pair and `TargetHost: a.Host`; a catch-all's derived entry publishes `TargetHost` too), `internal/edge/homemap.go` (`edgesHostView` and `homeEntry` gain the three fields; `dialBeats` compares `(incarnation, servingSeq)` first, then the existing alias and stamp rules; for an alias host, an entry whose `targetHost` differs from the previous winner's wins outright when it is a ready route, before the pair comparison) |
| +- Modify: `internal/host/edgesfile.go` (`edgeHost` gains `Incarnation uint64 \`json:"incarnation,omitempty"\``, `ServingSeq uint64 \`json:"serving_seq,omitempty"\``, `TargetHost string \`json:"target_host,omitempty"\``; a route publishes its row's pair; an explicit alias entry publishes its primary's pair and `TargetHost: a.Host`; a catch-all's derived alias entry (the `catchallServed` branch near line 624) publishes `TargetHost: wd.Host` and the primary's pair), `internal/edge/homemap.go` (`edgesHostView` and `homeEntry` gain `incarnation`, `servingSeq`, `targetHost`; `foldHomes` copies all three with the dial winner; `dialBeats` becomes: `!in.dial` → false; `!prev.dial` → true; `prev.alias != in.alias` → the real route wins as today; then higher `(incarnation, servingSeq)` wins; equal → `in.stamp > prev.stamp` as today) |
| - Test: `internal/host/edgesfile_test.go`, `internal/edge/homemap_test.go` |
| **Interfaces:** |
| -- Produces: the three fields on the wire; `dialBeats` semantics per spec §6.4: higher pair wins; equal pairs fall to today's rules; an alias ready route for a different `target_host` wins over an entry for another target. |
| +- Produces: the three fields on the wire and in the fold; the precedence order above. The different-target alias rule is plan 3's (it needs holding entries); plan 1 only publishes `target_host`. |
| -- [ ] **Step 1: Write the failing tests**: |
| - - `TestEdgesFilePublishesThePairAndTargetHost`: a row `(2, 3)` publishes `incarnation 2, serving_seq 3`; its alias publishes the same pair with `target_host` the row's host; a derived catch-all alias publishes `target_host`. |
| - - `TestFoldPrefersTheHigherPair`: two files for one host, `(1, 2)` and `(1, 1)` → the former; `(2, 0)` beats `(1, 9)`; `(0, 0)` loses to any pair; equal pairs → today's stamp rule (a test case with equal pairs and different stamps). |
| - - `TestFoldAliasDifferentTargetWins`: alias `customer.example.com` published by box A for target `a.example` at `(1, 9)` and by box B for target `b.example` at `(1, 0)` → B, because the targets differ and B's is a ready route (the later publication); same targets → the pair rule. |
| - - `TestFoldLegacyPublicationsUnchanged`: files without the fields fold exactly as before (reuse an existing fold test's fixtures). |
| +- [ ] **Step 1: Write the failing tests**: `TestEdgesFilePublishesThePairAndTargetHost` (a row `(2, 3)`; its explicit alias; a derived catch-all alias); `TestFoldPrefersTheHigherPair` (`(1, 2)` over `(1, 1)`; `(2, 0)` over `(1, 9)`; `(0, 0)` loses to any pair; equal pairs → the stamp rule); `TestFoldKeepsRealRouteOverAHigherPairAlias`; `TestFoldNeverDialsACertOnlyEntryForItsPair`; `TestFoldCopiesTheWinnersPairAndTarget` (three candidates, the final winner's fields intact); `TestFoldLegacyPublicationsUnchanged` (existing fixtures without the fields fold as before; `TestHomeMapPicksTheSameHomeOnADuplicateClaim` still passes). |
| - [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement.** For the different-target rule, `foldHomes` must know which entry is newer; with no sequence between targets, the rule is: when two alias entries name different targets, the one whose `target_host` has a ready route in the same fold wins; if both do, the higher pair; this is the "ready route wins an alias for one refresh cycle" rule, and the test pins it. |
| -- [ ] **Step 4: Run** `go test ./internal/host -run 'EdgesFile' -count=1` and `go test ./internal/edge -run 'Fold|HomeMap' -count=1`, then the census tests (the edges writer is a changed unit). |
| -- [ ] **Step 5: Commit** `edges: publish (incarnation, serving_seq) and target_host; the home map orders by the pair`. |
| +- [ ] **Step 3: Implement.** |
| +- [ ] **Step 4: Run** `go test ./internal/host -run 'EdgesFile' -count=1`, `go test ./internal/edge -run 'Fold|HomeMap' -count=1`, then the census tests (the edges writer's changed unit). |
| +- [ ] **Step 5: Commit** `edges: publish (incarnation, serving_seq) and target_host; the home map orders comparable entries by the pair`. |
| ### Task 9: Lifetimes end to end |
| **Files:** |
| -- Test: `internal/host/movesoffline_lifetime_test.go` (the two-box offline witness, extended), `internal/console/lifetimes_e2e_test.go` |
| +- Test: `internal/host/movesoffline_lifetime_test.go` (extending `TestOfflineMoveE2E`'s fixtures), `internal/console/lifetimes_e2e_test.go` |
| + |
| +These are §11's witnesses plan 1 owns; the ones that need holding publications are plan 3's and are not claimed here. |
| -- [ ] **Step 1: Write the tests** (each is one of §11's witnesses that Plan 1 owns): |
| - - `TestLifetimeAcrossAMoveAndAnAbort`: create → `(1, 0)` served; start → source held `(1, 1)` and the fold prefers the hold's publication over a retained `(1, 0)` route; open → target `(1, 2)` wins; abort from landed instead → source reopened `(1, 2)` wins over a retained target hold publication `(1, 1)`. |
| - - `TestReconstructedRowCarriesTheRecordsPair`: after a finished move (`move_seq 3`), delete the target's registry row, reconcile, and the new row is `(1, 3)`; it beats a retained `(1, 1)` publication. |
| - - `TestRecreatedHostOutranksItsPreviousLifetime`: retire (slot `{1, retired}`), create again → `(2, 0)` beats a retained `(1, 9)`. |
| - - `TestLateLegacyCreateIsInert`: slot `{1, retired}`; a fieldless record written under the key by the test; reconcile mints nothing; the door 404s; the next create replaces it. |
| +- [ ] **Step 1: Write the tests**: `TestLifetimeAcrossAMoveAndAnAbort` (create `(1, 0)`; start → source held `(1, 1)` and a fold with the hold's publication and a retained `(1, 0)` route prefers the hold's publisher for the dial; open → target `(1, 2)` wins; abort from landed → source `(1, 2)` wins over a retained target publication at `(1, 1)`); `TestReconstructedRowCarriesTheRecordsPair` (after a finished move at `move_seq 3`, delete the target's row, reconcile → `(1, 3)`, beats a retained `(1, 1)`); `TestReconstructedSourceAfterAbortWins` (abort from landed, finish, delete the source row, reconcile → `(1, 2)`, beats the target's retained `(1, 1)`); `TestRecreatedHostOutranksItsPreviousLifetime` (retire, create → `(2, 0)` beats a retained `(1, 9)`); `TestLateLegacyCreateIsInert` (slot `{1, retired}`; a fieldless record written by the test; reconcile mints nothing, the door 404s, the next create replaces it); `TestDelayedCreateReplayMoveAndRetirementInterleave` (pending create replayed; a move completes; retirement crashes between the tombstone and the slot; the late original create fails on the tombstone; recovery retires the slot and removes the tombstone). |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| -- [ ] **Step 3: Run the census tests and both cross-builds** (`GOOS=darwin go build ./internal/host/`, `GOOS=windows go vet ./internal/host/`). |
| -- [ ] **Step 4: Commit** `lifetimes: end-to-end witnesses across a move, an abort, a reconstruction and a recreation`; write the census note; push. |
| +- [ ] **Step 3: Run the census tests and both cross-builds** (`GOOS=darwin go build ./internal/host/`, `GOOS=windows go vet ./internal/host/`); write the census note. |
| +- [ ] **Step 4: Commit** `lifetimes: end-to-end witnesses across a move, an abort, a reconstruction, a recreation and an interleaved retirement`; push. |
| --- |
| ## Self-review notes |
| -- Spec coverage for §6.4's lifetime and ordering paragraphs: slot (T1, T3, T4), recovery branches (T3, T4, T5), validity rule and consumers (T1, T6), `move_seq` and the pair (T7), publication and fold (T8), legacy and migration (T5, T6), witnesses (T9). Holding entries, their five-minute validity and the probe mode are Plan 3; the move door, job and reservations Plan 2; cleanup Plan 4. |
| -- Names used across tasks: `RecordLifetimeValid`, `ReadHostIncarnation`, `WriteHostIncarnation`, `HostIncarnation{Incarnation, State, Intent}`, `createRecordWithLifetime`, `retireRecordWithLifetime`, `migrateHostLifetime`, `GetRecordRaw`, `Tombstone`, `SlotCache.Valid`, `SetHoldSeq`, `ReleaseHoldSeq`, `Route.Incarnation`, `Route.ServingSeq`, `edgeHost.Incarnation/ServingSeq/TargetHost`. |
| -- Review Focus 1 to 5 are pinned by T3, T5, T8, T6+T3 and T7 respectively. |
| +- §6.4 coverage: slot and tombstone form (T1, T4), create and recovery (T3), retirement and recovery (T4), migration, its fence and recovery (T5), validity in every claims-off consumer (T1, T6), `move_seq` and the row pair with the reconcile installation rule (T7), publication and the fold's comparable-entry order (T8), witnesses (T9). Deferred by name: holding entries and their five-minute validity, the probe mode and the alias different-target rule (plan 3); the door, token, job, reservations and reports (plan 2); cleanup (plan 4). |
| +- Names used across tasks: `RecordLifetimeValid(rec, etag, slot)`, `ReadHostIncarnation`, `WriteHostIncarnation`, `HostIncarnation{Incarnation, State, Intent, RecordETag}`, `TombstoneBody`, `GetRecordRaw`, `createRecordWithLifetime`, `retireRecordWithLifetime`, `migrateHostLifetime`, `errLifetimeSettling`, `errRetireRefused`, `ErrRecordMigrating`, `WithMigration`, `SlotCache.Valid`, `ListRecordsValid`, `registry.Pair`, `SetHoldIfSeq`, `ReleaseHoldSeq`, `InstallLifetime`, `Route.Incarnation`, `Route.ServingSeq`, `MoveState.Seq`, `edgeHost.Incarnation/ServingSeq/TargetHost`. |
| +- Review Focus 1 to 5 are pinned by T3, T5, T6+T3, T7 (two tests) and T7. |
| +- Store primitives, as verified in rev 1's review: `Store.Put(..., "")` is create-only on both backends; `WriteRecord` with `before={}` and `etag=""` is the create; `DeleteIfMatch` treats absence as success and refuses unsupported backends. |