amadan
plan: self-serve moves 1, rev 5 (Astra plan review 4: findings 1 to 9; no slot cache, atomic successor hold, provenance mark)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+138
−126
docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md+138 −126
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (1 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (139 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.466s
ok github.com/carlosframework/platform/cmd/moveprobe 11.907s
ok github.com/carlosframework/platform/cmd/zddprobe 0.761s
ok github.com/carlosframework/platform/internal/activator 58.732s
ok github.com/carlosframework/platform/internal/agentsock 0.019s
ok github.com/carlosframework/platform/internal/alerting 0.006s
ok github.com/carlosframework/platform/internal/analytics 0.019s
ok github.com/carlosframework/platform/internal/appconfig 2.987s
ok github.com/carlosframework/platform/internal/apperr 0.127s
ok github.com/carlosframework/platform/internal/applog 0.376s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.007s
ok github.com/carlosframework/platform/internal/boxid 0.864s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.010s
ok github.com/carlosframework/platform/internal/caps 0.017s
ok github.com/carlosframework/platform/internal/certwarm 0.002s
ok github.com/carlosframework/platform/internal/cgroup 0.734s
ok github.com/carlosframework/platform/internal/changelog 0.478s
ok github.com/carlosframework/platform/internal/claimactivation 1.285s
ok github.com/carlosframework/platform/internal/clamd 0.059s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.707s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.022s
ok github.com/carlosframework/platform/internal/dbverify 0.006s
ok github.com/carlosframework/platform/internal/dnsrec 0.011s
ok github.com/carlosframework/platform/internal/docsite 0.025s
ok github.com/carlosframework/platform/internal/domains 2.761s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.009s
ok github.com/carlosframework/platform/internal/edge 45.328s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.016s
ok github.com/carlosframework/platform/internal/fleetapi 0.182s
ok github.com/carlosframework/platform/internal/gate 1.852s
ok github.com/carlosframework/platform/internal/genselect 0.023s
ok github.com/carlosframework/platform/internal/geo 0.023s
ok github.com/carlosframework/platform/internal/geoip 0.040s
ok github.com/carlosframework/platform/internal/historyreadback 0.266s
ok github.com/carlosframework/platform/internal/host 135.619s
ok github.com/carlosframework/platform/internal/hostactors 6.857s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.007s
ok github.com/carlosframework/platform/internal/hostbox 3.018s
ok github.com/carlosframework/platform/internal/hostclaim 0.746s
ok github.com/carlosframework/platform/internal/hostenv 0.007s
ok github.com/carlosframework/platform/internal/hostevidence 0.007s
ok github.com/carlosframework/platform/internal/hostexec 5.404s
ok github.com/carlosframework/platform/internal/hostfence 0.048s
ok github.com/carlosframework/platform/internal/hostfiles 10.304s
ok github.com/carlosframework/platform/internal/hostfloor 0.035s
ok github.com/carlosframework/platform/internal/hostguard 1.098s
ok github.com/carlosframework/platform/internal/hostops 2.127s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.002s
ok github.com/carlosframework/platform/internal/hostquarantine 0.002s
ok github.com/carlosframework/platform/internal/hostrelocate 0.633s
ok github.com/carlosframework/platform/internal/hostresolve 0.241s
ok github.com/carlosframework/platform/internal/hostwrite 0.004s
ok github.com/carlosframework/platform/internal/infraview 0.004s
ok github.com/carlosframework/platform/internal/instancectl 5.238s
ok github.com/carlosframework/platform/internal/instanceexec 2.766s
ok github.com/carlosframework/platform/internal/instancefs 0.023s
ok github.com/carlosframework/platform/internal/instancekey 0.003s
ok github.com/carlosframework/platform/internal/instanceupdates 0.011s
ok github.com/carlosframework/platform/internal/layout 1.935s
ok github.com/carlosframework/platform/internal/lbhealth 0.005s
ok github.com/carlosframework/platform/internal/lease 10.142s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.550s
ok github.com/carlosframework/platform/internal/ledger 0.017s
ok github.com/carlosframework/platform/internal/mail 0.272s
ok github.com/carlosframework/platform/internal/maintenance 0.098s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.569s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.106s
ok github.com/carlosframework/platform/internal/metrics 0.013s
ok github.com/carlosframework/platform/internal/moves 21.404s
ok github.com/carlosframework/platform/internal/moves/archive 1.080s
ok github.com/carlosframework/platform/internal/objstore 0.246s
ok github.com/carlosframework/platform/internal/ops 0.177s
ok github.com/carlosframework/platform/internal/park 0.272s
ok github.com/carlosframework/platform/internal/plans 0.007s
ok github.com/carlosframework/platform/internal/privatefile 0.003s
ok github.com/carlosframework/platform/internal/provision 13.211s
ok github.com/carlosframework/platform/internal/publicrelease 0.008s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.576s
ok github.com/carlosframework/platform/internal/relay 4.222s
ok github.com/carlosframework/platform/internal/release 25.064s
ok github.com/carlosframework/platform/internal/rollout 3.751s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.008s
ok github.com/carlosframework/platform/internal/rootparticipation 0.063s
ok github.com/carlosframework/platform/internal/rootstate 0.029s
ok github.com/carlosframework/platform/internal/runs 0.596s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.740s
ok github.com/carlosframework/platform/internal/seeds 0.012s
ok github.com/carlosframework/platform/internal/sim/hostlife 10.601s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.705s
ok github.com/carlosframework/platform/internal/status 0.086s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.167s
ok github.com/carlosframework/platform/internal/svcrole 0.005s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.009s
ok github.com/carlosframework/platform/internal/unixdial 0.005s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.634s
=== internal_console@0 (77 s)
ok github.com/carlosframework/platform/internal/console 74.587s
=== internal_console@1 (37 s)
ok github.com/carlosframework/platform/internal/console 34.751s
=== internal_console@2 (43 s)
ok github.com/carlosframework/platform/internal/console 40.690s
=== internal_console@3 (38 s)
ok github.com/carlosframework/platform/internal/console 35.514s
=== internal_console@4 (37 s)
ok github.com/carlosframework/platform/internal/console 34.920s
=== internal_console@5 (35 s)
ok github.com/carlosframework/platform/internal/console 32.850s
=== internal_console@6 (36 s)
ok github.com/carlosframework/platform/internal/console 33.951s
=== internal_console@7 (43 s)
ok github.com/carlosframework/platform/internal/console 40.924s
=== root@0 (32 s)
ok github.com/carlosframework/platform 30.075s
=== root@1 (44 s)
ok github.com/carlosframework/platform 42.027s
=== root@2 (41 s)
ok github.com/carlosframework/platform 38.825s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.022s
=== RUN TestT4StorageProbeMinio
2026/10/22 07:24:42 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0C9354187144F, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.00s)
PASS
ok github.com/carlosframework/platform/internal/host 0.020s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md| index ebb266a97..7b8667404 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| @@ -2,26 +2,26 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 4, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan1-review.md`, 16 findings), rev 2 (`selfserve-plan1-review2.md`, F1 to F12), rev 3 (`selfserve-plan1-review3.md`, R1 to R10). Rev 4 answers R1 to R10: the DELETE door's absent-record path goes through retirement with an "expect absent" precondition; a slot CAS conflict re-classifies and only positive successor evidence supersedes, with a complete error table after the allocation is durable; a legacy retirement's migration is bound to the caller's observed ETag; the retirement recovery table is complete with exact predicates; the slot cache returns the observation it validated and refreshes both objects before believing `stale`; the box's raw rereads are judged; the hold setters install the lifetime together with the hold on a legacy row and a held predecessor row is retired by its op before the successor is minted; the fold compares pairs only between unstamped entries and the claims-off writer alone publishes them; one authoritative transition table; the fixtures corrected. |
| +Rev 5, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan1-review.md`, 16 findings), rev 2 (`selfserve-plan1-review2.md`, F1 to F12), rev 3 (`selfserve-plan1-review3.md`, R1 to R10), rev 4 (`selfserve-plan1-review4.md`, findings 1 to 9). Rev 5 answers 1 to 9 and removes two sources of interleavings rather than patching them: **there is no slot cache** (every judgement reads the slot fresh, after the record, in bounded rounds), and **a successor lifetime that needs a hold is installed with the hold in one row write** (the setters accept an older unheld lifetime). Also: a bound migration's recovery resumes against its persisted ETag and supersedes only when that ETag is gone; stale or corrupt cleanup under a live owned slot retires the slot before exposing absence; absence found on a re-read is an "expect absent" retirement; the edges writer takes the lifetimes mode explicitly and marks its entries' provenance; the runs, steering, retention and seed readers are judged; a positively stale record overrides the offline keep view. |
| **Goal:** Give every claims-off instance record a durable lifetime (the host incarnation slot) and every served row a serving order `(incarnation, serving_seq)` that boxes publish and edges honour, with crash-safe create, retirement and migration, so a later plan's holding pages and moves can never be out-ordered by a stale publication. |
| -**Architecture:** A per-hostname slot object in the deployment bucket (`control/host-incarnations/<host>.json`), created once and changed only by conditional writes, is the authority for a record's lifetime and names the record key that owns it; the console allocates, retires and migrates through persisted intents in SQLite under a per-hostname exclusion and recovers them at start. Records gain `incarnation`, `incarnation_migrating`, `move_seq` and a tombstone form; registry rows gain the pair; the claims-off edges writer publishes it with each alias's target; the home map orders unstamped entries of one lifetime by it before the existing claim stamp rule. Retirement replaces the record with a tombstone before retiring the slot. Lazy migration marks a record by CAS before binding a slot to it, and the write primitive refuses every other writer while the marker stands. |
| +**Architecture:** A per-hostname slot object in the deployment bucket (`control/host-incarnations/<host>.json`), created once and changed only by conditional writes, is the authority for a record's lifetime and names the record key that owns it; the console allocates, retires and migrates through persisted intents in SQLite under a per-hostname exclusion and recovers them at start. Records gain `incarnation`, `incarnation_migrating`, `move_seq` and a tombstone form; registry rows gain the pair; the claims-off edges writer publishes it, with each alias's target and a provenance mark; the home map orders marked, unstamped entries of one lifetime by it before the existing claim stamp rule. Retirement replaces the record with a tombstone before retiring the slot. Lazy migration marks a record by CAS before binding a slot to it, and the write primitive refuses every other writer while the marker stands. |
| -**Tech Stack:** Go. `internal/provision` (records, sequence, reconcile, offline transitions), `internal/registry`, `internal/console` (doors, store, store gate, passes), `internal/host` (edges file, offline hold and release, the agent's config), `internal/edge` (home map). Tests use `release.NewDirStore`, real registries, the console's doors in-process, and the existing two-box offline witness `TestOfflineMoveE2E` (`internal/host/movesoffline_e2e_test.go`). |
| +**Tech Stack:** Go. `internal/provision` (records, sequence, reconcile, offline transitions), `internal/registry`, `internal/console` (doors, store, store gate, passes), `internal/host` (edges file, offline hold and release, runs, steering, retention, seeds, the agent's config), `internal/edge` (home map). Tests use `release.NewDirStore`, real registries, the console's doors in-process, and the existing two-box offline witness `TestOfflineMoveE2E` (`internal/host/movesoffline_e2e_test.go`). |
| -**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md`, §6.4 ("Ordering by a per-host sequence", "Lifetimes: the host incarnation slot", "Aliases", "Freshness") and §11. Plans 2 (door, token, job, reservations, reports), 3 (holding entries, probe mode, the alias rule) and 4 (cleanup) build on this one. |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md`, §6.4 ("Ordering by a per-host sequence", "Lifetimes: the host incarnation slot", "Aliases", "Freshness") and §11. Plans 2 (door, token, job, reservations, reports), 3 (holding entries, probe mode, the alias rule) and 4 (cleanup) build on this one. One deviation from §6.4's text, stated here: boxes read the slot fresh on every judgement rather than "cached by ETag"; the store has no conditional read, and a cached positive answer is exactly what lets a retired lifetime's late record pass. |
| ## Global Constraints |
| -- **Scope: claims-off, every record.** On the console the protocol runs only when `!s.cfg.HostClaim`; on a box only when `!hostwrite.Enabled(getenv)` (`CARLOS_HOST_CLAIM` unset). Where it runs it applies to **every** instance record, fieldless ones included (a fieldless record beside a slot is stale, Task 1). Claims-on records are created by `finishClaimCreate` and retired by the host-claim operations; this plan does not touch them, their consumers read no slot (the slot cache is nil there, pinned by `TestClaimsOnConsumersReadNoSlot`), the claims-on edges writer publishes no pair, and their rows keep the claim stamp order (Task 8). Self-serve moves refuse on a claims-on console (MV-O20). |
| -- Every claims-off consumer treats a record whose lifetime verdict is `stale` (Task 1) as absent, one whose verdict is `corrupt` as today's unreadable body (reconcile protects; the delete rule may remove it conditionally), and one whose verdict is `unreadable` as a record it cannot judge this pass: reconcile protects the row, a door answers 502, a listing reports the host unreadable. |
| -- The slot is never deleted and never advanced to abandon a create (spec §6.4). It never moves backwards: every slot write names the exact prior state it replaces. A malformed slot body (unparseable, no owner, zero incarnation, a state other than `live`/`retired`) is `unreadable`, never `stale`. |
| -- One per-hostname exclusion (`withHostname`, Task 3) around create, retirement, migration and their recovery in the console process. A persisted pending operation for a hostname refuses any create or retirement of that hostname by another operation (409 "another change to this instance is still settling. try again in a minute") until recovery resolves it; unreadable evidence leaves it pending and keeps refusing. After the allocation, retirement or migration row is durable, no failure returns an ordinary error to the caller: the outcome is `errLifetimePending` (the intent stays pending; recovery owns it), a persisted terminal state, or success. |
| -- Every record write stays on `provision.WriteRecord` with the exact bytes read and the sequence discipline of `record_sequence.go`; no new raw `Put` of a record key. Every record removal on a claims-off console is `retireRecordWithLifetime` (Task 4); `provision.DeleteRecord` gains no new caller and loses its claims-off instance callers. |
| +- **Scope: claims-off, every record.** On the console the protocol runs only when `!s.cfg.HostClaim`; on a box only when `!hostwrite.Enabled(getenv)` (`CARLOS_HOST_CLAIM` unset). Where it runs it applies to **every** instance record, fieldless ones included. Claims-on records are created by `finishClaimCreate` and retired by the host-claim operations; this plan does not touch them, their consumers read no slot (the judge is nil there, pinned by `TestClaimsOnConsumersReadNoSlot`), the claims-on edges writer publishes no pair and no provenance mark, and their rows keep the claim stamp order (Task 8). Self-serve moves refuse on a claims-on console (MV-O20). |
| +- Every claims-off consumer acts on an **observation** (Task 6): a record and the slot read after it, judged `valid`, `stale`, `corrupt` or `unreadable`, with `Present` saying whether a body was there. `stale` is absent; `corrupt` is today's unreadable body (reconcile protects; the delete rule may remove it conditionally); `unreadable` cannot be judged this pass (reconcile protects the row, a door answers 502, a listing reports the host unreadable). |
| +- The slot is never deleted and never advanced to abandon a create (spec §6.4). It never moves backwards: every slot write names the exact prior state it replaces. A malformed slot body is `unreadable`, never `stale`. |
| +- One per-hostname exclusion (`withHostname`, Task 3) around create, retirement, migration and their recovery in the console process. A persisted pending operation for a hostname refuses any create or retirement of that hostname by another operation (409 "another change to this instance is still settling. try again in a minute") until recovery resolves it; unreadable evidence leaves it pending and keeps refusing. After an allocation, retirement or migration row is durable, no failure returns an ordinary error to the caller: the outcome is `errLifetimePending` (the intent stays pending; recovery owns it), a persisted terminal state, or success. In every recovery table a re-read that fails is `unreadable` and leaves the row pending; it never selects a terminal branch. |
| +- Every record write stays on `provision.WriteRecord` with the exact bytes read and the sequence discipline of `record_sequence.go`; no new raw `Put` of a record key. Every record removal on a claims-off console is `retireRecordWithLifetime` (Task 4) with an explicit expectation (an ETag, or "absent"); `provision.DeleteRecord` gains no new caller and loses its claims-off instance callers. |
| - `release.DeleteIfMatch` never falls back to a plain delete; absence counts as success. |
| - No console write to a bucket outside the deployment bucket; no root-side change. |
| -- Census: `TestPlan14RootReadCensus` and `TestPlan9B8RecordWriteCensus` are the authority on what they cover (every production package, source envelopes, tracked fields, key sites and primitive digests). The root-read overlay is `internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv`, loaded last (before the comparison) by a new block at the end of the test in `internal/provision/plan14_root_reads_test.go`, `rev3=PROPOSED-self-serve-moves-1`, with an `@fields` line naming the new record fields and the registry columns, rows for every changed unit and envelope the test names (host, provision, registry, console, edge), reviewed dispositions for changed frozen units, and the note `docs/superpowers/notes/census-proposal-self-serve-moves-1.md`. The record-write inventory is `plan14SelfServeMoves1WriteCensusInventory()`, wrapping `plan14ClaimRosterBindingWriteCensusInventory()` with every added, changed and removed site the scanner reports (write sites, key sites, readers, primitive digests), each reviewed. Both tests green. |
| +- Census: `TestPlan14RootReadCensus` and `TestPlan9B8RecordWriteCensus` are the authority on what they cover. The root-read overlay is `internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv`, loaded last (before the comparison) by a new block at the end of the test, `rev3=PROPOSED-self-serve-moves-1`, with an `@fields` line naming the new record fields and registry columns, rows for every changed unit and envelope the test names, reviewed dispositions for changed frozen units, and the note `docs/superpowers/notes/census-proposal-self-serve-moves-1.md`. The record-write inventory is `plan14SelfServeMoves1WriteCensusInventory()`, wrapping `plan14ClaimRosterBindingWriteCensusInventory()` with every added, changed and removed site the scanner reports, each reviewed. Both tests green. |
| - The new SQLite tables are classified in `claimSQLTables` (`internal/console/store_gate.go`) so a claims-on store still opens. |
| - Copy: two new sentences are copy rows pending Paul's review: "the instance changed. read it again and try the delete once more" and "another change to this instance is still settling. try again in a minute". Existing sentences are reused where the plan says so. No tenant names: `shop.example.com`, account `bab`, app `shop`. |
| - Boxes roll before the console (the roll order rule); the migration pass is behind `CARLOS_LIFETIMES_MIGRATE=1`, default off, until every box of a deployment runs this code. |
| @@ -29,11 +29,11 @@ Rev 4, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan1-review.md`, 16 find |
| ## Review Focus |
| -1. A console crash between the slot write and the record write, a second console start, then the late write; and the mirror case where the original slot write lands after recovery read the predecessor: the replay and the late writes are harmless and the allocation is never abandoned (Task 3, `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`, `TestLateSlotWriteDoesNotAbandonTheAllocation`); a DELETE arriving while the record is absent and the allocation pending is refused as settling through the real door (`TestDeleteDuringCreateRecoveryIsRefused`). |
| -2. A box-side offline transition issued against the pre-marker ETag during migration: its CAS fails and its retry succeeds after the field is set (Task 5, `TestAnIssuedWriteLosesToTheMarkerAndRetries`); a consumer whose reads straddle the marker, in either order and with cached slot evidence, ends with the validated record and ETag, never a removal (Task 6, `TestJudgeAcrossAMigrationBothOrders`). |
| -3. A legacy fieldless record beside a retired slot (an old console's late create): the strict listings, the console doors, reconcile, the box's reassert and staging rereads and the offline keep view all treat it as absent, the edges file stops publishing its row after the next reconcile, and the next create replaces it (Task 6 `TestEveryClaimsOffConsumerIgnoresAFieldlessRecordBesideASlot`, Task 3 `TestCreateReplacesStaleRecord`). |
| +1. A console crash between the slot write and the record write, a second console start, then the late write; the mirror case where the original slot write lands after recovery read the predecessor; and a bound migration whose marker PUT lands after recovery observed the unmarked record: no intent is abandoned and every late write is harmless (Task 3 `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`, `TestLateSlotWriteDoesNotAbandonTheAllocation`; Task 5 `TestLateMarkerWriteIsCompletedNotStranded`). |
| +2. A judgement whose reads straddle a retirement, a recreation or a migration, in every order: the result is the current lifetime's answer, with the record and ETag the caller may act on, and never a removal of a valid successor (Task 6, `TestJudgeAcrossConcurrentLifetimeChanges`). |
| +3. A legacy fieldless record beside a retired slot (an old console's late create): the strict listings, every console door, reconcile (a populated offline keep view notwithstanding), the box's reassert, staging, runs, steering, retention and seed readers all treat it as absent, the edges file stops publishing its row after the next reconcile, and the next create replaces it (Task 6 `TestEveryClaimsOffConsumerIgnoresAFieldlessRecordBesideASlot`, Task 3 `TestCreateReplacesStaleRecord`). |
| 4. A reconstructed registry row after a finished move and after an abort from landed: reconcile installs `(1, 3)` on the minted row; an existing row that stayed through the finish keeps `(1, 2)` (Task 7, `TestReconcileInstallsTheRecordsPairOnAMintedRow`, `TestAnExistingRowKeepsOpenSequenceThroughFinish`). |
| -5. Existing rows across lifetimes: a legacy row held at incarnation 0 whose record migrates is released by the move's open or abort; a ready legacy row takes the lifetime and the hold in one write; a predecessor row held by a vanished move is retired by its op and the successor is minted ready at `(2, 0)` (Task 7, `TestExistingRowsAcrossLifetimes`). |
| +5. Existing rows across lifetimes: a legacy row held at incarnation 0 whose record migrates is released by the move's open or abort; a ready legacy row takes the lifetime and the hold in one write; an unheld predecessor row whose successor already has a move in flight takes the successor's lifetime, hold and pair in one write (never a ready row at the hold's pair); a predecessor row held by a vanished move is retired by its op and the successor minted (Task 7, `TestExistingRowsAcrossLifetimes`). |
| (The alias repointed during a move and the holding entry's fold are plan 3's Review Focus.) |
| @@ -48,15 +48,15 @@ Rev 4, 2026-10-22. Reviews: rev 1 (`after236/selfserve-plan1-review.md`, 16 find |
| **Interfaces:** |
| - Produces: |
| - - `Record.Incarnation uint64 \`json:"incarnation,omitempty"\``, `Record.IncarnationMigrating string \`json:"incarnation_migrating,omitempty"\``, `Record.MoveSeq uint64 \`json:"move_seq,omitempty"\``, `Record.Retired bool \`json:"retired,omitempty"\``. None of the four joins `protectedFieldsEqual` (`maintenance.go:81`): ordinary updates do not touch them, the migration's writes do, and the fence (Task 5) stops everyone else while a marker stands. |
| - - `type HostIncarnation struct { Incarnation uint64 \`json:"incarnation"\`; State string \`json:"state"\`; Account string \`json:"account"\`; App string \`json:"app"\`; Intent string \`json:"intent,omitempty"\`; RecordETag string \`json:"record_etag,omitempty"\` }`; `const SlotLive = "live"`, `SlotRetired = "retired"`. `Account`/`App` are the record key the lifetime belongs to; `RecordETag` is set only by a migration slot. `func (h HostIncarnation) Valid() bool` (incarnation > 0, state live or retired, account and app non-empty). |
| + - `Record.Incarnation uint64 \`json:"incarnation,omitempty"\``, `Record.IncarnationMigrating string \`json:"incarnation_migrating,omitempty"\``, `Record.MoveSeq uint64 \`json:"move_seq,omitempty"\``, `Record.Retired bool \`json:"retired,omitempty"\``. None of the four joins `protectedFieldsEqual` (`maintenance.go:81`). |
| + - `type HostIncarnation struct { Incarnation uint64 \`json:"incarnation"\`; State string \`json:"state"\`; Account string \`json:"account"\`; App string \`json:"app"\`; Intent string \`json:"intent,omitempty"\`; RecordETag string \`json:"record_etag,omitempty"\` }`; `const SlotLive = "live"`, `SlotRetired = "retired"`; `func (h HostIncarnation) Valid() bool` (incarnation > 0, state live or retired, account and app non-empty). `Account`/`App` are the record key the lifetime belongs to; `RecordETag` is set only by a migration slot. |
| - `func HostIncarnationKey(host string) string` → `control/host-incarnations/<host>.json`; "" for a host failing `ValidHostShape`. |
| - - `var ErrSlotMalformed`; `func ReadHostIncarnation(ctx context.Context, s release.Store, host string) (*HostIncarnation, string, error)`: nil, "", nil when absent; `ErrSlotMalformed` (wrapping the decode error) for a body that does not decode or fails `Valid`. |
| + - `var ErrSlotMalformed`; `func ReadHostIncarnation(ctx context.Context, s release.Store, host string) (*HostIncarnation, string, error)`: nil, "", nil when absent; `ErrSlotMalformed` (wrapping the cause) for a body that does not decode or fails `Valid`. |
| - `func WriteHostIncarnation(ctx context.Context, s release.Store, host string, slot HostIncarnation, ifMatch string) (string, error)`: refuses an invalid slot; `ifMatch == ""` is create-only (`Store.Put` with "" is `If-None-Match: *` on S3 and refuses an existing file on `DirStore`), otherwise `If-Match`. |
| - `type LifetimeVerdict int`; `const (LifetimeValid LifetimeVerdict = iota; LifetimeStale; LifetimeCorrupt; LifetimeUnreadable)`. |
| - - `func LifetimeOf(rec *Record, recordETag string, slot *HostIncarnation) LifetimeVerdict` (pure; never `LifetimeCorrupt` or `LifetimeUnreadable`, which only a failed decode or a failed read produce, Tasks 3 and 6). |
| - - `func TombstoneBody(before []byte) ([]byte, error)`: decodes `before` as JSON fields, keeps exactly `schema`, `host`, `account`, `app`, `incarnation` and `record_seq` (the last two only when present in `before`), adds `retired: true`, drops everything else. The caller writes it with `WriteRecord(ctx, s, key, before, body, etag)`, so `record_seq` advances by one when `before` carries it. |
| - - `func GetRecordRaw(ctx context.Context, s release.Store, acct, app, host string) (*Record, []byte, string, error)`: the decoded record (tombstones included), its exact bytes and ETag; a body that does not decode or disagrees with its path returns `ErrRecordCorrupt` (new, wrapping the cause) **with the raw bytes and ETag** so a caller may act on the observation (the delete rule, Task 4). |
| + - `func LifetimeOf(rec *Record, recordETag string, slot *HostIncarnation) LifetimeVerdict` (pure; only `valid` or `stale`). |
| + - `func TombstoneBody(before []byte) ([]byte, error)`: decodes `before` as JSON fields, keeps exactly `schema`, `host`, `account`, `app`, `incarnation` and `record_seq` (the last two only when present in `before`), adds `retired: true`, drops everything else. Written with `WriteRecord(ctx, s, key, before, body, etag)`, so `record_seq` advances by one when `before` carries it. |
| + - `var ErrRecordCorrupt`; `func GetRecordRaw(ctx context.Context, s release.Store, acct, app, host string) (rec *Record, raw []byte, etag string, err error)`: the decoded record (tombstones included), its exact bytes and ETag; a body that does not decode or disagrees with its path returns `ErrRecordCorrupt` (wrapping the cause) **with `raw` and `etag` set** so a caller may act on the observation. |
| - `GetRecordETag` (and so `GetRecord`) return `os.ErrNotExist` for a tombstone; `GetRecordRaw` is the only reader that sees one. |
| The verdict, exactly, with `owned := slot != nil && slot.Account == rec.Account && slot.App == rec.App`: |
| @@ -85,12 +85,12 @@ The verdict, exactly, with `owned := slot != nil && slot.Account == rec.Account |
| - `record_migrations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, expect_etag TEXT, marked_etag TEXT, state TEXT, created_at INTEGER)` |
| - Classification in `claimSQLTables`: `{Sqid: "account", App: "app"}` for all three. |
| - States: `pending`, `committed`, `superseded`, `void`, `refused`. |
| -- On `*Store`: `BeginAllocation(a Allocation) error`, `SetAllocationState(opID, state string) error`, `PendingAllocations() ([]Allocation, error)`, `LastIncarnation(host string) (uint64, error)` (the highest `incarnation` over every state but `void`; 0 when none); `BeginRetirement(r Retirement) error`, `SetRetirementState(opID, state string) error`, `PendingRetirements() ([]Retirement, error)`; `BeginMigration(m Migration) error`, `SetMigrationMarkedETag(opID, etag string) error`, `SetMigrationState(opID, state string) error`, `PendingMigrations() ([]Migration, error)`; `PendingForHost(host, exceptOpID string) (int, error)` (pending rows across the three tables, not counting `exceptOpID`). |
| +- On `*Store`: `BeginAllocation(a Allocation) error`, `SetAllocationState(opID, state string) error`, `PendingAllocations() ([]Allocation, error)`, `LastIncarnation(host string) (uint64, error)` (the highest `incarnation` over every state but `void`; 0 when none); `BeginRetirement(r Retirement) error`, `SetRetirementState(opID, state string) error`, `PendingRetirements() ([]Retirement, error)`; `BeginMigration(m Migration) error`, `SetMigrationMarkedETag(opID, etag string) error`, `SetMigrationState(opID, state string) error`, `PendingMigrations() ([]Migration, error)`; `PendingForHost(host, exceptOpID string) (int, error)`. |
| - `type Allocation struct { OpID, Account, App, Host string; Incarnation uint64; State string; Body []byte; PredecessorETag, StaleETag string }`, `type Retirement struct { OpID, Account, App, Host string; Incarnation uint64; RecordETag, SlotETag, SlotIntent, State string }`, `type Migration struct { OpID, Account, App, Host, ExpectETag, MarkedETag, State string }`. |
| -- [ ] **Step 1: Write the failing tests**: `TestLifetimeTablesRoundTrip` (begin, list pending, set state and the marked ETag, list again, for all three); `TestLastIncarnationSkipsVoid` (pending 2 and void 3 → 2); `TestPendingForHostCountsAllThreeExceptOwn`; in the store-gate tests `TestClaimStoreOpensWithLifetimeTables`. |
| +- [ ] **Step 1: Write the failing tests**: `TestLifetimeTablesRoundTrip`; `TestLastIncarnationSkipsVoid` (pending 2 and void 3 → 2); `TestPendingForHostCountsAllThreeExceptOwn`; `TestClaimStoreOpensWithLifetimeTables`. |
| - [ ] **Step 2: Run to verify they fail.** |
| -- [ ] **Step 3: Implement** with the store's `CREATE TABLE IF NOT EXISTS` pattern and the classification entries. |
| +- [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run** `go test ./internal/console -run 'LifetimeTables|LastIncarnation|PendingForHost|ClaimStoreOpensWithLifetimeTables' -count=1`. |
| - [ ] **Step 5: Commit** `console: tables for pending allocations, retirements and migrations, classified for the gate`. |
| @@ -98,48 +98,46 @@ The verdict, exactly, with `owned := slot != nil && slot.Account == rec.Account |
| **Files:** |
| - Create: `internal/console/lifetimes.go` (the exclusion, admission and shared classification), `internal/console/lifetimes_create.go`, `internal/console/lifetimes_recovery.go` |
| -- Modify: `internal/console/server.go` (`NewServer`: `s.slots = provision.NewSlotCache()` when `!cfg.HostClaim`), `internal/console/apiinstances.go` (the claims-off branch at line 1664: `recordWriteErr = provision.CreateRecord(...)` becomes the lifetime create; the sibling undo at line 1684 stays on `retireInstanceRecord`, which Task 4 routes), `internal/console/apiinstanceseed.go` (line 341), `internal/console/apiinstanceimport.go` (line 172), `internal/console/domainattach.go` (the first-route create near line 237 and the two `rollbackAttach` calls that follow it), `internal/console/claim_workers.go` (register `"StartLifetimeRecoveryPass"`), `cmd/console/main.go` (start it beside the other passes) |
| +- Modify: `internal/console/server.go` (`NewServer`: `s.judge = provision.NewLifetimeJudge()` when `!cfg.HostClaim`, nil otherwise), `internal/console/apiinstances.go` (the claims-off branch at line 1664 becomes the lifetime create; the sibling undo at line 1684 stays on `retireInstanceRecord`), `internal/console/apiinstanceseed.go` (line 341), `internal/console/apiinstanceimport.go` (line 172), `internal/console/domainattach.go` (the first-route create near line 237, the `rollbackAttach` calls that follow it, and `rollbackAttach` itself), `internal/console/claim_workers.go` (register `"StartLifetimeRecoveryPass"`), `cmd/console/main.go` |
| - Test: `internal/console/lifetimes_create_test.go` |
| **Interfaces:** |
| -- Consumes: Tasks 1 and 2. |
| +- Consumes: Tasks 1 and 2; Task 6's `provision.LifetimeJudge` (its `Observe`) for `classifyLocked`. |
| - Produces: |
| - `func (s *Server) lifetimesOn() bool` (`!s.cfg.HostClaim && s.cfg.Releases != nil`). |
| - `func (s *Server) withHostname(host string, fn func() error) error` (a `sync.Map` of `*sync.Mutex`; not reentrant; every exported helper below takes it, every `...Locked` helper assumes it). |
| - `func (s *Server) admit(host, ownOpID string) error`: `errLifetimeSettling` when `PendingForHost(host, ownOpID) > 0`. |
| - - `func (s *Server) classifyLocked(ctx context.Context, acct, app, host string) (lifetimeClass, error)` where `type lifetimeClass struct { Rec *provision.Record; Raw []byte; RecordETag string; Slot *provision.HostIncarnation; SlotETag string; Verdict provision.LifetimeVerdict }`: `GetRecordRaw` and `ReadHostIncarnation`; a record read that fails other than absence or corruption, or a slot read that fails other than absence (`ErrSlotMalformed` included) → `Verdict: LifetimeUnreadable` and the error; a corrupt record → `Verdict: LifetimeCorrupt` with its raw bytes and ETag; otherwise `LifetimeOf`. After any lifetime write under the exclusion the caller runs `s.slots.Forget(host)` (Task 6). |
| - - `func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record) (*provision.Record, error)`: the one function every claims-off creation path calls; returns the record as written (incarnation set, `RecordSeq` set). Errors: `release.ErrCASConflict` (a valid record under this key; the doors keep their 409), `*hostHeldError` (`errors.As`; carries `*apexHold` for `hostHeldSentence`; a live slot owned by another record key is `&apexHold{Holder: slot.App, OtherAccount: slot.Account if ≠ rec.Account}`), `errLifetimeSettling` (409, the settling sentence), `errLifetimePending` (the allocation is persisted and the outcome of a later step is unknown: recovery completes it; the door answers its existing "could not create" sentence and **does not** undo anything), or a read error from step 2 (502, nothing persisted). |
| + - `func (s *Server) classifyLocked(ctx context.Context, acct, app, host string) (provision.Observation, error)`: `s.judge.Observe(ctx, s.cfg.Releases, acct, app, host)` (Task 6: record first, slot after, bounded rounds); the error is non-nil only with `Verdict == LifetimeUnreadable`. |
| + - `func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record) (*provision.Record, error)`: the one function every claims-off creation path calls; returns the record as written. Errors: `release.ErrCASConflict` (a valid record under this key), `*hostHeldError` (`errors.As`; carries `*apexHold` for `hostHeldSentence`: a live slot owned by another key gives `&apexHold{Holder: slot.App, OtherAccount: slot.Account if ≠ rec.Account}`), `errLifetimeSettling`, `errLifetimePending`, or a read error from step 2 (nothing persisted). |
| - `func (s *Server) recoverLifetimes(ctx context.Context) error`, `func (s *Server) StartLifetimeRecoveryPass(ctx context.Context)` (returns at once unless `lifetimesOn()`; once at start, then every 60 s). |
| The protocol, inside `withHostname(host)`: |
| 1. `admit(host, "")`. |
| -2. `classifyLocked`. Unreadable → the read error. Valid record under this key → `ErrCASConflict`; the slot is untouched. A live slot owned by another key → `*hostHeldError`. Then `s.hostHolder(ctx, acct, host, app)` with `acct` from `s.store.AccountBySqid(rec.Account)` (a lookup failure is a read error), under the lock (the pre-lock call at `apiinstances.go:1164` stays): a holder → `*hostHeldError`. A stale or corrupt record under this key is remembered: `StaleETag`. |
| +2. `classifyLocked`. Unreadable → the error. Valid record under this key → `ErrCASConflict`; the slot is untouched. A live slot owned by another key → `*hostHeldError`. Then `acct, err := s.store.AccountBySqid(rec.Account)`: an error, or `acct == nil`, is a read error (502, nothing persisted); then `s.hostHolder(ctx, acct, host, app)` under the lock (the pre-lock call at `apiinstances.go:1164` stays): a holder → `*hostHeldError`. A stale or corrupt present record under this key is remembered: `StaleETag` (its ETag, never empty for a present body). |
| 3. `N = max(LastIncarnation(host), slot.Incarnation) + 1`; `rec.Incarnation = N`; `body = json.Marshal(rec)`; `BeginAllocation({OpID, Account, App, Host, N, pending, body, PredecessorETag: slot's ETag or "", StaleETag})`. A SQLite failure here → the error (nothing durable). |
| -4. `WriteHostIncarnation({N, live, rec.Account, rec.App, "create " + OpID}, PredecessorETag)`. On `ErrCASConflict`: re-read the slot; `{N, live, this key, "create " + OpID}` (this allocation's own write landed) → continue at 5; a slot at an incarnation > N, or at N owned by another key, or at N with another intent (positive successor evidence) → `superseded`, `errLifetimeSettling`; anything else (unreadable, or the predecessor again) → `pending`, `errLifetimePending`. Any other error → `pending`, `errLifetimePending`. |
| +4. `WriteHostIncarnation({N, live, rec.Account, rec.App, "create " + OpID}, PredecessorETag)`. On `ErrCASConflict`: re-read the slot (unreadable → `pending`, `errLifetimePending`); `{N, live, this key, "create " + OpID}` → continue at 5; a slot at an incarnation > N, or at N owned by another key, or at N and this key with another intent → `superseded`, `errLifetimeSettling`; the predecessor again → `pending`, `errLifetimePending`. Any other error → `pending`, `errLifetimePending`. |
| 5. If `StaleETag != ""`: `DeleteIfMatch` the record on it; any error (conflict included) → `pending`, `errLifetimePending`. |
| -6. `WriteRecord(ctx, s, key, []byte("{}"), body, "")`. Success → 7. `ErrCASConflict` → re-read raw: a record at N under this key → 7 (a replay landed); otherwise `pending`, `errLifetimePending`. Any other error → `pending`, `errLifetimePending`. |
| +6. `WriteRecord(ctx, s, key, []byte("{}"), body, "")`. Success → 7. `ErrCASConflict` → re-read raw (unreadable → pending): a record at N under this key → 7 (a replay landed); otherwise `pending`, `errLifetimePending`. Any other error → `pending`, `errLifetimePending`. |
| 7. `SetAllocationState(committed)`; a SQLite failure → `errLifetimePending` (the record stands; recovery marks it committed). Return the written record. |
| -Recovery (`recoverLifetimes`), per pending allocation, inside `withHostname` with `admit(host, OpID)` (its own operation does not refuse itself), after `classifyLocked`, the first matching branch: |
| +Recovery (`recoverLifetimes`), per pending allocation, inside `withHostname` with `admit(host, OpID)`, after `classifyLocked`, the first matching branch: |
| - unreadable → leave pending; |
| -- slot at an incarnation > N, or at N owned by another key, or at N and this key with an intent that is neither `"create " + OpID` nor the predecessor's (positive successor evidence) → `superseded`; |
| +- slot at an incarnation > N, or at N owned by another key, or at N and this key with an intent other than `"create " + OpID` → `superseded`; |
| - slot `{N, live, this key, "create " + OpID}` and a record at N under this key → `committed`; |
| -- slot as above and a stale or corrupt record under the key → step 5 with the current ETag, then 6, 7; |
| +- slot as above and a stale or corrupt present record → step 5 with the current ETag, then 6, 7; |
| - slot as above and no record → step 6, 7; |
| -- slot absent (and `PredecessorETag == ""`), or the slot's ETag equals `PredecessorETag` → step 4 onward; |
| -- anything else → leave pending and log once per hour (an operator reads the row). |
| +- slot absent and `PredecessorETag == ""`, or the slot's ETag equals `PredecessorETag` → step 4 onward; |
| +- anything else → leave pending and log once per hour. |
| Then the retirements (Task 4) and the migrations (Task 5), in that order. |
| -Door mappings: the create door (`apiinstances.go`): `*hostHeldError` → 409 `hostHeldSentence(host, hold)`; `errLifetimeSettling` → 409 settling; `errLifetimePending` → 502 "could not create the instance — try again" (the existing sentence), no undo. Seed (`apiinstanceseed.go:341`) and import (`apiinstanceimport.go:172`): the same three mappings onto their existing failure sentences; on `errLifetimePending` the door writes nothing further, and the record recovery completes carries the seed state the body had, which the existing seed sweep then owns (a begun seed with no upload times out as today). Attach (`domainattach.go`): `ErrCASConflict` keeps today's adoption branch; `errLifetimePending` → `rollbackAttach(ctx, acct, app, host, false, false)`: the claim is **kept** (it is this app's; the record recovery completes is this app's first-route record under its own claim, the case the door already adopts on retry), and the answer is the existing "could not declare the route for this domain — nothing was attached, try again"; `errLifetimeSettling` and `*hostHeldError` → 409 with their sentences and `rollbackAttach(..., false, created)` as today's refusal branch does; a step-2 read error → the existing ambiguous branch (nothing was persisted, so `rollbackAttach(..., true, created)` is today's behaviour and its retirement finds nothing). |
| +Door mappings: the create door: `*hostHeldError` → 409 `hostHeldSentence(host, hold)`; `errLifetimeSettling` → 409 settling; `errLifetimePending` → 502 "could not create the instance — try again", no undo. Seed and import: the same three onto their existing failure sentences; on `errLifetimePending` the door writes nothing further and the record recovery completes carries the seed state the body had, which the existing seed sweep then owns. Attach: `ErrCASConflict` keeps today's adoption branch; `errLifetimePending`, `errLifetimeSettling` and `*hostHeldError` → the answer for each (the ambiguous sentence; 409 settling; 409 held) with `rollbackAttach(ctx, acct, app, host, false, keepClaim)` where the claim is dropped only when this call minted it **and** nothing was persisted (settling and held); a pending allocation keeps the claim. `rollbackAttach` itself changes: when `dropRecord` and the retirement returns anything but nil (pending, settling, refused, an error), the claim is **kept** and the failure logged; only a nil retirement result proceeds to `detachDomainClaim`. |
| - [ ] **Step 1: Write the failing tests** against an in-process console with `release.NewDirStore` and `HostClaim` off: |
| - - `TestCreateAllocatesSlotThenRecord` (slot `{1, live, bab, shop, "create <id>"}`, record `Incarnation 1`, the returned record has it), `TestCreateAfterRetirementGetsNextIncarnation` (`{1, retired}` → `{2, live}`), `TestCreateReplacesStaleRecord` (a fieldless record beside `{1, retired}` is deleted on its ETag; incarnation 2 written), `TestCreateOfALiveInstanceIsTheExistingConflict` (slot unchanged, no allocation row), `TestCreateRefusesWhileAnOperationIsPending`, `TestCreateRefusesALiveSlotOfAnotherApp` (slot `{1, live, bab, other}` → `hostHeldSentence` names `other`, slot unchanged), `TestTwoAppsCreatingOneHostnameSerialise` (two goroutines, apps `shop` and `other`, one host: exactly one slot write, one record, the loser gets the held or the existing conflict, never a second lifetime). |
| - - `TestEveryPostAllocationFailureIsPending` (a failing store wrapper injects a non-CAS error at step 4, at step 5, at step 6, and a failing SQLite at step 7: each leaves the row `pending` and returns `errLifetimePending`; recovery then completes each). |
| - - `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`: a pending allocation with the slot written and no record; recovery writes the record and commits; an injected late create-only write of the same body fails `ErrCASConflict` and the record's ETag is unchanged. |
| - - `TestLateSlotWriteDoesNotAbandonTheAllocation`: recovery reads the predecessor; the original slot write lands `{N, live, create A}` between the read and recovery's CAS; recovery's write loses, re-reads, recognises its own slot, writes the record, commits. The mirror: the original slot write lands after recovery's succeeded → the original fails CAS and nothing else happens. |
| - - `TestDeleteDuringCreateRecoveryIsRefused` (through the real DELETE door while the record is absent and the allocation pending → 409 settling, nothing written; Task 4's absent path). |
| - - `TestRecoveryWritesTheSlotWhenMissing`, `TestRecoveryContinuesFromAPredecessorSlot`, `TestRecoveryLeavesALandedCreateServing`, `TestRecoveryRemovesAStaleRecordThenWrites`, `TestRecoverySupersedesOnlyOnSuccessorEvidence` (slot past N → superseded; slot at N of another key → superseded; the predecessor still → replay, never superseded), `TestRecoveryLeavesPendingWhenUnreadable` (a failing slot read; a malformed slot body), `TestRecoverySurvivesASecondCrash` (a crash injected after step 4 on the first recovery run), `TestRecoveryUsesOnlyPersistedState` (the SQLite store is closed and reopened between the crash and the recovery). |
| - - `TestAttachKeepsItsClaimOverAPendingAllocation` (failure injected after the slot write, and separately a SQLite failure at step 7, in the attach's first-route create: the claim stands, the answer is the ambiguous sentence, recovery completes the record, a second attach adopts it). |
| + - `TestCreateAllocatesSlotThenRecord`, `TestCreateAfterRetirementGetsNextIncarnation`, `TestCreateReplacesStaleRecord`, `TestCreateOfALiveInstanceIsTheExistingConflict`, `TestCreateRefusesWhileAnOperationIsPending`, `TestCreateRefusesALiveSlotOfAnotherApp`, `TestCreateRefusesWhenTheAccountIsUnknown` (`AccountBySqid` nil → 502, nothing persisted), `TestTwoAppsCreatingOneHostnameSerialise`. |
| + - `TestEveryPostAllocationFailureIsPending` (non-CAS errors injected at steps 4, 5, 6 and a SQLite failure at 7: each `pending` + `errLifetimePending`; recovery completes each). |
| + - `TestRecoveryReplaysCreateAndTheLateWriteIsHarmless`; `TestLateSlotWriteDoesNotAbandonTheAllocation` (both orders); `TestDeleteDuringCreateRecoveryIsRefused` (through the real DELETE door while the record is absent → 409 settling). |
| + - `TestRecoveryWritesTheSlotWhenMissing`, `TestRecoveryContinuesFromAPredecessorSlot`, `TestRecoveryLeavesALandedCreateServing`, `TestRecoveryRemovesAStaleRecordThenWrites`, `TestRecoverySupersedesOnlyOnSuccessorEvidence`, `TestRecoveryLeavesPendingWhenUnreadable` (a failing slot read; a malformed slot body), `TestRecoverySurvivesASecondCrash`, `TestRecoveryUsesOnlyPersistedState`. |
| + - `TestAttachKeepsItsClaimOverAPendingAllocation` (failure after the slot write; a SQLite failure at step 7); `TestAttachRollbackKeepsTheClaimWhenRetirementIsNotProven` (the ambiguous branch's retirement returns pending, settling, refused and an injected error: the claim stands in each; a nil retirement drops it as today). |
| - `TestSeedCreatePendingIsOwnedByTheSweep`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement**; route the four claims-off creation paths through `createRecordWithLifetime`. |
| @@ -150,108 +148,113 @@ Door mappings: the create door (`apiinstances.go`): `*hostHeldError` → 409 `ho |
| **Files:** |
| - Create: `internal/console/lifetimes_retire.go` |
| -- Modify: `internal/console/hostclaimops.go` (`retireHostRecord`, claims-off branch: `ref.Family == "instance"` → `retireRecordWithLifetime(ctx, ref.Account, ref.App, ref.Host, "")`; sidecars and everything else unchanged on `provision.RetireRecord`), `internal/console/apiinstanceimport.go` (`readDeletedRecord` returns `(obs lifetimeClass, err error)` through `s.classifyObservation` (Task 6: judged, with the raw ETag and the corrupt case kept); `importDeleteRule`: the claims-off `os.ErrNotExist` branch becomes `retireRecordWithLifetime(ctx, acct.Sqid, app, host, retireExpectAbsent)` and answers finished on nil, 409 on settling; a `LifetimeStale` or `LifetimeCorrupt` observation → `deleteRead` with its ETag (the import logic needs a decoded, valid record and is skipped for these); `deleteRead`: `release.DeleteIfMatch(...)` becomes `retireRecordWithLifetime(ctx, acct.Sqid, app, host, etag)`; `ErrCASConflict` keeps the `again` re-decision; `errLifetimeSettling` → 409 settling; `errRetireRefused` → 409 its sentence), `internal/console/seedfinish.go` (`finishSeedOnce`'s final `DeleteIfMatch` becomes `retireRecordWithLifetime(ctx, sqid, app, host, etag)` with the ETag `deleteSeedReplica` returned; the seed resource teardown before it is unchanged; `ErrCASConflict` keeps `finishSeed`'s one retry), `internal/console/apiinstances.go` (`handleAPIInstanceDelete`: `errRetireRefused` and `errLifetimeSettling` → 409 on the `retireInstanceRecord` branch), `internal/console/domainattach.go` and `internal/console/apidomains.go` (the detach callers: `errLifetimeSettling` → 409 `claimDetachInProgressMsg`; `errRetireRefused` → 409 its sentence; `rollbackAttach` logs either and keeps the claim), `internal/console/lifetimes_recovery.go` (the retirement branches) |
| +- Modify: `internal/console/hostclaimops.go` (claims-off `ref.Family == "instance"` → `retireRecordWithLifetime(ctx, ref.Account, ref.App, ref.Host, retireAny)`), `internal/console/apiinstanceimport.go` (`readDeletedRecord` returns `(provision.Observation, error)` through `s.judge.Observe`; `importDeleteRule`: `!obs.Present` → `retireRecordWithLifetime(..., retireExpectAbsent)`; `obs.Verdict` stale or corrupt → `deleteRead(..., retireExpectETag(obs.ETag))`; valid → the import logic as today, and its ordinary delete `deleteRead(..., retireExpectETag(obs.ETag))`; `deleteRead` calls `retireRecordWithLifetime` with the expectation it is given: `ErrCASConflict` keeps the `again` re-decision; `errLifetimeSettling` → 409 settling; `errRetireRefused` → 409 its sentence), `internal/console/seedfinish.go` (`finishSeedOnce`'s final delete → `retireRecordWithLifetime(ctx, sqid, app, host, retireExpectETag(etag))`), `internal/console/apiinstances.go` (`handleAPIInstanceDelete`: `errRetireRefused`, `errLifetimeSettling` → 409), `internal/console/domainattach.go` and `internal/console/apidomains.go` (detach: `errLifetimeSettling` → 409 `claimDetachInProgressMsg`; `errRetireRefused` → 409 its sentence), `internal/console/lifetimes_recovery.go` |
| - Test: `internal/console/lifetimes_retire_test.go` |
| **Interfaces:** |
| -- Consumes: Tasks 1 to 3; Task 5's `migrateHostLifetimeLocked` (build Task 5's function first or stub it behind the same signature). |
| -- Produces: `const retireExpectAbsent = "\x00absent"`; `func (s *Server) retireRecordWithLifetime(ctx context.Context, acct, app, host, expectETag string) error`; `var errRetireRefused`. A nil return means the key is absent (or was left untouched on purpose, below) and the slot, if any, is retired or someone else's: the callers' "deleted" and "already gone" answers. |
| - |
| -Inside `withHostname(host)`: `admit(host, "")`; `classifyLocked` (unreadable → the error, 502 at the doors). Preconditions: `expectETag == retireExpectAbsent` and a record is present → nil, nothing touched (the import rule's protection: a record created since the rule observed absence is not deleted); `expectETag` a real ETag and the record absent or at another ETag → `release.ErrCASConflict` (the caller re-decides, as `deleteRead` does today). Then the first matching case: |
| -- (a) record absent, slot absent → nil. |
| -- (b) record absent, slot `{N, live}` owned by this key → an **orphan slot**: `BeginRetirement({..., N, RecordETag: "", SlotETag: S, SlotIntent: slot.Intent})`, then step (2) below, `committed`. (A crashed create is a pending allocation and was refused at `admit`; what remains is an old console's plain delete, and retiring the slot makes its late record stale.) |
| -- (c) record absent, slot owned by another key, or retired → nil. |
| -- (d) record present, verdict stale (a tombstone, a fieldless or behind record) → `DeleteIfMatch` on its ETag (a conflict → `errLifetimeSettling`); the slot is untouched; nil. |
| -- (g) record present, verdict corrupt (today's "deleted the ordinary way" case) → `DeleteIfMatch` on its ETag (conflict → `errLifetimeSettling`); the slot is untouched; nil. |
| -- (e) record present, fieldless, slot absent (legacy) → `rec, raw, E, state, err := s.migrateHostLifetimeLocked(ctx, acct, app, host, RecordETag)` (Task 5; the marker CAS is bound to the caller's observed ETag): `ErrCASConflict` → `ErrCASConflict` to the caller (an ordinary change won; the caller re-decides); `state != committed` → `errLifetimeSettling`; committed → continue at (f) with the returned `rec, raw, E` and a fresh slot read (its ETag S). |
| -- (f) record valid at N under slot `{N, live, this key}` at ETag S → `BeginRetirement({OpID, Account, App, Host, N, E, S, SlotIntent: slot.Intent, pending})`, then: |
| - - (1) `WriteRecord(ctx, s, key, raw, TombstoneBody(raw), E)`. `ErrCASConflict` → re-classify: a non-tombstone record at N → `refused` (`errRetireRefused`); a tombstone at N → continue at (2); anything else → `void`, nil. Any other error → `pending`, `errLifetimePending`. |
| - - (2) `WriteHostIncarnation({N, retired, Account, App, "retire " + OpID}, S)`. `ErrCASConflict` → re-read the slot: `{N, retired}` → continue; a slot at > N or of another key → `void` (the tombstone is left for (d) of a later call or for recovery); anything else → `pending`, `errLifetimePending`. Any other error → `pending`, `errLifetimePending`. |
| +- Consumes: Tasks 1 to 3; Task 5's `migrateHostLifetimeLocked`; Task 6's `Observation`. |
| +- Produces: `type retireExpectation struct { absent bool; etag string }`; `var retireExpectAbsent = retireExpectation{absent: true}`, `var retireAny = retireExpectation{}` (no precondition: the `retireInstanceRecord` callers, which decide from a judged read they just made), `func retireExpectETag(etag string) retireExpectation` (panics on ""); `func (s *Server) retireRecordWithLifetime(ctx context.Context, acct, app, host string, expect retireExpectation) error`; `var errRetireRefused`. A nil return means the key is absent (or was left untouched on purpose) and the slot, if any, is retired or someone else's. |
| + |
| +Inside `withHostname(host)`: `admit(host, "")`; `obs := classifyLocked` (unreadable → the error). Preconditions: `expect.absent && obs.Present` → nil, nothing touched (a record created since the caller observed absence is not deleted); `expect.etag != "" && (!obs.Present || obs.ETag != expect.etag)` → `release.ErrCASConflict`. Then the first matching case: |
| +- (a) absent, slot absent → nil. |
| +- (b) absent, slot `{N, live}` owned by this key → **orphan slot**: `BeginRetirement({..., N, RecordETag: "", SlotETag: S, SlotIntent})`, step (2), `committed`. |
| +- (c) absent, slot owned by another key, or retired → nil. |
| +- (d)/(g) present and stale or corrupt: if the slot is `{N, live}` owned by this key → **cleanup under a live owned slot**: `BeginRetirement({..., N, RecordETag: obs.ETag, SlotETag: S, SlotIntent})`, then (2) retire the slot against S, then `DeleteIfMatch` the body on `obs.ETag` (a conflict is logged and left: the body is non-servable and stale under the retired slot), `committed`; otherwise (no slot, a retired slot, or another key's) → `DeleteIfMatch` on `obs.ETag` (a conflict → `errLifetimeSettling`), nil. |
| +- (e) present, fieldless, slot absent (legacy) → `rec, raw, E, state, err := s.migrateHostLifetimeLocked(ctx, acct, app, host, obs.ETag)`: `ErrCASConflict` → `ErrCASConflict`; `state != committed` → `errLifetimeSettling`; committed → continue at (f) with the returned `rec, raw, E` and a fresh slot read (S). |
| +- (f) present and valid at N under slot `{N, live, this key}` at S → `BeginRetirement({OpID, Account, App, Host, N, E, S, SlotIntent, pending})`, then: |
| + - (1) `WriteRecord(ctx, s, key, raw, TombstoneBody(raw), E)`. `ErrCASConflict` → re-classify (unreadable → `pending`, `errLifetimePending`): a non-tombstone record at N → `refused` (`errRetireRefused`); a tombstone at N → (2); anything else → `void`, nil. Any other error → `pending`, `errLifetimePending`. |
| + - (2) `WriteHostIncarnation({N, retired, Account, App, "retire " + OpID}, S)`. `ErrCASConflict` → re-read the slot (unreadable → pending): `{N, retired, this key}` → continue; `> N` or another key → `void`; otherwise `pending`, `errLifetimePending`. Any other error → `pending`, `errLifetimePending`. |
| - (3) `DeleteIfMatch` the tombstone on its ETag (absence is success; any other error → `pending`, `errLifetimePending`). |
| - `committed` (a SQLite failure → `errLifetimePending`). |
| -Recovery, per pending retirement, inside `withHostname` with `admit(host, OpID)`, after `classifyLocked`, the first matching branch, where `succ := slot != nil && (slot.Incarnation > N || !owned)` and `pred := slot != nil && slot.Incarnation == N && slot.State == live && owned && slot.Intent == SlotIntent` and `retired := slot != nil && slot.Incarnation == N && slot.State == retired && owned`: |
| -- unreadable → pending; |
| +Recovery, per pending retirement, inside `withHostname` with `admit(host, OpID)`, after `classifyLocked` (unreadable → pending), with `owned := slot owned by this key`, `succ := slot != nil && (slot.Incarnation > N || !owned)`, `pred := slot != nil && owned && slot.Incarnation == N && slot.State == live && slot.Intent == SlotIntent`, `retired := slot != nil && owned && slot.Incarnation == N && slot.State == retired`, the first matching branch: |
| - `succ` → `void`; |
| -- `retired` and record absent → `committed`; |
| -- `retired` and a tombstone at N → (3), `committed`; |
| -- `retired` and a non-tombstone record at N (a late write landed after the tombstone) → `committed`, and the record is deleted conditionally on its ETag (stale under a retired slot; a conflict leaves it for the next create); |
| -- `pred`, `RecordETag != ""`, a non-tombstone record present at `RecordETag` → (1), (2), (3); |
| +- `retired` and absent → `committed`; |
| +- `retired` and a present body (a tombstone at N, a late record at N, a stale fieldless body, a corrupt body) → `DeleteIfMatch` on its ETag (a conflict leaves it for the next create), `committed`; |
| +- `pred` and a present record whose `Incarnation != N` and is not a tombstone (a different lifetime's body under this slot; §6.4's different-incarnation observation) → `void`; |
| +- `pred`, `RecordETag != ""`, a present non-tombstone record at `RecordETag` → (1), (2), (3); |
| - `pred`, a tombstone at N → (2), (3); |
| -- `pred`, `RecordETag == ""`, record absent (an orphan slot retirement) → (2), `committed`; |
| -- `pred`, `RecordETag != ""`, a non-tombstone record at N with another ETag → `refused`; |
| -- `pred`, record absent, `RecordETag != ""` (someone else removed it) → (2), `committed`; |
| -- anything else → pending, logged once per hour. |
| - |
| -- [ ] **Step 1: Write the failing tests** through the real doors (`DELETE /api/cli/.../instances/{host}` and the seed sweep) and the function: |
| - - `TestDeleteDoorRetiresThroughTheTombstone` (the import rule's ordinary path: after a hook at (1) the key holds a tombstone, `GetRecord` says not found, `record_seq` advanced by one; at the end the slot is `{1, retired}` and the key absent; the door answers "deleted"); `TestSlotRetiresBeforeTheKeyIsAbsent` (a hook between (2) and (3): the slot is already retired while the tombstone stands, and a create-only write of a record at incarnation 1 is stale under it). |
| - - `TestDeleteDoorAbsentPathAdmitsAndRetiresOrphans` (record absent, slot `{1, live}`, no pending op → the door answers "deleted" and the slot is `{1, retired}`; a pending allocation → 409 settling; a record created between the rule's read and the retirement → untouched, "deleted" as today). |
| - - `TestDeleteDoorRemovesStaleAndCorruptRecordsConditionally` (a stale fieldless record beside a retired slot → removed, slot untouched; a body that does not decode → removed on its ETag as today). |
| +- `pred`, `RecordETag == ""` (an orphan retirement), absent or any present body → (2), then the body if present is deleted conditionally, `committed`; |
| +- `pred`, `RecordETag != ""`, a present non-tombstone record at N with another ETag → `refused`; |
| +- `pred`, `RecordETag != ""`, absent → (2), `committed`; |
| +- a slot that is neither `succ`, `pred` nor `retired` (an owned live slot at N with another intent, or absent) → leave pending and log once per hour. |
| + |
| +- [ ] **Step 1: Write the failing tests** through the real doors and the function: |
| + - `TestDeleteDoorRetiresThroughTheTombstone`; `TestSlotRetiresBeforeTheKeyIsAbsent`. |
| + - `TestDeleteDoorAbsentPathAdmitsAndRetiresOrphans` (initial absence; and absence found on the judge's re-read: both admit and retire the orphan slot; a record created since is untouched and the door still answers "deleted"). |
| + - `TestStaleCleanupRetiresALiveOwnedSlotFirst` (a fieldless late body beside `{1, live, this key}`: the slot is `{1, retired}` before the body is removed; a blocked incarnation-1 create-only write released afterwards lands stale; the same for a corrupt body). |
| + - `TestDeleteDoorRemovesStaleAndCorruptRecordsConditionally` (under a retired or foreign slot: body removed, slot untouched). |
| - `TestSeedFinishRetiresThroughTheTombstone`. |
| - - `TestRetireRefusesWhenTheRecordChangedInItsLifetime` (a `SetRecordRegion` between the read and (1) → 409 with the sentence; record kept; state refused); `TestRetireVoidsAgainstASuccessor`; `TestRetireNeverMovesTheSlotBackwards` (every case of a record at 1 beside slots `{2, live}` and `{2, retired}`: the slot's ETag is unchanged afterwards). |
| - - `TestOrphanSlotIsRetired`. |
| - - `TestLegacyRetirementIsBoundToTheCallersRead` (an ordinary update lands between `deleteRead`'s read and the marker CAS → the door answers the `again` path's re-decision, the updated record survives; with no interference the migration completes and the tombstone is written against the migrated record's ETag; no plain delete). |
| - - `TestDelayedDeleteCannotEraseASuccessor`: block the conditional tombstone write (1) of a retirement with a hook; let recovery run after a restart (it finds the record at `RecordETag` and completes (1)–(3)); create the host again (`{2, live}`, record at 2); release the blocked write: it fails `ErrCASConflict`; the successor's record and slot are intact. |
| - - `TestRetirementRecoveryBranches` (one subtest per branch above with realistic predecessor intents `"create <x>"` and `"migration <x>"`, including the restored-record branch and its conditional cleanup; a second crash injected during recovery in two of them). |
| + - `TestRetireRefusesWhenTheRecordChangedInItsLifetime`; `TestRetireVoidsAgainstASuccessor`; `TestRetireNeverMovesTheSlotBackwards`. |
| + - `TestOrphanSlotIsRetired`; `TestOrphanRetirementSurvivesARestoredRecord` (crash after Begin; a late record at N lands; recovery retires the slot and removes the body). |
| + - `TestLegacyRetirementIsBoundToTheCallersRead` (the helper: an ordinary update between the caller's read and the marker CAS → `ErrCASConflict`, the updated record intact, the migration row superseded; the door: the `again` re-decision reads the updated record and decides it afresh, pinned as today's behaviour). |
| + - `TestDelayedDeleteCannotEraseASuccessor`. |
| + - `TestRetirementRecoveryBranches` (one subtest per branch above, with realistic predecessor intents; a second crash during recovery in two of them). |
| - `TestDetachMapsLifetimeErrors`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/console -run 'Retire|Tombstone|Lifetime|InstanceDelete|SeedFinish|DeleteDoor|Detach' -count=1` and `go test ./internal/provision -run 'Retire' -count=1`. |
| -- [ ] **Step 5: Commit** `console: every claims-off record removal retires through a tombstone, bound to the caller's read and classifying the record and its slot together`. |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'Retire|Tombstone|Lifetime|InstanceDelete|SeedFinish|DeleteDoor|Detach|StaleCleanup|Orphan' -count=1` and `go test ./internal/provision -run 'Retire' -count=1`. |
| +- [ ] **Step 5: Commit** `console: every claims-off record removal retires through a tombstone, with an explicit expectation and the owned slot retired before any absence shows`. |
| ### Task 5: Lazy migration with a marker fenced in the write primitive |
| **Files:** |
| -- Modify: `internal/provision/record_sequence.go` (`WriteRecord`: after `RecordSequence(before)`, if `before`'s `incarnation_migrating` is non-empty and `ctx` does not carry `WithMigration(ctx, thatID)`, return `ErrRecordMigrating`; a create-only write, `before == {}`, is unaffected), `internal/provision/provision.go` (`updateRecordSeeding`: when `update()` returns an error that `Is` `ErrRecordMigrating`, retry every second up to `recordMigrationWait` (30 s; a test hook shortens it), re-reading each time, then return it; `claimRawUpdateRecord`: returns `ErrRecordMigrating` as it comes, which `Is` `release.ErrCASConflict`, so every hostops writer re-reads as it does for a lost race), `internal/console/claim_workers.go` (register `"StartIncarnationMigrationPass"`), `cmd/console/main.go` |
| -- Create: `internal/provision/lifetime_migration.go` (`ErrRecordMigrating`, `WithMigration`), `internal/console/lifetimes_migrate.go` |
| +- Modify: `internal/provision/record_sequence.go` (`WriteRecord`: after `RecordSequence(before)`, if `before`'s `incarnation_migrating` is non-empty and `ctx` does not carry `WithMigration(ctx, thatID)`, return `ErrRecordMigrating`; a create-only write is unaffected), `internal/provision/provision.go` (`updateRecordSeeding`: an error that `Is` `ErrRecordMigrating` is retried every second up to `recordMigrationWait` (30 s; a test hook shortens it), re-reading each time; `claimRawUpdateRecord` returns it as it comes, which `Is` `release.ErrCASConflict`), `internal/console/claim_workers.go` (register `"StartIncarnationMigrationPass"`), `cmd/console/main.go` |
| +- Create: `internal/provision/lifetime_migration.go`, `internal/console/lifetimes_migrate.go` |
| - Test: `internal/provision/record_sequence_test.go`, `internal/console/lifetimes_migrate_test.go` |
| **Interfaces:** |
| -- Produces: `var ErrRecordMigrating = fmt.Errorf("%w: the record is being migrated; read it again", release.ErrCASConflict)`; `func WithMigration(ctx context.Context, id string) context.Context`; `func (s *Server) migrateHostLifetimeLocked(ctx context.Context, acct, app, host, expectETag string) (rec *provision.Record, raw []byte, etag string, state string, err error)` (assumes `withHostname`; `expectETag == ""` means "from the current record, retrying from an updated one"; a real ETag binds the marker CAS to it and a conflict returns `release.ErrCASConflict` with no retry; on `committed` the returned record, bytes and ETag are the migrated record as written by step (3)); `func (s *Server) StartIncarnationMigrationPass(ctx context.Context)` (behind `CARLOS_LIFETIMES_MIGRATE=1` and `lifetimesOn()`, every 5 minutes, at most 50 records per pass, oldest `created_at` first, each inside `withHostname` and `admit`, only fieldless unmarked records whose slot is absent, with `expectETag == ""`). |
| +- Produces: `var ErrRecordMigrating = fmt.Errorf("%w: the record is being migrated; read it again", release.ErrCASConflict)`; `func WithMigration(ctx context.Context, id string) context.Context`; `func (s *Server) migrateHostLifetimeLocked(ctx context.Context, acct, app, host, expectETag string) (rec *provision.Record, raw []byte, etag string, state string, err error)` (`expectETag == ""` is unbound: retry from the current record; a real ETag binds the marker CAS to it); `func (s *Server) StartIncarnationMigrationPass(ctx context.Context)` (behind `CARLOS_LIFETIMES_MIGRATE=1` and `lifetimesOn()`, every 5 minutes, at most 50 records per pass, oldest first, each inside `withHostname` and `admit`, only fieldless unmarked records whose slot is absent, unbound). |
| -The protocol (inside the exclusion): `admit(host, "")`; `BeginMigration({OpID, Account, App, Host, ExpectETag, "", pending})`; (1) `classifyLocked`; the record must be fieldless, unmarked, slot absent (else `superseded`, and for a bound call `ErrCASConflict` when the record's ETag differs from `ExpectETag`); `WriteRecord(WithMigration(ctx, OpID), s, key, raw, raw with IncarnationMigrating = OpID, E)`; on `ErrCASConflict`: bound → `superseded`, `ErrCASConflict`; unbound → re-read and retry from the current record, at most 5 times, then leave pending; `SetMigrationMarkedETag(OpID, M)` with the returned ETag (a SQLite failure → pending, `errLifetimePending`); (2) `WriteHostIncarnation({1, live, Account, App, "migration " + OpID, RecordETag: M}, "")` create-only, or `If-Match` when a slot `{1, live, this key, "migration " + OpID}` already exists (an earlier attempt's); a CAS conflict → re-read: this migration's slot → continue; otherwise `superseded`; (3) `WriteRecord(WithMigration(ctx, OpID), ..., Incarnation = 1, IncarnationMigrating = "", against M)` (a conflict → re-read: a record at `Incarnation 1` under this slot → continue; otherwise pending); `committed`; `s.slots.Forget(host)`. Every other error after `BeginMigration` → pending, `errLifetimePending`. |
| +The protocol (inside the exclusion): `admit(host, "")`; `BeginMigration({OpID, Account, App, Host, ExpectETag, "", pending})`; (1) `classifyLocked`: the record must be present, fieldless, unmarked, slot absent, and for a bound call at `ExpectETag` (else `superseded`, and for a bound call `ErrCASConflict`); `WriteRecord(WithMigration(ctx, OpID), s, key, raw, raw with IncarnationMigrating = OpID, E)`; on `ErrCASConflict`: bound → re-read; a record marked with OpID (this write landed) → continue; otherwise `superseded`, `ErrCASConflict`; unbound → re-read and retry from the current record, at most 5 times, then leave pending; `SetMigrationMarkedETag(OpID, M)` (a SQLite failure → pending, `errLifetimePending`); (2) `WriteHostIncarnation({1, live, Account, App, "migration " + OpID, RecordETag: M}, "")` create-only, or `If-Match` when a slot `{1, live, this key, "migration " + OpID}` exists; a CAS conflict → re-read: this migration's slot → continue; otherwise `superseded`; (3) `WriteRecord(WithMigration(ctx, OpID), ..., Incarnation = 1, IncarnationMigrating = "", against M)` (a conflict → re-read: a record at `Incarnation 1` under this slot → continue; otherwise pending); `committed`. Every other error after `BeginMigration` → pending, `errLifetimePending`; every re-read that fails → pending. |
| -Recovery, per pending migration, inside the exclusion with `admit(host, OpID)`: unreadable → pending; record carries `Incarnation 1` and slot `{1, live, this key, "migration " + OpID}` → `committed`; record marked with OpID, slot bound to the record's current ETag → (3); record marked with OpID and `MarkedETag == ""` → `SetMigrationMarkedETag(OpID, current ETag)` then (2); record marked with OpID, slot absent → (2); record fieldless and unmarked, slot absent → (1) (unbound: a retirement's bound migration that did not reach the marker is simply retried by the caller, so its row is `superseded` here); record gone, another lifetime, a marker of another op, or a slot not of this migration → `superseded`. |
| +Recovery, per pending migration, inside the exclusion with `admit(host, OpID)`, after `classifyLocked` (unreadable → pending): |
| +- record at `Incarnation 1` and slot `{1, live, this key, "migration " + OpID}` → `committed`; |
| +- record marked with OpID, slot bound to the record's current ETag → (3); |
| +- record marked with OpID and `MarkedETag == ""` → `SetMigrationMarkedETag(OpID, current ETag)` then (2); |
| +- record marked with OpID, slot absent → (2); |
| +- record present, fieldless, unmarked, slot absent: **bound** (`ExpectETag != ""`) and the record's ETag equals `ExpectETag` → resume (1) against `ExpectETag` (the outstanding original write either lands first, in which case our CAS fails and the re-read shows our marker, or ours lands and the original fails: both harmless); bound and the ETag differs → `superseded` (the original conditional write can no longer land); unbound → (1); |
| +- record gone, another lifetime, a marker of another op, or a slot not of this migration → `superseded`. |
| - [ ] **Step 1: Write the failing tests**: |
| - - `record_sequence_test.go`: `TestWriteRecordRefusesAMarkedRecordWithoutTheMigrationContext` (the error `Is` both `ErrRecordMigrating` and `release.ErrCASConflict`), `TestWriteRecordAllowsTheMigrationsOwnWrite`, `TestCreateOnlyWriteIgnoresTheMarker`, `TestClaimRawUpdateRecordSurfacesTheMarker`. |
| - - `TestMigrationMarksThenSlotsThenSets` (final: slot `{1, live, bab, shop, "migration <id>", RecordETag M}`, record `Incarnation 1`, no marker, committed; the returned record and ETag equal a fresh read). |
| - - `TestMigrationRestartsFromAnUpdateThatWon` (unbound: an update between the read and (1); the marker CAS fails; the pass re-reads and completes; the update's field survives) and `TestBoundMigrationReturnsConflictToItsCaller` (bound: the same interference → `ErrCASConflict`, row `superseded`, record untouched). |
| - - `TestAnIssuedWriteLosesToTheMarkerAndRetries` (a box-shaped `SetRecordMove` goroutine blocked on a hook with the pre-marker ETag: its `WriteRecord` fails `ErrCASConflict`, its re-read gets `ErrRecordMigrating`, it waits, and succeeds after (3)). |
| - - `TestWriterGivesUpAfterTheBound`. |
| - - `TestMigrationRecoveryBranches` (one subtest per branch, a restart after the marker write and before `MarkedETag` is persisted, and a second crash during recovery). |
| + - `record_sequence_test.go`: `TestWriteRecordRefusesAMarkedRecordWithoutTheMigrationContext`, `TestWriteRecordAllowsTheMigrationsOwnWrite`, `TestCreateOnlyWriteIgnoresTheMarker`, `TestClaimRawUpdateRecordSurfacesTheMarker`. |
| + - `TestMigrationMarksThenSlotsThenSets`; `TestMigrationRestartsFromAnUpdateThatWon` (unbound); `TestBoundMigrationReturnsConflictToItsCaller`. |
| + - `TestAnIssuedWriteLosesToTheMarkerAndRetries`; `TestWriterGivesUpAfterTheBound`. |
| + - `TestLateMarkerWriteIsCompletedNotStranded` (bound migration M against E; the marker PUT is paused; crash; recovery observes the unmarked record at E and resumes (1); release the paused PUT: whichever lands second fails CAS; the migration reaches committed; no marker is left without an owner); the same with the paused PUT landing before recovery's read. |
| + - `TestMigrationRecoveryBranches` (one subtest per branch, a restart after the marker write and before `MarkedETag` is persisted, a second crash during recovery). |
| - `TestMigrationPassIsOffWithoutTheFlag`, `TestMigrationSkipsAFieldlessRecordBesideAnUnrelatedSlot`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run** `go test ./internal/provision -run 'WriteRecord|Migrat|Record|ClaimRawUpdate' -count=1` and `go test ./internal/console -run 'Migrat' -count=1`. |
| -- [ ] **Step 5: Commit** `lifetimes: lazy migration marks the record by CAS, fenced in the write primitive, bound to a caller's read when it asks`. |
| +- [ ] **Step 5: Commit** `lifetimes: lazy migration marks the record by CAS, fenced in the write primitive, and never strands a marker it issued`. |
| -### Task 6: Consumers judge the lifetime |
| +### Task 6: Consumers observe the lifetime |
| **Files:** |
| -- Create: `internal/provision/lifetime_cache.go`, `internal/console/lifetimes_read.go` |
| -- Modify: `internal/provision/reconcile.go` (the decode loop at lines 855–941: `getEach` and `fetched` gain `etag`; after the path check and before `ValidHostShape`, the judgement, which may replace `rec`/`raw` with the observation it returns; `cfg.Slots.Begin()` at the top of the pass; `ListAllRecordsStrict` and `ListAllRecordsTolerant` gain judged siblings and `readAllRecords` takes the cache), `internal/provision/provision.go` (`GetRecordETag`'s judged sibling; `ListRecordsStrict`, `ListRecordsComplete`, `ListRecords` gain judged siblings), `internal/host/agent.go` (one `boxSlots *provision.SlotCache`, built once beside `provisionTickConfig` when `!hostwrite.Enabled(getenv)` and nil otherwise, handed to `ReconcileConfig.Slots`, `moveCfg.Slots` and the offline view), `internal/host/movespass.go` (line 1079: the judged strict listing), `internal/host/movesoffline.go` (`refreshOfflineView`: the judged listing; `offlineReassert` line 420: `ReadRecordJudged`), `internal/host/movesoffline_stage.go` (line 75 and the deletion gate at line 162: `ReadRecordJudged`), `internal/provision/reconcile.go` (`holdExisting` line 1716: `ReadRecordJudged` with `cfg.Slots`, and the pair for Task 7 from that fresh record), `internal/console/*.go` (every instance-record read in the inventory below goes through `lifetimes_read.go`) |
| -- Test: `internal/provision/reconcile_lifetime_test.go`, `internal/provision/lifetime_cache_test.go`, `internal/console/lifetimes_read_test.go`, `internal/host/lifetime_gating_test.go` |
| +- Create: `internal/provision/lifetime_judge.go`, `internal/console/lifetimes_read.go` |
| +- Modify: `internal/provision/reconcile.go` (the decode loop at lines 855–941: `getEach` and `fetched` gain `etag`; after the path check and before `ValidHostShape`, `cfg.Judge.ObserveWith(...)`, replacing `rec`/`raw` with the observation's; stale hosts are recorded in `stale map[string]bool`; the removal loop: a row whose host is in `stale` skips the `KeepParticipant` check (a positively stale record overrides an obsolete keep decision); `ListAllRecordsStrict` and `ListAllRecordsTolerant` gain judged siblings and `readAllRecords` takes the judge; `holdExisting` line 1716: `cfg.Judge.Observe`), `internal/provision/provision.go` (judged siblings of `GetRecordETag`, `ListRecordsStrict`, `ListRecordsComplete`, `ListRecords`), `internal/host/agent.go` (one `boxJudge *provision.LifetimeJudge`, built once beside `provisionTickConfig` when `!hostwrite.Enabled(getenv)` and nil otherwise, handed to `ReconcileConfig.Judge`, `moveCfg.Judge`, the offline view, the runs worker, the steering loop, retention and the seed wiring), `internal/host/movespass.go:1079`, `internal/host/movesoffline.go` (`refreshOfflineView`; `offlineReassert` line 420), `internal/host/movesoffline_stage.go` (lines 75 and 162), `internal/host/runs.go` (`bootPass` line 122: the judged tolerant listing; the pickup re-read at line 278), `internal/host/runexec.go` (lines 318 and 421), `internal/host/steering_loop.go` (line 100), `internal/host/retention.go` (line 308), `internal/host/seedwiring.go` (line 60), `internal/console/*.go` (the inventory below; `placeSeedBlock` and `importInFlight` gain a `*provision.LifetimeJudge` parameter from their callers) |
| +- Test: `internal/provision/reconcile_lifetime_test.go`, `internal/provision/lifetime_judge_test.go`, `internal/console/lifetimes_read_test.go`, `internal/host/lifetime_gating_test.go` |
| **Interfaces:** |
| - Produces: |
| - - `type Observation struct { Rec *Record; Raw []byte; ETag string; Slot *HostIncarnation; SlotETag string; Verdict LifetimeVerdict }`. |
| - - `type SlotCache struct{...}`; `func NewSlotCache() *SlotCache`; `func (c *SlotCache) Begin()` (drops every entry: a reconcile pass calls it first, so one pass reuses slot reads and no pass reuses another's); `func (c *SlotCache) Forget(host string)` (the console calls it after every lifetime write under the exclusion); `func (c *SlotCache) Judge(ctx context.Context, s release.Store, rec *Record, raw []byte, etag string) (Observation, error)`: round 1 judges `(rec, etag)` against the cached slot (an entry younger than 5 s; a miss reads the slot, caching its value or its absence); `valid` → return. Otherwise round 2: a fresh slot read bypassing the cache and a fresh `GetRecordRaw`; judge the fresh pair; `valid` → return the fresh observation (the caller uses **it**, not its first read); `stale` → return `stale` with the fresh observation; a corrupt fresh record → `corrupt`; absent → `stale` with `Rec == nil`; a read error in either round → `unreadable` with the error. The cache is refreshed with round 2's slot. A nil `*SlotCache` judges everything valid and returns the inputs unchanged. |
| - - `func ReadRecordJudged(ctx context.Context, s release.Store, c *SlotCache, acct, app, host string) (*Record, string, error)`: `GetRecordRaw` then `Judge`; stale or a tombstone → `os.ErrNotExist`; corrupt → `ErrRecordCorrupt`; unreadable → the error; valid → the observation's record and ETag. |
| - - `func ListRecordsStrictJudged(ctx, s, c, acct, app string, max int) (recs []Record, unreadable map[string]bool, err error)`, `ListRecordsCompleteJudged(ctx, s, c, acct, app, max) (recs []Record, truncated bool, skipped []string, err error)`, `ListRecordsJudged(ctx, s, c, acct, app, max) (recs []Record, truncated bool, err error)`, `ListAllRecordsStrictJudged(ctx, s, c) (recs []Record, unreadable map[string]bool, err error)`, `ListAllRecordsTolerantJudged(ctx, s, c) (recs []Record, complete bool, err error)`: each is its sibling with stale records dropped, records replaced by their validated observation, and an unreadable slot reported as that host unreadable (strict: the listing error, as a failed record read is today; complete: `skipped` names the host; tolerant: `complete = false`). |
| - - `ReconcileConfig.Slots *SlotCache`. In the decode loop: verdict stale → `continue` (not in `recs`, not protected: the removal phase retires the row as "record gone"); corrupt → `protect[host] = true` as today's malformed body; unreadable → `protect[host] = true` and `readFailed = true`, as a failed read is today; valid → the observation's record is the one judged by the shape checks that follow. |
| - - Console: `s.slots` (`NewServer`); `func (s *Server) classifyObservation(ctx, acct, app, host) (lifetimeClass, error)` (the unlocked read for doors: `GetRecordRaw`, `Judge`), `func (s *Server) readRecord(ctx, acct, app, host) (*Record, string, error)` and `listRecords`, `listRecordsStrict`, `listRecordsComplete`, `listAllRecordsStrict`, `listAllRecordsTolerant` wrappers passing `s.slots` (nil when `s.cfg.HostClaim`). Inventory to route (every instance-record read in these files): `apiinstancemove.go`, `apiinstances.go`, `seedfinish.go`, `channelmove.go`, `apiruns.go`, `apiinstancemove_offline.go`, `apiinstanceimport.go` (`readImportRecord`; `readDeletedRecord` through `classifyObservation`, Task 4), `updatescore.go`, `movesrecovery.go`, `domainattach.go`, `appsettings.go`, `apirunsfollow.go`, `apiretry.go`, `apidb.go`, `apex.go` (`hostHolder`), `apidomains.go`, `apiinstanceseed.go`, `apilegacyroutes.go`, `relayops.go`, `seedsweep.go`, `server.go`, `regiondrift.go`, `runsweep.go` (`ListAllRecordsTolerant`), `seedmove.go`. The walk test `TestNoConsoleReadBypassesTheJudgedReaders` fails on any of: `provision.GetRecord(`, `provision.GetRecordETag(`, `provision.GetRecordRaw(`, `provision.ListRecords`, `provision.ListAllRecords`, `provision.DecodeRecord(`, or `.Get(ctx, provision.RecordKey(` in `internal/console` non-test files outside `lifetimes*.go` and the claims-on files it lists by name. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestSlotCacheJudgesAndCaches` (two judgements, one slot read; absence cached; `Begin` and `Forget` drop entries); `TestJudgeAcrossAMigrationBothOrders` (record read before the marker, slot after → round 2 returns the migrated record and its ETag, verdict valid; slot cached as absent, migration completes, the incarnation-1 record is read → round 2 refreshes the slot → valid; cached `{N, live}`, retirement completes, a delayed record at N is read → round 2 → stale); `TestJudgeInconsistentTwiceIsUnreadable` (a slot that changes between the two rounds and a record that keeps disagreeing → unreadable, not stale); `TestNilSlotCacheJudgesEverythingValid`; `TestReconcileRemovesARecordBehindItsSlot`; `TestReconcileRemovesATombstone` (the tombstone has no channel: it leaves before the channel check, so the row is retired, not protected); `TestReconcileKeepsALegacyRecordWithNoSlot`; `TestReconcileKeepsAMarkedRecordUnderItsMigrationSlot`; `TestReconcileProtectsWhenTheSlotIsUnreadable` (a failing read; a malformed body); `TestReconcileUsesTheValidatedObservation` (the fresh record's region, not the first read's, drives the pass); `TestEveryClaimsOffConsumerIgnoresAFieldlessRecordBesideASlot` (slot `{1, retired}`, fieldless record carrying an offline move: `ReadRecordJudged` not found; the five judged listings omit it; the console GET answers 404; reconcile mints nothing and retires its row; the edges file stops listing the host on the next write; `offlineReassert` and the staging deletion gate refuse to act for it; the offline keep view after `refreshOfflineView` does not keep it); `TestClaimsOnConsumersReadNoSlot` (`HostClaim` on the console, `CARLOS_HOST_CLAIM` on the box: a counting store sees no `control/host-incarnations/` read); `TestBoxSlotCacheIsSharedAndGated` (`CARLOS_HOST_CLAIM` set → every consumer's cache is nil; unset → one cache instance in reconcile, the moves worker and the offline view); `TestNoConsoleReadBypassesTheJudgedReaders`. |
| + - `type Observation struct { Present bool; Rec *Record; Raw []byte; ETag string; Slot *HostIncarnation; SlotETag string; Verdict LifetimeVerdict; Err error }` (`Rec` is nil when absent or corrupt; `Raw`/`ETag` are set for a corrupt present body). |
| + - `type LifetimeJudge struct{}` (no state; nil means claims-on: every observation is `valid` with the inputs unchanged); `func NewLifetimeJudge() *LifetimeJudge`. |
| + - `func (j *LifetimeJudge) Observe(ctx context.Context, s release.Store, acct, app, host string) Observation` and `func (j *LifetimeJudge) ObserveWith(ctx context.Context, s release.Store, acct, app, host string, raw []byte, etag string, readErr error) Observation` (round 1 uses the given record read). The protocol, up to 3 rounds: read the record (`GetRecordRaw`; absence, corruption and errors noted), **then** read the slot; a read error in either → `unreadable` with the error; judge: a present, decodable record with `LifetimeOf == valid` → return `valid`; absent → return `{Present: false, Verdict: stale}` when a slot exists and `{Present: false, Verdict: valid}` when none (an absent record is simply absent; the verdict tells a caller whether a slot stands); corrupt → return `corrupt` with `Raw`, `ETag`; stale → if this is round 1, or the slot's ETag differs from the previous round's, loop; stale with the slot ETag unchanged since the previous round → return `stale` (the authority did not move between two observations, so the record is behind it). Exhausted (3 rounds, the slot still changing) → `unreadable` with an error naming the host. The slot is always read after the record, so a slot that validates a record is at least as new as it. |
| + - `func ReadRecordJudged(ctx, s, j *LifetimeJudge, acct, app, host string) (*Record, string, error)`: stale, absent or a tombstone → `os.ErrNotExist`; corrupt → `ErrRecordCorrupt`; unreadable → the error. |
| + - Listings (typed): `func ListRecordsStrictJudged(ctx context.Context, s release.Store, j *LifetimeJudge, acct, app string, max int) ([]Record, map[string]bool, error)`; `func ListRecordsCompleteJudged(ctx context.Context, s release.Store, j *LifetimeJudge, acct, app string, max int) (recs []Record, truncated bool, skipped []string, err error)`; `func ListRecordsJudged(ctx context.Context, s release.Store, j *LifetimeJudge, acct, app string, max int) (recs []Record, truncated bool, err error)`; `func ListAllRecordsStrictJudged(ctx context.Context, s release.Store, j *LifetimeJudge) (recs []Record, unreadable map[string]bool, err error)`; `func ListAllRecordsTolerantJudged(ctx context.Context, s release.Store, j *LifetimeJudge) (recs []Record, complete bool, err error)`: stale dropped, records replaced by their observation, an unreadable observation reported as that host unreadable (strict: the listing error; complete: `skipped`; tolerant: `complete = false`). |
| + - `ReconcileConfig.Judge *LifetimeJudge`; `moveCfg.Judge *provision.LifetimeJudge`; `runsWorker.judge`, the steering loop's and retention's parameters likewise. |
| + - Console: `s.judge`; `func (s *Server) readRecord(ctx context.Context, acct, app, host string) (*provision.Record, string, error)`; `func (s *Server) observeRecord(ctx context.Context, acct, app, host string) provision.Observation` (the delete rule); `listRecords`, `listRecordsStrict`, `listRecordsComplete`, `listAllRecordsStrict`, `listAllRecordsTolerant` wrappers. Inventory to route: every instance-record read in `apiinstancemove.go`, `apiinstances.go`, `seedfinish.go`, `channelmove.go`, `apiruns.go`, `apiinstancemove_offline.go`, `apiinstanceimport.go`, `updatescore.go`, `movesrecovery.go`, `domainattach.go`, `appsettings.go`, `apirunsfollow.go`, `apiretry.go`, `apidb.go`, `apex.go`, `apidomains.go`, `apiinstanceseed.go`, `apilegacyroutes.go`, `relayops.go`, `seedsweep.go`, `server.go`, `regiondrift.go`, `runsweep.go`, `seedmove.go`. `TestNoConsoleReadBypassesTheJudgedReaders` walks `internal/console` (non-test) and fails on `provision.GetRecord(`, `provision.GetRecordETag(`, `provision.GetRecordRaw(`, `provision.ListRecords`, `provision.ListAllRecords`, `provision.DecodeRecord(` or `.Get(ctx, provision.RecordKey(` outside `lifetimes*.go` and an explicit list of claims-on files (`claim_*.go`, `hostclaimops.go`, `movesrecovery.go`'s claims-on branches named by function). `TestNoBoxReadBypassesTheJudge` does the same for `internal/host` outside `lifetime*.go` and the claims-on files (`claim*.go`, `root*.go`, `hostlife*.go`). |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestJudgeReadsRecordThenSlot` (a counting store: one record read and one slot read for a valid record; order pinned); `TestJudgeAcrossConcurrentLifetimeChanges` (hooks between the reads: record read, then retirement completes, then slot read → round 2 → absent/stale with no record; record obsolete, slot `{N, retired}`, recreation at N+1 completes, round 2 reads the new record then the new slot → valid with the successor's record and ETag; record read before a migration's marker, slot after → round 2 valid with the migrated record and ETag; a slot that changes on every round → unreadable after 3); `TestNilJudgeIsClaimsOn`; `TestReconcileRemovesARecordBehindItsSlot`; `TestReconcileRemovesATombstone`; `TestReconcileKeepsALegacyRecordWithNoSlot`; `TestReconcileKeepsAMarkedRecordUnderItsMigrationSlot`; `TestReconcileProtectsWhenTheSlotIsUnreadable`; `TestReconcileUsesTheObservation`; `TestStaleOverridesAPopulatedKeepView` (the offline view holds a staging move for the host from an earlier listing; the record is now stale; the real `reconcileSerializedWith` path (no manual refresh) removes the row and the next edges write omits the host); `TestEveryClaimsOffConsumerIgnoresAFieldlessRecordBesideASlot` (slot `{1, retired}`, a fieldless record carrying an offline move and a queued run: `ReadRecordJudged` not found; the five judged listings omit it; the console GET answers 404; reconcile mints nothing and retires its row; the edges file stops listing it; `offlineReassert`, the staging deletion gate, the runs boot pass and pickup, `launchable`, the steering loop's desired state, retention's app records and the seed hook all ignore it); `TestClaimsOnConsumersReadNoSlot`; `TestBoxJudgeIsSharedAndGated`; `TestNoConsoleReadBypassesTheJudgedReaders`; `TestNoBoxReadBypassesTheJudge`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run** the touched packages' targeted tests, then the two census tests and add the rows they name. |
| -- [ ] **Step 5: Commit** `lifetimes: every claims-off read judges a record's lifetime and acts on the observation it validated`. |
| +- [ ] **Step 5: Commit** `lifetimes: every claims-off read observes the record then the slot, in bounded rounds, and acts on the observation`. |
| ### Task 7: `move_seq` on the record and the pair on rows |
| **Files:** |
| -- Modify: `internal/provision/offline_move.go` (the transition table below), `internal/provision/provision.go` (`MoveState.Seq uint64 \`json:"seq,omitempty"\``), `internal/provision/offline_move_test.go` (`TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo`: `move_seq` joins the allowlist and nothing else changes; `TestOfflineAbort` keeps pinning that a repeated abort is a success without a write), `internal/registry/registry.go` (columns `incarnation INTEGER NOT NULL DEFAULT 0`, `serving_seq INTEGER NOT NULL DEFAULT 0` in the `ALTER TABLE` list; `Route.Incarnation uint64`, `Route.ServingSeq uint64`; `routeCols`, `routeColCount` (+2), `scanRoute`, `routeArgs`, `normalizeRoute`; `Upsert`'s conflict clause gains `incarnation=routes.incarnation, serving_seq=routes.serving_seq` (an ordinary upsert never moves the pair; the claim-ownership `WHERE` predicate is unchanged); `Fingerprint`'s column list gains both; `remote_dispatch.go` gains the new methods; `SetHold`, `SetHoldIf`, `HandHold`, `ReleaseHold`, `DeleteIfHold` are untouched), `internal/provision/reconcile.go` (`RouteFor` copies `rec.Incarnation` and `rec.MoveSeq` into a minted row, so `mintHeld`'s mint carries them too; `ReconcileConfig.PairOf func(rec Record) registry.Pair` (nil → `holdExisting` and the release paths call the existing `SetHoldIf`/`ReleaseHold`, untouched; non-nil → the `Seq` setters); `holdExisting` passes `cfg.PairOf(fresh)` from the judged fresh record to `SetHoldIfSeq`; the existing-row rule after the converge of a provisioned row of this pass's app and account: **(i) migration, unheld**: `existing.Incarnation == 0 && rec.Incarnation > 0 && existing.HoldOp == ""` → `InstallLifetimeIf(host, Pair{0, existing.ServingSeq}, Pair{rec.Incarnation, existing.ServingSeq})` (the row's own sequence is kept); **(ii) recreation, unheld**: `0 < existing.Incarnation < rec.Incarnation && existing.HoldOp == ""` → `InstallLifetimeIf(host, Pair{existing.Incarnation, existing.ServingSeq}, Pair{rec.Incarnation, rec.MoveSeq})`; **(iii) recreation, held**: `existing.Incarnation < rec.Incarnation && existing.HoldOp != ""` and `rec.Move == nil || rec.Move.ID != existing.HoldOp` (the hold's op is not this record's move) → under `WithHostHoldLock`, `cfg.DeleteIfHold(host, existing.HoldOp)` (the agent's `DeleteIfHoldWithTimers`), then the ordinary mint of the record on this pass; **(iv) same incarnation**: nothing; a held row whose op is the record's own move is left to the move), `internal/host/movesoffline.go` (`withOfflineHolds` sets `cfg.PairOf = func(rec) Pair{rec.Incarnation, rec.Move.Seq}` (nil-safe); `moveCfg` gains `ReleaseHoldSeq func(host, op string, pair registry.Pair) (bool, error)`, backed by a new `releaseHeldSeq(reg, written)` beside `releaseHeld`; the **source abort/revert** path at `movesoffline.go:1077` calls it with the fresh judged record's `{Incarnation, Move.Seq}`), `internal/host/movesoffline_land.go` (the **target open** path at line 165 likewise), `internal/host/movespass.go` (`armOffline`, line 371: `cfg.ReleaseHoldSeq = releaseHeldSeq(reg, holdStageRequester(dataDir))` beside `cfg.ReleaseHold`), `internal/host/movesoffline_test.go:207` and `internal/host/movesoffline_land_test.go:206` (the fixture overrides gain `cfg.ReleaseHoldSeq = releaseHeldSeq(b.reg, nil)`) |
| +- Modify: `internal/provision/offline_move.go` (the transition table below), `internal/provision/provision.go` (`MoveState.Seq uint64 \`json:"seq,omitempty"\``), `internal/provision/offline_move_test.go` (`TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo`: `move_seq` joins the allowlist; `TestOfflineAbort` keeps pinning the repeated abort), `internal/registry/registry.go` (columns `incarnation INTEGER NOT NULL DEFAULT 0`, `serving_seq INTEGER NOT NULL DEFAULT 0`; `Route.Incarnation`, `Route.ServingSeq`; `routeCols`, `routeColCount` (+2), `scanRoute`, `routeArgs`, `normalizeRoute`; `Upsert`'s conflict clause gains `incarnation=routes.incarnation, serving_seq=routes.serving_seq`; `Fingerprint` gains both; `remote_dispatch.go` gains the new methods; `SetHold`, `SetHoldIf`, `HandHold`, `ReleaseHold`, `DeleteIfHold` untouched), `internal/provision/reconcile.go` (`RouteFor` copies `rec.Incarnation` and `rec.MoveSeq`; `ReconcileConfig.PairOf func(rec Record) registry.Pair` (nil → the untouched `SetHoldIf`/`ReleaseHold`) and `ReconcileConfig.DeleteIfHold func(host, op string) (bool, error)` (nil → rule (iv) logs and skips); the existing-row rules below), `internal/host/agent.go` (`provisionTickConfig`: `DeleteIfHold: reg.DeleteIfHoldWithTimers`), `internal/host/movesoffline.go` (`withOfflineHolds` sets `cfg.PairOf = func(rec) Pair{rec.Incarnation, rec.Move.Seq}` (nil-safe); `moveCfg.ReleaseHoldSeq func(host, op string, pair registry.Pair) (bool, error)` backed by `releaseHeldSeq(reg, written)`; the **source abort/revert** path at `movesoffline.go:1077`: under the existing locks, `cfg.Judge.Observe` the record; `Move != nil && Move.ID == m.ID && Move.State == OfflineReverting` (else skip this pass, logged); `ReleaseHoldSeq(host, m.ID, {rec.Incarnation, rec.Move.Seq})` from that observation), `internal/host/movesoffline_land.go` (the **target open** path at line 165 likewise, expecting `OfflineOpen`), `internal/host/movespass.go` (`armOffline`, line 371: `cfg.ReleaseHoldSeq = releaseHeldSeq(reg, holdStageRequester(dataDir))`), `internal/host/movesoffline_test.go:207` and `internal/host/movesoffline_land_test.go:206` (the overrides gain `cfg.ReleaseHoldSeq = releaseHeldSeq(b.reg, nil)`) |
| - Test: `internal/provision/offline_move_test.go`, `internal/registry/registry_test.go`, `internal/provision/reconcile_lifetime_test.go`, `internal/host/movesoffline_e2e_test.go` |
| **The transition table (authoritative):** |
| @@ -272,44 +275,53 @@ Recovery, per pending migration, inside the exclusion with `admit(host, OpID)`: |
| Each `+1` sets `rec.MoveSeq++` and `rec.Move.Seq = rec.MoveSeq` (nil-safe) in the same mutation. |
| -**Interfaces:** |
| -- Produces: `type registry.Pair struct { Incarnation, ServingSeq uint64 }`; `func (r *Registry) SetHoldIfSeq(host, status, op, expectOp string, pair Pair) (bool, error)` (`SetHoldIf`'s validation of status, op and `expectOp` verbatim, then `UPDATE routes SET status=?, hold_op=?, incarnation=?, serving_seq=? WHERE host=? AND hold_op=? AND (incarnation=? OR incarnation=0)` with `pair.Incarnation` twice: a legacy row at 0 takes the lifetime with the hold in one write; a row of another lifetime changes nothing); `func (r *Registry) ReleaseHoldSeq(host, op string, pair Pair) (bool, error)` (`ReleaseHold`'s empty-op no-op, then `SET status='ready', hold_op='', incarnation=?, serving_seq=? WHERE host=? AND hold_op=? AND (incarnation=? OR incarnation=0)`); `func (r *Registry) InstallLifetimeIf(host string, expect, next Pair) (bool, error)` (`SET incarnation=?, serving_seq=? WHERE host=? AND incarnation=? AND serving_seq=? AND hold_op=''`); `Route.Incarnation`, `Route.ServingSeq`; `MoveState.Seq`; `Record.MoveSeq`; `ReconcileConfig.PairOf`; `moveCfg.ReleaseHoldSeq`. |
| +**The setters:** |
| +- `type registry.Pair struct { Incarnation, ServingSeq uint64 }`. |
| +- `func (r *Registry) SetHoldIfSeq(host, status, op, expectOp string, pair Pair) (bool, error)`: `SetHoldIf`'s validation verbatim, then `UPDATE routes SET status=?, hold_op=?, incarnation=?, serving_seq=? WHERE host=? AND hold_op=? AND incarnation <= ?` with `pair.Incarnation`: a hold of a lifetime installs that lifetime on an unheld row of the same or an **older** lifetime (a legacy row at 0, or a predecessor row left by a recreation) together with the status, op and pair, in one write; a row of a newer lifetime changes nothing. |
| +- `func (r *Registry) ReleaseHoldSeq(host, op string, pair Pair) (bool, error)`: `ReleaseHold`'s empty-op no-op, then `SET status='ready', hold_op='', incarnation=?, serving_seq=? WHERE host=? AND hold_op=? AND (incarnation=? OR incarnation=0)`: a release names its lifetime (or a legacy row) and never moves a successor's row. |
| +- `func (r *Registry) InstallLifetimeIf(host string, expect, next Pair) (bool, error)`: `SET incarnation=?, serving_seq=? WHERE host=? AND incarnation=? AND serving_seq=? AND hold_op=''`. |
| + |
| +**The existing-row rules** (reconcile, a provisioned row of this pass's app and account, in this order, each a separate write under `WithHostHoldLock`): |
| +- (i) **the record asks for a hold** (`cfg.Hold(rec)` ok; today's `holdExisting`): `SetHoldIfSeq(host, status, op, "", cfg.PairOf(fresh))` from the fresh observation. This covers a legacy row (0), the same lifetime and an unheld predecessor row (`incarnation <= rec.Incarnation`): the successor's lifetime, status and pair land together, so no ready row ever carries a hold's pair. A row already held by another op is left (today). |
| +- (ii) **migration, unheld, no hold asked**: `existing.Incarnation == 0 && rec.Incarnation > 0 && existing.HoldOp == ""` → `InstallLifetimeIf(host, {0, existing.ServingSeq}, {rec.Incarnation, existing.ServingSeq})` (the row's own sequence is kept). |
| +- (iii) **recreation, unheld, no hold asked**: `0 < existing.Incarnation < rec.Incarnation && existing.HoldOp == ""` → `InstallLifetimeIf(host, {existing.Incarnation, existing.ServingSeq}, {rec.Incarnation, rec.MoveSeq})`. |
| +- (iv) **recreation, held by a vanished op**: `existing.Incarnation < rec.Incarnation && existing.HoldOp != "" && (rec.Move == nil || rec.Move.ID != existing.HoldOp)` → `cfg.DeleteIfHold(host, existing.HoldOp)`: `true` → mint the record on this pass (held, if the record asks for a hold; ready otherwise); `false` or an error → log, skip this pass; nil callback → log, skip; dry-run → log only. |
| +- (v) **same incarnation, unheld**: nothing; a held row whose op is the record's own move is left to the move. |
| - [ ] **Step 1: Write the failing tests**: |
| - - `TestOfflineTransitionsAdvanceMoveSeq`: one subtest per row of the table (direct path 1, 2, 3; staged path 0, 1, 2, 3; abort after start 2 then finish 3; abort from staging 0 then finish 0; a repeated abort on a reverting record returns the record unchanged with no write; a record at `MoveSeq 5` starts at 6; `Move.Seq` equals the record's value at each advancing transition). |
| - - `TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo` passes with `move_seq` in its allowlist; `TestOfflineAbort` unchanged. |
| - - `TestRegistryPairColumns` (upsert with the pair; an ordinary upsert of another pair keeps the first; `Fingerprint` changes when only the pair changes); `TestSetHoldIfSeqInstallsOnALegacyRowAndRefusesAnotherLifetime` (row at 0 → status, op and `(1, 1)` in one write; row at 2 with a setter at 1 → unchanged; `SetHoldIf`'s validation errors reproduced); `TestReleaseHoldSeqLikewise` (and the empty-op no-op); `TestInstallLifetimeIfIsGuarded`. |
| - - `TestRouteForCopiesThePair`; `TestReconcileInstallsTheRecordsPairOnAMintedRow` (after a finished move at `move_seq 3`, delete the row, reconcile → `(1, 3)`; after an abort from landed and finish, likewise `(1, 3)`); `TestAnExistingRowKeepsOpenSequenceThroughFinish` (the target row at `(1, 2)` after open stays `(1, 2)` after finish and after any number of passes). |
| - - `TestExistingRowsAcrossLifetimes` (subtests, each run to its end state: **ready legacy row, migrated record** `(0, 0)` + `{1, 0}` → `(1, 0)` once, and a later record at `{1, 5}` leaves it; **ready row, recreation** `(1, 9)` + `{2, 0}` → `(2, 0)`; **held legacy row, migration during a move**: a row held `(0, 1)` by op X, the record migrates to incarnation 1 with the move still X → the row is untouched by the installer, and the move's open (`ReleaseHoldSeq(host, X, {1, 2})`) releases it to ready `(1, 2)`; the abort path likewise; **hold applied across migration**: a ready row at `(0, 0)`, a migrated record `{1, 1}` asking for a hold → `holdExisting` writes status, op and `(1, 1)` together; **held predecessor row, recreation**: a row held `(1, 1)` by op X, a record at `{2, 0}` with no move → `DeleteIfHold(host, X)` then a ready mint at `(2, 0)` on the same pass; **a stale setter**: `ReleaseHoldSeq` for incarnation 1 against the successor's row at 2 → no change; **Phase 1 unchanged**: `PairOf == nil` with a row carrying `(1, 3)` from an earlier claims-off lifetime → `holdExisting` calls `SetHoldIf` and the release calls `ReleaseHold`, the pair untouched). |
| - - In `TestOfflineMoveE2E`: the records are created through the lifetime (slot `{1, live}`, `Incarnation 1`) with judging on; after `start` the source row is held with `(1, 1)`; after `open` the target row is ready with `(1, 2)`; a second trace aborting from landed leaves the source ready with `(1, 2)` and no target row; `armOffline`'s wiring (not a test override) is what releases. |
| + - `TestOfflineTransitionsAdvanceMoveSeq` (one subtest per table row); `TestOfflineRecordDiffersOnlyWhereTheMoveSaysSo` with `move_seq`; `TestOfflineAbort` unchanged. |
| + - `TestRegistryPairColumns` (and `Fingerprint`); `TestSetHoldIfSeqInstallsOnOlderUnheldRows` (rows at 0, 1 and 2 with a hold at 2: 0 and 1 take status, op and `(2, 1)` in one write, 2 takes it; a row at 3 unchanged; an already-held row unchanged; `SetHoldIf`'s validation errors reproduced); `TestReleaseHoldSeqNamesItsLifetime` (release at 1 against rows at 1 and 0 → ready `(1, s)`; against 2 → unchanged; empty op → no-op); `TestInstallLifetimeIfIsGuarded`. |
| + - `TestRouteForCopiesThePair`; `TestReconcileInstallsTheRecordsPairOnAMintedRow` ((1, 3) after finish and after abort-then-finish); `TestAnExistingRowKeepsOpenSequenceThroughFinish`. |
| + - `TestExistingRowsAcrossLifetimes` (subtests run to their end state: **ready legacy row, migrated record** `(0, 0)` + `{1, 0}` → `(1, 0)` once, then unchanged by `{1, 5}`; **ready row, recreation, no move** `(1, 9)` + `{2, 0}` → `(2, 0)`; **ready predecessor row, recreation with a started move** `(1, 9)` unheld + record `{2, 1}` asking a hold → one write to held `(2, 1)`, never a ready row at `(2, 1)` (a hook after every registry write asserts it); **held legacy row, migration during a move** held `(0, 1)` by X, record migrates to 1 → untouched by the installer; open's `ReleaseHoldSeq(X, {1, 2})` releases it to ready `(1, 2)`; the abort path likewise; **hold across migration** ready `(0, 0)` + migrated `{1, 1}` asking a hold → one write; **held predecessor by a vanished op** held `(1, 1)` by X + record `{2, 0}` → `DeleteIfHold` then a ready mint at `(2, 0)` on the same pass; with the callback nil → logged, unchanged; **a stale setter** release for 1 against the successor's row at 2 → no change; **Phase 1 unchanged** `PairOf == nil`, a row carrying `(1, 3)` → `SetHoldIf`/`ReleaseHold`, pair untouched). |
| + - In `TestOfflineMoveE2E`: the records are created through the lifetime with the judge on; after `start` the source row is held `(1, 1)`; after `open` the target row is ready `(1, 2)`; a second trace aborting from landed leaves the source ready `(1, 2)` and no target row; the release comes from `armOffline`'s wiring and the re-observed record (a hook that moves the record on before the release makes the release skip). |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run** `go test ./internal/registry -run 'Pair|Hold|Lifetime|Fingerprint' -count=1`, `go test ./internal/provision -run 'MoveSeq|RouteFor|Offline|Reconcile|Lifetime' -count=1`, `go test ./internal/host -run 'OfflineMoveE2E|Offline' -count=1`, then both census tests. |
| -- [ ] **Step 5: Commit** `moves: the offline hold, open, abort and finish advance move_seq; rows carry (incarnation, serving_seq), installed with the hold on legacy rows and never moved by another lifetime`. |
| +- [ ] **Step 5: Commit** `moves: the offline hold, open, abort and finish advance move_seq; rows take a lifetime with its hold in one write and are never moved by another lifetime`. |
| -### Task 8: Publish the pair and order unstamped entries of one lifetime by it |
| +### Task 8: Publish the pair with its provenance and order marked entries by it |
| **Files:** |
| -- Modify: `internal/host/edgesfile.go` (`edgeHost` gains `Incarnation uint64 \`json:"incarnation,omitempty"\``, `ServingSeq uint64 \`json:"serving_seq,omitempty"\``, `TargetHost string \`json:"target_host,omitempty"\``; **only the claims-off writer publishes the pair** (`edgesFileWriteClaims` with `claims == nil`): a served route publishes its row's pair; the explicit alias entry at line 546 publishes `hosts[a.Host]`'s pair and `TargetHost: a.Host`; the wildcard dial entry at line 632, when `catchallServed[wd.Host]` names a `primary`, publishes `hosts[primary]`'s pair and `TargetHost: primary`; a certificate-only wildcard entry publishes none; with `claims != nil` the pair fields stay zero whatever the row carries, and `TargetHost` is still published), `internal/edge/homemap.go` (`edgesHostView` and `homeEntry` gain `incarnation`, `servingSeq`, `targetHost`; `foldHomes` copies all three with the dial winner; `dialBeats` becomes: `!in.dial` → false; `!prev.dial` → true; `prev.alias != in.alias` → the real route wins (today); **if either entry carries a stamp (`stamp != 0`) → `in.stamp > prev.stamp` (today, the claim authority)**; both unstamped and both real → higher `(incarnation, servingSeq)` wins, equal → `in.stamp > prev.stamp` (false: today's first-wins); both unstamped aliases with `prev.targetHost != "" && prev.targetHost == in.targetHost` → the same pair rule; both unstamped aliases otherwise → today's rule) |
| +- Modify: `internal/host/edgesfile.go` (`edgeHost` gains `Lifetimes bool \`json:"lifetimes,omitempty"\``, `Incarnation uint64 \`json:"incarnation,omitempty"\``, `ServingSeq uint64 \`json:"serving_seq,omitempty"\``, `TargetHost string \`json:"target_host,omitempty"\``; `edgesFileWriteClaims` gains `lifetimes bool`, passed from the agent as `!hostwrite.Enabled(getenv)` (**not** the `claims` argument, which is the region-HA switch); with `lifetimes` every entry this box publishes carries `Lifetimes: true`: a served route publishes its row's pair; the explicit alias entry at line 546 publishes `hosts[a.Host]`'s pair and `TargetHost: a.Host`; the wildcard dial entry at line 632 publishes `hosts[primary]`'s pair and `TargetHost: primary`; a certificate-only wildcard entry publishes none; without `lifetimes` no entry carries the mark or a pair, and `TargetHost` is still published), `internal/edge/homemap.go` (`edgesHostView` and `homeEntry` gain `lifetimes`, `incarnation`, `servingSeq`, `targetHost`; `foldHomes` copies all four with the dial winner; `dialBeats`: `!in.dial` → false; `!prev.dial` → true; `prev.alias != in.alias` → the real route wins (today); if either entry is stamped → `in.stamp > prev.stamp` (today); both unstamped: if exactly one carries `lifetimes` → the marked entry wins (a lifetimes publisher is newer than any unmarked, unstamped one); both marked and both real → the higher `(incarnation, servingSeq)`, equal → today's rule (`in.stamp > prev.stamp`, false); both marked aliases with `prev.targetHost != "" && prev.targetHost == in.targetHost` → the same pair rule; otherwise → today's rule) |
| - Test: `internal/host/edgesfile_test.go`, `internal/edge/homemap_test.go` |
| **Interfaces:** |
| -- Produces: the three fields on the wire and in the fold; the comparable rule above. A claims-on publication always carries a stamp (`applyEdgesClaims`), so a retained claims-off publication with a pair loses to it, and two claims-on publications fold exactly as today. The different-target alias rule is plan 3's. |
| +- Produces: the four fields on the wire and in the fold; the rule above. Mixed history, stated: two claims-on publications fold as today (stamp when relocatable, first-wins otherwise); a claims-on publication of a relocatable row is stamped and beats any retained claims-off entry; a claims-on publication of a non-relocatable route (unstamped, unmarked) against a retained claims-off marked entry for the same name loses to it until the retained file is refreshed, which is the carry-forward window of a failed fetch and today's behaviour for a duplicate unstamped claim; a claims-off deployment mid-roll: marked entries beat unmarked retained ones, as the design asks (the first move's hold `(1, 1)` beats a retained pair-less `(0, 0)`). |
| -- [ ] **Step 1: Write the failing tests**: `TestEdgesFilePublishesThePairAndTargetHost` (a row `(2, 3)`; its explicit alias; a wildcard dial entry whose parent name differs from the primary; a certificate-only wildcard entry publishes no target and no pair); `TestClaimsOnWriterPublishesNoPair` (`claims != nil`, a row carrying `(1, 3)` → the entry has a stamp and no pair fields); `TestFoldPrefersTheHigherPairOfOneLifetime` (`(1, 2)` over `(1, 1)`; `(2, 0)` over `(1, 9)`; `(0, 0)` loses to any pair; equal → today's rule); `TestFoldStampBeatsPair` (a retained unstamped `(1, 3)` against a stamped `(0, 0)` → the stamped entry; two stamped entries → the higher stamp as today, whatever pairs they carry); `TestFoldKeepsRealRouteOverAHigherPairAlias`; `TestFoldComparesAliasesOnlyForOneTarget`; `TestFoldNeverDialsACertOnlyEntryForItsPair`; `TestFoldCopiesTheWinnersPairAndTarget`; `TestFoldLegacyPublicationsUnchanged` (fixtures without the fields fold as before; `TestHomeMapPicksTheSameHomeOnADuplicateClaim` still passes). |
| +- [ ] **Step 1: Write the failing tests**: `TestEdgesFilePublishesThePairAndTargetHost` (with `lifetimes`: a row `(2, 3)`; its explicit alias; a wildcard dial entry whose parent differs from the primary; a certificate-only wildcard entry has no target, no pair, no mark); `TestLifetimesOffWriterPublishesNoMarkOrPair` (`lifetimes == false` with the region-HA `claims` both nil and non-nil: no mark, no pair, `TargetHost` present); `TestFoldPrefersTheHigherPairOfOneLifetime`; `TestFoldStampBeatsPair` (a retained marked `(1, 3)` vs a stamped unmarked `(0, 0)` → the stamped; two stamped → the higher stamp); `TestFoldMarkedBeatsUnmarkedUnstamped` (a marked `(1, 1)` hold-side route vs an unmarked `(0, 0)` → marked; a marked `(0, 0)` vs an unmarked `(0, 0)` → marked); `TestFoldTwoUnmarkedUnstampedIsToday`; `TestFoldKeepsRealRouteOverAHigherPairAlias`; `TestFoldComparesAliasesOnlyForOneTarget`; `TestFoldNeverDialsACertOnlyEntryForItsPair`; `TestFoldCopiesTheWinnersFields`; `TestFoldLegacyPublicationsUnchanged` (existing fixtures; `TestHomeMapPicksTheSameHomeOnADuplicateClaim` still passes). |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run** `go test ./internal/host -run 'EdgesFile|ClaimsOnWriter' -count=1`, `go test ./internal/edge -run 'Fold|HomeMap' -count=1`, then the census tests. |
| -- [ ] **Step 5: Commit** `edges: the claims-off writer publishes (incarnation, serving_seq) and target_host; the home map orders unstamped entries of one lifetime by the pair`. |
| +- [ ] **Step 4: Run** `go test ./internal/host -run 'EdgesFile|LifetimesOffWriter' -count=1`, `go test ./internal/edge -run 'Fold|HomeMap' -count=1`, then the census tests. |
| +- [ ] **Step 5: Commit** `edges: the lifetimes writer marks its entries and publishes (incarnation, serving_seq) and target_host; the home map orders marked, unstamped entries of one lifetime by the pair`. |
| ### Task 9: Lifetimes end to end |
| **Files:** |
| - Test: `internal/host/movesoffline_lifetime_test.go` (extending `TestOfflineMoveE2E`'s fixtures), `internal/console/lifetimes_e2e_test.go` |
| -These are §11's witnesses plan 1 owns: record and row transitions and the ordering of ready publications. Every witness that needs a holding publication is plan 3's. Where a fold input below is not something this plan's writer produces (a retained target publication at a hold's pair, a retained `(1, 9)`), the test builds it as a **synthetic fixture** and says so in its name. |
| +These are §11's witnesses plan 1 owns. Every witness that needs a holding publication is plan 3's. Where a fold input is not something this plan's writer produces, the test builds it as a **synthetic fixture** and says so in its name. |
| -- [ ] **Step 1: Write the tests**: `TestLifetimeAcrossAForwardMove` (create `(1, 0)`; start → source row held `(1, 1)`; open → target row ready `(1, 2)`; finish → the target row stays `(1, 2)`, the record's `move_seq` is 3; a fold of the target's ready publication at `(1, 2)` against a retained source publication at `(1, 0)` prefers the target); `TestLifetimeAcrossAnAbort` (a second trace: start, flip to landed, abort → source reopens at `(1, 2)`, finish → the row stays `(1, 2)`, `move_seq` 3; the source's publication at `(1, 2)` beats a synthetic retained target publication at `(1, 1)`); `TestReconstructedRowCarriesTheRecordsPair` (after a finished move at `move_seq 3`, delete the target's row, reconcile → `(1, 3)`, beats a synthetic `(1, 1)`); `TestReconstructedSourceAfterAbortWins` (abort from landed, finish, delete the source row, reconcile → `(1, 3)`); `TestRecreatedHostOutranksItsPreviousLifetime` (retire, create → `(2, 0)` beats a synthetic retained `(1, 9)`); `TestLateLegacyCreateIsInert` (slot `{1, retired}`; a fieldless record written by the test; reconcile mints nothing, the door 404s, the strict listings omit it, the edges file does not list it, the next create replaces it); `TestDelayedCreateReplayMoveAndRetirementInterleave` (pending create replayed; a move completes; retirement crashes between the tombstone and the slot; the late original create fails on the tombstone; recovery retires the slot and removes the tombstone); `TestLandedCreateThenSecondRecoveryCrash` (a create whose record landed and serves, a crash before `committed`, recovery marks it committed with the record's ETag unchanged, including a second crash during that recovery). |
| +- [ ] **Step 1: Write the tests**: `TestLifetimeAcrossAForwardMove` (create `(1, 0)`; start → source held `(1, 1)`; open → target ready `(1, 2)`; finish → the target row stays `(1, 2)`, `move_seq` 3; the target's `(1, 2)` publication beats a retained source `(1, 0)`); `TestLifetimeAcrossAnAbort` (start, flip to landed, abort → source `(1, 2)`, finish → stays, `move_seq` 3; beats a synthetic target `(1, 1)`); `TestReconstructedRowCarriesTheRecordsPair` (`(1, 3)`); `TestReconstructedSourceAfterAbortWins` (`(1, 3)`); `TestRecreatedHostOutranksItsPreviousLifetime` (`(2, 0)` beats a synthetic retained `(1, 9)`); `TestRecreatedHostWithAStartedMoveIsHeldAtOnce` (a box reconciling late into a recreation whose move already started: its row goes from `(1, 9)` ready to held `(2, 1)` in one write); `TestLateLegacyCreateIsInert`; `TestDelayedCreateReplayMoveAndRetirementInterleave`; `TestLandedCreateThenSecondRecoveryCrash`. |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| - [ ] **Step 3: Run the census tests and both cross-builds** (`GOOS=darwin go build ./internal/host/`, `GOOS=windows go vet ./internal/host/`); write the census note. |
| - [ ] **Step 4: Commit** `lifetimes: end-to-end witnesses across a move, an abort, a reconstruction, a recreation and an interleaved retirement`; push. |
| @@ -318,7 +330,7 @@ These are §11's witnesses plan 1 owns: record and row transitions and the order |
| ## Self-review notes |
| -- §6.4 coverage: slot, owner key and tombstone form (T1, T4), create and recovery (T3), every claims-off removal and its recovery (T4), migration, its fence and recovery (T5), the verdict in every claims-off consumer with the two-round judgement (T1, T6), `move_seq` and the row pair with lifetime-naming setters and the four-case existing-row rule (T7), publication by the claims-off writer and the unstamped one-lifetime fold order (T8), witnesses (T9). Deferred by name: holding entries and their five-minute validity, the probe mode, the different-target alias rule and every holding-dependent witness (plan 3); the door, token, job, reservations and reports (plan 2); cleanup (plan 4). |
| -- Names used across tasks: `LifetimeOf`, `LifetimeVerdict` (`LifetimeValid`, `LifetimeStale`, `LifetimeCorrupt`, `LifetimeUnreadable`), `HostIncarnation{Incarnation, State, Account, App, Intent, RecordETag}.Valid`, `ErrSlotMalformed`, `ErrRecordCorrupt`, `ReadHostIncarnation`, `WriteHostIncarnation`, `TombstoneBody`, `GetRecordRaw`, `withHostname`, `admit`, `classifyLocked`, `classifyObservation`, `lifetimeClass`, `createRecordWithLifetime`, `retireRecordWithLifetime`, `retireExpectAbsent`, `migrateHostLifetimeLocked`, `errLifetimeSettling`, `errLifetimePending`, `*hostHeldError`, `errRetireRefused`, `ErrRecordMigrating`, `WithMigration`, `Observation`, `SlotCache.Begin/Forget/Judge`, `ReadRecordJudged`, `List*Judged`, `ReconcileConfig.Slots`, `ReconcileConfig.PairOf`, `registry.Pair`, `SetHoldIfSeq`, `ReleaseHoldSeq`, `InstallLifetimeIf`, `moveCfg.ReleaseHoldSeq`, `releaseHeldSeq`, `Route.Incarnation`, `Route.ServingSeq`, `MoveState.Seq`, `edgeHost.Incarnation/ServingSeq/TargetHost`. |
| -- Review Focus 1 to 5 are pinned by T3 (three tests), T5+T6, T6+T3, T7 (two tests), T7. |
| -- Store primitives, as verified in the reviews: `Store.Put(..., "")` is create-only on both backends; `WriteRecord` with `before={}` and `etag=""` is the create and needs the real CAS base otherwise; `DeleteIfMatch` treats absence as success and has no unconditional fallback; `OfflineAbort` on a reverting record returns `errRecordUnchanged`, which `SetRecordMove` turns into a success without a write. |
| +- §6.4 coverage: slot, owner key and tombstone form (T1, T4), create and recovery (T3), every claims-off removal with an explicit expectation and its recovery (T4), migration, its fence and recovery (T5), the observation protocol in every claims-off consumer (T1, T6), `move_seq`, the setters that install a lifetime with its hold and the existing-row rules (T7), publication with provenance and the fold order (T8), witnesses (T9). Deferred by name: holding entries and their five-minute validity, the probe mode, the different-target alias rule and every holding-dependent witness (plan 3); the door, token, job, reservations and reports (plan 2); cleanup (plan 4). The one stated deviation from §6.4's text is the uncached slot read. |
| +- Names used across tasks: `LifetimeOf`, `LifetimeVerdict`, `HostIncarnation.Valid`, `ErrSlotMalformed`, `ErrRecordCorrupt`, `ReadHostIncarnation`, `WriteHostIncarnation`, `TombstoneBody`, `GetRecordRaw`, `Observation{Present, Rec, Raw, ETag, Slot, SlotETag, Verdict, Err}`, `LifetimeJudge.Observe/ObserveWith`, `ReadRecordJudged`, `List*Judged`, `withHostname`, `admit`, `classifyLocked`, `createRecordWithLifetime`, `retireRecordWithLifetime`, `retireExpectation` (`retireExpectAbsent`, `retireAny`, `retireExpectETag`), `migrateHostLifetimeLocked`, `errLifetimeSettling`, `errLifetimePending`, `*hostHeldError`, `errRetireRefused`, `ErrRecordMigrating`, `WithMigration`, `ReconcileConfig.Judge/PairOf/DeleteIfHold`, `moveCfg.Judge/ReleaseHoldSeq`, `releaseHeldSeq`, `registry.Pair`, `SetHoldIfSeq`, `ReleaseHoldSeq`, `InstallLifetimeIf`, `Route.Incarnation/ServingSeq`, `MoveState.Seq`, `edgeHost.Lifetimes/Incarnation/ServingSeq/TargetHost`. |
| +- Review Focus 1 to 5 are pinned by T3+T5, T6, T6+T3, T7, T7+T9. |
| +- Store primitives, as verified in the reviews: `Store.Put(..., "")` is create-only on both backends; `WriteRecord` with `before={}` and `etag=""` is the create and needs the real CAS base otherwise; `DeleteIfMatch` treats absence as success and has no unconditional fallback; `OfflineAbort` on a reverting record returns `errRecordUnchanged`, a success without a write; `AccountBySqid` returns `(nil, nil)` for an unknown sqid. |