amadan
plan: self-serve moves 1, rev 13 (Astra review 12: a durable lifetimes-on marker as the boxes' authority; purge excludes instance keys from the sweep, voids pending allocations durably; the box retires by CAS only, no migration or journal)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+13
−10
docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md+13 −10
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok amadan.net/carlos/platform.git/launchctx 0.014s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (1 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (149 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.858s
ok github.com/carlosframework/platform/cmd/moveprobe 11.871s
ok github.com/carlosframework/platform/cmd/zddprobe 0.749s
ok github.com/carlosframework/platform/internal/activator 60.139s
ok github.com/carlosframework/platform/internal/agentsock 0.040s
ok github.com/carlosframework/platform/internal/alerting 0.009s
ok github.com/carlosframework/platform/internal/analytics 0.028s
ok github.com/carlosframework/platform/internal/appconfig 2.994s
ok github.com/carlosframework/platform/internal/apperr 0.176s
ok github.com/carlosframework/platform/internal/applog 0.259s
ok github.com/carlosframework/platform/internal/awsinv 0.007s
ok github.com/carlosframework/platform/internal/boxchoice 0.007s
ok github.com/carlosframework/platform/internal/boxid 1.224s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.021s
ok github.com/carlosframework/platform/internal/certwarm 0.003s
ok github.com/carlosframework/platform/internal/cgroup 0.768s
ok github.com/carlosframework/platform/internal/changelog 0.592s
ok github.com/carlosframework/platform/internal/claimactivation 1.355s
ok github.com/carlosframework/platform/internal/clamd 0.062s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.019s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.020s
ok github.com/carlosframework/platform/internal/dbverify 0.009s
ok github.com/carlosframework/platform/internal/dnsrec 0.003s
ok github.com/carlosframework/platform/internal/docsite 0.013s
ok github.com/carlosframework/platform/internal/domains 2.910s
ok github.com/carlosframework/platform/internal/doorkey 0.014s
ok github.com/carlosframework/platform/internal/economics 0.004s
ok github.com/carlosframework/platform/internal/edge 45.945s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.020s
ok github.com/carlosframework/platform/internal/fleetapi 0.185s
ok github.com/carlosframework/platform/internal/gate 1.815s
ok github.com/carlosframework/platform/internal/genselect 0.045s
ok github.com/carlosframework/platform/internal/geo 0.034s
ok github.com/carlosframework/platform/internal/geoip 0.051s
ok github.com/carlosframework/platform/internal/historyreadback 0.293s
ok github.com/carlosframework/platform/internal/host 146.131s
ok github.com/carlosframework/platform/internal/hostactors 7.835s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.004s
ok github.com/carlosframework/platform/internal/hostbox 3.181s
ok github.com/carlosframework/platform/internal/hostclaim 0.877s
ok github.com/carlosframework/platform/internal/hostenv 0.004s
ok github.com/carlosframework/platform/internal/hostevidence 0.010s
ok github.com/carlosframework/platform/internal/hostexec 5.684s
ok github.com/carlosframework/platform/internal/hostfence 0.047s
ok github.com/carlosframework/platform/internal/hostfiles 10.306s
ok github.com/carlosframework/platform/internal/hostfloor 0.042s
ok github.com/carlosframework/platform/internal/hostguard 1.068s
ok github.com/carlosframework/platform/internal/hostops 2.299s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.004s
ok github.com/carlosframework/platform/internal/hostquarantine 0.004s
ok github.com/carlosframework/platform/internal/hostrelocate 0.738s
ok github.com/carlosframework/platform/internal/hostresolve 0.209s
ok github.com/carlosframework/platform/internal/hostwrite 0.007s
ok github.com/carlosframework/platform/internal/infraview 0.003s
ok github.com/carlosframework/platform/internal/instancectl 5.249s
ok github.com/carlosframework/platform/internal/instanceexec 2.785s
ok github.com/carlosframework/platform/internal/instancefs 0.007s
ok github.com/carlosframework/platform/internal/instancekey 0.002s
ok github.com/carlosframework/platform/internal/instanceupdates 0.011s
ok github.com/carlosframework/platform/internal/layout 2.026s
ok github.com/carlosframework/platform/internal/lbhealth 0.003s
ok github.com/carlosframework/platform/internal/lease 10.230s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.570s
ok github.com/carlosframework/platform/internal/ledger 0.035s
ok github.com/carlosframework/platform/internal/mail 0.237s
ok github.com/carlosframework/platform/internal/maintenance 0.080s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.583s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.120s
ok github.com/carlosframework/platform/internal/metrics 0.009s
ok github.com/carlosframework/platform/internal/moves 21.810s
ok github.com/carlosframework/platform/internal/moves/archive 1.173s
ok github.com/carlosframework/platform/internal/objstore 0.265s
ok github.com/carlosframework/platform/internal/ops 0.221s
ok github.com/carlosframework/platform/internal/park 0.278s
ok github.com/carlosframework/platform/internal/plans 0.006s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
ok github.com/carlosframework/platform/internal/provision 13.791s
ok github.com/carlosframework/platform/internal/publicrelease 0.008s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.741s
ok github.com/carlosframework/platform/internal/relay 7.242s
ok github.com/carlosframework/platform/internal/release 25.659s
ok github.com/carlosframework/platform/internal/rollout 3.858s
ok github.com/carlosframework/platform/internal/rolloutwords 0.010s
ok github.com/carlosframework/platform/internal/rootaws 0.009s
ok github.com/carlosframework/platform/internal/rootparticipation 0.062s
ok github.com/carlosframework/platform/internal/rootstate 0.028s
ok github.com/carlosframework/platform/internal/runs 0.570s
ok github.com/carlosframework/platform/internal/s3compat 0.006s
ok github.com/carlosframework/platform/internal/safefile 0.005s
ok github.com/carlosframework/platform/internal/schedule 2.736s
ok github.com/carlosframework/platform/internal/seeds 0.012s
ok github.com/carlosframework/platform/internal/sim/hostlife 11.597s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.754s
ok github.com/carlosframework/platform/internal/status 0.075s
ok github.com/carlosframework/platform/internal/steering 0.008s
ok github.com/carlosframework/platform/internal/stopevidence 0.170s
ok github.com/carlosframework/platform/internal/svcrole 0.004s
ok github.com/carlosframework/platform/internal/telemetry 0.005s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.006s
ok github.com/carlosframework/platform/internal/unixdial 0.005s
ok github.com/carlosframework/platform/internal/updatewords 0.016s
ok github.com/carlosframework/platform/internal/usage 0.010s
ok github.com/carlosframework/platform/internal/version 0.005s
ok github.com/carlosframework/platform/internal/vet 1.659s
=== internal_console@0 (82 s)
ok github.com/carlosframework/platform/internal/console 80.269s
=== internal_console@1 (38 s)
ok github.com/carlosframework/platform/internal/console 36.257s
=== internal_console@2 (43 s)
ok github.com/carlosframework/platform/internal/console 41.356s
=== internal_console@3 (39 s)
ok github.com/carlosframework/platform/internal/console 36.695s
=== internal_console@4 (38 s)
ok github.com/carlosframework/platform/internal/console 36.428s
=== internal_console@5 (37 s)
ok github.com/carlosframework/platform/internal/console 34.466s
=== internal_console@6 (39 s)
ok github.com/carlosframework/platform/internal/console 36.483s
=== internal_console@7 (45 s)
ok github.com/carlosframework/platform/internal/console 42.768s
=== root@0 (34 s)
ok github.com/carlosframework/platform 32.066s
=== root@1 (45 s)
ok github.com/carlosframework/platform 43.341s
=== root@2 (43 s)
ok github.com/carlosframework/platform 41.014s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.03s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.02s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.030s
=== RUN TestT4StorageProbeMinio
2026/10/22 09:49:04 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0D1162EEE2BA1, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.024s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md| index bb9eed807..16b71d07f 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| @@ -2,7 +2,7 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 12, 2026-10-22. Rev 11's review (`selfserve-plan1-review11.md`, findings 1 to 7) is answered: the storage settle is anchored **after the last `before` unit released admission** (and after the process start on a restart), so a legacy request that was fully received before its client timed out has completed before activation; activation requires a **validated marker read** (`markerSeen`), never uncertainty alone; app purge is **lifetime-aware** on every path (browser, expiry pass, box auto-purge) through a retirement core in `provision` that `release.PurgeApp` calls for every instance record before its sweep; an in-scope valid own record defeats only the record-gone reason and still reaches the app-trash decision; the dispatch of create, delete, detach and `readDeletedRecord` is stated per admitted unit; both detached continuations (`runImportUpload`, `createFrom → runFrom`) are wired; the Task 3 witness no longer says draining reads are legacy; `claimAliasUnbacked` is named in the audit. Rev 11, 2026-10-22. Rev 10's review (`selfserve-plan1-review10.md`, findings 1 to 4) is answered: the cutover gains an **admission gate**: every claims-off unit of record work (a request's handler, a detached continuation, a worker iteration) is admitted under a counted gate while the state is `before`; the cutover door closes admission synchronously, refuses new work with 503, **waits for the admitted count to reach zero**, then starts the storage settle, then activates; so no unit admitted before the marker ever runs beside the lifetime protocol, and a unit never changes mode mid-flight. `on` is **latched**: once observed it is never left; at startup the state is `unknown` until the first successful marker read and every claims-off record door (reads and mutations) answers 503 until then, so a marker-read failure never restores legacy interpretation. The removal rule separates lifetime validity from placement eligibility: a valid own record that is **in this pass's scope** protects the legacy collision; a valid own record **outside** the scope follows today's removal rules (`KeepParticipant`, the app-trash rule), so offline finishing removes the source row as today; absence or staleness authorises the new path. One worker lifecycle (start once, loop, check the state each iteration), one migration interval, one context contract (`recordCtx`), the CLI command, the retry error, the three root constructions and the copy inventory are stated. Rev 10, 2026-10-22. Rev 9's review (`selfserve-plan1-review9.md`, findings 1 to 5) is answered: the cutover is a **mode** with a drain: from the marker's time until the window ends the console refuses every claims-off record mutation (503), so no legacy write can be admitted into the window, and the gate, the judge's installation, the recovery pass and the marker's failure modes are one contract with an implementation task for the operator door; the removal loop decides from the row's **own** record (observed by its own key), so a valid legacy owner in a hostname collision is kept as today and only an established absent or stale own record retires the row, with the keep view bypassed on that evidence and replacement stated as two passes; `fenceRecord` returns a classification so an own tombstone or marker that won resumes the operation instead of terminating it; the audit's allowlist names the two remaining claims-on callers of `hostops.ReadRecord` and is pinned both ways; and the remaining contracts are settled (the detach sentence, cleanup recovery through C2, corrupt index candidates, the root restart readers' judge). Rev 8's review (`selfserve-plan1-review8.md`, findings 1 to 5) is answered: the rollout guarantee becomes an **enforced cutover** (an operator marker written after the old console has stopped, plus the settle window, with the storage assumption stated) rather than a timer alone; quarantine copies the corrupt bytes and **never deletes under a live slot**: the retirement converts durably to the cleanup kind and retires the slot first, and a conditional-delete conflict re-classifies instead of counting as absence; a provisioned row owned by another record key than the hostname's current record is retired as "record gone" so a cross-owner recreation cannot leave a lagging box serving the predecessor; `RecordIndex.Find` judges its candidate keys before deciding ambiguity; and the remaining contradictions are resolved (the root scope sentence, one detach chain, the locked retirement's callers, reconcile's one sequence, the audit's enumerated list, the fence witness split). Rev 7's review (`selfserve-plan1-review7.md`, findings 1 to 6) is answered: a stated rollout guarantee closes the late legacy delete (one console per deployment, a settle window before any termination, old consoles never reintroduced over lifetimes records); the first-route attach, its rollback and detach run **inside** the hostname exclusion from the claim write to the record write, through locked variants of the create and retirement helpers; root's proof and restart record readers take the box's judge (the scope statement is corrected: only `ClaimRootHTTP`'s claims-on readers are slot-free); a corrupt body is quarantined (its bytes kept under `control/quarantine/`, the key removed conditionally) so the retirement or migration proceeds as for absence instead of promising a fence that cannot parse it; `provision.WithLifetimes` is the one context helper that carries the judge and enables sequencing, the runs worker wraps its detached execution context, the audit distinguishes approved judged wrappers from raw readers, the fence witnesses state their timing, exhaustion is witnessed at every transition, and the sibling undo has one dispatch path. Earlier reviews: rev 1 (`after236/selfserve-plan1-review.md`, 16 findings), rev 2 (F1 to F12), rev 3 (R1 to R10), rev 4 (1 to 9), rev 5 (1 to 6), rev 6 (`selfserve-plan1-review6.md`, findings 1 to 5). Rev 7 answers rev 6's five: every claims-off record write runs with `WithRecordSequence`, so `record_seq` exists after a record's first write and an ETag never recurs, and recovery writes a **fence** (a sequencing rewrite) before it terminates an operation on evidence of a changed record; one migration dispatcher classifies the observation at every entry, so a resume can never supersede its own marker; the attach door's claim rollback runs inside the hostname exclusion and never releases a claim a pending operation or a valid record needs; the judge reaches every mutation context (the console's top-level handler wraps every request, browser and API alike; the seed callback wraps its own context; the full-record updater's re-read is judged; `RecordIndex.Find` and `FindRecordByHost` take the judge from the context), and the audit is by AST; the remaining contracts (the running N1 table, a slot-retirement helper, one expectation type, listed-key absence, exhaustion at read, the import witness's state, the producer-state publication tests) are answered. Rev 5 removed the slot cache (every judgement reads the slot fresh, after the record, in bounded rounds) and made a successor lifetime's hold one row write. Rev 6 answers rev 5's six: a retirement persists its **kind** (normal, orphan, cleanup) and each kind has its own steps and recovery table; a migration's marker-CAS conflict is classified by a re-read for both the bound and the unbound form, so an own marker that landed is resumed, never superseded; every `retireInstanceRecord` caller states its expectation (the sibling undo names the lifetime it created; detach deletes what is current; the attach door never asks for a record rollback over a lifetime failure); a judge carried on the context makes `updateRecordSeeding`, and so every `SetRecord*` writer, refuse a stale record, and every box reader (Phase 1's included) and the import journal's readers go through the judge; the provenance mark is dropped (the fold compares pairs between unstamped entries, a pair-less entry is `(0, 0)` and equal pairs fall to the existing rule, exactly §6.4); the remaining interface gaps are answered. |
| +Rev 13, 2026-10-22. Rev 12's review (`selfserve-plan1-review12.md`: findings 2, 3, 4, 6, 7 closed; 1 a witness contradiction; 5, 8, 9 the purge) is answered: the restart witness no longer activates "at once"; the console writes a durable **`control/lifetimes-on.json`** as the first act of `on`, and that marker, not judge availability, is a box's authority for any slot mutation; purge **excludes instance-record keys from its sweep** whenever a retire hook is installed; the console's purge **voids the app's pending allocations durably** (`purged_at`) and retires their slots before touching records, and recovery treats a purged pending row as `void`; the box's purge needs **no migration, no journal and no cross-process lock**: it retires by CAS alone (slot CAS to `retired`, then `DeleteIfMatch` on the observed record ETag; a marked record is skipped; any CAS loss returns an error and the next pass retries idempotently), and before the on-marker it deletes only by `DeleteIfMatch` on the listed ETag after re-reading the marker. Rev 12's review (`selfserve-plan1-review11.md`, findings 1 to 7) is answered: the storage settle is anchored **after the last `before` unit released admission** (and after the process start on a restart), so a legacy request that was fully received before its client timed out has completed before activation; activation requires a **validated marker read** (`markerSeen`), never uncertainty alone; app purge is **lifetime-aware** on every path (browser, expiry pass, box auto-purge) through a retirement core in `provision` that `release.PurgeApp` calls for every instance record before its sweep; an in-scope valid own record defeats only the record-gone reason and still reaches the app-trash decision; the dispatch of create, delete, detach and `readDeletedRecord` is stated per admitted unit; both detached continuations (`runImportUpload`, `createFrom → runFrom`) are wired; the Task 3 witness no longer says draining reads are legacy; `claimAliasUnbacked` is named in the audit. Rev 11, 2026-10-22. Rev 10's review (`selfserve-plan1-review10.md`, findings 1 to 4) is answered: the cutover gains an **admission gate**: every claims-off unit of record work (a request's handler, a detached continuation, a worker iteration) is admitted under a counted gate while the state is `before`; the cutover door closes admission synchronously, refuses new work with 503, **waits for the admitted count to reach zero**, then starts the storage settle, then activates; so no unit admitted before the marker ever runs beside the lifetime protocol, and a unit never changes mode mid-flight. `on` is **latched**: once observed it is never left; at startup the state is `unknown` until the first successful marker read and every claims-off record door (reads and mutations) answers 503 until then, so a marker-read failure never restores legacy interpretation. The removal rule separates lifetime validity from placement eligibility: a valid own record that is **in this pass's scope** protects the legacy collision; a valid own record **outside** the scope follows today's removal rules (`KeepParticipant`, the app-trash rule), so offline finishing removes the source row as today; absence or staleness authorises the new path. One worker lifecycle (start once, loop, check the state each iteration), one migration interval, one context contract (`recordCtx`), the CLI command, the retry error, the three root constructions and the copy inventory are stated. Rev 10, 2026-10-22. Rev 9's review (`selfserve-plan1-review9.md`, findings 1 to 5) is answered: the cutover is a **mode** with a drain: from the marker's time until the window ends the console refuses every claims-off record mutation (503), so no legacy write can be admitted into the window, and the gate, the judge's installation, the recovery pass and the marker's failure modes are one contract with an implementation task for the operator door; the removal loop decides from the row's **own** record (observed by its own key), so a valid legacy owner in a hostname collision is kept as today and only an established absent or stale own record retires the row, with the keep view bypassed on that evidence and replacement stated as two passes; `fenceRecord` returns a classification so an own tombstone or marker that won resumes the operation instead of terminating it; the audit's allowlist names the two remaining claims-on callers of `hostops.ReadRecord` and is pinned both ways; and the remaining contracts are settled (the detach sentence, cleanup recovery through C2, corrupt index candidates, the root restart readers' judge). Rev 8's review (`selfserve-plan1-review8.md`, findings 1 to 5) is answered: the rollout guarantee becomes an **enforced cutover** (an operator marker written after the old console has stopped, plus the settle window, with the storage assumption stated) rather than a timer alone; quarantine copies the corrupt bytes and **never deletes under a live slot**: the retirement converts durably to the cleanup kind and retires the slot first, and a conditional-delete conflict re-classifies instead of counting as absence; a provisioned row owned by another record key than the hostname's current record is retired as "record gone" so a cross-owner recreation cannot leave a lagging box serving the predecessor; `RecordIndex.Find` judges its candidate keys before deciding ambiguity; and the remaining contradictions are resolved (the root scope sentence, one detach chain, the locked retirement's callers, reconcile's one sequence, the audit's enumerated list, the fence witness split). Rev 7's review (`selfserve-plan1-review7.md`, findings 1 to 6) is answered: a stated rollout guarantee closes the late legacy delete (one console per deployment, a settle window before any termination, old consoles never reintroduced over lifetimes records); the first-route attach, its rollback and detach run **inside** the hostname exclusion from the claim write to the record write, through locked variants of the create and retirement helpers; root's proof and restart record readers take the box's judge (the scope statement is corrected: only `ClaimRootHTTP`'s claims-on readers are slot-free); a corrupt body is quarantined (its bytes kept under `control/quarantine/`, the key removed conditionally) so the retirement or migration proceeds as for absence instead of promising a fence that cannot parse it; `provision.WithLifetimes` is the one context helper that carries the judge and enables sequencing, the runs worker wraps its detached execution context, the audit distinguishes approved judged wrappers from raw readers, the fence witnesses state their timing, exhaustion is witnessed at every transition, and the sibling undo has one dispatch path. Earlier reviews: rev 1 (`after236/selfserve-plan1-review.md`, 16 findings), rev 2 (F1 to F12), rev 3 (R1 to R10), rev 4 (1 to 9), rev 5 (1 to 6), rev 6 (`selfserve-plan1-review6.md`, findings 1 to 5). Rev 7 answers rev 6's five: every claims-off record write runs with `WithRecordSequence`, so `record_seq` exists after a record's first write and an ETag never recurs, and recovery writes a **fence** (a sequencing rewrite) before it terminates an operation on evidence of a changed record; one migration dispatcher classifies the observation at every entry, so a resume can never supersede its own marker; the attach door's claim rollback runs inside the hostname exclusion and never releases a claim a pending operation or a valid record needs; the judge reaches every mutation context (the console's top-level handler wraps every request, browser and API alike; the seed callback wraps its own context; the full-record updater's re-read is judged; `RecordIndex.Find` and `FindRecordByHost` take the judge from the context), and the audit is by AST; the remaining contracts (the running N1 table, a slot-retirement helper, one expectation type, listed-key absence, exhaustion at read, the import witness's state, the producer-state publication tests) are answered. Rev 5 removed the slot cache (every judgement reads the slot fresh, after the record, in bounded rounds) and made a successor lifetime's hold one row write. Rev 6 answers rev 5's six: a retirement persists its **kind** (normal, orphan, cleanup) and each kind has its own steps and recovery table; a migration's marker-CAS conflict is classified by a re-read for both the bound and the unbound form, so an own marker that landed is resumed, never superseded; every `retireInstanceRecord` caller states its expectation (the sibling undo names the lifetime it created; detach deletes what is current; the attach door never asks for a record rollback over a lifetime failure); a judge carried on the context makes `updateRecordSeeding`, and so every `SetRecord*` writer, refuse a stale record, and every box reader (Phase 1's included) and the import journal's readers go through the judge; the provenance mark is dropped (the fold compares pairs between unstamped entries, a pair-less entry is `(0, 0)` and equal pairs fall to the existing rule, exactly §6.4); the remaining interface gaps are answered. |
| **Goal:** Give every claims-off instance record a durable lifetime (the host incarnation slot) and every served row a serving order `(incarnation, serving_seq)` that boxes publish and edges honour, with crash-safe create, retirement and migration, so a later plan's holding pages and moves can never be out-ordered by a stale publication. |
| @@ -51,13 +51,13 @@ Rev 12, 2026-10-22. Rev 11's review (`selfserve-plan1-review11.md`, findings 1 t |
| **Interfaces:** |
| - `type cutoverMarker struct { At time.Time \`json:"at"\`; By string \`json:"by"\` }`; `const lifetimesCutoverKey = "control/lifetimes-cutover.json"`; `const lifetimeSettle = 60 * time.Second`; `type cutoverState int` (`cutoverUnknown`, `cutoverBefore`, `cutoverDraining`, `cutoverOn`). |
| -- `type cutoverGate struct { mu sync.Mutex; state cutoverState; markerSeen bool; at time.Time; started time.Time; lastDone time.Time; admitted int; drained chan struct{}; pageAt time.Time }`: `func (g *cutoverGate) admit(now time.Time, mutating bool) (unit *admittedUnit, ok bool, refusal string)` (`unknown` → false, the starting sentence; `before` → a unit with `mode: before`, `admitted++`; `draining` → `mutating` → false, the draining sentence; a plain read → a unit with `mode: on` (judged; reads never write); `on` → a unit with `mode: on`); `func (u *admittedUnit) done()` (`admitted--` for a `before` unit; the last one records `lastDone = now` and closes `drained`; exactly one `done()` per unit, enforced by a `sync.Once`); `func (g *cutoverGate) observe(m *cutoverMarker, readErr error, now time.Time)` (the watcher's and the door's input: the transitions of the Global Constraint; `on` is never left; `before`→`draining` closes admission synchronously); `func (g *cutoverGate) activateWhenDrained(now time.Time) bool` (`draining` and `markerSeen` and `admitted == 0` and `now >= max(at, started, lastDone) + lifetimeSettle` → `on`; a read failure sets `draining` without `markerSeen`, which never activates); `func (g *cutoverGate) status() cutoverStatus` (`{State string; At time.Time; Pending int}`). |
| +- `type cutoverGate struct { mu sync.Mutex; state cutoverState; markerSeen bool; at time.Time; started time.Time; lastDone time.Time; admitted int; drained chan struct{}; pageAt time.Time }`: `func (g *cutoverGate) admit(now time.Time, mutating bool) (unit *admittedUnit, ok bool, refusal string)` (`unknown` → false, the starting sentence; `before` → a unit with `mode: before`, `admitted++`; `draining` → `mutating` → false, the draining sentence; a plain read → a unit with `mode: on` (judged; reads never write); `on` → a unit with `mode: on`); `func (u *admittedUnit) done()` (`admitted--` for a `before` unit; the last one records `lastDone = now` and closes `drained`; exactly one `done()` per unit, enforced by a `sync.Once`); `func (g *cutoverGate) observe(m *cutoverMarker, readErr error, now time.Time)` (the watcher's and the door's input: the transitions of the Global Constraint; `on` is never left; `before`→`draining` closes admission synchronously); `func (g *cutoverGate) activateWhenDrained(now time.Time) bool` (`draining` and `markerSeen` and `admitted == 0` and `now >= max(at, started, lastDone) + lifetimeSettle` → `on`; a read failure sets `draining` without `markerSeen`, which never activates); **the on-marker**: the first act of `on`, before any lifetime write, is `Put(control/lifetimes-on.json, {"at", "console"}, create-only)` (`lifetimesOnKey`); an existing one is left; a failed write keeps the state `draining` and retries each watcher tick. The on-marker is the **boxes'** authority (Task 4's purge): a box performs a slot mutation only after it has read `lifetimes-on.json` (latched once seen; read each pass until then); the cutover marker alone never authorises a box. `func (g *cutoverGate) status() cutoverStatus` (`{State string; At time.Time; Pending int}`). |
| - `func (s *Server) lifetimesAdmit(ctx context.Context, mutating bool) (*admittedUnit, bool, string)`; `func (s *Server) recordCtx(ctx context.Context, u *admittedUnit) context.Context` (`u.mode == on` → `provision.WithLifetimes(ctx, s.judge)`, else `ctx` unchanged); `func (s *Server) lifetimesOn() bool` (`!cfg.HostClaim && cfg.Releases != nil && s.cutover.state == cutoverOn`). Every claims-off record door: `u, ok, why := s.lifetimesAdmit(ctx, mutating); if !ok { 503 why }; defer u.done(); ctx = s.recordCtx(ctx, u)`. **Dispatch by unit:** every claims-off create, delete, detach, purge and `readDeletedRecord` site selects its path from `u.mode`: `before` → today's legacy call (the plain create, the unconditional `DeleteRecord`, the raw read) with no slot and no judge; `on` → the lifetime helper of Tasks 3 to 5; `!cfg.HostClaim` alone never selects the lifetime path. **Detached continuations** (`runImportUpload` and `createFrom → runFrom`): the handler transfers `u` to the job **before** launching the goroutine, the handler does not call `done()`, and the job calls it exactly once after its deferred abort or cleanup writes have returned. Every worker iteration: admit (non-mutating workers as reads; the passes that write as mutating), skip the iteration when refused, `done()` at its end. |
| - `func (s *Server) handleAPILifetimesCutover(w, r)`: an operator's own CLI token (else 404, as the move family); `cfg.HostClaim` → 409 "this console runs host claims; lifetimes are not for it"; writes the marker create-only with `At: now`, `By: addr`, then `s.cutover.observe(&m, nil, now)` **synchronously** (admission closes before the response); `ErrCASConflict` → read the existing marker: readable → `observe` it and answer with the status; unreadable or malformed → 502 "the cutover marker exists but cannot be read. repair control/lifetimes-cutover.json by hand" (never a success); the audit line "switched instance lifetimes on (cli)"; the response is `cutoverStatus` (200). The CLI prints "instance lifetimes: switching on; N unit(s) of earlier work still finishing" while `Pending > 0` or the settle has not passed, and "instance lifetimes: on since <time>" when `on`; `carlos system lifetimes cutover` polls the door every 5 s until `on` (the door is idempotent) or `--no-wait`. |
| - `StartLifetimesCutoverWatch`: reads the marker at start and every 10 s and calls `observe`, then `activateWhenDrained`; the door's synchronous `observe` is what makes refusal immediate at `At`; the watcher covers a restart and a marker written by hand (documented: a hand-written marker closes admission within 10 s). |
| - **Worker lifecycle (one rule):** every worker starts once at process start and loops for the life of the process; each iteration admits, does its work only when admitted in the mode it requires (`on` for the recovery and migration passes), and ends its unit. No worker returns early because the state was `before` or `unknown` at start. |
| -- [ ] **Step 1: Write the failing tests**: `TestCutoverDoorIsOperatorOnlyAndIdempotent` (second call → the status; an unreadable existing marker → 502, not a success); `TestCutoverStates` (unknown at start → 503 on every door incl. GET; a read marker absent → before; a marker written → draining at once; `on` after drain and settle; `on` is kept across a later read failure and a corrupt marker; a restart into a marker older than 60 s → on at once; a restart into a marker 10 s old → on at 60 s after the process start); `TestDrainingRefusesEveryClaimsOffMutation` (one subtest per door → 503, nothing written; GETs answer, judged); `TestActivationWaitsForAdmittedWork` (a real creation handler paused before its fresh record-write context, and a detached import continuation paused mid-body, both admitted before the marker; the marker written; 60 s pass; the state is still draining with `Pending == 2`; each finishes → its legacy write lands under `before` rules; the count reaches zero → `on` after the settle; a creation arriving at `At + 1 s` → 503); `TestNoUnitChangesModeMidFlight` (a unit admitted in `before` observes no judge after activation would otherwise have happened; one admitted in `on` has it); `TestWorkersLoopAcrossTheCutover` (the recovery and migration passes started before the marker do nothing, then run after activation without a restart; the seed sweep's iteration admitted before the marker finishes under legacy rules and the next iteration is judged); `TestMarkerFailureAfterActivationKeepsJudgedReads` (a retired slot and an obsolete body; the marker read fails; GET and listings refuse the body); `TestCutoverCLI` (the dispatch, the client call, the two output lines, `--no-wait`); `TestSettleStartsAfterTheLastLegacyUnit` (a `before` unit whose final store call returns an ambiguous error while the server-side effect is still outstanding (a fake store that applies the delete 5 s after returning a timeout); the unit calls `done()` at `T`; the marker is 10 min old; activation happens no earlier than `T + 60 s`; the late delete lands before activation and after it no legacy write exists); `TestActivationRequiresAValidatedMarker` (absent marker; a read failure; no units; 60 s pass → still draining, not on; a successful absent read → before; a successful present read → draining then on after the settle; a restart into a 10 min old marker → on 60 s after the process start, not at once); `TestDispatchFollowsTheUnit` (a `before` unit's create, delete, detach and `readDeletedRecord` take the legacy path even after the marker; an `on` unit's take the lifetime helpers); `TestDetachedContinuationsOwnTheirUnit` (import upload and `runFrom`: the handler returns, the unit stays admitted until the job's last write, exactly one `done()`). |
| +- [ ] **Step 1: Write the failing tests**: `TestCutoverDoorIsOperatorOnlyAndIdempotent` (second call → the status; an unreadable existing marker → 502, not a success); `TestCutoverStates` (unknown at start → 503 on every door incl. GET; a read marker absent → before; a marker written → draining at once; `on` after drain and settle; `on` is kept across a later read failure and a corrupt marker; a restart into a marker older than 60 s → draining then on 60 s after the process start; a restart into a marker 10 s old → the same); `TestDrainingRefusesEveryClaimsOffMutation` (one subtest per door → 503, nothing written; GETs answer, judged); `TestActivationWaitsForAdmittedWork` (a real creation handler paused before its fresh record-write context, and a detached import continuation paused mid-body, both admitted before the marker; the marker written; 60 s pass; the state is still draining with `Pending == 2`; each finishes → its legacy write lands under `before` rules; the count reaches zero → `on` after the settle; a creation arriving at `At + 1 s` → 503); `TestNoUnitChangesModeMidFlight` (a unit admitted in `before` observes no judge after activation would otherwise have happened; one admitted in `on` has it); `TestWorkersLoopAcrossTheCutover` (the recovery and migration passes started before the marker do nothing, then run after activation without a restart; the seed sweep's iteration admitted before the marker finishes under legacy rules and the next iteration is judged); `TestMarkerFailureAfterActivationKeepsJudgedReads` (a retired slot and an obsolete body; the marker read fails; GET and listings refuse the body); `TestCutoverCLI` (the dispatch, the client call, the two output lines, `--no-wait`); `TestSettleStartsAfterTheLastLegacyUnit` (a `before` unit whose final store call returns an ambiguous error while the server-side effect is still outstanding (a fake store that applies the delete 5 s after returning a timeout); the unit calls `done()` at `T`; the marker is 10 min old; activation happens no earlier than `T + 60 s`; the late delete lands before activation and after it no legacy write exists); `TestActivationRequiresAValidatedMarker` (absent marker; a read failure; no units; 60 s pass → still draining, not on; a successful absent read → before; a successful present read → draining then on after the settle; a restart into a 10 min old marker → draining, then on 60 s after the process start, not at once; on activation `control/lifetimes-on.json` is written create-only before any lifetime write); `TestDispatchFollowsTheUnit` (a `before` unit's create, delete, detach and `readDeletedRecord` take the legacy path even after the marker; an `on` unit's take the lifetime helpers); `TestDetachedContinuationsOwnTheirUnit` (import upload and `runFrom`: the handler returns, the unit stays admitted until the job's last write, exactly one `done()`). |
| - [ ] **Step 2: Run to verify they fail**: `go test ./internal/console -run 'Cutover|Draining|Activation|MidFlight|WorkersLoop|MarkerFailure' -count=1`, `go test . -run 'SystemLifetimes' -count=1`. |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run to verify they pass.** |
| @@ -104,7 +104,7 @@ The verdict, exactly, with `owned := slot != nil && slot.Account == rec.Account |
| **Interfaces:** |
| - Tables (`account` is the sqid): |
| - - `host_allocations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, state TEXT, body BLOB, predecessor_etag TEXT, stale_etag TEXT, created_at INTEGER)` |
| + - `host_allocations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, state TEXT, body BLOB, predecessor_etag TEXT, stale_etag TEXT, purged_at INTEGER, created_at INTEGER)` (`purged_at` is Task 4's purge intent: set durably before the purge touches any slot or record; `PurgePendingLifetimes(account, app, host, now)` and `PurgePendingLifetimesForApp(account, app, now)` set it on every non-terminal row) |
| - `record_retirements (op_id TEXT PRIMARY KEY, kind TEXT, account TEXT, app TEXT, host TEXT, incarnation INTEGER, record_etag TEXT, slot_etag TEXT, slot_intent TEXT, state TEXT, created_at INTEGER)` (`kind` is `normal`, `orphan` or `cleanup`) |
| - `record_migrations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, expect_etag TEXT, marked_etag TEXT, state TEXT, created_at INTEGER)` |
| - Classification in `claimSQLTables`: `{Sqid: "account", App: "app"}` for all three. |
| @@ -112,7 +112,7 @@ The verdict, exactly, with `owned := slot != nil && slot.Account == rec.Account |
| - On `*Store`: `BeginAllocation(a Allocation) error`, `SetAllocationState(opID, state string) error`, `PendingAllocations() ([]Allocation, error)`, `LastIncarnation(host string) (uint64, error)` (the highest `incarnation` over every state but `void`; 0 when none); `BeginRetirement(r Retirement) error`, `SetRetirementState(opID, state string) error`, `PendingRetirements() ([]Retirement, error)`; `BeginMigration(m Migration) error`, `SetMigrationMarkedETag(opID, etag string) error`, `SetMigrationState(opID, state string) error`, `PendingMigrations() ([]Migration, error)`; `PendingForHost(host, exceptOpID string) (int, error)`. |
| - `type Allocation struct { OpID, Account, App, Host string; Incarnation uint64; State string; Body []byte; PredecessorETag, StaleETag string }`, `type Retirement struct { OpID, Kind, Account, App, Host string; Incarnation uint64; RecordETag, SlotETag, SlotIntent, State string }`, `type Migration struct { OpID, Account, App, Host, ExpectETag, MarkedETag, State string }`. |
| -- [ ] **Step 1: Write the failing tests**: `TestLifetimeTablesRoundTrip`; `TestLastIncarnationSkipsVoid` (pending 2 and void 3 → 2); `TestPendingForHostCountsAllThreeExceptOwn`; `TestClaimStoreOpensWithLifetimeTables`. |
| +- [ ] **Step 1: Write the failing tests**: `TestLifetimeTablesRoundTrip`; `TestLastIncarnationSkipsVoid` (pending 2 and void 3 → 2); `TestPendingForHostCountsAllThreeExceptOwn`; `TestPurgedAtMarksEveryPendingRow`; `TestClaimStoreOpensWithLifetimeTables`. |
| - [ ] **Step 2: Run to verify they fail.** |
| - [ ] **Step 3: Implement.** |
| - [ ] **Step 4: Run** `go test ./internal/console -run 'LifetimeTables|LastIncarnation|PendingForHost|ClaimStoreOpensWithLifetimeTables' -count=1`. |
| @@ -133,7 +133,7 @@ The verdict, exactly, with `owned := slot != nil && slot.Account == rec.Account |
| - `func (s *Server) admit(host, ownOpID string) error`: `errLifetimeSettling` when `PendingForHost(host, ownOpID) > 0`. |
| - `func (s *Server) classifyLocked(ctx context.Context, acct, app, host string) (provision.Observation, error)`: `s.judge.Observe(ctx, s.cfg.Releases, acct, app, host)` (Task 6: record first, slot after, bounded rounds); the error is non-nil only with `Verdict == LifetimeUnreadable`. |
| - `func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record) (*provision.Record, error)` (takes `withHostname`) and `func (s *Server) createRecordWithLifetimeLocked(ctx context.Context, rec provision.Record) (*provision.Record, error)` (assumes it): the one protocol every claims-off creation path runs; the create, seed and import doors call the locking form; the attach door calls the locked form inside its own exclusion (below). Returns the record as written. Errors: `release.ErrCASConflict` (a valid record under this key), `*hostHeldError` (`errors.As`; carries `*apexHold` for `hostHeldSentence`: a live slot owned by another key gives `&apexHold{Holder: slot.App, OtherAccount: slot.Account if ≠ rec.Account}`), `errLifetimeSettling`, `errLifetimePending`, or a read error from step 2 (nothing persisted). |
| - - `func (s *Server) fenceRecord(ctx context.Context, acct, app, host string, op fenceOwner) (res fenceResult, obs provision.Observation, err error)` with `type fenceOwner struct { Tombstone bool; MarkerID string }` (what this operation's own outstanding write would look like) and `fenceResult` one of `fenced` (a body that is neither this operation's tombstone nor its marker was rewritten with its sequence introduced or advanced: the caller terminates on `obs`), `ownWriteWon` (the re-read found this operation's tombstone, or a record marked with `MarkerID`: the caller does **not** terminate; it dispatches again from `obs` and the operation resumes, retirement at N2, migration through its dispatcher), `absent` (nothing to fence: the slot is the fence for an absent key), or an error after 3 CAS retries (the caller leaves its operation pending). Every recovery branch below and in Tasks 4 and 5 that would mark an operation `refused`, `void` or `superseded` because the record is not what the operation captured calls `fenceRecord` first and acts on its result. |
| + - `func (s *Server) fenceRecord(ctx context.Context, acct, app, host string, op fenceOwner) (res fenceResult, obs provision.Observation, err error)` with `type fenceOwner struct { Tombstone bool; MarkerID string }` (what this operation's own outstanding write would look like) and `fenceResult` one of `fenced` (a body that is neither this operation's tombstone nor its marker was rewritten with its sequence introduced or advanced: the caller terminates on `obs`), `ownWriteWon` (the re-read found this operation's tombstone, or a record marked with `MarkerID`: the caller does **not** terminate; it dispatches again from `obs` and the operation resumes, retirement at N2, migration through its dispatcher), `absent` (nothing to fence: the slot is the fence for an absent key), or an error after 3 CAS retries (the caller leaves its operation pending). Every recovery branch below and in Tasks 4 and 5 that would mark an operation `refused`, `void` or `superseded` because the record is not what the operation captured calls `fenceRecord` first and acts on its result. **A purged allocation** (`purged_at` set, Task 4) is classified before any other rule: whatever the slot and record say, the row becomes `void` without a create replay; if the slot is live at this row's incarnation it is retired by CAS (kind `normal`, the retirement row written by the purge), and a record at that incarnation is left to the purge's own retirement; `TestPurgedAllocationNeverReplays` pins it (a live slot and no record; a slot and a record; no slot). |
| - `func (s *Server) recoverLifetimes(ctx context.Context) error`, `func (s *Server) StartLifetimeRecoveryPass(ctx context.Context)` (Task 0's worker lifecycle: started once, loops every 60 s for the life of the process, each iteration admitted as a mutating unit and doing work only in `on`). |
| The protocol, inside `withHostname(host)`: |
| @@ -176,8 +176,11 @@ Door mappings: the create door: `*hostHeldError` → 409 `hostHeldSentence(host, |
| ### Task 4: Retirement through a tombstone, from every claims-off removal |
| **Files:** |
| -- Create: `internal/console/lifetimes_retire.go`, `internal/provision/lifetimes_retire.go` (the **retirement core** `provision.RetireRecordWithLifetime(ctx, s release.Store, judge *LifetimeJudge, acct, app, host string, kind RetirementKind, expect RetireExpectation) error`: observe, tombstone, retire the slot, delete, as this task specifies; the console's `retireRecordWithLifetime` wraps it in the hostname exclusion; a box calls it under its own per-host hold lock) |
| -- Modify: `internal/release/trash.go` (`purgeApp` gains `retire func(ctx context.Context, acct, app, host string) error`; **before** the key sweep it calls `retire` for every key under the app prefix of the instance-record shape (`provision.RecordKey`), in sorted order; nil → today's unconditional sweep; `PurgeApp` and `PurgeAppIfTrashedAt` take the parameter), `internal/console/ownership.go` (`handlePurgeApp`: `retire` = `retireRecordWithLifetime(..., normal, retireAny)` for an `on` unit, nil for a `before` unit), `internal/console/expirypass.go` (`purgeOne`: the same by its iteration's unit), `internal/domains/reconcile.go` (the box's auto-purge at line 271: `retire` = `provision.RetireRecordWithLifetime(ctx, s, cfg.Judge, ...)` under `WithHostHoldLock`; nil when `cfg.Judge == nil`); a purge interrupted after k retirements is re-run as today (retirement with `retireAny` is idempotent: absent → committed); a pending allocation of a purged host finds its slot retired and voids (Task 3's recovery); the hostname is reusable by another app at incarnation N+1, |
| +- Create: `internal/console/lifetimes_retire.go`, `internal/host/purge_retire.go` (the box's CAS-only retirement, below) |
| +- Modify: `internal/release/trash.go` (`purgeApp` gains `retire func(ctx context.Context, acct, app, host string) error`; when `retire != nil` every key of the instance-record shape (`provision.RecordKey`) under the app prefix is **excluded from the `s.Delete` sweep** and handed to `retire` instead, in sorted order, **before** the sweep of the remaining keys; a `retire` error aborts the purge before any sweep (the next attempt re-runs it; every step is idempotent); nil → today's unconditional sweep; `PurgeApp` and `PurgeAppIfTrashedAt` take the parameter), `internal/console/ownership.go` and `internal/console/expirypass.go` (`handlePurgeApp` and `purgeOne`: for an `on` unit `retire = s.purgeRetire` (below); for a `before` unit nil), `internal/domains/reconcile.go` (`Config.Retire func(ctx, acct, app, host string) error` and `Config.OnMarker func(ctx) bool`; the box's auto-purge at line 271 passes `cfg.Retire` when `cfg.OnMarker(ctx)` is true, else a `retire` that deletes the record **conditionally** (`DeleteIfMatch` on the ETag the listing returned, after re-reading `lifetimes-on.json` once more; the marker now present → return an error so the pass retries lifetime-aware); the agent wires both from `host.purgeRetire`) |
| +- **The console's purge retirement** `func (s *Server) purgeRetire(ctx, acct, app, host) error`, under `withHostname`: (0) **pending allocations first**: `s.store.PurgePendingLifetimes(acct, app, host, now)` sets `purged_at` on every pending allocation row of this hostname (durable; Task 3's recovery treats a row with `purged_at` as **`void`** whatever its slot says, never replays its create, and retires its slot if the slot is live at that row's incarnation); then each such slot: `retireSlotLocked` by CAS; (1) the record: `retireRecordWithLifetime(ctx, acct, app, host, normal, retireAny)` (absent → committed). Because the hook is called per **listed record key** only, `purgeApp`'s console caller first runs `s.store.PurgePendingLifetimesForApp(acct, app)` over every pending row of the app (hosts with no record yet included), so an allocation interrupted before its record, or before its slot, is voided too. |
| +- **The box's purge retirement** `func purgeRetire(ctx, s release.Store, judge *provision.LifetimeJudge, acct, app, host string) error` (`internal/host/purge_retire.go`): no migration, no journal, no lock beyond the process-local `WithHostHoldLock`; one judged observation, then: a record marked `incarnation_migrating` → return `errPurgeDeferred` (the console owns the migration; the next pass retries); a live slot owned by this record → `WriteHostIncarnation` CAS (`SlotETag`) to `{N, retired}` then `DeleteIfMatch(record, ETag)`; a retired slot with a present body → `DeleteIfMatch`; no slot (a legacy fieldless record) → `DeleteIfMatch(record, ETag)` (a console migration that started meanwhile changed the ETag, so the delete fails and the next pass sees the marker); absent → done. **Any CAS loss returns an error**: the purge aborts before its sweep and re-runs on the next `domains.Reconcile` pass; every branch is idempotent, so a box crash at any point is recovered by the next pass, and a console retirement or recreation that wins a CAS simply makes the box re-observe. Nothing here needs the console's journal. |
| +- **Witness of the purge rules:** a purge interrupted after k retirements resumes; a box that retired H and paused before its sweep while H was recreated at N+1 by another app does not touch H (instance keys are not in the sweep); a pending allocation with a slot and no record is voided and its slot retired; one with no slot is voided; a pre-on box purge of a record the console touched after the listing fails its CAS and is retried lifetime-aware; the hostname is reusable by another app at incarnation N+1, |
| - Modify: `internal/console/hostclaimops.go` (claims-off `ref.Family == "instance"` → `retireRecordWithLifetime(ctx, ref.Account, ref.App, ref.Host, ref.Expect)`; `provision.RetireRef` gains `Expect retireExpectation`-shaped fields through a new `provision.RetireExpectation{Absent bool; ETag string; Incarnation uint64}` so each caller states its own), `internal/console/claim_retire.go` (`retireInstanceRecord(ctx, account, app, host, by, incarnation string, expect provision.RetireExpectation)`; its claims-off callers: `apiinstances.go:1684` (the sibling undo) with `{Incarnation: created.Incarnation}` through the locking `retireRecordWithLifetime`; **detach does not go through it on the claims-off path**: `apidomains.go:989` and `domainattach.go:349` call Task 3's `detachFirstRouteLocked`, which calls `retireRecordWithLifetimeLocked(..., retireAny)` inside the exclusion it already holds; the claims-on detach keeps today's `retireInstanceRecord` chain; `domainattach.go:557` (`rollbackAttach`) never retires on the claims-off path; `apiinstances.go:2245` is not reached on claims-off: `importDeleteRule` answers every claims-off delete), `internal/console/apiinstanceimport.go` (`readDeletedRecord` returns `(provision.Observation, error)` through `s.judge.Observe`; `importDeleteRule`: `!obs.Present` → `retireRecordWithLifetime(..., retireExpectAbsent)`; `obs.Verdict` stale or corrupt → `deleteRead(..., retireExpectETag(obs.ETag))`; valid → the import logic as today, and its ordinary delete `deleteRead(..., retireExpectETag(obs.ETag))`; `deleteRead` passes the expectation through: `ErrCASConflict` keeps the `again` re-decision; `errLifetimeSettling` → 409 settling; `errRetireRefused` → 409 its sentence), `internal/console/seedfinish.go` (`finishSeedOnce`'s final delete → `retireRecordWithLifetime(ctx, sqid, app, host, retireExpectETag(etag))`), `internal/console/apiinstances.go` (`handleAPIInstanceDelete`: `errRetireRefused`, `errLifetimeSettling` → 409), `internal/console/domainattach.go` and `internal/console/apidomains.go` (detach: `errLifetimeSettling` → 409 `claimDetachInProgressMsg`; `errRetireRefused` → 409 its sentence), `internal/console/lifetimes_recovery.go` |
| - Test: `internal/console/lifetimes_retire_test.go` |
| @@ -207,7 +210,7 @@ Inside `withHostname(host)`: `admit(host, "")`; `obs := classifyLocked` (unreada |
| - kind **orphan**, `pred`: absent or any present body → O1 (the slot, then the body if present), `committed`. |
| - kind **cleanup**, `pred`: any observation (absent, corrupt, stale, or a valid body at N: the DELETE this operation answers was for this lifetime, and under the retired slot every body is stale) → C1 then C2 on the current body's ETag when present, `committed`. |
| -- [ ] **Step 1: Write the failing tests** through the real doors and the function: `TestPurgeRetiresEveryInstanceRecord` (browser purge, the expiry pass and the box auto-purge: every record under the app has its slot retired and a tombstone written before the sweep; a purge interrupted after k retirements resumes and completes; a pending allocation of a purged host voids on recovery; another app then creates the hostname at N+1; a `before` unit's purge is today's sweep); |
| +- [ ] **Step 1: Write the failing tests** through the real doors and the function: `TestPurgeRetiresEveryInstanceRecord` (browser purge, the expiry pass and the box auto-purge: every record under the app has its slot retired and a tombstone written, and no instance key is in the sweep; a purge interrupted after k retirements resumes and completes; a pending allocation with a live slot and no record is voided with `purged_at` and its slot retired, recovery never replays it; one interrupted before its slot is voided; a box that retired H then paused while another app recreated H at N+1 does not delete the successor; a box purge before the on-marker deletes only by `DeleteIfMatch` and loses to a console write; a box crash mid-retirement is completed by its next pass; a box and the console retiring the same record concurrently end with one tombstone and one retired slot; a `before` unit's console purge is today's sweep); |
| - `TestDeleteDoorRetiresThroughTheTombstone`; `TestSlotRetiresBeforeTheKeyIsAbsent`. |
| - `TestDeleteDoorAbsentPathAdmitsAndRetiresOrphans` (initial absence; absence found on the judge's re-read; a record created since is untouched and the door still answers "deleted"). |
| - `TestStaleCleanupRetiresALiveOwnedSlotFirst` (a fieldless late body beside `{1, live, this key}`: the slot is `{1, retired}` before the body is removed; a blocked incarnation-1 create-only write released afterwards lands stale; the same for a corrupt body). |
| @@ -357,6 +360,6 @@ These are §11's witnesses plan 1 owns. Every witness that needs a holding publi |
| ## Self-review notes |
| - §6.4 coverage: slot, owner key and tombstone form (T1, T4), create and recovery (T3), every claims-off removal with an explicit expectation and its recovery (T4), migration, its fence and recovery (T5), the observation protocol in every claims-off consumer (T1, T6), `move_seq`, the setters that install a lifetime with its hold and the existing-row rules (T7), publication by the lifetimes writer and §6.4's fold order (T8), witnesses (T9). Deferred by name: holding entries and their five-minute validity, the probe mode, the different-target alias rule and every holding-dependent witness (plan 3); the door, token, job, reservations and reports (plan 2); cleanup (plan 4). The stated additions to §6.4's text are the uncached slot read and the cutover (an operator step with an admission gate, a rollout prerequisite); the fold rule is §6.4's own. |
| -- Names used across tasks: `LifetimeOf`, `LifetimeVerdict`, `HostIncarnation.Valid`, `ErrSlotMalformed`, `ErrRecordCorrupt`, `ReadHostIncarnation`, `WriteHostIncarnation`, `TombstoneBody`, `GetRecordRaw`, `Observation{Present, Rec, Raw, ETag, Slot, SlotETag, Verdict, Err}`, `LifetimeJudge.Observe/ObserveWith`, `ReadRecordJudged`, `List*Judged`, `withHostname`, `admit`, `classifyLocked`, `createRecordWithLifetime`, `retireRecordWithLifetime`, `provision.RetireExpectation` (`retireExpectAbsent`, `retireAny`, `retireExpectETag`, `retireExpectIncarnation`), `Retirement.Kind` (`normal`, `orphan`, `cleanup`), `WithJudge`, `JudgeFrom`, `ErrLifetimeExhausted`, `fenceRecord`/`fenceResult`/`fenceOwner`, `provision.RetireRecordWithLifetime`, `purgeApp(..., retire)`, `quarantineCopy`, `ConvertRetirementKind`, `cutoverMarker`, `cutoverState` (`cutoverUnknown/Before/Draining/On`), `cutoverGate` (`markerSeen`, `lastDone`), `admittedUnit`, `lifetimesAdmit`, `recordCtx`, `activateWhenDrained`, `cutoverStatus`, `StartLifetimesCutoverWatch`, `handleAPILifetimesCutover`, `SystemLifetimesCutover`, `retireSlotLocked`, `slotDisposition`, `migrationDispatch`, `WithLifetimes`, `WithRecordSequence` (existing), `createRecordWithLifetimeLocked`, `retireRecordWithLifetimeLocked`, `attachFirstRouteLocked`, `rollbackAttachLocked`, `detachFirstRouteLocked`, `lifetimeSettle`, `migrateHostLifetimeLocked`, `errLifetimeSettling`, `errLifetimePending`, `*hostHeldError`, `errRetireRefused`, `ErrRecordMigrating`, `WithMigration`, `ReconcileConfig.Judge/PairOf/DeleteIfHold`, `moveCfg.Judge/ReleaseHoldSeq`, `releaseHeldSeq`, `registry.Pair`, `SetHoldIfSeq`, `ReleaseHoldSeq`, `InstallLifetimeIf`, `Route.Incarnation/ServingSeq`, `MoveState.Seq`, `edgeHost.Incarnation/ServingSeq/TargetHost`. |
| +- Names used across tasks: `LifetimeOf`, `LifetimeVerdict`, `HostIncarnation.Valid`, `ErrSlotMalformed`, `ErrRecordCorrupt`, `ReadHostIncarnation`, `WriteHostIncarnation`, `TombstoneBody`, `GetRecordRaw`, `Observation{Present, Rec, Raw, ETag, Slot, SlotETag, Verdict, Err}`, `LifetimeJudge.Observe/ObserveWith`, `ReadRecordJudged`, `List*Judged`, `withHostname`, `admit`, `classifyLocked`, `createRecordWithLifetime`, `retireRecordWithLifetime`, `provision.RetireExpectation` (`retireExpectAbsent`, `retireAny`, `retireExpectETag`, `retireExpectIncarnation`), `Retirement.Kind` (`normal`, `orphan`, `cleanup`), `WithJudge`, `JudgeFrom`, `ErrLifetimeExhausted`, `fenceRecord`/`fenceResult`/`fenceOwner`, `purgeApp(..., retire)`, `(*console.Server).purgeRetire`, `host.purgeRetire`, `errPurgeDeferred`, `PurgePendingLifetimes`/`PurgePendingLifetimesForApp`, `purged_at`, `lifetimesOnKey`, `domains.Config.Retire/OnMarker`, `quarantineCopy`, `ConvertRetirementKind`, `cutoverMarker`, `cutoverState` (`cutoverUnknown/Before/Draining/On`), `cutoverGate` (`markerSeen`, `lastDone`), `admittedUnit`, `lifetimesAdmit`, `recordCtx`, `activateWhenDrained`, `cutoverStatus`, `StartLifetimesCutoverWatch`, `handleAPILifetimesCutover`, `SystemLifetimesCutover`, `retireSlotLocked`, `slotDisposition`, `migrationDispatch`, `WithLifetimes`, `WithRecordSequence` (existing), `createRecordWithLifetimeLocked`, `retireRecordWithLifetimeLocked`, `attachFirstRouteLocked`, `rollbackAttachLocked`, `detachFirstRouteLocked`, `lifetimeSettle`, `migrateHostLifetimeLocked`, `errLifetimeSettling`, `errLifetimePending`, `*hostHeldError`, `errRetireRefused`, `ErrRecordMigrating`, `WithMigration`, `ReconcileConfig.Judge/PairOf/DeleteIfHold`, `moveCfg.Judge/ReleaseHoldSeq`, `releaseHeldSeq`, `registry.Pair`, `SetHoldIfSeq`, `ReleaseHoldSeq`, `InstallLifetimeIf`, `Route.Incarnation/ServingSeq`, `MoveState.Seq`, `edgeHost.Incarnation/ServingSeq/TargetHost`. |
| - Review Focus 1 to 5 are pinned by T3+T5, T6, T6+T3, T7, T7+T9. |
| - Store primitives, as verified in the reviews: `Store.Put(..., "")` is create-only on both backends; `WriteRecord` with `before={}` and `etag=""` is the create and needs the real CAS base otherwise; `DeleteIfMatch` treats absence as success and has no unconditional fallback; `OfflineAbort` on a reverting record returns `errRecordUnchanged`, a success without a write; `AccountBySqid` returns `(nil, nil)` for an unknown sqid. |