amadan
plan: self-serve moves 1, rev 18 (Astra review 17: every box sweep step fenced at the marker, one app-row insertion helper mints the lifetime, marker checked under the create lock, best-effort cross-key cleanup, lock table completed, admitted startup materialisation, three-valued restore evidence)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+8
−8
docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md+8 −8
CI cancelled — superseded by a newer push — run details
CI log
=== gate === .amadan/test-gate.sh amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md| index ce176da8e..d511be5a3 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| @@ -2,7 +2,7 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 17, 2026-10-22. Rev 16's review (`selfserve-plan1-review16.md`: 5, 10, 17, 18, 19 narrowed; 20, 21, 22 new) is answered: the create protocol checks the app's lifecycle **under the app lock** (present, not trashed, the same `lifetime` the door read) before allocating (5); a box purge re-reads the cutover marker **immediately before every conditional delete** and abandons the purge when more than 30 s passed since that check, so a sweep paused across the cutover deletes nothing numbered (10); cancellation writes the orphan retirement row and the allocation's `void` in **one SQLite transaction** and the retirement names the allocation it cancels, so no two pending rows block each other (17); the app lifetime is a non-repeating `apps.lifetime` id captured by `BeginAllocation` (18); the stale-body promise is narrowed to the same key and the create's step 5 also deletes a stale body under the retired slot's owner key (19); a **lock table** states what every wrapper acquires and every `Locked` helper assumes, and the attach door takes the app lock outside its hostname exclusion (20); the purge's absent-record branch acts only on a slot **owned by the purged app** (21); `settleApp` and `MaterializeApps` count only judged-valid instance records as restore evidence (22). Rev 15's review (`selfserve-plan1-review15.md`: 13, 14, 15, 16 closed; 5, 10, 17, 18, 19 open) is answered with **one lock order** instead of a snapshot: the create protocol, allocation recovery and the console's purge all take `s.lockApp(acct, app)` and then `withHostname(host)`, so a purge and a create or recovery of the same app never interleave, and the purge observes and retires each instance **inside** the same locked step (no delayed callback can see a successor; a legacy record is retired by **ETag**); an allocation is bound to the **app lifetime** that authorised it (`app_lifetime` = the apps row's `created_at`), so a reused name never replays an old allocation; **boxes stop auto-purging once the cutover marker exists** and the console's expiry pass runs the trash sweep under the app lock, so after the cutover the only purger is the one that can see pending allocations; a slot-bound orphan entry point (`retireSlotOrphanLocked`) exists and the void path's cleanup is stated honestly (a late stale body stays until the next create of the hostname removes it in step 5; a late slot PUT leaves a live orphan the orphan rule reclaims). Rev 14's review (`selfserve-plan1-review14.md`: 9, 11, 12 closed; 5, 10, 13, 14, 15, 16 the purge protocol) is answered by **removing the purge protocol**: four rounds of a lifetime-aware purge each found the next race (snapshot vs recovery, in-flight box sweeps, legacy incarnation 0, the app lock, guard ordering, the schema), so purge goes back to **today's sweep** with one conditional step (an instance record is deleted by `DeleteIfMatch` on an ETag a `Get` captured just before), on the console under the existing `s.lockApp` mutex its callers already hold, on the box as today; and the lifetime protocol **absorbs the consequences** with two rules it already nearly had: a live slot whose record is absent and for which this console holds no pending operation is an **orphan**, retired as kind `orphan` by the next create that meets it (so a purged app's slots never block a hostname); and allocation recovery **never replays a create for a trashed or purged app** (it retires the slot, and anything that landed at its incarnation, as `orphan`, and voids the row), so a late record or slot PUT of a purged app's allocation is cleaned by the same recovery. No generation table, no slot fence, no on-marker, no console trash sweep. Rev 13's review (`selfserve-plan1-review13.md`: findings 1 and 8 closed; 5, 9, 10, 11, 12 the purge) is answered by **taking boxes out of purging once lifetimes are on**: a box that has seen `lifetimes-on.json` never purges an app again (the console's expiry pass gains the store-driven trash sweep the box runs today), so no purger exists that cannot see the console's pending allocations; before the on-marker a box purge deletes an instance record only by `DeleteIfMatch` on an ETag it captured **before** its final marker check. The console's purge runs under the app mutation lock, records the **generation** it observed for every host (slot incarnation or record ETag) before any destructive step, and every retirement states that expectation (never `retireAny`), so a delayed callback cannot touch a successor; a pending allocation is terminalised by **fencing its slot** (a CAS write of a retired slot at its incarnation, against absence or the predecessor's ETag) so an outstanding allocation PUT can never land, then `void`; the purge calls the **locked** retirement under its own `withHostname`; the box has no retirement code at all; the witnesses are split into console and box expectations with their real terminal states. Rev 12's review (`selfserve-plan1-review12.md`: findings 2, 3, 4, 6, 7 closed; 1 a witness contradiction; 5, 8, 9 the purge) is answered: the restart witness no longer activates "at once"; the console writes a durable **`control/lifetimes-on.json`** as the first act of `on`, and that marker, not judge availability, is a box's authority for any slot mutation; purge **excludes instance-record keys from its sweep** whenever a retire hook is installed; the console's purge **voids the app's pending allocations durably** (`purged_at`) and retires their slots before touching records, and recovery treats a purged pending row as `void`; the box's purge needs **no migration, no journal and no cross-process lock**: it retires by CAS alone (slot CAS to `retired`, then `DeleteIfMatch` on the observed record ETag; a marked record is skipped; any CAS loss returns an error and the next pass retries idempotently), and before the on-marker it deletes only by `DeleteIfMatch` on the listed ETag after re-reading the marker. Rev 12's review (`selfserve-plan1-review11.md`, findings 1 to 7) is answered: the storage settle is anchored **after the last `before` unit released admission** (and after the process start on a restart), so a legacy request that was fully received before its client timed out has completed before activation; activation requires a **validated marker read** (`markerSeen`), never uncertainty alone; app purge is **lifetime-aware** on every path (browser, expiry pass, box auto-purge) through a retirement core in `provision` that `release.PurgeApp` calls for every instance record before its sweep; an in-scope valid own record defeats only the record-gone reason and still reaches the app-trash decision; the dispatch of create, delete, detach and `readDeletedRecord` is stated per admitted unit; both detached continuations (`runImportUpload`, `createFrom → runFrom`) are wired; the Task 3 witness no longer says draining reads are legacy; `claimAliasUnbacked` is named in the audit. Rev 11, 2026-10-22. Rev 10's review (`selfserve-plan1-review10.md`, findings 1 to 4) is answered: the cutover gains an **admission gate**: every claims-off unit of record work (a request's handler, a detached continuation, a worker iteration) is admitted under a counted gate while the state is `before`; the cutover door closes admission synchronously, refuses new work with 503, **waits for the admitted count to reach zero**, then starts the storage settle, then activates; so no unit admitted before the marker ever runs beside the lifetime protocol, and a unit never changes mode mid-flight. `on` is **latched**: once observed it is never left; at startup the state is `unknown` until the first successful marker read and every claims-off record door (reads and mutations) answers 503 until then, so a marker-read failure never restores legacy interpretation. The removal rule separates lifetime validity from placement eligibility: a valid own record that is **in this pass's scope** protects the legacy collision; a valid own record **outside** the scope follows today's removal rules (`KeepParticipant`, the app-trash rule), so offline finishing removes the source row as today; absence or staleness authorises the new path. One worker lifecycle (start once, loop, check the state each iteration), one migration interval, one context contract (`recordCtx`), the CLI command, the retry error, the three root constructions and the copy inventory are stated. Rev 10, 2026-10-22. Rev 9's review (`selfserve-plan1-review9.md`, findings 1 to 5) is answered: the cutover is a **mode** with a drain: from the marker's time until the window ends the console refuses every claims-off record mutation (503), so no legacy write can be admitted into the window, and the gate, the judge's installation, the recovery pass and the marker's failure modes are one contract with an implementation task for the operator door; the removal loop decides from the row's **own** record (observed by its own key), so a valid legacy owner in a hostname collision is kept as today and only an established absent or stale own record retires the row, with the keep view bypassed on that evidence and replacement stated as two passes; `fenceRecord` returns a classification so an own tombstone or marker that won resumes the operation instead of terminating it; the audit's allowlist names the two remaining claims-on callers of `hostops.ReadRecord` and is pinned both ways; and the remaining contracts are settled (the detach sentence, cleanup recovery through C2, corrupt index candidates, the root restart readers' judge). Rev 8's review (`selfserve-plan1-review8.md`, findings 1 to 5) is answered: the rollout guarantee becomes an **enforced cutover** (an operator marker written after the old console has stopped, plus the settle window, with the storage assumption stated) rather than a timer alone; quarantine copies the corrupt bytes and **never deletes under a live slot**: the retirement converts durably to the cleanup kind and retires the slot first, and a conditional-delete conflict re-classifies instead of counting as absence; a provisioned row owned by another record key than the hostname's current record is retired as "record gone" so a cross-owner recreation cannot leave a lagging box serving the predecessor; `RecordIndex.Find` judges its candidate keys before deciding ambiguity; and the remaining contradictions are resolved (the root scope sentence, one detach chain, the locked retirement's callers, reconcile's one sequence, the audit's enumerated list, the fence witness split). Rev 7's review (`selfserve-plan1-review7.md`, findings 1 to 6) is answered: a stated rollout guarantee closes the late legacy delete (one console per deployment, a settle window before any termination, old consoles never reintroduced over lifetimes records); the first-route attach, its rollback and detach run **inside** the hostname exclusion from the claim write to the record write, through locked variants of the create and retirement helpers; root's proof and restart record readers take the box's judge (the scope statement is corrected: only `ClaimRootHTTP`'s claims-on readers are slot-free); a corrupt body is quarantined (its bytes kept under `control/quarantine/`, the key removed conditionally) so the retirement or migration proceeds as for absence instead of promising a fence that cannot parse it; `provision.WithLifetimes` is the one context helper that carries the judge and enables sequencing, the runs worker wraps its detached execution context, the audit distinguishes approved judged wrappers from raw readers, the fence witnesses state their timing, exhaustion is witnessed at every transition, and the sibling undo has one dispatch path. Earlier reviews: rev 1 (`after236/selfserve-plan1-review.md`, 16 findings), rev 2 (F1 to F12), rev 3 (R1 to R10), rev 4 (1 to 9), rev 5 (1 to 6), rev 6 (`selfserve-plan1-review6.md`, findings 1 to 5). Rev 7 answers rev 6's five: every claims-off record write runs with `WithRecordSequence`, so `record_seq` exists after a record's first write and an ETag never recurs, and recovery writes a **fence** (a sequencing rewrite) before it terminates an operation on evidence of a changed record; one migration dispatcher classifies the observation at every entry, so a resume can never supersede its own marker; the attach door's claim rollback runs inside the hostname exclusion and never releases a claim a pending operation or a valid record needs; the judge reaches every mutation context (the console's top-level handler wraps every request, browser and API alike; the seed callback wraps its own context; the full-record updater's re-read is judged; `RecordIndex.Find` and `FindRecordByHost` take the judge from the context), and the audit is by AST; the remaining contracts (the running N1 table, a slot-retirement helper, one expectation type, listed-key absence, exhaustion at read, the import witness's state, the producer-state publication tests) are answered. Rev 5 removed the slot cache (every judgement reads the slot fresh, after the record, in bounded rounds) and made a successor lifetime's hold one row write. Rev 6 answers rev 5's six: a retirement persists its **kind** (normal, orphan, cleanup) and each kind has its own steps and recovery table; a migration's marker-CAS conflict is classified by a re-read for both the bound and the unbound form, so an own marker that landed is resumed, never superseded; every `retireInstanceRecord` caller states its expectation (the sibling undo names the lifetime it created; detach deletes what is current; the attach door never asks for a record rollback over a lifetime failure); a judge carried on the context makes `updateRecordSeeding`, and so every `SetRecord*` writer, refuse a stale record, and every box reader (Phase 1's included) and the import journal's readers go through the judge; the provenance mark is dropped (the fold compares pairs between unstamped entries, a pair-less entry is `(0, 0)` and equal pairs fall to the existing rule, exactly §6.4); the remaining interface gaps are answered. |
| +Rev 18, 2026-10-22. Rev 17's review (`selfserve-plan1-review17.md`: 10, 17, 21 closed; 5, 18, 19, 20, 22 narrowed; 23, 24 new) is answered: a box purge is fenced at **every** destructive step (`purgeApp` gains a `before func(ctx) error` hook the box fills with the marker re-read and the 30 s clock, called before each `Delete`, the final marker delete included) (23); the app lifetime is minted by **one insertion helper** every app-row insert uses (`CreateAppInRegion`, `CreateTemporaryAppInRegion`, `BackfillOpsMemberships`, the migration's `apps_new` copy, the open-time backfill), pinned by a test that scans every `INSERT INTO apps`, and the declarations agree (`TEXT`, `Allocation.AppLifetime`, the recovery wording) (18); the create's step 0 also requires a successful read of the store's trash marker as absent (5); the cross-key stale-body cleanup is **best effort within the running create** and the promise is narrowed to that (19); the initial migration pass takes app → hostname and the cross-owner orphan retirement needs only the hostname lock and its captured slot ETag, stated with its reason (20); startup materialisation runs as an admitted worker iteration, never before the gate has a state (22); restore evidence has an explicit **unknown** outcome that defers the decision (24). Rev 16's review (`selfserve-plan1-review16.md`: 5, 10, 17, 18, 19 narrowed; 20, 21, 22 new) is answered: the create protocol checks the app's lifecycle **under the app lock** (present, not trashed, the same `lifetime` the door read) before allocating (5); a box purge re-reads the cutover marker **immediately before every conditional delete** and abandons the purge when more than 30 s passed since that check, so a sweep paused across the cutover deletes nothing numbered (10); cancellation writes the orphan retirement row and the allocation's `void` in **one SQLite transaction** and the retirement names the allocation it cancels, so no two pending rows block each other (17); the app lifetime is a non-repeating `apps.lifetime` id captured by `BeginAllocation` (18); the stale-body promise is narrowed to the same key and the create's step 5 also deletes a stale body under the retired slot's owner key (19); a **lock table** states what every wrapper acquires and every `Locked` helper assumes, and the attach door takes the app lock outside its hostname exclusion (20); the purge's absent-record branch acts only on a slot **owned by the purged app** (21); `settleApp` and `MaterializeApps` count only judged-valid instance records as restore evidence (22). Rev 15's review (`selfserve-plan1-review15.md`: 13, 14, 15, 16 closed; 5, 10, 17, 18, 19 open) is answered with **one lock order** instead of a snapshot: the create protocol, allocation recovery and the console's purge all take `s.lockApp(acct, app)` and then `withHostname(host)`, so a purge and a create or recovery of the same app never interleave, and the purge observes and retires each instance **inside** the same locked step (no delayed callback can see a successor; a legacy record is retired by **ETag**); an allocation is bound to the **app lifetime** that authorised it (`app_lifetime` = the apps row's `created_at`), so a reused name never replays an old allocation; **boxes stop auto-purging once the cutover marker exists** and the console's expiry pass runs the trash sweep under the app lock, so after the cutover the only purger is the one that can see pending allocations; a slot-bound orphan entry point (`retireSlotOrphanLocked`) exists and the void path's cleanup is stated honestly (a late stale body stays until the next create of the hostname removes it in step 5; a late slot PUT leaves a live orphan the orphan rule reclaims). Rev 14's review (`selfserve-plan1-review14.md`: 9, 11, 12 closed; 5, 10, 13, 14, 15, 16 the purge protocol) is answered by **removing the purge protocol**: four rounds of a lifetime-aware purge each found the next race (snapshot vs recovery, in-flight box sweeps, legacy incarnation 0, the app lock, guard ordering, the schema), so purge goes back to **today's sweep** with one conditional step (an instance record is deleted by `DeleteIfMatch` on an ETag a `Get` captured just before), on the console under the existing `s.lockApp` mutex its callers already hold, on the box as today; and the lifetime protocol **absorbs the consequences** with two rules it already nearly had: a live slot whose record is absent and for which this console holds no pending operation is an **orphan**, retired as kind `orphan` by the next create that meets it (so a purged app's slots never block a hostname); and allocation recovery **never replays a create for a trashed or purged app** (it retires the slot, and anything that landed at its incarnation, as `orphan`, and voids the row), so a late record or slot PUT of a purged app's allocation is cleaned by the same recovery. No generation table, no slot fence, no on-marker, no console trash sweep. Rev 13's review (`selfserve-plan1-review13.md`: findings 1 and 8 closed; 5, 9, 10, 11, 12 the purge) is answered by **taking boxes out of purging once lifetimes are on**: a box that has seen `lifetimes-on.json` never purges an app again (the console's expiry pass gains the store-driven trash sweep the box runs today), so no purger exists that cannot see the console's pending allocations; before the on-marker a box purge deletes an instance record only by `DeleteIfMatch` on an ETag it captured **before** its final marker check. The console's purge runs under the app mutation lock, records the **generation** it observed for every host (slot incarnation or record ETag) before any destructive step, and every retirement states that expectation (never `retireAny`), so a delayed callback cannot touch a successor; a pending allocation is terminalised by **fencing its slot** (a CAS write of a retired slot at its incarnation, against absence or the predecessor's ETag) so an outstanding allocation PUT can never land, then `void`; the purge calls the **locked** retirement under its own `withHostname`; the box has no retirement code at all; the witnesses are split into console and box expectations with their real terminal states. Rev 12's review (`selfserve-plan1-review12.md`: findings 2, 3, 4, 6, 7 closed; 1 a witness contradiction; 5, 8, 9 the purge) is answered: the restart witness no longer activates "at once"; the console writes a durable **`control/lifetimes-on.json`** as the first act of `on`, and that marker, not judge availability, is a box's authority for any slot mutation; purge **excludes instance-record keys from its sweep** whenever a retire hook is installed; the console's purge **voids the app's pending allocations durably** (`purged_at`) and retires their slots before touching records, and recovery treats a purged pending row as `void`; the box's purge needs **no migration, no journal and no cross-process lock**: it retires by CAS alone (slot CAS to `retired`, then `DeleteIfMatch` on the observed record ETag; a marked record is skipped; any CAS loss returns an error and the next pass retries idempotently), and before the on-marker it deletes only by `DeleteIfMatch` on the listed ETag after re-reading the marker. Rev 12's review (`selfserve-plan1-review11.md`, findings 1 to 7) is answered: the storage settle is anchored **after the last `before` unit released admission** (and after the process start on a restart), so a legacy request that was fully received before its client timed out has completed before activation; activation requires a **validated marker read** (`markerSeen`), never uncertainty alone; app purge is **lifetime-aware** on every path (browser, expiry pass, box auto-purge) through a retirement core in `provision` that `release.PurgeApp` calls for every instance record before its sweep; an in-scope valid own record defeats only the record-gone reason and still reaches the app-trash decision; the dispatch of create, delete, detach and `readDeletedRecord` is stated per admitted unit; both detached continuations (`runImportUpload`, `createFrom → runFrom`) are wired; the Task 3 witness no longer says draining reads are legacy; `claimAliasUnbacked` is named in the audit. Rev 11, 2026-10-22. Rev 10's review (`selfserve-plan1-review10.md`, findings 1 to 4) is answered: the cutover gains an **admission gate**: every claims-off unit of record work (a request's handler, a detached continuation, a worker iteration) is admitted under a counted gate while the state is `before`; the cutover door closes admission synchronously, refuses new work with 503, **waits for the admitted count to reach zero**, then starts the storage settle, then activates; so no unit admitted before the marker ever runs beside the lifetime protocol, and a unit never changes mode mid-flight. `on` is **latched**: once observed it is never left; at startup the state is `unknown` until the first successful marker read and every claims-off record door (reads and mutations) answers 503 until then, so a marker-read failure never restores legacy interpretation. The removal rule separates lifetime validity from placement eligibility: a valid own record that is **in this pass's scope** protects the legacy collision; a valid own record **outside** the scope follows today's removal rules (`KeepParticipant`, the app-trash rule), so offline finishing removes the source row as today; absence or staleness authorises the new path. One worker lifecycle (start once, loop, check the state each iteration), one migration interval, one context contract (`recordCtx`), the CLI command, the retry error, the three root constructions and the copy inventory are stated. Rev 10, 2026-10-22. Rev 9's review (`selfserve-plan1-review9.md`, findings 1 to 5) is answered: the cutover is a **mode** with a drain: from the marker's time until the window ends the console refuses every claims-off record mutation (503), so no legacy write can be admitted into the window, and the gate, the judge's installation, the recovery pass and the marker's failure modes are one contract with an implementation task for the operator door; the removal loop decides from the row's **own** record (observed by its own key), so a valid legacy owner in a hostname collision is kept as today and only an established absent or stale own record retires the row, with the keep view bypassed on that evidence and replacement stated as two passes; `fenceRecord` returns a classification so an own tombstone or marker that won resumes the operation instead of terminating it; the audit's allowlist names the two remaining claims-on callers of `hostops.ReadRecord` and is pinned both ways; and the remaining contracts are settled (the detach sentence, cleanup recovery through C2, corrupt index candidates, the root restart readers' judge). Rev 8's review (`selfserve-plan1-review8.md`, findings 1 to 5) is answered: the rollout guarantee becomes an **enforced cutover** (an operator marker written after the old console has stopped, plus the settle window, with the storage assumption stated) rather than a timer alone; quarantine copies the corrupt bytes and **never deletes under a live slot**: the retirement converts durably to the cleanup kind and retires the slot first, and a conditional-delete conflict re-classifies instead of counting as absence; a provisioned row owned by another record key than the hostname's current record is retired as "record gone" so a cross-owner recreation cannot leave a lagging box serving the predecessor; `RecordIndex.Find` judges its candidate keys before deciding ambiguity; and the remaining contradictions are resolved (the root scope sentence, one detach chain, the locked retirement's callers, reconcile's one sequence, the audit's enumerated list, the fence witness split). Rev 7's review (`selfserve-plan1-review7.md`, findings 1 to 6) is answered: a stated rollout guarantee closes the late legacy delete (one console per deployment, a settle window before any termination, old consoles never reintroduced over lifetimes records); the first-route attach, its rollback and detach run **inside** the hostname exclusion from the claim write to the record write, through locked variants of the create and retirement helpers; root's proof and restart record readers take the box's judge (the scope statement is corrected: only `ClaimRootHTTP`'s claims-on readers are slot-free); a corrupt body is quarantined (its bytes kept under `control/quarantine/`, the key removed conditionally) so the retirement or migration proceeds as for absence instead of promising a fence that cannot parse it; `provision.WithLifetimes` is the one context helper that carries the judge and enables sequencing, the runs worker wraps its detached execution context, the audit distinguishes approved judged wrappers from raw readers, the fence witnesses state their timing, exhaustion is witnessed at every transition, and the sibling undo has one dispatch path. Earlier reviews: rev 1 (`after236/selfserve-plan1-review.md`, 16 findings), rev 2 (F1 to F12), rev 3 (R1 to R10), rev 4 (1 to 9), rev 5 (1 to 6), rev 6 (`selfserve-plan1-review6.md`, findings 1 to 5). Rev 7 answers rev 6's five: every claims-off record write runs with `WithRecordSequence`, so `record_seq` exists after a record's first write and an ETag never recurs, and recovery writes a **fence** (a sequencing rewrite) before it terminates an operation on evidence of a changed record; one migration dispatcher classifies the observation at every entry, so a resume can never supersede its own marker; the attach door's claim rollback runs inside the hostname exclusion and never releases a claim a pending operation or a valid record needs; the judge reaches every mutation context (the console's top-level handler wraps every request, browser and API alike; the seed callback wraps its own context; the full-record updater's re-read is judged; `RecordIndex.Find` and `FindRecordByHost` take the judge from the context), and the audit is by AST; the remaining contracts (the running N1 table, a slot-retirement helper, one expectation type, listed-key absence, exhaustion at read, the import witness's state, the producer-state publication tests) are answered. Rev 5 removed the slot cache (every judgement reads the slot fresh, after the record, in bounded rounds) and made a successor lifetime's hold one row write. Rev 6 answers rev 5's six: a retirement persists its **kind** (normal, orphan, cleanup) and each kind has its own steps and recovery table; a migration's marker-CAS conflict is classified by a re-read for both the bound and the unbound form, so an own marker that landed is resumed, never superseded; every `retireInstanceRecord` caller states its expectation (the sibling undo names the lifetime it created; detach deletes what is current; the attach door never asks for a record rollback over a lifetime failure); a judge carried on the context makes `updateRecordSeeding`, and so every `SetRecord*` writer, refuse a stale record, and every box reader (Phase 1's included) and the import journal's readers go through the judge; the provenance mark is dropped (the fold compares pairs between unstamped entries, a pair-less entry is `(0, 0)` and equal pairs fall to the existing rule, exactly §6.4); the remaining interface gaps are answered. |
| **Goal:** Give every claims-off instance record a durable lifetime (the host incarnation slot) and every served row a serving order `(incarnation, serving_seq)` that boxes publish and edges honour, with crash-safe create, retirement and migration, so a later plan's holding pages and moves can never be out-ordered by a stale publication. |
| @@ -104,7 +104,7 @@ The verdict, exactly, with `owned := slot != nil && slot.Account == rec.Account |
| **Interfaces:** |
| - Tables (`account` is the sqid): |
| - - `host_allocations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, state TEXT, body BLOB, predecessor_etag TEXT, stale_etag TEXT, app_lifetime INTEGER, created_at INTEGER)` (`app_lifetime TEXT`: the apps row's **`lifetime`** at allocation time, a new column `apps.lifetime TEXT NOT NULL DEFAULT ''` set to a random 16-hex id by `CreateAppInRegion`, kept by trash and restore, and backfilled once at store open for rows with `''` (so an old app has one stable value); a purge removes the row and a recreate mints a new id, so equal values mean the same app lifetime; `created_at` is seconds and is not used for this) |
| + - `host_allocations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, incarnation INTEGER, state TEXT, body BLOB, predecessor_etag TEXT, stale_etag TEXT, app_lifetime INTEGER, created_at INTEGER)` (`app_lifetime TEXT`: the apps row's **`lifetime`** at allocation time, a new column `apps.lifetime TEXT NOT NULL DEFAULT ''`: a random 16-hex id minted by **one helper `insertAppRow`** that every app-row insertion goes through (`CreateAppInRegion`, `CreateTemporaryAppInRegion`, `BackfillOpsMemberships`, the `apps_new` migration copy, which mints for rows without one), plus a one-time backfill at store open for rows still `''`; kept by trash, restore and keep; a purge removes the row and a recreate mints a new id, so equal values mean the same app lifetime; `created_at` is seconds and is not used; `TestEveryAppInsertMintsALifetime` scans the console package's SQL for `INSERT INTO apps` and fails on any statement outside `insertAppRow`, and creates an app by every constructor asserting a non-empty, distinct lifetime); `Allocation.AppLifetime string` is the struct field |
| - `record_retirements (op_id TEXT PRIMARY KEY, kind TEXT, account TEXT, app TEXT, host TEXT, incarnation INTEGER, record_etag TEXT, slot_etag TEXT, slot_intent TEXT, state TEXT, created_at INTEGER)` (`kind` is `normal`, `orphan` or `cleanup`) |
| - `record_migrations (op_id TEXT PRIMARY KEY, account TEXT, app TEXT, host TEXT, expect_etag TEXT, marked_etag TEXT, state TEXT, created_at INTEGER)` |
| - Classification in `claimSQLTables`: `{Sqid: "account", App: "app"}` for all three. |
| @@ -129,15 +129,15 @@ The verdict, exactly, with `owned := slot != nil && slot.Account == rec.Account |
| - Consumes: Tasks 1 and 2; Task 6's `provision.LifetimeJudge` (its `Observe`) for `classifyLocked`. |
| - Produces: |
| - `lifetimesOn()` is Task 0's. |
| - - `func (s *Server) withHostname(host string, fn func() error) error` (a `sync.Map` of `*sync.Mutex`; not reentrant). **The lock table** (neither mutex is reentrant; the order is always account lock (where a door already holds one) → `s.lockApp` → `withHostname`): the create, seed and import doors' `createRecordWithLifetime` acquires `s.lockApp` then `withHostname`; `createRecordWithLifetimeLocked` assumes **both**; the attach door acquires `s.lockApp` **before** entering its hostname exclusion and `attachFirstRouteLocked` assumes both; detach and rollback run inside the hostname exclusion (their door acquires the app lock first) and call `Locked` helpers that acquire nothing; `retireRecordWithLifetime` acquires `s.lockApp` then `withHostname`, `retireRecordWithLifetimeLocked` and `retireSlotOrphanLocked` assume both; allocation, retirement and migration recovery acquire `s.lockApp` then `withHostname` per row; the purge's callers hold `s.lockApp` and `purgeAppWithLifetimes` acquires `withHostname` per host; trash, restore, keep and ttl hold `s.lockApp` and acquire no hostname lock; `TestLockOrderIsRespected` runs every caller under a lock-order checker (a test hook on both mutexes) and fails on an inversion (every exported helper below takes the locks its row says, every `...Locked` helper assumes them). |
| + - `func (s *Server) withHostname(host string, fn func() error) error` (a `sync.Map` of `*sync.Mutex`; not reentrant). **The lock table** (neither mutex is reentrant; the order is always account lock (where a door already holds one) → `s.lockApp` → `withHostname`): the create, seed and import doors' `createRecordWithLifetime` acquires `s.lockApp` then `withHostname`; `createRecordWithLifetimeLocked` assumes **both**; the attach door acquires `s.lockApp` **before** entering its hostname exclusion and `attachFirstRouteLocked` assumes both; detach and rollback run inside the hostname exclusion (their door acquires the app lock first) and call `Locked` helpers that acquire nothing; `retireRecordWithLifetime` acquires `s.lockApp` then `withHostname`, `retireRecordWithLifetimeLocked` and `retireSlotOrphanLocked` assume both; allocation, retirement and migration recovery acquire `s.lockApp` then `withHostname` per row; the purge's callers hold `s.lockApp` and `purgeAppWithLifetimes` acquires `withHostname` per host; trash, restore, keep and ttl hold `s.lockApp` and acquire no hostname lock; the initial migration pass acquires `s.lockApp` then `withHostname` per record; **`retireSlotOrphanLocked` needs only the hostname lock and its captured `SlotETag`**, never the slot owner's app lock (a create under key B retiring A's orphan slot holds `app(B)` and H; A's purge, if any, holds `app(A)` and waits for H; whichever CAS on the slot lands first wins and the other re-observes under H, so no owner lock is needed and acquiring one would invert the order), stated in the table and covered by the witness; `TestLockOrderIsRespected` runs every caller under a lock-order checker (a test hook on both mutexes) and fails on an inversion (every exported helper below takes the locks its row says, every `...Locked` helper assumes them). |
| - `func (s *Server) admit(host, ownOpID string) error`: `errLifetimeSettling` when `PendingForHost(host, ownOpID) > 0`. |
| - `func (s *Server) classifyLocked(ctx context.Context, acct, app, host string) (provision.Observation, error)`: `s.judge.Observe(ctx, s.cfg.Releases, acct, app, host)` (Task 6: record first, slot after, bounded rounds); the error is non-nil only with `Verdict == LifetimeUnreadable`. |
| - `func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record) (*provision.Record, error)` (takes `withHostname`) and `func (s *Server) createRecordWithLifetimeLocked(ctx context.Context, rec provision.Record) (*provision.Record, error)` (assumes it): the one protocol every claims-off creation path runs; the create, seed and import doors call the locking form; the attach door calls the locked form inside its own exclusion (below). Returns the record as written. Errors: `release.ErrCASConflict` (a valid record under this key), `*hostHeldError` (`errors.As`; carries `*apexHold` for `hostHeldSentence`: a live slot owned by another key gives `&apexHold{Holder: slot.App, OtherAccount: slot.Account if ≠ rec.Account}`), `errLifetimeSettling`, `errLifetimePending`, or a read error from step 2 (nothing persisted). |
| - - `func (s *Server) fenceRecord(ctx context.Context, acct, app, host string, op fenceOwner) (res fenceResult, obs provision.Observation, err error)` with `type fenceOwner struct { Tombstone bool; MarkerID string }` (what this operation's own outstanding write would look like) and `fenceResult` one of `fenced` (a body that is neither this operation's tombstone nor its marker was rewritten with its sequence introduced or advanced: the caller terminates on `obs`), `ownWriteWon` (the re-read found this operation's tombstone, or a record marked with `MarkerID`: the caller does **not** terminate; it dispatches again from `obs` and the operation resumes, retirement at N2, migration through its dispatcher), `absent` (nothing to fence: the slot is the fence for an absent key), or an error after 3 CAS retries (the caller leaves its operation pending). Every recovery branch below and in Tasks 4 and 5 that would mark an operation `refused`, `void` or `superseded` because the record is not what the operation captured calls `fenceRecord` first and acts on its result. **An allocation is replayed only for the app lifetime that authorised it**: recovery, holding `s.lockApp` then `withHostname`, reads the apps row (`AppLife(acct, app)`: absent, or `created_at != row.app_lifetime`, or trashed) and the store's app tombstone marker; any of those → the row's slot, when it is `{N, live, this key, "create " + OpID}`, is retired through `retireSlotOrphanLocked` **cancelling this allocation**: the retirement row (`record_retirements` with `kind: orphan`, `cancels: OpID`) and the allocation's `void` are written in **one SQLite transaction**, so there is never a moment with both pending; `admit(host, R)` for the retirement then has nothing to exclude, and O1 proceeds (any body at N under the key deleted on its ETag); a retirement row with `cancels` set is recovered before any allocation row of its host; a slot already superseded → `void`; a slot absent → `void` without a slot write (an outstanding slot PUT that lands later leaves a **live orphan slot**, which the orphan rule reclaims on the next create; an outstanding record PUT that lands after O1 leaves a **stale body under a retired slot**, which §6.4 tolerates: the next create of the hostname deletes it in step 5 whatever key it is under (step 2 records `StaleETag` for a stale body under this create's key **and** `OwnerStaleETag`/key for a stale body under the retired slot's owner key when that differs); recovery does not revisit terminal rows, and the plan claims no more than that); `TestAllocationBoundToAppLifetime` pins it (a live slot and no record; a slot and a record that landed late; no slot; the app purged and recreated under the same name before the pass; a record PUT after the void → stale, removed by the next create). |
| + - `func (s *Server) fenceRecord(ctx context.Context, acct, app, host string, op fenceOwner) (res fenceResult, obs provision.Observation, err error)` with `type fenceOwner struct { Tombstone bool; MarkerID string }` (what this operation's own outstanding write would look like) and `fenceResult` one of `fenced` (a body that is neither this operation's tombstone nor its marker was rewritten with its sequence introduced or advanced: the caller terminates on `obs`), `ownWriteWon` (the re-read found this operation's tombstone, or a record marked with `MarkerID`: the caller does **not** terminate; it dispatches again from `obs` and the operation resumes, retirement at N2, migration through its dispatcher), `absent` (nothing to fence: the slot is the fence for an absent key), or an error after 3 CAS retries (the caller leaves its operation pending). Every recovery branch below and in Tasks 4 and 5 that would mark an operation `refused`, `void` or `superseded` because the record is not what the operation captured calls `fenceRecord` first and acts on its result. **An allocation is replayed only for the app lifetime that authorised it**: recovery, holding `s.lockApp` then `withHostname`, reads the apps row (`AppLife(acct, app)`: absent, or `life.Lifetime != row.AppLifetime`, or trashed) and the store's trash marker (unreadable → the row stays pending); any of those → the row's slot, when it is `{N, live, this key, "create " + OpID}`, is retired through `retireSlotOrphanLocked` **cancelling this allocation**: the retirement row (`record_retirements` with `kind: orphan`, `cancels: OpID`) and the allocation's `void` are written in **one SQLite transaction**, so there is never a moment with both pending; `admit(host, R)` for the retirement then has nothing to exclude, and O1 proceeds (any body at N under the key deleted on its ETag); a retirement row with `cancels` set is recovered before any allocation row of its host; a slot already superseded → `void`; a slot absent → `void` without a slot write (an outstanding slot PUT that lands later leaves a **live orphan slot**, which the orphan rule reclaims on the next create; an outstanding record PUT that lands after O1 leaves a **stale body under a retired slot**, which §6.4 tolerates: the next create of the hostname deletes it in step 5 when it is under **the same key**; a stale body under **another** key (the retired slot's previous owner) is deleted by the running create as **best effort** (step 2 records `OwnerStaleETag`/key in memory only; step 5 deletes it; a crash before step 5 leaves it, since recovery has no record of it and the plan does not promise more: §6.4 tolerates the body, every judged consumer ignores it, and only bytes remain); recovery does not revisit terminal rows); `TestAllocationBoundToAppLifetime` pins it (a live slot and no record; a slot and a record that landed late; no slot; the app purged and recreated under the same name before the pass; a record PUT after the void → stale, removed by the next create). |
| - `func (s *Server) recoverLifetimes(ctx context.Context) error`, `func (s *Server) StartLifetimeRecoveryPass(ctx context.Context)` (Task 0's worker lifecycle: started once, loops every 60 s for the life of the process, each iteration admitted as a mutating unit and doing work only in `on`). |
| The protocol, inside `withHostname(host)`: |
| -0. `s.lockApp(acct.Sqid, app)` (the lifecycle mutex every app transition takes; **lock order everywhere in this plan: the app lock, then the hostname lock**; recovery and the purge follow the same order), then **the app's lifecycle under that lock**: `life, err := s.store.AppLife(acct.Sqid, app)`: absent → 404 (the app is gone), trashed → 409 "the app is in the trash", `life.Lifetime != the lifetime the door read before taking the lock` → 409 "the app changed. try again" (a purge and a recreate under the same name happened meanwhile); then `withHostname(host)`. |
| +0. `s.lockApp(acct.Sqid, app)` (the lifecycle mutex every app transition takes; **lock order everywhere in this plan: the app lock, then the hostname lock**; recovery and the purge follow the same order), then **the app's lifecycle under that lock**: `life, err := s.store.AppLife(acct.Sqid, app)` **and** `release.AppTrashMarker(ctx, s, acct.Sqid, app)` (the store's `deleted.json`): a marker read error → 502 "could not read the app. try again" (unreadable never admits); the marker present, or `AppLife` trashed → 409 "the app is in the trash"; absent → 404 (the app is gone), `life.Lifetime != the lifetime the door read before taking the lock` → 409 "the app changed. try again" (a purge and a recreate under the same name happened meanwhile); then `withHostname(host)`. |
| 1. `admit(host, "")`. |
| 2. `classifyLocked`. Unreadable → the error. Valid record under this key → `ErrCASConflict`; the slot is untouched. **The orphan rule**: a live slot (this key or another) with **no record under its key** and `PendingForHost(host, "") == 0` is an orphan (a purge deleted the record, or a durable allocation row was lost, which this plan treats as outside its guarantees and states so): `retireSlotOrphanLocked(ctx, host, slot)` runs first (the slot-bound entry: persists a `record_retirements` row of kind `orphan` with `SlotETag`, then O1: the slot by CAS from that ETag, then any body under the key deleted on its ETag), and the create continues from the retired slot; a live slot owned by another key **with** a record, or with a pending operation, → `*hostHeldError`. Then `acct, err := s.store.AccountBySqid(rec.Account)`: an error, or `acct == nil`, is a read error (502, nothing persisted); then `s.hostHolder(ctx, acct, host, app)` under the lock (the pre-lock call at `apiinstances.go:1164` stays): a holder → `*hostHeldError`. A stale or corrupt present record under this key is remembered: `StaleETag` (its ETag, never empty for a present body). |
| 3. `N = max(LastIncarnation(host), slot.Incarnation) + 1`; `rec.Incarnation = N`; `body = json.Marshal(rec)`; `BeginAllocation({OpID, Account, App, Host, N, pending, body, PredecessorETag: slot's ETag or "", StaleETag, AppLifetime: life.Lifetime})` (`Allocation.AppLifetime string`). A SQLite failure here → the error (nothing durable). |
| @@ -179,7 +179,7 @@ Door mappings: the create door: `*hostHeldError` → 409 `hostHeldSentence(host, |
| **Files:** |
| - Create: `internal/console/lifetimes_retire.go` |
| -- **App purge.** Two regimes, split by the cutover marker. **A box** (`domains.Reconcile` 1d) auto-purges only while it has **not** read `control/lifetimes-cutover.json` (read each pass; latched once seen; an unreadable marker → skip purging this pass, log once an hour; `cfg.CutoverSeen func(ctx) (bool, error)` wired by the agent); its purge is today's `PurgeApp` with instance-record keys deleted by `Get` → `DeleteIfMatch`, where **the marker is re-read immediately before each `Get`** (present → the purge aborts with nothing further deleted; the console owns the app from here) and the pair runs under a clock guard (more than 30 s since that marker read before the `DeleteIfMatch` → abort and retry next pass); because the console's activation is at least `lifetimeSettle` (60 s) after the marker appears, a box that read the marker absent cannot delete a numbered record (none exists until 60 s later and its own window is 30 s), and a sweep paused across the cutover sees the marker and stops; a conflict aborts before any other deletion; the next pass retries; within those bounds the listing-to-`Get` window is today's two-purger race on `main` and is not widened. **The console** (`handlePurgeApp`, `purgeOne`, and a new hourly `release.ExpiredTrashApps(ctx, s, now)` sweep in `StartExpiryPass` that runs whether or not the pass has rows, so headless apps are purged once boxes stop; `HostClaim` deployments keep `RequestClaimAutoPurge` unchanged) purges through `s.purgeAppWithLifetimes(ctx, acct, app, want deletedAt, now)`, every caller already holding or taking **`s.lockApp(acct, app)`**: (1) **the guards first**: `release.PurgeAppGuards(ctx, s, pub, acct, app, want, now)` (the marker, its clock, the production flag and the store-estate guard exactly as `PurgeAppIfTrashedAt` applies them today, factored out so they run before any lifetime step; a refusal ends the purge with today's sentence); (2) **per instance key** under the app prefix, and per pending allocation of the app (`PendingLifetimesForApp(acct, app)`), under `withHostname(host)`: observe (`classifyLocked`); a pending allocation of this app → the lifetime check of Task 3 fails by construction (the app is trashed) → `retireSlotOrphanLocked` when the slot is the allocation's, `void`; a present record → `retireRecordWithLifetimeLocked(ctx, acct, app, host, normal, RetireExpectation{Incarnation: obs.Rec.Incarnation})` when numbered, `RetireExpectation{ETag: obs.ETag}` when legacy (never `retireAny`; the observation and the retirement are one locked step, so nothing can be observed stale); absent with a live slot **owned by this app** (`slot.Account == acct && slot.App == app`) → `retireSlotOrphanLocked`; absent with a live slot owned by **another** key → nothing (a foreign slot is never purge's; whether it is an orphan is the create rule's question, asked against the slot owner's own key); absent with no slot → nothing; any `errLifetimePending` → the purge stops here and re-runs later (every step idempotent); (3) `purgeApp`'s sweep with instance keys **excluded** (the retire hook is the no-op marker that the keys were handled), then the trash marker. Because create, recovery and purge all take the app lock before the hostname lock, no create or recovery of this app runs while the purge does, and the app name is released (the marker deleted, the apps row removed) only after every instance of this app lifetime is retired, so a successor under the same name starts with no slot or record of its predecessor to collide with. **Restore evidence**: `settleApp` and `MaterializeApps` today count any object under the app prefix as evidence the app lives; in lifetimes mode an instance-record key counts only when the judge classifies it **valid** (`ListRecordsJudged`), so a stale body left under a retired slot never turns a finished purge into a restore; `TestStaleBodyIsNotRestoreEvidence` pins it (a purge whose `PurgeAppRow` write failed, a stale body under the prefix: `settleApp` releases the name instead of restoring). |
| +- **App purge.** Two regimes, split by the cutover marker. **A box** (`domains.Reconcile` 1d) auto-purges only while it has **not** read `control/lifetimes-cutover.json` (read each pass; latched once seen; an unreadable marker → skip purging this pass, log once an hour; `cfg.CutoverSeen func(ctx) (bool, error)` wired by the agent); its purge is today's `PurgeApp` with instance-record keys deleted by `Get` → `DeleteIfMatch`, where **the marker is re-read immediately before each `Get`** (present → the purge aborts with nothing further deleted; the console owns the app from here) and the pair runs under a clock guard (more than 30 s since that marker read before the `DeleteIfMatch` → abort and retry next pass); because the console's activation is at least `lifetimeSettle` (60 s) after the marker appears, a box that read the marker absent cannot delete a numbered record (none exists until 60 s later and its own window is 30 s), and a sweep paused across the cutover sees the marker and stops; a conflict aborts before any other deletion; the next pass retries. **Every other destructive step of the box's sweep is fenced the same way**: `purgeApp` gains `before func(ctx context.Context) error`, called immediately before **each** `Delete` (configuration keys, ledgers, promotion records and the final `deleted.json` included); the box's `before` re-reads the cutover marker (present → `errPurgeDeferred`, the sweep stops where it is; a re-run after the marker never happens because the box no longer purges) and refuses when more than 30 s passed since the previous check; the console passes nil. A box paused anywhere in its sweep across the cutover therefore deletes nothing further once it resumes; within those bounds the listing-to-delete window is today's two-purger race on `main` and is not widened. **The console** (`handlePurgeApp`, `purgeOne`, and a new hourly `release.ExpiredTrashApps(ctx, s, now)` sweep in `StartExpiryPass` that runs whether or not the pass has rows, so headless apps are purged once boxes stop; `HostClaim` deployments keep `RequestClaimAutoPurge` unchanged) purges through `s.purgeAppWithLifetimes(ctx, acct, app, want deletedAt, now)`, every caller already holding or taking **`s.lockApp(acct, app)`**: (1) **the guards first**: `release.PurgeAppGuards(ctx, s, pub, acct, app, want, now)` (the marker, its clock, the production flag and the store-estate guard exactly as `PurgeAppIfTrashedAt` applies them today, factored out so they run before any lifetime step; a refusal ends the purge with today's sentence); (2) **per instance key** under the app prefix, and per pending allocation of the app (`PendingLifetimesForApp(acct, app)`), under `withHostname(host)`: observe (`classifyLocked`); a pending allocation of this app → the lifetime check of Task 3 fails by construction (the app is trashed) → `retireSlotOrphanLocked` when the slot is the allocation's, `void`; a present record → `retireRecordWithLifetimeLocked(ctx, acct, app, host, normal, RetireExpectation{Incarnation: obs.Rec.Incarnation})` when numbered, `RetireExpectation{ETag: obs.ETag}` when legacy (never `retireAny`; the observation and the retirement are one locked step, so nothing can be observed stale); absent with a live slot **owned by this app** (`slot.Account == acct && slot.App == app`) → `retireSlotOrphanLocked`; absent with a live slot owned by **another** key → nothing (a foreign slot is never purge's; whether it is an orphan is the create rule's question, asked against the slot owner's own key); absent with no slot → nothing; any `errLifetimePending` → the purge stops here and re-runs later (every step idempotent); (3) `purgeApp`'s sweep with instance keys **excluded** (the retire hook is the no-op marker that the keys were handled), then the trash marker. Because create, recovery and purge all take the app lock before the hostname lock, no create or recovery of this app runs while the purge does, and the app name is released (the marker deleted, the apps row removed) only after every instance of this app lifetime is retired, so a successor under the same name starts with no slot or record of its predecessor to collide with. **Restore evidence**: `settleApp` and `MaterializeApps` today count any object under the app prefix as evidence the app lives; in lifetimes mode the evidence is **three-valued**: `ListRecordsComplete`'s judged sibling returns the valid records **and** whether any key was unreadable, corrupt or truncated; a valid record is evidence the app lives; an established stale or absent body contributes nothing; any unreadable, corrupt or truncated observation makes the outcome **unknown**, and `settleApp`/`MaterializeApps` then keep the row and defer (no `PurgeAppRow`, no `FinishRestoreRow`, logged once per hour), so a stale body never turns a finished purge into a restore and a failed read never releases a name whose records still exist (`TestRestoreEvidenceIsThreeValued`); **startup**: `NewServer` no longer calls `materializeQuietly` directly; materialisation is the first iteration of a worker that admits like every other (Task 0's lifecycle: `before` → legacy as today; `on` → judged; `unknown` → wait), so no nil-judge listing runs before the gate has a state, and claims-on keeps today's call; `TestStaleBodyIsNotRestoreEvidence` pins it (a purge whose `PurgeAppRow` write failed, a stale body under the prefix: `settleApp` releases the name instead of restoring). |
| - Modify: `internal/console/hostclaimops.go` (claims-off `ref.Family == "instance"` → `retireRecordWithLifetime(ctx, ref.Account, ref.App, ref.Host, ref.Expect)`; `provision.RetireRef` gains `Expect retireExpectation`-shaped fields through a new `provision.RetireExpectation{Absent bool; ETag string; Incarnation uint64}` so each caller states its own), `internal/console/claim_retire.go` (`retireInstanceRecord(ctx, account, app, host, by, incarnation string, expect provision.RetireExpectation)`; its claims-off callers: `apiinstances.go:1684` (the sibling undo) with `{Incarnation: created.Incarnation}` through the locking `retireRecordWithLifetime`; **detach does not go through it on the claims-off path**: `apidomains.go:989` and `domainattach.go:349` call Task 3's `detachFirstRouteLocked`, which calls `retireRecordWithLifetimeLocked(..., retireAny)` inside the exclusion it already holds; the claims-on detach keeps today's `retireInstanceRecord` chain; `domainattach.go:557` (`rollbackAttach`) never retires on the claims-off path; `apiinstances.go:2245` is not reached on claims-off: `importDeleteRule` answers every claims-off delete), `internal/console/apiinstanceimport.go` (`readDeletedRecord` returns `(provision.Observation, error)` through `s.judge.Observe`; `importDeleteRule`: `!obs.Present` → `retireRecordWithLifetime(..., retireExpectAbsent)`; `obs.Verdict` stale or corrupt → `deleteRead(..., retireExpectETag(obs.ETag))`; valid → the import logic as today, and its ordinary delete `deleteRead(..., retireExpectETag(obs.ETag))`; `deleteRead` passes the expectation through: `ErrCASConflict` keeps the `again` re-decision; `errLifetimeSettling` → 409 settling; `errRetireRefused` → 409 its sentence), `internal/console/seedfinish.go` (`finishSeedOnce`'s final delete → `retireRecordWithLifetime(ctx, sqid, app, host, retireExpectETag(etag))`), `internal/console/apiinstances.go` (`handleAPIInstanceDelete`: `errRetireRefused`, `errLifetimeSettling` → 409), `internal/console/domainattach.go` and `internal/console/apidomains.go` (detach: `errLifetimeSettling` → 409 `claimDetachInProgressMsg`; `errRetireRefused` → 409 its sentence), `internal/console/lifetimes_recovery.go` |
| - Test: `internal/console/lifetimes_retire_test.go` |
| @@ -209,7 +209,7 @@ Inside `withHostname(host)`: `admit(host, "")`; `obs := classifyLocked` (unreada |
| - kind **orphan**, `pred`: absent or any present body → O1 (the slot, then the body if present), `committed`. |
| - kind **cleanup**, `pred`: any observation (absent, corrupt, stale, or a valid body at N: the DELETE this operation answers was for this lifetime, and under the retired slot every body is stale) → C1 then C2 on the current body's ETag when present, `committed`. |
| -- [ ] **Step 1: Write the failing tests** through the real doors and the function: `TestConsolePurgeRetiresEveryLifetime` (browser purge, the expiry pass's row purge and its hourly store sweep (an app with no console row), all holding `s.lockApp`: the guards run before any lifetime step and a refused purge retires nothing; every record ends absent with a retired slot (numbered by incarnation, legacy by ETag); a pending allocation with a live slot and no record is voided and its slot retired; one with no slot is voided; a create of the same app blocked on the app lock during the purge finds the app trashed and refuses; a recovery pass blocked on the app lock finds the lifetime mismatch and voids; the hostname is then created by another app at N+1; an `errLifetimePending` mid-purge leaves the marker and the re-run completes); `TestBoxPurgeStopsAtTheCutoverMarker` (a box that has read the marker leaves every trashed app alone; an unreadable marker → skip and log; before the marker its instance deletes are conditional and a record changed after the `Get` survives); `TestOrphanSlotReclaimedByTheNextCreate` (a live slot with no record and no pending op, from a lost allocation row, is retired as `orphan` and the create allocates N+1; with a pending op it is `*hostHeldError`; a stale body under the owner key is deleted in step 5 by a create under another key); `TestCreateChecksTheAppUnderTheLock` (a create that read a live app, blocked on the app lock across a trash → 409 trash; across a purge → 404; across a purge and a recreate under the same name → 409 changed, nothing allocated); `TestCancellationIsOneTransaction` (a crash anywhere between the void and O1 leaves exactly one pending row, the retirement, which completes); `TestBoxPurgeAbandonsAcrossTheCutover` (a box paused between its listing and its `Get` while the marker appears and the console activates and recreates the host: the re-read marker stops the purge, nothing deleted; the 30 s clock guard likewise); |
| +- [ ] **Step 1: Write the failing tests** through the real doors and the function: `TestConsolePurgeRetiresEveryLifetime` (browser purge, the expiry pass's row purge and its hourly store sweep (an app with no console row), all holding `s.lockApp`: the guards run before any lifetime step and a refused purge retires nothing; every record ends absent with a retired slot (numbered by incarnation, legacy by ETag); a pending allocation with a live slot and no record is voided and its slot retired; one with no slot is voided; a create of the same app blocked on the app lock during the purge finds the app trashed and refuses; a recovery pass blocked on the app lock finds the lifetime mismatch and voids; the hostname is then created by another app at N+1; an `errLifetimePending` mid-purge leaves the marker and the re-run completes); `TestBoxPurgeStopsAtTheCutoverMarker` (a box that has read the marker leaves every trashed app alone; an unreadable marker → skip and log; before the marker its instance deletes are conditional and a record changed after the `Get` survives); `TestOrphanSlotReclaimedByTheNextCreate` (a live slot with no record and no pending op, from a lost allocation row, is retired as `orphan` and the create allocates N+1; with a pending op it is `*hostHeldError`; a stale body under the owner key is deleted in step 5 by a create under another key); `TestCreateChecksTheAppUnderTheLock` (a create that read a live app, blocked on the app lock across a trash → 409 trash, including a trash whose SQLite row update failed after the marker landed; across a purge → 404; across a purge and a recreate under the same name, by the permanent and by the temporary constructor → 409 changed, nothing allocated; an unreadable marker → 502, nothing allocated); `TestCancellationIsOneTransaction` (a crash anywhere between the void and O1 leaves exactly one pending row, the retirement, which completes); `TestBoxPurgeAbandonsAcrossTheCutover` (a box paused between its listing and its `Get`, and a box paused before a configuration-key `Delete` after its last instance check, and one paused before the final marker delete, while the marker appears and the console activates and recreates the app and host: the re-read marker stops every further delete; the 30 s clock guard likewise); |
| - `TestDeleteDoorRetiresThroughTheTombstone`; `TestSlotRetiresBeforeTheKeyIsAbsent`. |
| - `TestDeleteDoorAbsentPathAdmitsAndRetiresOrphans` (initial absence; absence found on the judge's re-read; a record created since is untouched and the door still answers "deleted"). |
| - `TestStaleCleanupRetiresALiveOwnedSlotFirst` (a fieldless late body beside `{1, live, this key}`: the slot is `{1, retired}` before the body is removed; a blocked incarnation-1 create-only write released afterwards lands stale; the same for a corrupt body). |
| @@ -235,7 +235,7 @@ Inside `withHostname(host)`: `admit(host, "")`; `obs := classifyLocked` (unreada |
| - Test: `internal/provision/record_sequence_test.go`, `internal/console/lifetimes_migrate_test.go` |
| **Interfaces:** |
| -- Produces: `var ErrRecordMigrating = fmt.Errorf("%w: the record is being migrated; read it again", release.ErrCASConflict)`; `func WithMigration(ctx context.Context, id string) context.Context`; `func (s *Server) migrateHostLifetimeLocked(ctx context.Context, acct, app, host, expectETag string) (rec *provision.Record, raw []byte, etag string, state string, err error)` (`expectETag == ""` is unbound: retry from the current record; a real ETag binds the marker CAS to it); `func (s *Server) StartIncarnationMigrationPass(ctx context.Context)` (behind `CARLOS_LIFETIMES_MIGRATE=1`; Task 0's worker lifecycle: started once, loops every 60 s, each iteration admitted as a mutating unit and doing work only in `on`; at most 50 records per pass, oldest first, each inside `withHostname` and `admit`, only fieldless unmarked records whose slot is absent, unbound). |
| +- Produces: `var ErrRecordMigrating = fmt.Errorf("%w: the record is being migrated; read it again", release.ErrCASConflict)`; `func WithMigration(ctx context.Context, id string) context.Context`; `func (s *Server) migrateHostLifetimeLocked(ctx context.Context, acct, app, host, expectETag string) (rec *provision.Record, raw []byte, etag string, state string, err error)` (`expectETag == ""` is unbound: retry from the current record; a real ETag binds the marker CAS to it); `func (s *Server) StartIncarnationMigrationPass(ctx context.Context)` (behind `CARLOS_LIFETIMES_MIGRATE=1`; Task 0's worker lifecycle: started once, loops every 60 s, each iteration admitted as a mutating unit and doing work only in `on`; at most 50 records per pass, oldest first, each inside `s.lockApp` then `withHostname` and `admit`, only fieldless unmarked records whose slot is absent, unbound). |
| The protocol (inside the exclusion): `admit(host, "")`; `BeginMigration({OpID, Account, App, Host, ExpectETag, "", pending})`; then **the dispatcher**, `migrationDispatch(obs, m) action`, is the only way in, at the first entry, after every CAS conflict and at every recovery: `own marker` (the record carries `IncarnationMigrating == OpID`) → resume at `SetMigrationMarkedETag` then (2); `own slot and done` (record at `Incarnation 1`, slot `{1, live, this key, "migration " + OpID}`) → `committed`; `eligible` (present, fieldless, unmarked, slot absent; and for a bound call at `ExpectETag`) → (1); `bound and moved` (bound, eligible but another ETag) → `fenceRecord`, `superseded`, `ErrCASConflict`; `unreadable` → pending; a corrupt body (slot absent, so no live slot is exposed) → `quarantineCopy`, then `DeleteIfMatch` on its ETag (a conflict → dispatch again), `superseded`; anything else (gone, another lifetime, another op's marker, a slot not of this migration) → `fenceRecord` when a record is present, `superseded`. "Resume (1)" anywhere below means "dispatch again". (1): `WriteRecord(WithMigration(WithRecordSequence(ctx), OpID), s, key, raw, raw with IncarnationMigrating = OpID, E)` (the marked record carries `record_seq`); on `ErrCASConflict` → dispatch again (an unbound `eligible` result retries (1) from the fresh record, at most 5 times, then leaves pending); `SetMigrationMarkedETag(OpID, M)` (a SQLite failure → pending, `errLifetimePending`); (2) `WriteHostIncarnation({1, live, Account, App, "migration " + OpID, RecordETag: M}, "")` create-only, or `If-Match` when a slot `{1, live, this key, "migration " + OpID}` exists; a CAS conflict → re-read: this migration's slot → continue; otherwise `superseded`; (3) `WriteRecord(WithMigration(ctx, OpID), ..., Incarnation = 1, IncarnationMigrating = "", against M)` (a conflict → re-read: a record at `Incarnation 1` under this slot → continue; otherwise pending); `committed`. Every other error after `BeginMigration` → pending, `errLifetimePending`; every re-read that fails → pending. |