amadan
plan: self-serve moves 1, record lifetimes and serving order
Nine tasks: record fields and the slot type; console tables for pending lifetimes; the create protocol and recovery; tombstone retirement; marked lazy migration; consumers checking the lifetime; move_seq and the row pair; publication and the home map's order; end-to-end witnesses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+237
−0
docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md+237 −0
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (1 s)
ok amadan.net/carlos/platform.git/launchctx 0.016s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (2 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (141 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.945s
ok github.com/carlosframework/platform/cmd/moveprobe 11.921s
ok github.com/carlosframework/platform/cmd/zddprobe 0.718s
ok github.com/carlosframework/platform/internal/activator 59.062s
ok github.com/carlosframework/platform/internal/agentsock 0.022s
ok github.com/carlosframework/platform/internal/alerting 0.004s
ok github.com/carlosframework/platform/internal/analytics 0.034s
ok github.com/carlosframework/platform/internal/appconfig 3.036s
ok github.com/carlosframework/platform/internal/apperr 0.138s
ok github.com/carlosframework/platform/internal/applog 0.219s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.006s
ok github.com/carlosframework/platform/internal/boxid 1.036s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.009s
ok github.com/carlosframework/platform/internal/caps 0.020s
ok github.com/carlosframework/platform/internal/certwarm 0.004s
ok github.com/carlosframework/platform/internal/cgroup 0.788s
ok github.com/carlosframework/platform/internal/changelog 0.549s
ok github.com/carlosframework/platform/internal/claimactivation 1.316s
ok github.com/carlosframework/platform/internal/clamd 0.068s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.677s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.032s
ok github.com/carlosframework/platform/internal/dbverify 0.006s
ok github.com/carlosframework/platform/internal/dnsrec 0.003s
ok github.com/carlosframework/platform/internal/docsite 0.010s
ok github.com/carlosframework/platform/internal/domains 2.928s
ok github.com/carlosframework/platform/internal/doorkey 0.007s
ok github.com/carlosframework/platform/internal/economics 0.009s
ok github.com/carlosframework/platform/internal/edge 45.478s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.018s
ok github.com/carlosframework/platform/internal/fleetapi 0.280s
ok github.com/carlosframework/platform/internal/gate 1.859s
ok github.com/carlosframework/platform/internal/genselect 0.025s
ok github.com/carlosframework/platform/internal/geo 0.056s
ok github.com/carlosframework/platform/internal/geoip 0.074s
ok github.com/carlosframework/platform/internal/historyreadback 0.314s
ok github.com/carlosframework/platform/internal/host 138.023s
ok github.com/carlosframework/platform/internal/hostactors 5.646s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.006s
ok github.com/carlosframework/platform/internal/hostbox 3.397s
ok github.com/carlosframework/platform/internal/hostclaim 0.869s
ok github.com/carlosframework/platform/internal/hostenv 0.008s
ok github.com/carlosframework/platform/internal/hostevidence 0.010s
ok github.com/carlosframework/platform/internal/hostexec 5.615s
ok github.com/carlosframework/platform/internal/hostfence 0.059s
ok github.com/carlosframework/platform/internal/hostfiles 10.394s
ok github.com/carlosframework/platform/internal/hostfloor 0.015s
ok github.com/carlosframework/platform/internal/hostguard 1.041s
ok github.com/carlosframework/platform/internal/hostops 2.315s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.006s
ok github.com/carlosframework/platform/internal/hostquarantine 0.007s
ok github.com/carlosframework/platform/internal/hostrelocate 0.686s
ok github.com/carlosframework/platform/internal/hostresolve 0.224s
ok github.com/carlosframework/platform/internal/hostwrite 0.003s
ok github.com/carlosframework/platform/internal/infraview 0.003s
ok github.com/carlosframework/platform/internal/instancectl 5.257s
ok github.com/carlosframework/platform/internal/instanceexec 2.832s
ok github.com/carlosframework/platform/internal/instancefs 0.008s
ok github.com/carlosframework/platform/internal/instancekey 0.004s
ok github.com/carlosframework/platform/internal/instanceupdates 0.009s
ok github.com/carlosframework/platform/internal/layout 2.013s
ok github.com/carlosframework/platform/internal/lbhealth 0.005s
ok github.com/carlosframework/platform/internal/lease 10.265s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.574s
ok github.com/carlosframework/platform/internal/ledger 0.017s
ok github.com/carlosframework/platform/internal/mail 0.201s
ok github.com/carlosframework/platform/internal/maintenance 0.085s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.591s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.092s
ok github.com/carlosframework/platform/internal/metrics 0.005s
ok github.com/carlosframework/platform/internal/moves 21.491s
ok github.com/carlosframework/platform/internal/moves/archive 1.130s
ok github.com/carlosframework/platform/internal/objstore 0.327s
ok github.com/carlosframework/platform/internal/ops 0.183s
ok github.com/carlosframework/platform/internal/park 0.256s
ok github.com/carlosframework/platform/internal/plans 0.007s
ok github.com/carlosframework/platform/internal/privatefile 0.005s
ok github.com/carlosframework/platform/internal/provision 13.625s
ok github.com/carlosframework/platform/internal/publicrelease 0.012s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.680s
ok github.com/carlosframework/platform/internal/relay 7.244s
ok github.com/carlosframework/platform/internal/release 25.144s
ok github.com/carlosframework/platform/internal/rollout 3.830s
ok github.com/carlosframework/platform/internal/rolloutwords 0.005s
ok github.com/carlosframework/platform/internal/rootaws 0.008s
ok github.com/carlosframework/platform/internal/rootparticipation 0.065s
ok github.com/carlosframework/platform/internal/rootstate 0.030s
ok github.com/carlosframework/platform/internal/runs 0.615s
ok github.com/carlosframework/platform/internal/s3compat 0.006s
ok github.com/carlosframework/platform/internal/safefile 0.006s
ok github.com/carlosframework/platform/internal/schedule 2.789s
ok github.com/carlosframework/platform/internal/seeds 0.013s
ok github.com/carlosframework/platform/internal/sim/hostlife 11.186s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.936s
ok github.com/carlosframework/platform/internal/status 0.092s
ok github.com/carlosframework/platform/internal/steering 0.008s
ok github.com/carlosframework/platform/internal/stopevidence 0.191s
ok github.com/carlosframework/platform/internal/svcrole 0.003s
ok github.com/carlosframework/platform/internal/telemetry 0.005s
ok github.com/carlosframework/platform/internal/termfence 0.007s
ok github.com/carlosframework/platform/internal/tunnelclient 0.005s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.007s
ok github.com/carlosframework/platform/internal/usage 0.010s
ok github.com/carlosframework/platform/internal/version 0.004s
ok github.com/carlosframework/platform/internal/vet 1.678s
=== internal_console@0 (82 s)
ok github.com/carlosframework/platform/internal/console 79.980s
=== internal_console@1 (38 s)
ok github.com/carlosframework/platform/internal/console 35.594s
=== internal_console@2 (43 s)
ok github.com/carlosframework/platform/internal/console 40.736s
=== internal_console@3 (38 s)
ok github.com/carlosframework/platform/internal/console 35.379s
=== internal_console@4 (38 s)
ok github.com/carlosframework/platform/internal/console 35.370s
=== internal_console@5 (36 s)
ok github.com/carlosframework/platform/internal/console 33.327s
=== internal_console@6 (37 s)
ok github.com/carlosframework/platform/internal/console 34.423s
=== internal_console@7 (44 s)
ok github.com/carlosframework/platform/internal/console 41.337s
=== root@0 (32 s)
ok github.com/carlosframework/platform 30.771s
=== root@1 (44 s)
ok github.com/carlosframework/platform 42.579s
=== root@2 (41 s)
ok github.com/carlosframework/platform 39.882s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.025s
=== RUN TestT4StorageProbeMinio
2026/10/21 23:43:54 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0B01004D850ED, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.023s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md| new file mode 100644 |
| index 000000000..cab648472 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-1-lifetimes-and-order.md |
| @@ -0,0 +1,237 @@ |
| +# Self-serve moves, plan 1: record lifetimes and serving order |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Give every instance record a durable lifetime (the host incarnation slot) and every served row a serving order `(incarnation, serving_seq)` that boxes publish and edges honour, with crash-safe create, retirement and migration, so a later plan's holding pages and moves can never be out-ordered by a stale publication. |
| + |
| +**Architecture:** A per-hostname slot object in the deployment bucket (`control/host-incarnations/<host>.json`), created once and changed only by conditional writes, is the authority for a record's lifetime; the console allocates and retires it through persisted intents in SQLite and recovers them at start. Records gain `incarnation` and `move_seq`; registry rows gain the pair; the edges file publishes it; the home map orders by it before the existing claim stamp. Retirement replaces the record with a tombstone before retiring the slot. Lazy migration marks a record by CAS before binding a slot to it. |
| + |
| +**Tech Stack:** Go. `internal/provision` (records, reconcile), `internal/registry`, `internal/console` (doors, store, passes), `internal/host` (edges file, offline move steps), `internal/edge` (home map). Tests use `release.DirStore`, real registries, the console's doors in-process, and the existing offline two-box witness. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md`, §6.4 ("Ordering by a per-host sequence", "Lifetimes: the host incarnation slot", "Aliases", "Freshness") and §11. Plans 2 to 4 build on this one. |
| + |
| +## Global Constraints |
| + |
| +- Every record consumer treats a record that fails `RecordLifetimeValid` (Task 1) as absent: reconcile, the console doors, the boxes' publication. |
| +- The slot is never deleted and never advanced to abandon a create; a `pending` allocation is replayed or superseded, never dropped (spec §6.4). |
| +- No console write to a bucket outside the deployment bucket; no root-side change in this plan. |
| +- Every new root-state read or changed unit in `internal/host` or `internal/provision` takes a row in a new Plan 14 census overlay `plan14-root-reads-self-serve-moves-1.tsv`, loaded last, `rev3=PROPOSED-self-serve-moves-1`, with a note `docs/superpowers/notes/census-proposal-self-serve-moves-1.md`; run `TestPlan14RootReadCensus` and `TestPlan9B8RecordWriteCensus`. |
| +- Every user-facing sentence is a copy row, pending Paul's review; the one in this plan is the delete door's "the instance changed. read it again and try the delete once more". |
| +- No tenant names in code, tests or copy: `shop.example.com`, account `bab`, app `shop`. |
| +- Boxes roll before the console (the roll order rule) so every box understands the migration marker before any migration runs; the migration pass therefore checks `CARLOS_LIFETIMES_MIGRATE=1` on the console, default off, until the fleet is rolled. |
| +- Local tests: only the touched packages with `-run`; CI runs the full suite. Go needs the sandbox off; never set `GOCACHE`. |
| + |
| +## Review Focus |
| + |
| +1. A console crash between the slot write and the record write, with a second console start before the first process's write has landed: the replay and the late write must both be harmless (Task 3's recovery test with an injected late write). |
| +2. A record updated by a box-side offline transition during migration: the marker must make the box's CAS fail and its retry succeed after the field is set (Task 5's concurrent-writer test). |
| +3. An alias repointed to another instance while the first is held: the different-target rule must choose the new target whatever the pairs (Task 8's alias test). |
| +4. A legacy record with no incarnation beside a retired slot (an old console's late create): every consumer must treat it as absent and the next create must replace it (Task 6's validity test and Task 3's stale-record branch). |
| +5. A reconstructed registry after a finished move: reconcile must write the record's pair with the minted row so the row outranks a stale holding publication (Task 7's reconciliation test). |
| + |
| +--- |
| + |
| +### Task 1: Record fields, the slot type and the validity rule |
| + |
| +**Files:** |
| +- Modify: `internal/provision/provision.go` (the `Record` struct, after `Gate`) |
| +- Create: `internal/provision/lifetime.go` |
| +- Test: `internal/provision/lifetime_test.go`, `internal/provision/provision_test.go` (the additivity test) |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `Record.Incarnation uint64 \`json:"incarnation,omitempty"\``, `Record.IncarnationMigrating string \`json:"incarnation_migrating,omitempty"\``, `Record.MoveSeq uint64 \`json:"move_seq,omitempty"\``, `Record.Retired bool \`json:"retired,omitempty"\``. |
| + - `type HostIncarnation struct { Incarnation uint64; State string; Intent string }` with `json` names `incarnation`, `state`, `intent`; `const SlotLive = "live"`, `SlotRetired = "retired"`. |
| + - `func HostIncarnationKey(host string) string` returning `control/host-incarnations/<host>.json`. |
| + - `func ReadHostIncarnation(ctx, s release.Store, host string) (*HostIncarnation, string, error)`: nil, "", nil when absent. |
| + - `func WriteHostIncarnation(ctx, s release.Store, host string, slot HostIncarnation, ifMatch string) (string, error)`: `ifMatch == ""` is create-only (`If-None-Match: *`), otherwise `If-Match`. |
| + - `func RecordLifetimeValid(rec *Record, slot *HostIncarnation) bool`. |
| + |
| +- [ ] **Step 1: Write the failing tests** in `lifetime_test.go`: |
| + - `TestRecordLifetimeValid`: a table over (record, slot) with the spec's rules: tombstone → false; fieldless and no slot → true; fieldless with marker `m1` and slot `{1, live, "migration m1"}` → true; fieldless with marker and slot `{1, live, "create c9"}` → false; fieldless, no marker, slot `{1, live, ...}` → false; fieldless and slot `{1, retired}` → false; `Incarnation 2` and slot `{2, live}` → true; `Incarnation 1` and slot `{2, live}` → false; `Incarnation 2` and slot `{2, retired}` → false; `Incarnation 2` and no slot → false. |
| + - `TestHostIncarnationWrites`: against `release.NewDirStore(t.TempDir())`: create-only succeeds once and fails the second time with `release.ErrCASConflict`; `If-Match` with the returned ETag succeeds; with a stale ETag fails; `ReadHostIncarnation` on an absent key returns nil, "", nil. |
| + - In `provision_test.go`, extend `TestRecordWireShapeIsAdditive` so a record without the four new fields round-trips byte-identical and one with them keeps them. |
| +- [ ] **Step 2: Run them to verify they fail**: `go test ./internal/provision -run 'RecordLifetimeValid|HostIncarnationWrites|RecordWireShapeIsAdditive' -count=1` fails with undefined symbols. |
| +- [ ] **Step 3: Implement** the fields and `lifetime.go` as named above; `WriteHostIncarnation` marshals with `json.Marshal` and calls `s.Put(ctx, key, body, ifMatch)`; `ReadHostIncarnation` maps the store's not-found to nil. |
| +- [ ] **Step 4: Run the tests to verify they pass.** |
| +- [ ] **Step 5: Commit** `provision: record lifetime fields, the host incarnation slot and the validity rule`. |
| + |
| +### Task 2: Console tables for pending lifetimes |
| + |
| +**Files:** |
| +- Modify: `internal/console/store.go` (`migrate`, after the `account_plans` table) |
| +- Create: `internal/console/lifetimes_store.go` |
| +- Test: `internal/console/lifetimes_store_test.go` |
| + |
| +**Interfaces:** |
| +- Produces, on `*Store`: |
| + - tables `host_allocations (host TEXT, incarnation INTEGER, create_id TEXT PRIMARY KEY, state TEXT, body BLOB, predecessor_etag TEXT, created_at INTEGER)`, `record_retirements (host TEXT, incarnation INTEGER, retire_id TEXT PRIMARY KEY, record_etag TEXT, slot_etag TEXT, state TEXT, created_at INTEGER)`, `record_migrations (host TEXT PRIMARY KEY, migration_id TEXT, state TEXT, created_at INTEGER)`. |
| + - `func (s *Store) BeginAllocation(host string, incarnation uint64, createID string, body []byte, predecessorETag string) error`, `func (s *Store) SetAllocationState(createID, state string) error`, `func (s *Store) PendingAllocations() ([]Allocation, error)`, `func (s *Store) LastIncarnation(host string) (uint64, error)` (the highest committed or pending for the host, 0 when none). |
| + - `type Allocation struct { Host string; Incarnation uint64; CreateID string; Body []byte; PredecessorETag string; State string }`. |
| + - `func (s *Store) BeginRetirement(host string, incarnation uint64, retireID, recordETag, slotETag string) error`, `func (s *Store) SetRetirementState(retireID, state string) error`, `func (s *Store) PendingRetirements() ([]Retirement, error)`; `type Retirement struct { Host string; Incarnation uint64; RetireID, RecordETag, SlotETag, State string }`. |
| + - `func (s *Store) BeginMigration(host, migrationID string) error`, `func (s *Store) SetMigrationState(host, state string) error`, `func (s *Store) PendingMigrations() ([]Migration, error)`; `type Migration struct { Host, MigrationID, State string }`. |
| + - States: `pending`, `committed`, `superseded`, `void`, `refused`. |
| +- [ ] **Step 1: Write the failing tests**: `TestLifetimeTablesRoundTrip` (begin, list pending, set state, list again) for all three; `TestLastIncarnationCountsPending` (a pending allocation at 2 makes `LastIncarnation` 2). |
| +- [ ] **Step 2: Run to verify they fail.** |
| +- [ ] **Step 3: Implement** with the store's existing `migrate` pattern (`CREATE TABLE IF NOT EXISTS`) and plain SQL. |
| +- [ ] **Step 4: Run to verify they pass**: `go test ./internal/console -run 'LifetimeTables|LastIncarnation' -count=1`. |
| +- [ ] **Step 5: Commit** `console: tables for pending allocations, retirements and migrations`. |
| + |
| +### Task 3: The create protocol and its recovery |
| + |
| +**Files:** |
| +- Create: `internal/console/lifetimes_create.go`, `internal/console/lifetimes_recovery.go` |
| +- Modify: `internal/console/apiinstances.go` (the create door around line 1664, `provision.CreateRecord`), `internal/console/apiinstanceseed.go:341`, `internal/console/apiinstanceimport.go:172`, `cmd/console/main.go` (start the recovery pass beside the other `RunClaimWorker` passes) |
| +- Test: `internal/console/lifetimes_create_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's slot helpers and fields; Task 2's store methods; `provision.WriteRecord` (create-only when the ETag is "") and `release.DeleteIfMatch`. |
| +- Produces: `func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record) error`, the one function every creation path calls; `func (s *Server) hostnameMutex(host string) *sync.Mutex`; `func (s *Server) recoverLifetimes(ctx context.Context) error` and `StartLifetimeRecoveryPass` (runs `recoverLifetimes` once at start and then every 60 s). |
| + |
| +The protocol, in `createRecordWithLifetime`: lock the hostname; read the slot (ETag P or absent); N = `LastIncarnation(host) + 1` (and at least slot.Incarnation + 1); `BeginAllocation(host, N, createID, body, P)`; `WriteHostIncarnation({N, live, "create " + createID}, P)` (create-only when absent); set `rec.Incarnation = N`; create-only `WriteRecord`; if the key holds a stale record (a tombstone, or a record whose lifetime is not valid against the new slot) delete it with `DeleteIfMatch` on its ETag and write again; `SetAllocationState(committed)`. Recovery, per pending allocation, exactly the branches of spec §6.4: slot absent or at the predecessor → write the slot; slot `{N, live, this create}` and record absent → write the record; slot at N and record at N → committed; slot at N and stale record → conditional delete then write; slot past N → superseded; unreadable → leave pending. |
| + |
| +- [ ] **Step 1: Write the failing tests** against an in-process console with `release.NewDirStore`: |
| + - `TestCreateAllocatesSlotThenRecord`: after a create, the slot is `{1, live, "create <id>"}`, the record carries `Incarnation 1`, the allocation is `committed`. |
| + - `TestCreateAfterRetirementGetsNextIncarnation`: slot `{1, retired}` seeded; a create yields `{2, live}` and a record at 2. |
| + - `TestCreateReplacesStaleRecord`: a fieldless record under the key beside slot `{1, retired}`; a create deletes it conditionally and writes incarnation 2. |
| + - `TestRecoveryReplaysCreateAtItsIncarnation`: a pending allocation at 1 with the slot written and no record; recovery writes the record at 1 and commits; then an injected "late original write" (create-only, same body) fails with `release.ErrCASConflict` and changes nothing. |
| + - `TestRecoveryWritesTheSlotWhenMissing`: pending allocation, slot absent; recovery writes the slot then the record. |
| + - `TestRecoveryLeavesALandedCreateServing`: slot and record at 1, allocation still pending; recovery marks it committed and rewrites nothing (the record ETag is unchanged). |
| + - `TestRecoverySupersedesWhenSlotMovedPast`: slot at 2, pending allocation at 1 → superseded; no record write. |
| + - `TestRecoveryLeavesPendingWhenUnreadable`: a store whose `Get` of the slot fails → state stays pending, no writes. |
| + - `TestConcurrentCreatesOfOneHostnameSerialise`: two creates in parallel; one succeeds at 1, the other fails the create-only record write with the existing 409 and allocates nothing committed. |
| +- [ ] **Step 2: Run to verify they fail.** |
| +- [ ] **Step 3: Implement** `lifetimes_create.go` and `lifetimes_recovery.go`; route the three creation paths through `createRecordWithLifetime`; start the pass in `cmd/console/main.go`. |
| +- [ ] **Step 4: Run the tests and the existing create-door tests** (`go test ./internal/console -run 'Create|Lifetime|Recovery' -count=1`). |
| +- [ ] **Step 5: Commit** `console: creates allocate a host incarnation through a persisted intent, with recovery`. |
| + |
| +### Task 4: Retirement through a tombstone |
| + |
| +**Files:** |
| +- Create: `internal/console/lifetimes_retire.go` |
| +- Modify: `internal/console/apiinstances.go` (`retireInstanceRecord`, called from `handleAPIInstanceDelete`), `internal/provision/provision.go` (`GetRecord`, `GetRecordETag`: a tombstone reads as not found for every caller except the lifetime code, which uses a new `GetRecordRaw`), `internal/console/lifetimes_recovery.go` (the retirement branches) |
| +- Test: `internal/console/lifetimes_retire_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 2's retirement rows; Task 1's slot helpers; `release.DeleteIfMatch`; `provision.WriteRecord` with an `If-Match` ETag. |
| +- Produces: `func (s *Server) retireRecordWithLifetime(ctx context.Context, acct, app, host string) error` replacing the plain delete inside `retireInstanceRecord`; `func provision.GetRecordRaw(ctx, s, acct, app, host) (*Record, string, error)` (tombstones included); `func provision.Tombstone(host string, incarnation uint64) Record` (`Host`, `Incarnation`, `Retired: true`, nothing else). |
| + |
| +The protocol: read the record and the slot (ETags E, S); `BeginRetirement(host, N, id, E, S)`; (1) `WriteRecord` the tombstone with `If-Match: E`; on a CAS failure reread: a record at N that is not a tombstone → `refused` with the copy sentence; another incarnation or slot past N → `void`; (2) `WriteHostIncarnation({N, retired, "retire " + id}, S)`; (3) `DeleteIfMatch` the tombstone on its ETag; `committed`. A record with no `Incarnation` (legacy, no slot) keeps today's plain delete, after writing a slot `{1, retired}` create-only so the hostname's next create starts at 2. Recovery branches exactly as spec §6.4: record at E → step 1; tombstone present and slot at S → step 2; tombstone present and slot retired → step 3; record absent and slot retired → committed; slot retired and a non-tombstone record at N → committed after a conditional delete of that record; slot live and a non-tombstone record at N with another ETag → `refused`; another incarnation or slot past N with no tombstone → `void`; unreadable → pending. |
| + |
| +- [ ] **Step 1: Write the failing tests**: |
| + - `TestRetireWritesTombstoneThenRetiresSlotThenRemovesTombstone`: after the door, the record key is absent, the slot is `{1, retired}`, the retirement is `committed`; a probe `GetRecord` mid-way (after step 1, with a hook) returns not found while `GetRecordRaw` returns the tombstone. |
| + - `TestRetireRefusesWhenTheRecordChangedInItsLifetime`: the record's ETag moves between read and step 1 (a `SetRecordRegion` in between); the door answers 409 with the sentence; the record is kept; the retirement is `refused`. |
| + - `TestRetireVoidsAgainstASuccessor`: slot at 2 when the door read 1 → `void`, nothing written. |
| + - `TestLateCreateFailsOnTheTombstone`: with the tombstone in place, a create-only write fails; after step 3 with the slot retired, a late record at 1 is written by the test and `RecordLifetimeValid` is false for it. |
| + - `TestRetirementRecoveryBranches`: one subtest per branch above, each asserting the final slot, key state and retirement state. |
| + - `TestLegacyRecordRetiresWithASlot`: a fieldless record with no slot; the door deletes it and leaves `{1, retired}`. |
| +- [ ] **Step 2: Run to verify they fail.** |
| +- [ ] **Step 3: Implement.** `GetRecord`/`GetRecordETag` return the store's not-found error for a tombstone. |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'Retire|Tombstone|Lifetime' -count=1` and `go test ./internal/provision -run 'Record' -count=1`. |
| +- [ ] **Step 5: Commit** `console: retirement replaces the record with a tombstone before retiring the slot`. |
| + |
| +### Task 5: Lazy migration with a marker |
| + |
| +**Files:** |
| +- Create: `internal/console/lifetimes_migrate.go` |
| +- Modify: `internal/provision/provision.go` (`updateRecordSeeding`: a record carrying `IncarnationMigrating` makes the mutation wait and retry, bounded to 30 s, then returns `ErrRecordMigrating`), `cmd/console/main.go` (start `StartIncarnationMigrationPass` behind `CARLOS_LIFETIMES_MIGRATE=1`) |
| +- Test: `internal/console/lifetimes_migrate_test.go`, `internal/provision/provision_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 2's migration rows; Task 1's helpers. |
| +- Produces: `func (s *Server) migrateHostLifetime(ctx context.Context, acct, app, host string) error`; `StartIncarnationMigrationPass` (every 5 minutes, up to 50 hosts per pass, oldest records first, only records with no `Incarnation`); `var provision.ErrRecordMigrating = errors.New("provision: the record is being migrated; try again")`. |
| + |
| +The protocol: `BeginMigration(host, id)`; (1) CAS `IncarnationMigrating = id` against the ETag read; on conflict reread and retry from the current record (a legitimate update won); (2) `WriteHostIncarnation({1, live, "migration " + id}, "")` create-only, or `If-Match` when a slot at 1 with this intent exists from an earlier attempt; (3) CAS `Incarnation = 1`, `IncarnationMigrating = ""`; `committed`. Recovery branches as spec §6.4. |
| + |
| +- [ ] **Step 1: Write the failing tests**: |
| + - `TestMigrationMarksThenSlotsThenSets`: a fieldless record; after the pass the slot is `{1, live, "migration <id>"}`, the record has `Incarnation 1` and no marker. |
| + - `TestMigrationRestartsFromAnUpdateThatWon`: an update changes the record between the read and step 1; the marker CAS fails; the pass rereads and completes on the updated record; the update's field survives. |
| + - `TestWriterWaitsOnTheMarker`: with the marker set and no slot yet, `updateRecordSeeding` blocks until a goroutine completes the migration, then applies; with the marker never cleared it returns `ErrRecordMigrating` after 30 s (use a short bound via a test hook). |
| + - `TestMigrationRecoveryBranches`: marked and no slot → slot then set; marked and slot bound → set; `Incarnation 1` present → committed; record gone → superseded. |
| + - `TestMigrationPassIsOffWithoutTheFlag`. |
| +- [ ] **Step 2: Run to verify they fail.** |
| +- [ ] **Step 3: Implement.** |
| +- [ ] **Step 4: Run** `go test ./internal/console -run 'Migration|Migrat' -count=1` and `go test ./internal/provision -run 'Migrating|updateRecord' -count=1`. |
| +- [ ] **Step 5: Commit** `console: lazy incarnation migration marks the record by CAS before binding its slot`. |
| + |
| +### Task 6: Consumers check the lifetime |
| + |
| +**Files:** |
| +- Modify: `internal/provision/reconcile.go` (the ready-row minting loop near line 1258, `holdExisting`, `mintHeld`), `internal/console/apiinstances.go` (`instanceGate`'s record read), `internal/host/edgesfile.go` (the publication loop: a row whose record fails validity is not published) |
| +- Create: `internal/provision/lifetime_cache.go` (`type SlotCache` keyed by host with the slot's ETag, refreshed per reconcile pass) |
| +- Test: `internal/provision/reconcile_lifetime_test.go`, `internal/console/apiinstances_lifetime_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 1's `RecordLifetimeValid`, `ReadHostIncarnation`. |
| +- Produces: `func (c *SlotCache) Valid(ctx context.Context, s release.Store, rec *Record) (bool, error)`; `ReconcileConfig.Slots *SlotCache` (nil keeps today's behaviour for tests that predate this; production sets it). |
| + |
| +- [ ] **Step 1: Write the failing tests**: |
| + - `TestReconcileSkipsARecordBehindItsSlot`: a record at 1 with slot `{2, live}` mints no row and retires an existing row as "record gone". |
| + - `TestReconcileSkipsATombstone`. |
| + - `TestReconcileKeepsALegacyRecordWithNoSlot` and `...AMarkedRecordUnderItsMigrationSlot`. |
| + - `TestInstanceGateAnswers404ForAStaleRecord` (the console door). |
| + - `TestEdgesFileOmitsAStaleRecordsRow`. |
| +- [ ] **Step 2: Run to verify they fail.** |
| +- [ ] **Step 3: Implement**, reading the slot only for records that carry `Incarnation` or a marker (legacy records cost no read). |
| +- [ ] **Step 4: Run** the touched packages' targeted tests, then `go test ./internal/provision -run 'TestPlan14RootReadCensus|TestPlan9B8RecordWriteCensus' -count=1`; add the census overlay rows the test names (the slot read inside reconcile is a bucket read, not a root read; expect envelope re-pins for `internal/host`). |
| +- [ ] **Step 5: Commit** `lifetimes: reconcile, the doors and the edges file treat a record behind its slot as absent`. |
| + |
| +### Task 7: `move_seq` on the record and the pair on rows |
| + |
| +**Files:** |
| +- Modify: `internal/provision/offline_move.go` (`offlineBegin`, `OfflineFlip`, `offlineOpenMut`'s transition, `OfflineAbort`'s reopening, `OfflineFinish`: each increments `rec.MoveSeq`, initialising to 1 when 0; the move state gains `Seq uint64 \`json:"seq,omitempty"\`` carrying the sequence of the transition that produced it), `internal/registry/registry.go` (columns `incarnation INTEGER NOT NULL DEFAULT 0`, `serving_seq INTEGER NOT NULL DEFAULT 0`; `Route.Incarnation uint64`, `Route.ServingSeq uint64`; `Upsert` writes them when non-zero; `SetHold` and `ReleaseHold` gain variants `SetHoldSeq(host, status, op string, incarnation, seq uint64)` and `ReleaseHoldSeq(host, op string, incarnation, seq uint64)` that write the pair in the same statement), `internal/provision/reconcile.go` (`RouteFor` copies `rec.Incarnation` and `rec.MoveSeq` into the minted row), `internal/host/movesoffline.go` (the hold reconcile hook passes the move's `Seq`), `internal/host/movesoffline_land.go` (the target's held mint and its open carry `open`'s seq), the abort path's reopening (the sealer's `ReleaseHoldSeq` with the abort's seq) |
| +- Test: `internal/provision/offline_move_test.go`, `internal/registry/registry_test.go`, `internal/host/movesoffline_test.go` (the two-box witness) |
| + |
| +**Interfaces:** |
| +- Produces: the record's `MoveSeq` and `Move.Seq`; `registry.Route.Incarnation`, `registry.Route.ServingSeq`; `SetHoldSeq`, `ReleaseHoldSeq`. |
| + |
| +- [ ] **Step 1: Write the failing tests**: |
| + - `TestOfflineTransitionsAdvanceMoveSeq`: start → 1, open → 2, finish → 3 on one record; abort after start → 2 (the reopening) and finish → 3; a record with `MoveSeq 5` starts at 6. |
| + - `TestRegistryPairColumns`: upsert with the pair, read back; `SetHoldSeq` writes status, op and pair in one statement; `ReleaseHoldSeq` the same. |
| + - `TestRouteForCopiesThePair` (a record `{Incarnation 2, MoveSeq 3}` mints a row `(2, 3)`; a record with neither mints `(0, 0)`). |
| + - In the two-box witness: after `start` the source row is held with `(1, 1)`; after `open` the target row is ready with `(1, 2)`; after an abort from landed the source row is ready with `(1, 2)` and the target has no row. |
| +- [ ] **Step 2: Run to verify they fail.** |
| +- [ ] **Step 3: Implement.** The `Upsert` conflict clause keeps the pair when the incoming value is 0 (`CASE WHEN excluded.serving_seq = 0 THEN routes.serving_seq ELSE excluded.serving_seq END`, and the same for `incarnation`). |
| +- [ ] **Step 4: Run** `go test ./internal/registry -run 'Pair|Hold' -count=1`, `go test ./internal/provision -run 'MoveSeq|RouteFor|Offline' -count=1`, `go test ./internal/host -run 'Offline' -count=1`, then the two census tests; add rows for the changed `internal/host` units (`offlineHold`, the landing's mint, the abort's release) at their existing classes. |
| +- [ ] **Step 5: Commit** `moves: the offline transitions advance move_seq and rows carry (incarnation, serving_seq)`. |
| + |
| +### Task 8: Publish the pair and order by it |
| + |
| +**Files:** |
| +- Modify: `internal/host/edgesfile.go` (`edgeHost` gains `Incarnation uint64 \`json:"incarnation,omitempty"\``, `ServingSeq uint64 \`json:"serving_seq,omitempty"\``, `TargetHost string \`json:"target_host,omitempty"\``; routes publish the row's pair; an alias entry publishes its primary's pair and `TargetHost: a.Host`; a catch-all's derived entry publishes `TargetHost` too), `internal/edge/homemap.go` (`edgesHostView` and `homeEntry` gain the three fields; `dialBeats` compares `(incarnation, servingSeq)` first, then the existing alias and stamp rules; for an alias host, an entry whose `targetHost` differs from the previous winner's wins outright when it is a ready route, before the pair comparison) |
| +- Test: `internal/host/edgesfile_test.go`, `internal/edge/homemap_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: the three fields on the wire; `dialBeats` semantics per spec §6.4: higher pair wins; equal pairs fall to today's rules; an alias ready route for a different `target_host` wins over an entry for another target. |
| + |
| +- [ ] **Step 1: Write the failing tests**: |
| + - `TestEdgesFilePublishesThePairAndTargetHost`: a row `(2, 3)` publishes `incarnation 2, serving_seq 3`; its alias publishes the same pair with `target_host` the row's host; a derived catch-all alias publishes `target_host`. |
| + - `TestFoldPrefersTheHigherPair`: two files for one host, `(1, 2)` and `(1, 1)` → the former; `(2, 0)` beats `(1, 9)`; `(0, 0)` loses to any pair; equal pairs → today's stamp rule (a test case with equal pairs and different stamps). |
| + - `TestFoldAliasDifferentTargetWins`: alias `customer.example.com` published by box A for target `a.example` at `(1, 9)` and by box B for target `b.example` at `(1, 0)` → B, because the targets differ and B's is a ready route (the later publication); same targets → the pair rule. |
| + - `TestFoldLegacyPublicationsUnchanged`: files without the fields fold exactly as before (reuse an existing fold test's fixtures). |
| +- [ ] **Step 2: Run to verify they fail.** |
| +- [ ] **Step 3: Implement.** For the different-target rule, `foldHomes` must know which entry is newer; with no sequence between targets, the rule is: when two alias entries name different targets, the one whose `target_host` has a ready route in the same fold wins; if both do, the higher pair; this is the "ready route wins an alias for one refresh cycle" rule, and the test pins it. |
| +- [ ] **Step 4: Run** `go test ./internal/host -run 'EdgesFile' -count=1` and `go test ./internal/edge -run 'Fold|HomeMap' -count=1`, then the census tests (the edges writer is a changed unit). |
| +- [ ] **Step 5: Commit** `edges: publish (incarnation, serving_seq) and target_host; the home map orders by the pair`. |
| + |
| +### Task 9: Lifetimes end to end |
| + |
| +**Files:** |
| +- Test: `internal/host/movesoffline_lifetime_test.go` (the two-box offline witness, extended), `internal/console/lifetimes_e2e_test.go` |
| + |
| +- [ ] **Step 1: Write the tests** (each is one of §11's witnesses that Plan 1 owns): |
| + - `TestLifetimeAcrossAMoveAndAnAbort`: create → `(1, 0)` served; start → source held `(1, 1)` and the fold prefers the hold's publication over a retained `(1, 0)` route; open → target `(1, 2)` wins; abort from landed instead → source reopened `(1, 2)` wins over a retained target hold publication `(1, 1)`. |
| + - `TestReconstructedRowCarriesTheRecordsPair`: after a finished move (`move_seq 3`), delete the target's registry row, reconcile, and the new row is `(1, 3)`; it beats a retained `(1, 1)` publication. |
| + - `TestRecreatedHostOutranksItsPreviousLifetime`: retire (slot `{1, retired}`), create again → `(2, 0)` beats a retained `(1, 9)`. |
| + - `TestLateLegacyCreateIsInert`: slot `{1, retired}`; a fieldless record written under the key by the test; reconcile mints nothing; the door 404s; the next create replaces it. |
| +- [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| +- [ ] **Step 3: Run the census tests and both cross-builds** (`GOOS=darwin go build ./internal/host/`, `GOOS=windows go vet ./internal/host/`). |
| +- [ ] **Step 4: Commit** `lifetimes: end-to-end witnesses across a move, an abort, a reconstruction and a recreation`; write the census note; push. |
| + |
| +--- |
| + |
| +## Self-review notes |
| + |
| +- Spec coverage for §6.4's lifetime and ordering paragraphs: slot (T1, T3, T4), recovery branches (T3, T4, T5), validity rule and consumers (T1, T6), `move_seq` and the pair (T7), publication and fold (T8), legacy and migration (T5, T6), witnesses (T9). Holding entries, their five-minute validity and the probe mode are Plan 3; the move door, job and reservations Plan 2; cleanup Plan 4. |
| +- Names used across tasks: `RecordLifetimeValid`, `ReadHostIncarnation`, `WriteHostIncarnation`, `HostIncarnation{Incarnation, State, Intent}`, `createRecordWithLifetime`, `retireRecordWithLifetime`, `migrateHostLifetime`, `GetRecordRaw`, `Tombstone`, `SlotCache.Valid`, `SetHoldSeq`, `ReleaseHoldSeq`, `Route.Incarnation`, `Route.ServingSeq`, `edgeHost.Incarnation/ServingSeq/TargetHost`. |
| +- Review Focus 1 to 5 are pinned by T3, T5, T8, T6+T3 and T7 respectively. |