amadan
design: self-serve offline moves, rev 16 (Astra round 15)
Lazy migration first CAS-writes a marker into the record, which fences every console and box writer and makes an already-issued write fail its CAS; the slot is bound to the marked record; writers wait on the marker and retry; a fieldless record is valid unmigrated, or marked during migration, and stale otherwise; recovery branches for every migration boundary.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+4
−4
docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md+4 −4
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (1 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (2 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (146 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.768s
ok github.com/carlosframework/platform/cmd/moveprobe 11.951s
ok github.com/carlosframework/platform/cmd/zddprobe 0.787s
ok github.com/carlosframework/platform/internal/activator 58.435s
ok github.com/carlosframework/platform/internal/agentsock 0.020s
ok github.com/carlosframework/platform/internal/alerting 0.006s
ok github.com/carlosframework/platform/internal/analytics 0.021s
ok github.com/carlosframework/platform/internal/appconfig 3.064s
ok github.com/carlosframework/platform/internal/apperr 0.117s
ok github.com/carlosframework/platform/internal/applog 0.300s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.005s
ok github.com/carlosframework/platform/internal/boxid 0.743s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.009s
ok github.com/carlosframework/platform/internal/caps 0.013s
ok github.com/carlosframework/platform/internal/certwarm 0.002s
ok github.com/carlosframework/platform/internal/cgroup 0.788s
ok github.com/carlosframework/platform/internal/changelog 0.589s
ok github.com/carlosframework/platform/internal/claimactivation 1.363s
ok github.com/carlosframework/platform/internal/clamd 0.061s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.991s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.010s
ok github.com/carlosframework/platform/internal/dbverify 0.004s
ok github.com/carlosframework/platform/internal/dnsrec 0.002s
ok github.com/carlosframework/platform/internal/docsite 0.009s
ok github.com/carlosframework/platform/internal/domains 2.936s
ok github.com/carlosframework/platform/internal/doorkey 0.008s
ok github.com/carlosframework/platform/internal/economics 0.006s
ok github.com/carlosframework/platform/internal/edge 45.591s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.011s
ok github.com/carlosframework/platform/internal/fleetapi 0.166s
ok github.com/carlosframework/platform/internal/gate 1.949s
ok github.com/carlosframework/platform/internal/genselect 0.020s
ok github.com/carlosframework/platform/internal/geo 0.017s
ok github.com/carlosframework/platform/internal/geoip 0.053s
ok github.com/carlosframework/platform/internal/historyreadback 0.276s
ok github.com/carlosframework/platform/internal/host 142.587s
ok github.com/carlosframework/platform/internal/hostactors 6.199s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.005s
ok github.com/carlosframework/platform/internal/hostbox 3.289s
ok github.com/carlosframework/platform/internal/hostclaim 0.818s
ok github.com/carlosframework/platform/internal/hostenv 0.005s
ok github.com/carlosframework/platform/internal/hostevidence 0.007s
ok github.com/carlosframework/platform/internal/hostexec 5.631s
ok github.com/carlosframework/platform/internal/hostfence 0.058s
ok github.com/carlosframework/platform/internal/hostfiles 10.326s
ok github.com/carlosframework/platform/internal/hostfloor 0.024s
ok github.com/carlosframework/platform/internal/hostguard 1.057s
ok github.com/carlosframework/platform/internal/hostops 2.143s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.012s
ok github.com/carlosframework/platform/internal/hostquarantine 0.003s
ok github.com/carlosframework/platform/internal/hostrelocate 0.671s
ok github.com/carlosframework/platform/internal/hostresolve 0.239s
ok github.com/carlosframework/platform/internal/hostwrite 0.009s
ok github.com/carlosframework/platform/internal/infraview 0.009s
ok github.com/carlosframework/platform/internal/instancectl 5.248s
ok github.com/carlosframework/platform/internal/instanceexec 2.800s
ok github.com/carlosframework/platform/internal/instancefs 0.009s
ok github.com/carlosframework/platform/internal/instancekey 0.011s
ok github.com/carlosframework/platform/internal/instanceupdates 0.015s
ok github.com/carlosframework/platform/internal/layout 1.940s
ok github.com/carlosframework/platform/internal/lbhealth 0.003s
ok github.com/carlosframework/platform/internal/lease 10.108s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.602s
ok github.com/carlosframework/platform/internal/ledger 0.014s
ok github.com/carlosframework/platform/internal/mail 0.254s
ok github.com/carlosframework/platform/internal/maintenance 0.082s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.601s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.102s
ok github.com/carlosframework/platform/internal/metrics 0.005s
ok github.com/carlosframework/platform/internal/moves 21.375s
ok github.com/carlosframework/platform/internal/moves/archive 1.124s
ok github.com/carlosframework/platform/internal/objstore 0.285s
ok github.com/carlosframework/platform/internal/ops 0.184s
ok github.com/carlosframework/platform/internal/park 0.276s
ok github.com/carlosframework/platform/internal/plans 0.014s
ok github.com/carlosframework/platform/internal/privatefile 0.005s
ok github.com/carlosframework/platform/internal/provision 13.552s
ok github.com/carlosframework/platform/internal/publicrelease 0.008s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.570s
ok github.com/carlosframework/platform/internal/relay 7.233s
ok github.com/carlosframework/platform/internal/release 25.335s
ok github.com/carlosframework/platform/internal/rollout 3.826s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootparticipation 0.059s
ok github.com/carlosframework/platform/internal/rootstate 0.029s
ok github.com/carlosframework/platform/internal/runs 0.504s
ok github.com/carlosframework/platform/internal/s3compat 0.006s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.659s
ok github.com/carlosframework/platform/internal/seeds 0.011s
ok github.com/carlosframework/platform/internal/sim/hostlife 11.337s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.525s
ok github.com/carlosframework/platform/internal/status 0.077s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.108s
ok github.com/carlosframework/platform/internal/svcrole 0.005s
ok github.com/carlosframework/platform/internal/telemetry 0.007s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.005s
ok github.com/carlosframework/platform/internal/unixdial 0.006s
ok github.com/carlosframework/platform/internal/updatewords 0.005s
ok github.com/carlosframework/platform/internal/usage 0.006s
ok github.com/carlosframework/platform/internal/version 0.003s
ok github.com/carlosframework/platform/internal/vet 1.653s
=== internal_console@0 (78 s)
ok github.com/carlosframework/platform/internal/console 75.754s
=== internal_console@1 (38 s)
ok github.com/carlosframework/platform/internal/console 35.532s
=== internal_console@2 (43 s)
ok github.com/carlosframework/platform/internal/console 40.561s
=== internal_console@3 (37 s)
ok github.com/carlosframework/platform/internal/console 35.135s
=== internal_console@4 (38 s)
ok github.com/carlosframework/platform/internal/console 35.283s
=== internal_console@5 (36 s)
ok github.com/carlosframework/platform/internal/console 33.533s
=== internal_console@6 (37 s)
ok github.com/carlosframework/platform/internal/console 34.485s
=== internal_console@7 (44 s)
ok github.com/carlosframework/platform/internal/console 41.932s
=== root@0 (32 s)
ok github.com/carlosframework/platform 30.777s
=== root@1 (44 s)
ok github.com/carlosframework/platform 42.565s
=== root@2 (41 s)
ok github.com/carlosframework/platform 39.963s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50069666 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.03s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.02s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.030s
=== RUN TestT4StorageProbeMinio
2026/10/21 23:35:02 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E0AF942B142A9E, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.026s
minio contract: PASS
diff --git a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md| index 8b944f4e2..2ee162eda 100644 |
| --- a/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| +++ b/docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md |
| @@ -1,6 +1,6 @@ |
| # Self-serve offline moves: a tenant's customer moves their own instance |
| -Rev 15, 2026-10-22. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. Rev 1 (062387a1) was reviewed by Astra (`after236/selfserve-spec-review.md`, NOT READY, 14 findings) and by a cold builder check (7 questions); rev 2 (0e58e51d) answered them and was reviewed again (`selfserve-spec-review2.md`, NOT READY, 12 findings). Rev 2's automatic return after `open` was the source of most of round 2: whether the new region ever accepted a write cannot be proven from the bucket (litestream releases the lease even when its final sync failed; file-only changes leave no LTX; an absent lock does not tell "never taken" from "released"), and a safe post-open revert would need a new admission barrier and stop protocol for the target, which is Phase 1's work. **Rev 3 narrows the promise to what is provable**: every failure before `open` returns the account to the old region automatically and losslessly, everything checkable is checked before `open`, and a failure after `open` is paged with the holding page kept until the app answers, the return being an operator's run of the same offline move the other way. This changes one of Paul's rulings and is §10 Q3. Rev 3 (b77c01a9) was reviewed again (`selfserve-spec-review3.md`, NOT READY, 8 findings, the abort boundary accepted); rev 4 answers them: which failures stay after `open` (§7.5), the job's durable admission order (§7), cleanup's own reservation and the root handoff (§7.6), the holding entries' causal ordering (§6.4), the probe mode (§7.4), deadlines as initiate-and-retry with blocked states (§7.2), and the stale contradictions. Rev 4 (20fdcf18) was reviewed again (`selfserve-spec-review4.md`, NOT READY, 2 P1 and 2 P2: holding entries ordered by move identity cannot order consecutive moves or show an abort completed; a timed-out root cleanup request could run after its reservation was released; the queued handshake's producers; stale text). Rev 5 answers them with a per-host `move_seq` published with every route and holding entry (§6.4), a root request that ends only with root's `done` or acknowledged `cancelled` (§7.6), queued defined as no copying with the participants' reports allowed (§7), and the stale passages aligned. Rev 5 (89e0f37a) was reviewed again (`selfserve-spec-review5.md`, NOT READY, 1 P1 and 2 P2: a serving row rebuilt by ordinary reconciliation after a finished move would carry sequence 0 and lose to a stale holding entry, and a recreated host's publications need separating from a previous life's; the queued table order; two stale sentences). Rev 6 answers them: reconciliation persists the record's sequence with a row it mints after a move (§6.4), a per-record `move_lineage` separates lives (§6.4), the table order and the two sentences are fixed. Rev 6 (bc5e2d97) was reviewed again (`selfserve-spec-review6.md`, NOT READY, 1 P1: a random lineage identifies but does not order lifetimes, and a lineage-less publication retained by an edge through a read failure can coexist with the first move's holding entry). Rev 7 replaces the lineage with a durable per-hostname incarnation number the console allocates and never reuses, so every publication orders as (incarnation, sequence) and a publication without one loses (§6.4). Rev 7 (f2feed6c) was reviewed again (`selfserve-spec-review7.md`, NOT READY, 1 P1: a custom domain repointed from one instance to another compares two unrelated counters). Rev 8 gives alias bindings their own durable revision from the domains record, published with the target hostname; edges pick the newest binding first and compare the target's pair within it (§6.4). Rev 8 (05f4c353) was reviewed again (`selfserve-spec-review8.md`, NOT READY, 1 P1: aliases derived from a wildcard catch-all have no binding record to carry a revision; 1 P2: retaining a detached domains record changes what a claim listing means). Rev 9 narrows: holding entries list explicit aliases only and derived aliases keep today's answer during a move; the binding revision lives in a console table that survives the domains record, so detach stays a delete (§6.4). Rev 9 (3bcea70e) was reviewed again (`selfserve-spec-review9.md`, NOT READY: a detached explicit alias can fall back to a derived one beneath a retained explicit holding entry; the revision table and the bucket write need a crash-safe protocol). Every remaining round-8 and round-9 finding was about a holding entry an edge keeps as last-good through a read failure and later orders against a new binding. **Rev 10 stops tolerating that case instead of ordering it:** a holding entry is never last-good, it is honoured only from a publication fetched within the last five minutes, and the alias revision machinery is dropped; the incarnation number is allocated durably before it is exposed (§6.4). Rev 10 (82023a80) was reviewed again (`selfserve-spec-review10.md`, NOT READY, 2 P1 and 1 P2: two concurrent creates of one hostname can commit incarnations out of order; a fresh hold must not displace a retained ready route for an alias that now names another instance; alias publications need the target hostname). Rev 11 answers them: creates and deletes of a hostname are serialised through a pending allocation that only a completed record write commits (§6.4); for an alias, any eligible ready route naming a different target wins over a holding entry, pairs are compared only within one target, and the five-minute bound is stated as a bound on reader disconnection, not publisher death (§6.4); every alias publication carries its target hostname (§6.4). Rev 11 (9ae3902f) was reviewed again (`selfserve-spec-review11.md`, NOT READY, 1 P1: recovery after a console crash could abandon an outstanding create whose record write later lands, because the record's "absent" precondition recurs after a deletion). Rev 12 makes the incarnation an authoritative bucket slot that never returns to absent and that every consumer checks the record against, so a late write materialises nothing (§6.4). Rev 12 (35bd391d) was reviewed again (`selfserve-spec-review12.md`, NOT READY, 2 P1: recovery that advances the slot would invalidate a create whose record did land; the ordinary delete is unconditional and a delayed one could erase a successor). Rev 13 answers them: recovery replays the create's own intent to completion at its allocated number and never advances a slot to abandon; deletes are conditional on the observed record ETag and retire only the captured slot revision, with persisted intent (§6.4). Rev 13 (696cb374) was reviewed again (`selfserve-spec-review13.md`, NOT READY, 1 P1: a retirement that exposes the record's absence before the slot is retired lets a late original create restore an obsolete record, and an ETag difference alone was read as a successor; 1 P2: allocation recovery omitted the crash before the slot write and a stale record under the key). Rev 14 answers them: retirement first replaces the record with a non-servable tombstone, so absence is never exposed while the slot is live, and every recovery branch of the create and the retirement is listed (§6.4). Rev 14 (bb05348f) was reviewed again (`selfserve-spec-review14.md`, NOT READY, 1 P1: a record without the incarnation field was accepted beside any existing slot, so an old console's outstanding create could land after a retirement; 1 P2: two retirement recovery states had no terminal disposition). Rev 15 answers them: a fieldless record beside a slot is valid only under a live migration intent bound to that record's key and ETag, and both retirement states get terminal outcomes (§6.4). |
| +Rev 16, 2026-10-22. Status: design for review (Paul). Branch `design-self-serve-offline-moves`. Rev 1 (062387a1) was reviewed by Astra (`after236/selfserve-spec-review.md`, NOT READY, 14 findings) and by a cold builder check (7 questions); rev 2 (0e58e51d) answered them and was reviewed again (`selfserve-spec-review2.md`, NOT READY, 12 findings). Rev 2's automatic return after `open` was the source of most of round 2: whether the new region ever accepted a write cannot be proven from the bucket (litestream releases the lease even when its final sync failed; file-only changes leave no LTX; an absent lock does not tell "never taken" from "released"), and a safe post-open revert would need a new admission barrier and stop protocol for the target, which is Phase 1's work. **Rev 3 narrows the promise to what is provable**: every failure before `open` returns the account to the old region automatically and losslessly, everything checkable is checked before `open`, and a failure after `open` is paged with the holding page kept until the app answers, the return being an operator's run of the same offline move the other way. This changes one of Paul's rulings and is §10 Q3. Rev 3 (b77c01a9) was reviewed again (`selfserve-spec-review3.md`, NOT READY, 8 findings, the abort boundary accepted); rev 4 answers them: which failures stay after `open` (§7.5), the job's durable admission order (§7), cleanup's own reservation and the root handoff (§7.6), the holding entries' causal ordering (§6.4), the probe mode (§7.4), deadlines as initiate-and-retry with blocked states (§7.2), and the stale contradictions. Rev 4 (20fdcf18) was reviewed again (`selfserve-spec-review4.md`, NOT READY, 2 P1 and 2 P2: holding entries ordered by move identity cannot order consecutive moves or show an abort completed; a timed-out root cleanup request could run after its reservation was released; the queued handshake's producers; stale text). Rev 5 answers them with a per-host `move_seq` published with every route and holding entry (§6.4), a root request that ends only with root's `done` or acknowledged `cancelled` (§7.6), queued defined as no copying with the participants' reports allowed (§7), and the stale passages aligned. Rev 5 (89e0f37a) was reviewed again (`selfserve-spec-review5.md`, NOT READY, 1 P1 and 2 P2: a serving row rebuilt by ordinary reconciliation after a finished move would carry sequence 0 and lose to a stale holding entry, and a recreated host's publications need separating from a previous life's; the queued table order; two stale sentences). Rev 6 answers them: reconciliation persists the record's sequence with a row it mints after a move (§6.4), a per-record `move_lineage` separates lives (§6.4), the table order and the two sentences are fixed. Rev 6 (bc5e2d97) was reviewed again (`selfserve-spec-review6.md`, NOT READY, 1 P1: a random lineage identifies but does not order lifetimes, and a lineage-less publication retained by an edge through a read failure can coexist with the first move's holding entry). Rev 7 replaces the lineage with a durable per-hostname incarnation number the console allocates and never reuses, so every publication orders as (incarnation, sequence) and a publication without one loses (§6.4). Rev 7 (f2feed6c) was reviewed again (`selfserve-spec-review7.md`, NOT READY, 1 P1: a custom domain repointed from one instance to another compares two unrelated counters). Rev 8 gives alias bindings their own durable revision from the domains record, published with the target hostname; edges pick the newest binding first and compare the target's pair within it (§6.4). Rev 8 (05f4c353) was reviewed again (`selfserve-spec-review8.md`, NOT READY, 1 P1: aliases derived from a wildcard catch-all have no binding record to carry a revision; 1 P2: retaining a detached domains record changes what a claim listing means). Rev 9 narrows: holding entries list explicit aliases only and derived aliases keep today's answer during a move; the binding revision lives in a console table that survives the domains record, so detach stays a delete (§6.4). Rev 9 (3bcea70e) was reviewed again (`selfserve-spec-review9.md`, NOT READY: a detached explicit alias can fall back to a derived one beneath a retained explicit holding entry; the revision table and the bucket write need a crash-safe protocol). Every remaining round-8 and round-9 finding was about a holding entry an edge keeps as last-good through a read failure and later orders against a new binding. **Rev 10 stops tolerating that case instead of ordering it:** a holding entry is never last-good, it is honoured only from a publication fetched within the last five minutes, and the alias revision machinery is dropped; the incarnation number is allocated durably before it is exposed (§6.4). Rev 10 (82023a80) was reviewed again (`selfserve-spec-review10.md`, NOT READY, 2 P1 and 1 P2: two concurrent creates of one hostname can commit incarnations out of order; a fresh hold must not displace a retained ready route for an alias that now names another instance; alias publications need the target hostname). Rev 11 answers them: creates and deletes of a hostname are serialised through a pending allocation that only a completed record write commits (§6.4); for an alias, any eligible ready route naming a different target wins over a holding entry, pairs are compared only within one target, and the five-minute bound is stated as a bound on reader disconnection, not publisher death (§6.4); every alias publication carries its target hostname (§6.4). Rev 11 (9ae3902f) was reviewed again (`selfserve-spec-review11.md`, NOT READY, 1 P1: recovery after a console crash could abandon an outstanding create whose record write later lands, because the record's "absent" precondition recurs after a deletion). Rev 12 makes the incarnation an authoritative bucket slot that never returns to absent and that every consumer checks the record against, so a late write materialises nothing (§6.4). Rev 12 (35bd391d) was reviewed again (`selfserve-spec-review12.md`, NOT READY, 2 P1: recovery that advances the slot would invalidate a create whose record did land; the ordinary delete is unconditional and a delayed one could erase a successor). Rev 13 answers them: recovery replays the create's own intent to completion at its allocated number and never advances a slot to abandon; deletes are conditional on the observed record ETag and retire only the captured slot revision, with persisted intent (§6.4). Rev 13 (696cb374) was reviewed again (`selfserve-spec-review13.md`, NOT READY, 1 P1: a retirement that exposes the record's absence before the slot is retired lets a late original create restore an obsolete record, and an ETag difference alone was read as a successor; 1 P2: allocation recovery omitted the crash before the slot write and a stale record under the key). Rev 14 answers them: retirement first replaces the record with a non-servable tombstone, so absence is never exposed while the slot is live, and every recovery branch of the create and the retirement is listed (§6.4). Rev 14 (bb05348f) was reviewed again (`selfserve-spec-review14.md`, NOT READY, 1 P1: a record without the incarnation field was accepted beside any existing slot, so an old console's outstanding create could land after a retirement; 1 P2: two retirement recovery states had no terminal disposition). Rev 15 answers them: a fieldless record beside a slot is valid only under a live migration intent bound to that record's key and ETag, and both retirement states get terminal outcomes (§6.4). Rev 15 (dd094a27) was reviewed again (`selfserve-spec-review15.md`, NOT READY, 1 P1: an ordinary record write that wins between the migration's read and its CAS left a legitimate fieldless record that the new rule treated as stale). Rev 16 answers it: migration first marks the record by CAS, which fences every writer, and binds the slot to the marked record (§6.4). |
| Builds on the offline operator move (`docs/superpowers/plans/2026-10-20-offline-operator-move.md`, shipped in v0.28.5 to v0.28.7) and its runbook (`docs/superpowers/plans/2026-10-21-offline-move-runbook.md`). Supersedes nothing: Phase 1 (`docs/superpowers/specs/2026-10-02-move-any-app-design.md`) stays the live-move path for claims-on deployments, and the M1 to M18 rulings stand. M18 ("Tito waits; move mutations are operator-only") is answered here for one door, not lifted: §5 says exactly what the customer's request may do. |
| @@ -120,7 +120,7 @@ From `start` until the target's row is ready, the host is held on every source b |
| - **TLS.** A host or alias in a holding entry is allowed for TLS exactly as it would be as a served route (the same `Allowed` decision, wildcard and mirrored certificates included), so the page is reachable on HTTPS; the entry never widens what the host could already present. |
| - **The display name.** Tito supplies it per request; the console validates it (at most 80 characters, printable, no control characters, no newlines) and stores it in the move; the edge renders it through contextual template escaping. It is the one piece of request-supplied text the page shows, by design (the customer named their own account), and it is shown to unauthenticated visitors, as the account's own pages are. Nothing else from the request, no header, path, diagnostic or `requested_by`, reaches the page. |
| - **Ordering by a per-host sequence.** The record gains `move_seq`, a per-host counter the console initialises to 1 on a host's first offline move and increments on every offline transition that changes who serves the host: `start` (the hold), `open` (the target serves), the abort's reopening of the source, and `finish`. Every box persists on its registry row the `move_seq` of the transition that produced the row's current state (`serving_seq`: the hold's for a held row, `open`'s for a landed ready row, the abort's for a reopened source row), written in the same row write as the status, and publishes it with the route and with any holding entry. **Rows minted outside a move:** when ordinary reconciliation mints or re-mints a ready row for a host from a record that carries no move (a rebuilt registry, a box returning from an outage, a row removed and restored), it writes `serving_seq` = the record's current `move_seq` (the last finished or aborted move's final value) in the same write; a record with no `move_seq` (a host with no move history) gives 0. Zero therefore means "no move history", never "a new row". Observing a newer record does not by itself move a row's sequence; only applying a transition, or minting the row from the record, does. |
| -- **Lifetimes: the host incarnation slot.** Each hostname has one authoritative object, `control/host-incarnations/<host>.json`, the **slot**: `{incarnation, state: live|retired, intent}`. It is created once and never deleted; every change is a conditional write (`If-None-Match: *` the first time, `If-Match: <etag>` after), so it never returns to absent. The create door: takes the console's per-hostname mutex; records a `pending` allocation in SQLite; advances the slot to `{incarnation: N, live, intent: <create id>}` (N = the slot's previous value plus one, or 1 for a new slot); writes the record carrying `incarnation: N`; marks the allocation `committed`. **The delete door retires in three conditional steps and never exposes the record's absence while the slot is live.** It records a `pending` retirement in SQLite `{host, incarnation N, record ETag as read (E), slot ETag as read (S)}`, then: (1) replaces the record, conditionally on E, with a **tombstone** record `{host, incarnation N, retired: true}` (non-servable: every consumer treats a tombstone as absent; a create's `If-None-Match: *` fails while it exists); (2) writes the slot `{N, retired}` conditionally on S; (3) deletes the tombstone conditionally on its own ETag; then marks the retirement `committed`. A conditional failure at step 1 (the record's ETag is not E) is **not** read as a successor: the door rereads, and if the record still carries incarnation N and is not a tombstone, that is the same lifetime changed since the read (a move, a repoint) and the door refuses the delete with "the instance changed; read it again" rather than guessing; only a record with a different incarnation, or a slot already past N, makes the retirement `void`. Recovery of a `pending` retirement, from the persisted E and S: record present with ETag E: resume at step 1; tombstone for N present: resume at step 2 (if the slot is still at S) or step 3 (if the slot is `{N, retired}`); record absent and slot `{N, retired}`: step 3 is done, `committed`; record present with another incarnation, or slot past N and no tombstone: `void`; **slot `{N, retired}` and a non-tombstone record at N, whatever its ETag** (a late write landed after the tombstone went): `committed`, the record is stale to every consumer under the retired slot and is removed conditionally on its ETag, by recovery or by the next create; **slot still live at S and a non-tombstone record at N with an ETag other than E** (the lifetime changed under the read): the terminal outcome `refused`, persisted, the record preserved and the hostname's exclusion released, the same answer the running door gives; anything unreadable: stays `pending`, and the hostname's creates and deletes wait. A late original create cannot restore an obsolete record: while the tombstone exists its `If-None-Match` fails, and once the tombstone is gone the slot is already retired, so a record at N is stale to every consumer and the next create replaces it (below). **Every consumer of a record that carries an incarnation checks it against the slot** before acting on it: reconcile mints and keeps a row only for a record whose incarnation equals the slot's and whose slot is `live`; the doors refuse a record that fails that check as absent; a box publishes nothing for it. **Recovery completes the create, it never abandons it.** The `pending` allocation in SQLite carries everything needed to replay: the create's id, account, app, host, the allocated N, the exact record body, and the slot's predecessor condition (its ETag, or "absent" for a new hostname). For each `pending` allocation the restarted console reads the slot and the record and takes exactly one branch: **slot absent or at the predecessor:** replay the slot write (`If-None-Match: *` or `If-Match` the predecessor) then continue; **slot `{N, live, intent: this create}`, record absent:** replay the record write (`If-None-Match: *`; a late original write that lands first is the same content and whichever is second fails harmlessly), then `committed`; **slot at N, record present with incarnation N:** `committed`; **slot at N, record key occupied by a stale record** (incarnation below N, or a tombstone of an earlier lifetime): delete it conditionally on its ETag, then replay the record write; **slot past N:** `superseded` (the per-hostname mutex makes this an operator's hand move), and any record at N is stale to every consumer; **anything unreadable:** stays `pending`, and the hostname's creates and deletes wait on it. The slot is never advanced in order to abandon a create, so a create that materialised keeps serving through any number of console crashes. The ordinary create door runs the same branches: it finds a stale record under the key only after the slot shows a later lifetime, deletes it conditionally on its ETag and proceeds. Existing hosts' lazy migration is itself a persisted operation: the console records a `pending` migration `{host, record key, record ETag as read}`, writes the slot `{1, live, intent: migration}`, writes the record's `incarnation: 1` by CAS on that ETag, and marks it `committed`. **A record without the field is valid beside an existing slot only while that slot's intent is a live migration bound to this record's key and ETag**; a fieldless record beside any other slot (retired, or live from a create) is stale: every consumer treats it as absent and the next create removes it conditionally on its ETag. A record without the field and no slot at all is legacy and valid until migrated. Every creation path after this ships uses the incarnation protocol; an old console's create that lands after the upgrade (its `If-None-Match: *` succeeding on an absent key) is therefore either the migration's own record (valid) or stale (inert), never a resurrected lifetime. Boxes read the slot alongside the record (one small object per host per pass, cached by ETag). `move_seq` stays on the record; `serving_seq` is persisted and published as before, as `(incarnation, serving_seq)`; publications without the pair are `(0, 0)`. The order among publications is as before: the first move's hold `(1, 1)` beats a retained pair-less ready `(0, 0)`; `open` `(1, 2)` beats the hold; a recreated host's first row `(2, 0)` beats every publication of lifetime 1; a current hold beats a previous lifetime's ready route; two ready routes from different lifetimes resolve to the higher incarnation; equal pairs fall to the existing home-map rule. |
| +- **Lifetimes: the host incarnation slot.** Each hostname has one authoritative object, `control/host-incarnations/<host>.json`, the **slot**: `{incarnation, state: live|retired, intent}`. It is created once and never deleted; every change is a conditional write (`If-None-Match: *` the first time, `If-Match: <etag>` after), so it never returns to absent. The create door: takes the console's per-hostname mutex; records a `pending` allocation in SQLite; advances the slot to `{incarnation: N, live, intent: <create id>}` (N = the slot's previous value plus one, or 1 for a new slot); writes the record carrying `incarnation: N`; marks the allocation `committed`. **The delete door retires in three conditional steps and never exposes the record's absence while the slot is live.** It records a `pending` retirement in SQLite `{host, incarnation N, record ETag as read (E), slot ETag as read (S)}`, then: (1) replaces the record, conditionally on E, with a **tombstone** record `{host, incarnation N, retired: true}` (non-servable: every consumer treats a tombstone as absent; a create's `If-None-Match: *` fails while it exists); (2) writes the slot `{N, retired}` conditionally on S; (3) deletes the tombstone conditionally on its own ETag; then marks the retirement `committed`. A conditional failure at step 1 (the record's ETag is not E) is **not** read as a successor: the door rereads, and if the record still carries incarnation N and is not a tombstone, that is the same lifetime changed since the read (a move, a repoint) and the door refuses the delete with "the instance changed; read it again" rather than guessing; only a record with a different incarnation, or a slot already past N, makes the retirement `void`. Recovery of a `pending` retirement, from the persisted E and S: record present with ETag E: resume at step 1; tombstone for N present: resume at step 2 (if the slot is still at S) or step 3 (if the slot is `{N, retired}`); record absent and slot `{N, retired}`: step 3 is done, `committed`; record present with another incarnation, or slot past N and no tombstone: `void`; **slot `{N, retired}` and a non-tombstone record at N, whatever its ETag** (a late write landed after the tombstone went): `committed`, the record is stale to every consumer under the retired slot and is removed conditionally on its ETag, by recovery or by the next create; **slot still live at S and a non-tombstone record at N with an ETag other than E** (the lifetime changed under the read): the terminal outcome `refused`, persisted, the record preserved and the hostname's exclusion released, the same answer the running door gives; anything unreadable: stays `pending`, and the hostname's creates and deletes wait. A late original create cannot restore an obsolete record: while the tombstone exists its `If-None-Match` fails, and once the tombstone is gone the slot is already retired, so a record at N is stale to every consumer and the next create replaces it (below). **Every consumer of a record that carries an incarnation checks it against the slot** before acting on it: reconcile mints and keeps a row only for a record whose incarnation equals the slot's and whose slot is `live`; the doors refuse a record that fails that check as absent; a box publishes nothing for it. **Recovery completes the create, it never abandons it.** The `pending` allocation in SQLite carries everything needed to replay: the create's id, account, app, host, the allocated N, the exact record body, and the slot's predecessor condition (its ETag, or "absent" for a new hostname). For each `pending` allocation the restarted console reads the slot and the record and takes exactly one branch: **slot absent or at the predecessor:** replay the slot write (`If-None-Match: *` or `If-Match` the predecessor) then continue; **slot `{N, live, intent: this create}`, record absent:** replay the record write (`If-None-Match: *`; a late original write that lands first is the same content and whichever is second fails harmlessly), then `committed`; **slot at N, record present with incarnation N:** `committed`; **slot at N, record key occupied by a stale record** (incarnation below N, or a tombstone of an earlier lifetime): delete it conditionally on its ETag, then replay the record write; **slot past N:** `superseded` (the per-hostname mutex makes this an operator's hand move), and any record at N is stale to every consumer; **anything unreadable:** stays `pending`, and the hostname's creates and deletes wait on it. The slot is never advanced in order to abandon a create, so a create that materialised keeps serving through any number of console crashes. The ordinary create door runs the same branches: it finds a stale record under the key only after the slot shows a later lifetime, deletes it conditionally on its ETag and proceeds. Existing hosts' lazy migration is itself a persisted operation that **fences every record writer before it binds anything to an ETag**: the console records a `pending` migration `{host, record key, migration id}`; (1) CAS-writes a marker into the record, `incarnation_migrating: <migration id>`, against the ETag it read (a writer that won in between makes this CAS fail; the console rereads and retries from a valid record, so a legitimate change is never lost and never stranded); (2) writes the slot `{1, live, intent: migration <id>}` bound to the marked record's key and ETag; (3) CAS-writes `incarnation: 1` and removes the marker; marks `committed`. Every record writer, console or box, that finds the marker on its reread waits for the migration (it retries its own write once the field is set; the marker is short-lived), and an already-issued write against the pre-marker ETag fails its CAS as any lost race does. Boxes roll before the console (the roll order rule), so every box understands the marker before any migration runs. **Validity of a record without `incarnation`:** valid with no slot (legacy, unmigrated); valid with the marker while the slot is absent or is the live migration slot bound to this record (migration in flight); stale otherwise (a retired slot, a create's slot, or no marker beside a slot): every consumer treats it as absent and the next create removes it conditionally on its ETag. Recovery of a `pending` migration from the persisted id: record fieldless and unmarked, slot absent: restart at (1); record marked with this id, slot absent: (2); record marked, slot bound to it: (3); record carries `incarnation: 1`: `committed`; record gone or carrying another lifetime: `superseded`; unreadable: `pending`. Every creation path after this ships uses the incarnation protocol; an old console's create that lands after the upgrade is either the migration's own record (valid) or stale (inert), never a resurrected lifetime. Boxes read the slot alongside the record (one small object per host per pass, cached by ETag). `move_seq` stays on the record; `serving_seq` is persisted and published as before, as `(incarnation, serving_seq)`; publications without the pair are `(0, 0)`. The order among publications is as before: the first move's hold `(1, 1)` beats a retained pair-less ready `(0, 0)`; `open` `(1, 2)` beats the hold; a recreated host's first row `(2, 0)` beats every publication of lifetime 1; a current hold beats a previous lifetime's ready route; two ready routes from different lifetimes resolve to the higher incarnation; equal pairs fall to the existing home-map rule. |
| - **Aliases.** A holding entry lists the host's explicit aliases (those with their own domains record naming the host) as the edges writer would publish them for a served route, under the same eligibility rules; derived aliases from a wildcard catch-all are not listed, and answer during a move as they do today for a held row (the pending page on a platform domain, the plain 404 elsewhere). **Every alias publication, ready routes included and derived aliases included, carries `target_host`**, the primary hostname it resolves to; holding entries identify their target through `host`. The edge's rule for an alias: **any ready route for the alias that is eligible under ordinary routing (fresh or last-good, remote or local) and names a different `target_host` than a holding entry wins over that holding entry**, whatever the two targets' pairs, because a binding change is a console decision and the pairs of different instances are unrelated; a holding entry and a ready route for the **same** `target_host` are compared by the target's `(incarnation, serving_seq)` as for the primary hostname (the stale ready route of the instance being moved loses to its own hold). A legacy publication without `target_host` is treated as a different target (the ready route wins), which is today's behaviour. No alias revision is kept. The five-minute bound on holding entries bounds a reader's disconnection from a publisher, not a publisher's death: a box that stops updating but whose last object stays readable keeps its holding entry eligible, which is exactly the case the different-target rule decides. Expiry is enforced from the entry's last successful fetch time, independent of any later refresh succeeding; a refresh that fails as a whole leaves the old snapshot's routes installed as today and its holding entries expiring on their own clock. "Within one refresh" describes healthy convergence after a repoint, not a deadline. |
| - **Reopening.** The public reopens when the target's row is ready and its publication, at `open`'s sequence, reaches the edges: visitors' first requests wake the app as any cold wake does. The console's health check (§7.4) is detection, not a gate on reopening. A failure after that is visible to visitors as the app's own answers, as any instance failure is. |
| - **The health probe** (§7.4) goes to the target box directly, by its address from its edges file with the host as SNI, so it reaches the app whatever the edges show. |
| @@ -257,10 +257,10 @@ Infra: both staging lanes for every region pair the console arms (today IE to UK |
| - The estimate's storage and the rolling median; the phase timings' recording. |
| - The `moves` column migration, the mint flag and the instance-bound refusal. |
| - The return's leftover acceptance in the operator start door's target-ready contract, keyed by the forward move id. |
| -- the incarnation slot `control/host-incarnations/<host>.json` (never deleted, conditional writes, advanced on create and by recovery, retired on delete), the record's `incarnation` checked against it by every consumer, `move_seq` on the record, `(incarnation, serving_seq)` on rows (written with the status, by moves and by reconciliation), both published with routes and holding entries; the edges' highest-pair rule with `(0, 0)` for publications without the pair; the lazy migration for existing hosts; the five-minute validity of holding entries at the edge (never last-good, enforced from the fetch time), with the fallback answer; `target_host` on alias publications and the different-target rule; derived aliases excluded from holding entries; the per-hostname pending allocation with the create's request, its replay on restart, the pending retirement with captured ETags, the tombstone and the three-step resolution, and the stale-record cleanup in the create door and in recovery. the probe mode (`X-Carlos-Probe`, `X-Carlos-Served-By`); the "taking longer" sentence's trigger. |
| +- the incarnation slot `control/host-incarnations/<host>.json` (never deleted, conditional writes, advanced on create and by recovery, retired on delete), the record's `incarnation` checked against it by every consumer, `move_seq` on the record, `(incarnation, serving_seq)` on rows (written with the status, by moves and by reconciliation), both published with routes and holding entries; the edges' highest-pair rule with `(0, 0)` for publications without the pair; the lazy migration for existing hosts; the five-minute validity of holding entries at the edge (never last-good, enforced from the fetch time), with the fallback answer; `target_host` on alias publications and the different-target rule; derived aliases excluded from holding entries; the per-hostname pending allocation with the create's request, its replay on restart, the pending retirement with captured ETags, the tombstone and the three-step resolution, the stale-record cleanup in the create door and in recovery, and the migration marker with its writer wait and its recovery branches. the probe mode (`X-Carlos-Probe`, `X-Carlos-Served-By`); the "taking longer" sentence's trigger. |
| - The root cleanup request, done and cancel files and root's check before each deletion. |
| - The `offline-queued` state and the `self_serve_moves` row's phases; the cleanup reservation kind in `active.json` and the create door's check of it. |
| -- Tests: the two-box end-to-end witness of the offline plan (Task 8) extended to the job: forward; cancel in ready; an abort after landing followed by the source row's reconstruction by reconciliation while the target's holding publication is retained, with the reconstructed route winning; the first move with the source's pair-less ready publication retained by an edge through the hold and the open; a host recreated after a move with the old lifetime's holding and ready publications retained, the new lifetime's first row winning; a current hold beating a previous lifetime's ready route; an edge restarted into those mixtures; a holding entry from a publication last fetched more than five minutes ago ignored, with the fallback answer, including after a whole refresh fails; an alias repointed from a held instance to another, the new target's ready route winning as a fresh and as a last-good publication; a detached explicit alias falling back to a derived one whose ready route names another target; the stale ready alias of the instance being moved losing to its own hold; a create whose record landed and is served, a crash before the allocation commits, and recovery marking it committed with the account still serving, including a second crash during recovery; an outstanding create replayed by recovery with the late original write failing harmlessly; an outstanding delete, a restart, a completed retry, a recreation and the late original delete failing on its ETag with the successor intact; a replayed create, a completed move, a delete that crashes between the tombstone and the slot, and the late original create failing on the tombstone, then the slot retired and the tombstone removed; a crash before the slot write and recovery replaying it; a stale record under the key at create and at recovery, removed conditionally; an old console's fieldless create landing after an upgrade, a retry, a move and a retirement, ignored by every consumer and removed by the next create; a retirement crashing after the tombstone is removed with a late record restored, resolved as committed with the record removed; a retirement finding the record changed in the same lifetime, resolved as refused with the record kept; a late stale record (incarnation behind the slot) ignored by reconcile, the doors and the boxes, and replaced by the next create; an edge restarted into mixed publications; a cut naming a stale id; each pre-check refusal (a stale live link, a non-empty extra directory, an ineligible archive, a failed lane probe); each deadline before open ending in a lossless abort; a post-open failure to answer ending in `done` with a warning and a page; the holding page on a third edge and a custom domain; a ready route winning over a stale holding entry and a holding entry winning over a stale route; two moves at once on disjoint boxes and two queued on one; a reservation reclaimed only on evidence; cleanup refusing when the record moved again, when a current version differs from the inventory, and when the reservation is held by a move. |
| +- Tests: the two-box end-to-end witness of the offline plan (Task 8) extended to the job: forward; cancel in ready; an abort after landing followed by the source row's reconstruction by reconciliation while the target's holding publication is retained, with the reconstructed route winning; the first move with the source's pair-less ready publication retained by an edge through the hold and the open; a host recreated after a move with the old lifetime's holding and ready publications retained, the new lifetime's first row winning; a current hold beating a previous lifetime's ready route; an edge restarted into those mixtures; a holding entry from a publication last fetched more than five minutes ago ignored, with the fallback answer, including after a whole refresh fails; an alias repointed from a held instance to another, the new target's ready route winning as a fresh and as a last-good publication; a detached explicit alias falling back to a derived one whose ready route names another target; the stale ready alias of the instance being moved losing to its own hold; a create whose record landed and is served, a crash before the allocation commits, and recovery marking it committed with the account still serving, including a second crash during recovery; an outstanding create replayed by recovery with the late original write failing harmlessly; an outstanding delete, a restart, a completed retry, a recreation and the late original delete failing on its ETag with the successor intact; a replayed create, a completed move, a delete that crashes between the tombstone and the slot, and the late original create failing on the tombstone, then the slot retired and the tombstone removed; a crash before the slot write and recovery replaying it; a stale record under the key at create and at recovery, removed conditionally; an old console's fieldless create landing after an upgrade, a retry, a move and a retirement, ignored by every consumer and removed by the next create; an ordinary update winning between the migration's read and its marker CAS, the migration restarting from the updated record; a box's write issued before the marker failing its CAS and succeeding on retry after the field is set; a restart before and after each migration write, and a second crash during recovery; a retirement crashing after the tombstone is removed with a late record restored, resolved as committed with the record removed; a retirement finding the record changed in the same lifetime, resolved as refused with the record kept; a late stale record (incarnation behind the slot) ignored by reconcile, the doors and the boxes, and replaced by the next create; an edge restarted into mixed publications; a cut naming a stale id; each pre-check refusal (a stale live link, a non-empty extra directory, an ineligible archive, a failed lane probe); each deadline before open ending in a lossless abort; a post-open failure to answer ending in `done` with a warning and a page; the holding page on a third edge and a custom domain; a ready route winning over a stale holding entry and a holding entry winning over a stale route; two moves at once on disjoint boxes and two queued on one; a reservation reclaimed only on evidence; cleanup refusing when the record moved again, when a current version differs from the inventory, and when the reservation is held by a move. |
| ## 12. Risks |