carlos / platform Public

console: claims-off creates allocate an owned host incarnation through a persisted intent, with recovery that never abandons it

A claims-off create admitted with lifetimes on (dispatch by the admitted
unit; a unit admitted before the cutover keeps today's calls) runs one
protocol under the app lock and then a per-hostname exclusion: the app's
lifecycle is checked under the lock, other pending operations of the
hostname refuse it, a live orphan slot is retired first, then a
host_allocations intent, the slot by CAS from its predecessor, the stale
body, the record with record_seq introduced, and the commit. Once the
intent is durable every failure is pending, terminal or the record.

The recovery pass (StartLifetimeRecoveryPass, every 60 s, on only)
recovers orphan retirements (cancelling ones first), then allocations,
from SQLite and fresh reads alone; an allocation is replayed only for the
app lifetime that authorised it. fenceRecord and terminateFenced make an
outstanding conditional write harmless before a branch terminates on
record evidence.

The create, seed, import and first-route attach doors route through it;
attach holds both locks from its claim to its record and its rollback
never retires a record; detach runs inside the same exclusion; the delete
door refuses while the hostname is settling. Census rows and the note
updated; all PROPOSED-self-serve-moves-1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 2 days ago a2960376f2244cec67e28f0be912b26ddc410c16 parent a27ae8a

22 files changed, +3683 −14

cmd/console/main.go +5 −0
docs/superpowers/notes/census-proposal-self-serve-moves-1.md +2 −0
internal/console/apex.go +4 −1
internal/console/apidomains.go +6 −0
internal/console/apiinstanceimport.go +9 −1
internal/console/apiinstances.go +46 −5
internal/console/apiinstanceseed.go +23 −1
internal/console/applocks.go +13 −1
internal/console/claim_invariant_fixtures_test.go +1 −0
internal/console/claim_invariant_probes_worker_test.go +1 −0
internal/console/claim_workers.go +1 −0
internal/console/domainattach.go +125 −0
internal/console/lifetimes.go +276 −0
internal/console/lifetimes_create.go +537 −0
internal/console/lifetimes_create_test.go +2355 −0
internal/console/lifetimes_cutover_test.go +5 −0
internal/console/lifetimes_recovery.go +216 −0
internal/console/server.go +5 −0
internal/console/testdata/claim-routes.txt +1 −0
internal/provision/plan14_root_reads_test.go +3 −1
internal/provision/self_serve_moves1_census_test.go +40 −0
internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv +9 −4

CI failed — run details

CI log

=== gate ===
.amadan/test-gate.sh
=== launchctx (2 s)
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
=== vet (17 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (166 s)
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.508s
ok  	github.com/carlosframework/platform/cmd/moveprobe	11.614s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.744s
ok  	github.com/carlosframework/platform/internal/activator	60.104s
ok  	github.com/carlosframework/platform/internal/agentsock	0.023s
ok  	github.com/carlosframework/platform/internal/alerting	0.010s
ok  	github.com/carlosframework/platform/internal/analytics	0.021s
ok  	github.com/carlosframework/platform/internal/appconfig	2.781s
ok  	github.com/carlosframework/platform/internal/apperr	0.086s
ok  	github.com/carlosframework/platform/internal/applog	0.210s
ok  	github.com/carlosframework/platform/internal/awsinv	0.006s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.006s
ok  	github.com/carlosframework/platform/internal/boxid	0.538s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.009s
ok  	github.com/carlosframework/platform/internal/caps	0.017s
ok  	github.com/carlosframework/platform/internal/certwarm	0.006s
ok  	github.com/carlosframework/platform/internal/cgroup	0.775s
ok  	github.com/carlosframework/platform/internal/changelog	0.423s
ok  	github.com/carlosframework/platform/internal/claimactivation	1.076s
ok  	github.com/carlosframework/platform/internal/clamd	0.060s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	5.813s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.013s
ok  	github.com/carlosframework/platform/internal/dbverify	0.007s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.007s
--- FAIL: TestFenceLanguagesAreDeclared (0.00s)
    fences_test.go:32: cli.md:2345: fence declares no language
    fences_test.go:32: cli.md:2351: fence declares no language
FAIL
FAIL	github.com/carlosframework/platform/internal/docsite	0.009s
ok  	github.com/carlosframework/platform/internal/domains	2.777s
ok  	github.com/carlosframework/platform/internal/doorkey	0.007s
ok  	github.com/carlosframework/platform/internal/economics	0.007s
ok  	github.com/carlosframework/platform/internal/edge	45.333s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.011s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.230s
ok  	github.com/carlosframework/platform/internal/gate	1.693s
ok  	github.com/carlosframework/platform/internal/genselect	0.037s
ok  	github.com/carlosframework/platform/internal/geo	0.024s
ok  	github.com/carlosframework/platform/internal/geoip	0.057s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.225s
ok  	github.com/carlosframework/platform/internal/host	137.827s
ok  	github.com/carlosframework/platform/internal/hostactors	5.559s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.014s
ok  	github.com/carlosframework/platform/internal/hostbox	3.103s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.788s
ok  	github.com/carlosframework/platform/internal/hostenv	0.014s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.014s
ok  	github.com/carlosframework/platform/internal/hostexec	5.345s
ok  	github.com/carlosframework/platform/internal/hostfence	0.053s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.264s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.020s
ok  	github.com/carlosframework/platform/internal/hostguard	1.077s
ok  	github.com/carlosframework/platform/internal/hostops	2.009s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.015s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.014s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.587s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.184s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.015s
ok  	github.com/carlosframework/platform/internal/infraview	0.015s
ok  	github.com/carlosframework/platform/internal/instancectl	5.229s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.790s
ok  	github.com/carlosframework/platform/internal/instancefs	0.013s
ok  	github.com/carlosframework/platform/internal/instancekey	0.018s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.024s
ok  	github.com/carlosframework/platform/internal/layout	1.824s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.015s
ok  	github.com/carlosframework/platform/internal/lease	10.237s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.528s
ok  	github.com/carlosframework/platform/internal/ledger	0.033s
ok  	github.com/carlosframework/platform/internal/mail	0.240s
ok  	github.com/carlosframework/platform/internal/maintenance	0.090s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.561s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.088s
ok  	github.com/carlosframework/platform/internal/metrics	0.006s
ok  	github.com/carlosframework/platform/internal/moves	21.904s
ok  	github.com/carlosframework/platform/internal/moves/archive	1.001s
ok  	github.com/carlosframework/platform/internal/objstore	0.268s
ok  	github.com/carlosframework/platform/internal/ops	0.218s
ok  	github.com/carlosframework/platform/internal/park	0.308s
ok  	github.com/carlosframework/platform/internal/plans	0.007s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	21.371s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.012s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.633s
ok  	github.com/carlosframework/platform/internal/relay	4.232s
ok  	github.com/carlosframework/platform/internal/release	25.881s
ok  	github.com/carlosframework/platform/internal/rollout	3.850s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.006s
ok  	github.com/carlosframework/platform/internal/rootaws	0.009s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.061s
ok  	github.com/carlosframework/platform/internal/rootstate	0.027s
ok  	github.com/carlosframework/platform/internal/runs	0.624s
ok  	github.com/carlosframework/platform/internal/s3compat	0.007s
ok  	github.com/carlosframework/platform/internal/safefile	0.010s
ok  	github.com/carlosframework/platform/internal/schedule	2.734s
ok  	github.com/carlosframework/platform/internal/seeds	0.013s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	11.264s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	4.299s
ok  	github.com/carlosframework/platform/internal/status	0.073s
ok  	github.com/carlosframework/platform/internal/steering	0.006s
ok  	github.com/carlosframework/platform/internal/stopevidence	1.136s
ok  	github.com/carlosframework/platform/internal/svcrole	0.005s
ok  	github.com/carlosframework/platform/internal/telemetry	0.008s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.005s
ok  	github.com/carlosframework/platform/internal/usage	0.009s
ok  	github.com/carlosframework/platform/internal/version	0.002s
ok  	github.com/carlosframework/platform/internal/vet	1.741s
FAIL
=== internal_console@0 (79 s)
ok  	github.com/carlosframework/platform/internal/console	78.147s
=== internal_console@1 (40 s)
ok  	github.com/carlosframework/platform/internal/console	38.614s
=== internal_console@2 (37 s)
ok  	github.com/carlosframework/platform/internal/console	36.155s
=== internal_console@3 (39 s)
2026/10/22 21:27:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:47 console: alert watch: no keymail grant for ops-b@keymail.dev — not notified (they must sign in to the console once)
2026/10/22 21:27:47 console: alert watch: no keymail grant for ops-b@keymail.dev — not notified (they must sign in to the console once)
2026/10/22 21:27:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:47 console: alert watch: keymail box-a/disk to ops-b@keymail.dev: context deadline exceeded
2026/10/22 21:27:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:47 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:27:47 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:47 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:27:47 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: box ie-old has the legacy role (by paul@keymail.dev, roster revision 2)
2026/10/22 21:27:48 console: box ie-old has the default role (by paul@keymail.dev, roster revision 3)
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: app region bab/paul:  (default)
2026/10/22 21:27:48 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: api ship bab/console: read app flags: SQL logic error: no such column: build_probe (1)
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:27:48 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:27:48 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:27:48 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:27:48 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:27:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:48 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:27:48 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:27:48 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:27:49 console: api db bab/titogo/acme.go.tito.io: stream level 0 1-1: context canceled
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_ACME_DELEGATION_ZONE "*.bad zone" is not a valid domain — delegation off, wildcard-enable requests refuse with "no delegation zone configured" until it's fixed
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: import old.shop.example.com: write the record: the bucket refused the write
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: finish seed sd_urd6vyapqxicqryahgc7nvsrqu of bab/shop/old.bab.apps.example.net: tore down its store (bucket carlos-bab-shop-old)
2026/10/22 21:27:50 console: finished seed sd_urd6vyapqxicqryahgc7nvsrqu of bab/shop/old.bab.apps.example.net
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: import old.shop.example.com: the record went while seed sd_ergco4zhxtcahpxuegnw54x7la was checked; nothing put
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: import old.shop.example.com: check: not a database
2026/10/22 21:27:50 console: import old.shop.example.com: seed sd_mxmvi3cvmdxr2lhk3o55moirri moved on before its upload's the check was recorded; not aborted: provision: seed transition refused: seed sd_mxmvi3cvmdxr2lhk3o55moirri is uploaded
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: import old.shop.example.com: seed sd_fjwwg2on6oipzfwhcetkhc2f2i moved on while it was checked (&{ID:sd_fjwwg2on6oipzfwhcetkhc2f2i Kind:import From: TXID: SHA256:dd9b20ff50b5deb0b975a4206846b45e1635146b14b80ba2b91468246a47d29d Bytes:8192 State:uploaded Reason: By:alice@keymail.dev At:2026-10-22 21:27:50.347642586 +0000 UTC Replaces:127.0.0.1:8401 VerifyHash:522eba0de50688709977725cc856277bc4635735ad3877f3877c35307134f264}); nothing put
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: import old.shop.example.com: seed sd_h5pc6ulgbxn4hdzip7z4t2jhke moved on while it was checked (&{ID:sd_h5pc6ulgbxn4hdzip7z4t2jhke Kind:import From: TXID: SHA256:dd9b20ff50b5deb0b975a4206846b45e1635146b14b80ba2b91468246a47d29d Bytes:8192 State:aborted Reason: By:alice@keymail.dev At:2026-10-22 21:27:50.396760427 +0000 UTC Replaces:127.0.0.1:8401 VerifyHash:285118e77d7746f34d3a3069fadf33871b2e678788fdd123efe6d3b8ab02bfe8}); nothing put
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: import bad.shop.example.com: seed "not-a-seed-id": seeds: not a seed id
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: finished seed sd_rmljeaw4mhe3z7kxqhrins7ttu of bab/shop/old.shop.example.com
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: finished seed sd_744g3hcozmqb3bysiov77fh56q of bab/shop/early.shop.example.com
2026/10/22 21:27:50 console: finished seed sd_uuabathbtfdqf6bt7zd73m5scq of bab/shop/old.shop.example.com
2026/10/22 21:27:50 console: finished seed sd_xdkhric6djuqth7a4bt5qxrilm of bab/shop/refused.shop.example.com
2026/10/22 21:27:50 console: finished seed sd_tywl66nev4kgqs7mgadhicbhpq of bab/shop/rolled-back.shop.example.com
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:50 console: api instance move bab/titogo/acme.go.tito.io: the claim is moving under mv_hhwyonwaka3upuevkp754muzwe but the record start failed: secret-detail: record put failed
2026/10/22 21:27:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:51 console: api instances delete bab/titogo/acme.go.tito.io: provision: claim retirement unavailable
2026/10/22 21:27:51 console: api instances delete bab/titogo/acme.go.tito.io: provision: claim retirement unavailable
2026/10/22 21:27:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:51 console: api instance move bab/titogo/acme.go.tito.io: typed move write refused: hostops: identity mismatch
2026/10/22 21:27:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:52 console: api instance move bab/titogo/acme.go.tito.io: declared loss refused: hostresolve: evidence or operation does not authorize resolution
2026/10/22 21:27:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:52 edges: box "uk-box" has an edges file under control/edges/ but no status snapshot at all, so no region outside the home is automatically eligible until it reports; said once
2026/10/22 21:27:52 edges: 1 edges file(s) under control/edges/ but no fresh box snapshot matched any of them, so automatic region placement sees the home alone (are edges and status in different buckets? see CARLOS_STATUS_BUCKET/CARLOS_STATUS_PREFIX)
2026/10/22 21:27:52 console: an incomplete read of the edges files or box snapshots (edges complete true, snapshots complete false), so no region outside the home is automatically eligible this request; said once
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: api instances create bab/titogo: host claim check junk-titogo.bab.oncarlos.test: domains: claim junk-titogo.bab.oncarlos.test: invalid character 'o' in literal null (expecting 'u')
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: api instances create bab/titogo: pipeline read: pipeline wrapper: invalid character 'o' in literal null (expecting 'u')
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:27:54 console: from main.shop.example.com: upload canary.shop.example.com: access denied
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:27:54 console: from main.shop.example.com: publish canary.shop.example.com: the bucket refused the write
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:27:54 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:27:54 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: app region bab/widget:  (default)
2026/10/22 21:27:54 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: relay enabled bab/titogo in eu-west-1 by alice@keymail.dev (incarnation 46321473a6d0d447, op 1)
2026/10/22 21:27:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:54 console: relay enabled bab/titogo in eu-west-1 by alice@keymail.dev (incarnation dce8c44c5974cbb3, op 1)
2026/10/22 21:27:54 console: relay rotated eu-west-1 to kid k2, re-issued 1 app(s)
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: api run bab/shop/acme.example.com run_qcpu3vzl333ood23yxgurhijk4: clear after ack: put refused; record read: <nil>
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: api run bab/shop/acme.example.com run_hqva7xiuebcyg4pnvbof2waz24: clear after ack: <nil>; record read: get refused
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: api sidecars create bab/titogo: open registry: unable to open database file (14)
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: api store provision bab: read entitlement: objstore: store/entitlements/bab.json: invalid character 'p' looking for beginning of value
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: api store grant bab/titogo (bucket "other-uploads"): objstore: bucket or IAM user name belongs to another store: other-uploads
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: api system antivirus: read: release: system/antivirus.json: schema 99 is not 1
2026/10/22 21:27:57 console: api system antivirus: replacing an unreadable record on disable: release: system/antivirus.json: schema 99 is not 1
2026/10/22 21:27:57 console: object scanning disabled deployment-wide by alice@keymail.dev
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:57 console: could not notify paul@keymail.dev of a CLI token issue: Post "http://127.0.0.1:36375/api/app/send": dial tcp 127.0.0.1:36375: connect: connection refused
2026/10/22 21:27:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:27:58 console: api updates bab/km/a.example.com: enable refused, not every box applies update settings: ie-1: runs a binary that does not apply update settings
2026/10/22 21:27:58 console: api updates bab/km/a.example.com: enable refused, not every box applies update settings: ie-1: runs a binary that does not apply update settings
2026/10/22 21:27:58 console: api updates bab/km/a.example.com: enable refused, not every box applies update settings: ie-1: runs a binary that does not apply update settings
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:27:58 console: api updates bab/km policy: enable refused, not every box applies update settings: ie-1: runs a binary that does not apply update settings
2026/10/22 21:27:58 console: api updates bab/km policy: enable refused, not every box applies update settings: ie-1: runs a binary that does not apply update settings
2026/10/22 21:27:58 console: api updates bab/km policy: enable refused, not every box applies update settings: ie-1: runs a binary that does not apply update settings
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: app region bab/rot-a0:  (default)
2026/10/22 21:27:59 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:59 console: app region bab/rot-a1:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a2:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a3:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a4:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a5:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a6:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a7:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a8:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a9:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a10:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a11:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a12:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a13:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a14:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a15:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a16:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a17:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a18:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a19:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a20:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a21:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a22:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a23:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a24:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a25:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a26:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a27:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a28:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a29:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a30:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a31:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a32:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a33:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a34:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a35:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a36:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a37:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a38:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a39:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a40:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a41:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a42:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a43:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a44:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a45:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a46:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a47:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a48:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a49:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a50:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a51:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a52:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a53:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a54:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a55:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a56:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a57:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a58:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a59:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a60:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a61:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a62:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a63:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a64:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a65:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a66:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a67:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a68:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a69:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a70:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a71:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a72:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a73:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a74:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a75:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a76:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a77:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a78:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a79:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a80:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a81:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a82:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a83:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a84:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a85:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a86:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a87:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a88:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a89:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a90:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a91:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a92:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a93:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a94:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a95:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a96:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a97:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a98:  (default)
2026/10/22 21:27:59 console: app region bab/rot-a99:  (default)
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: app region bab/person-a0:  (default)
2026/10/22 21:27:59 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:59 console: app region bab/person-a1:  (default)
2026/10/22 21:27:59 console: app region bab/person-a2:  (default)
2026/10/22 21:27:59 console: app region bab/person-a3:  (default)
2026/10/22 21:27:59 console: app region bab/person-a4:  (default)
2026/10/22 21:27:59 console: app region bab/person-a5:  (default)
2026/10/22 21:27:59 console: app region bab/person-a6:  (default)
2026/10/22 21:27:59 console: app region bab/person-a7:  (default)
2026/10/22 21:27:59 console: app region bab/person-a8:  (default)
2026/10/22 21:27:59 console: app region bab/person-a9:  (default)
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: production flag kass: production flag wrapper: invalid character 'n' looking for beginning of object key string
2026/10/22 21:27:59 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:59 console: api updates bab/kass/web.kass.carlosapp.dev: enable refused, not every box applies update settings: linus-marquetry: runs a binary that does not apply update settings
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:27:59 console: backfilled account manifest for bab (ops)
2026/10/22 21:27:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:01 console: cli token revoke paul@keymail.dev: hostops: pending
2026/10/22 21:28:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:01 console: materialized account bac (Account) from the bucket
2026/10/22 21:28:01 console: materialized account bac (Account) from the bucket
2026/10/22 21:28:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:02 console: claim restore bab/kass: recovered (restored)
2026/10/22 21:28:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:02 console: claim restore bab/kass: recovered (restored)
2026/10/22 21:28:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 mail: disassociating tenant resource for bab/remove.example.com: lost disassociation response
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: materialized account bac (Testco) from the bucket
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:03 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:28:03 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:28:03 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:28:04 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:28:04 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: backfilled account manifest for bab (ops)
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: backfilled account manifest for bab (ops)
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: backfilled account manifest for bab (ops)
2026/10/22 21:28:05 console: geoip published ed972738 (GeoIP2-City, feed:test-feed)
2026/10/22 21:28:05 console: geoip feed test-feed: edition 2026-10 rejected: too large
2026/10/22 21:28:05 console: geoip database cccccccc (GeoIP2-City) followed from this box's edge
2026/10/22 21:28:05 console: geoip local state unreadable; keeping the loaded database
2026/10/22 21:28:05 console: geoip published ed972738 (GeoIP2-City, manual)
2026/10/22 21:28:05 console: geoip published ed972738 (GeoIP2-City, manual)
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: backfilled account manifest for bab (ops)
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: backfilled account manifest for bab (ops)
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: invite to sam@gmail.com: no keymail mailbox, trying email
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: invite to sam@keymail.dev by keymail: keymail: the recipient's proof-of-work price is above the console's cap
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: invite to sam@keymail.dev: the recipient blocked the console on keymail; not emailing
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: invite to sam@gmail.com: no keymail mailbox, trying email
2026/10/22 21:28:05 console: invite to sam@gmail.com by email: console mail: sending identity not verified yet
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: invite to sam@gmail.com: no keymail mailbox, trying email
2026/10/22 21:28:05 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:05 console: backfilled account manifest for bab (ops)
2026/10/22 21:28:06 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:06 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:06 console: backfilled account manifest for bab (ops)
2026/10/22 21:28:06 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:06 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: lifetimes: allocation 2e2be9f32b16f9f6ef601d894115865c of one.shop.example.com: remove the stale record: the bucket did not answer (pending)
2026/10/22 21:28:08 console: api instances create bab/shop/one.shop.example.com: console: the lifetime operation is pending; recovery finishes it
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: lifetimes: allocation 821c33156d48435c030d147ca0743af0 of one.shop.example.com: commit: console: host_allocations op "821c33156d48435c030d147ca0743af0" is committed and cannot move to committed (pending; the record stands)
2026/10/22 21:28:08 console: api instances create bab/shop/one.shop.example.com: console: the lifetime operation is pending; recovery finishes it
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: lifetimes: allocation ad30cfa18099a2b92791d5c0ad8fb36c of one.shop.example.com: commit: console: host_allocations op "ad30cfa18099a2b92791d5c0ad8fb36c" is committed and cannot move to committed (pending; the record stands)
2026/10/22 21:28:08 console: api instances create bab/shop/one.shop.example.com: console: the lifetime operation is pending; recovery finishes it
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:28:08 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:28:08 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:28:08 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:28:08 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:28:08 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:28:08 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:28:08 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:28:08 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: lifetimes: allocation c7dc254de9127bd62cd7fa8476aaf337 of copy.shop.example.com: write the record: the bucket did not answer (pending)
2026/10/22 21:28:08 console: from main.shop.example.com: reserve copy.shop.example.com: console: the lifetime operation is pending; recovery finishes it
2026/10/22 21:28:08 console: seeds pass: abandoned seed sd_iqwkbjbomhxoyl4ye3dq2pb43a of bab/shop/copy.shop.example.com (reserving since 2026-10-22T21:28:08Z)
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: lifetimes: allocation 86e760abed400f8645a8eb6d46130268 of old.shop.example.com: write the record: the bucket did not answer (pending)
2026/10/22 21:28:08 console: import old.shop.example.com: write the record: console: the lifetime operation is pending; recovery finishes it
2026/10/22 21:28:08 console: seeds pass: abandoned seed sd_4wj6uynfgobedcasvfug5key6q of bab/shop/old.shop.example.com (awaiting-upload since 2026-10-22T21:28:08Z)
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:08 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:08 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:08 console: lifetimes: before -> draining
2026/10/22 21:28:08 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: lifetimes: allocation 1804e7af55b6fd19fa52a354f3031736 of docs.example.com: write the slot: the bucket did not answer (pending)
2026/10/22 21:28:09 console: attach docs.example.com: record write: console: the lifetime operation is pending; recovery finishes it
2026/10/22 21:28:09 console: attach docs.example.com: the claim stays: console: another change to this hostname is still settling
2026/10/22 21:28:09 console: audit system-update platform: sql: database is closed
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: lifetimes: allocation c8749de1ed2026f215c0ea33fc786d93 of docs.example.com: write the record: the bucket did not answer (pending)
2026/10/22 21:28:09 console: attach docs.example.com: record write: console: the lifetime operation is pending; recovery finishes it
2026/10/22 21:28:09 console: attach docs.example.com: the claim stays: console: another change to this hostname is still settling
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: lifetimes: allocation 3d797edb1548b49f34a8d7c3b4f4826f of docs.example.com: commit: constraint failed: test: the commit is refused (1811) (pending; the record stands)
2026/10/22 21:28:09 console: attach docs.example.com: record write: console: the lifetime operation is pending; recovery finishes it
2026/10/22 21:28:09 console: attach docs.example.com: the claim stays: console: another change to this hostname is still settling
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: attach docs.example.com: record write: console: another change to this hostname is still settling
2026/10/22 21:28:09 console: attach docs.example.com: the claim stays: console: another change to this hostname is still settling
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: attach docs.example.com: record write: console: lifetimes: docs.example.com is already held by web; choose another address
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: attach docs.example.com: record write: console: lifetimes: observe bab/shop/docs.example.com: the bucket did not answer
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: attach docs.example.com: record write: console: lifetimes: the app is in the trash
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: could not notify paul@keymail.dev of a CLI token issue: keymail: lookup paul@keymail.dev: Get "http://127.0.0.1:35055/api/federation/lookup?addr=paul%40keymail.dev&from=console%40127.0.0.1%3A33293": dial tcp 127.0.0.1:35055: connect: connection refused
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: could not notify paul@keymail.dev of a CLI token issue: Post "http://127.0.0.1:44201/api/app/send": dial tcp 127.0.0.1:44201: connect: connection refused
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: lifetimes: before -> draining
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: lifetimes: bab/shop/one.shop.example.com: console: lifetimes: fence apps/bab/shop/instances/one.shop.example.com.json: the bucket did not answer (pending)
2026/10/22 21:28:09 console: could not notify paul@keymail.dev of a CLI token issue: Post "http://127.0.0.1:42245/api/app/send": dial tcp 127.0.0.1:42245: connect: connection refused
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: lifetimes: the cutover marker is written (by paul@keymail.dev); admission is closed to legacy work
2026/10/22 21:28:09 console: lifetimes: draining -> on
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: lifetimes: the cutover marker is written (by paul@keymail.dev); admission is closed to legacy work
2026/10/22 21:28:09 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:09 console: lifetimes: the cutover marker exists but cannot be read: control/lifetimes-cutover.json: invalid character 'n' looking for beginning of object key string
2026/10/22 21:28:10 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:10 console: lifetimes: the cutover marker is written (by paul@keymail.dev); admission is closed to legacy work
2026/10/22 21:28:10 console: lifetimes: draining -> on
2026/10/22 21:28:10 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:10 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:10 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:10 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:10 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:10 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:11 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:11 console: moves: warning: kass.carlosapp.dev: the log of move mv_b2nwmnwsvz6iwycrqrcolhakmm at elsewhere is held: the writer refuses it (hostops: identity mismatch); its entry is kept until its incarnation is purged
2026/10/22 21:28:11 console: moves: warning: acme.go.tito.io: the log of move mv_b2nwmnwsvz6iwycrqrcolhakmm at stale is held: the writer refuses it (hostops: identity mismatch); its entry is kept until its incarnation is purged
2026/10/22 21:28:11 console: moves: warning: acme.go.tito.io: the log of move mv_forged at forged is held: the writer refuses it (hostops: identity mismatch); its entry is kept until its incarnation is purged
2026/10/22 21:28:11 console: moves: warning: acme.go.tito.io: the log of move mv_b2nwmnwsvz6iwycrqrcolhakmm at retired is held: the writer refuses it (hostops: identity mismatch); its entry is kept until its incarnation is purged
2026/10/22 21:28:11 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:12 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:13 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:15 console: api instance move bab/titogo/acme.go.tito.io: declared loss refused: hostresolve: evidence or operation does not authorize resolution
2026/10/22 21:28:15 console: moves: warning: acme.go.tito.io: the log of move mv_3ibjwojobkutq6o7y22gjxcfgu at rolling_back-target-lost has been unwritten for over an hour: unconfirmed: no loss of the move's target has committed, and the move can still be declared lost
2026/10/22 21:28:15 console: moves: acme.go.tito.io: the log of move mv_3ibjwojobkutq6o7y22gjxcfgu at rolling_back-target-lost was never confirmed and the claim no longer holds the instance it was queued for; dropped
2026/10/22 21:28:15 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:17 console: api instance move bab/titogo/acme.go.tito.io: typed move write: moves: the declared loss cannot be queued in the move log: SQL logic error: no such table: move_log_outbox (1)
2026/10/22 21:28:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:18 console: claim request refused: moves: the declared loss cannot be queued in the move log: SQL logic error: no such table: move_log_outbox (1)
2026/10/22 21:28:19 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:19 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:19 console: refused: no eligible box in ie
2026/10/22 21:28:19 console: claim request refused: no eligible box in ie: hostops: region has no box that can take this
2026/10/22 21:28:19 console: refused: no eligible box in ie
2026/10/22 21:28:19 console: claim request refused: no eligible box in ie: hostops: region has no box that can take this
2026/10/22 21:28:19 console: refused: no eligible box in ie
2026/10/22 21:28:19 console: claim request refused: no eligible box in ie: hostops: region has no box that can take this
2026/10/22 21:28:19 console: refused: no eligible box in ie
2026/10/22 21:28:19 console: claim request refused: no eligible box in ie: hostops: region has no box that can take this
2026/10/22 21:28:20 console: refused: no eligible box in ie
2026/10/22 21:28:20 console: claim request refused: no eligible box in ie: hostops: region has no box that can take this
2026/10/22 21:28:20 console: refused: no eligible box in ie
2026/10/22 21:28:20 console: claim request refused: no eligible box in ie: hostops: region has no box that can take this
2026/10/22 21:28:20 console: refused: no eligible box in ie
2026/10/22 21:28:20 console: claim request refused: no eligible box in ie: hostops: region has no box that can take this
2026/10/22 21:28:21 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:21 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:21 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:28:21 console: api promote widget abc1234 -> production: promote widget abc1234 -> production: production is entered from beta only, and this build is on edge — use --hotfix to bypass the order
2026/10/22 21:28:21 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
--- FAIL: TestEveryClosedDoorRefusesAServiceCredential (0.05s)
    svcroutes_test.go:1011: walked 64 closed doors, want all 64 (and 63 today: instance-token, three door-key doors, the routes door, instances-manifest, the app update policy, sidecars-manifest, the route-channel move, three instance update settings doors, two db replica doors, six fleets, four services, two email credentials, email pause, the build probe, the swap policy, the live tenancy read, the layout read, the activation-cancel door, nine system doors, three deployment-wide antivirus, three platform geoip doors, two pointer digest doors, the channel resign, two release authorize doors, four store-entitlement doors, two account plan doors, and the five run doors)
FAIL
FAIL	github.com/carlosframework/platform/internal/console	38.013s
FAIL
=== internal_console@4 (42 s)
ok  	github.com/carlosframework/platform/internal/console	40.998s
=== internal_console@5 (38 s)
ok  	github.com/carlosframework/platform/internal/console	36.908s
=== internal_console@6 (44 s)
ok  	github.com/carlosframework/platform/internal/console	42.466s
=== internal_console@7 (38 s)
ok  	github.com/carlosframework/platform/internal/console	37.158s
=== root@0 (30 s)
ok  	github.com/carlosframework/platform	28.763s
=== root@1 (44 s)
ok  	github.com/carlosframework/platform	42.739s
=== root@2 (45 s)
ok  	github.com/carlosframework/platform	43.967s

FAILED: rest internal_console@3
make: *** [Makefile:19: test] Error 1

amadan: step gate failed: exit status 2

amadan: job failed at step gate — its output is above, not at the tail of this log

diff --git a/cmd/console/main.go b/cmd/console/main.go

index ed19a70ec..ef3141377 100644

--- a/cmd/console/main.go

+++ b/cmd/console/main.go

@@ -540,6 +540,11 @@ func serve() {

// claims-off app materialisation as its first admitted iteration. Off on

// a host-claim console.

go srv.RunClaimWorker(context.Background(), "StartLifetimesCutoverWatch")

+ // The lifetime recovery pass (internal/console/lifetimes_recovery.go):

+ // every 60 seconds, finishes or ends each create and orphan retirement

+ // a crash or a failed write left pending. It does work only once

+ // lifetimes are on, and is off on a host-claim console.

+ go srv.RunClaimWorker(context.Background(), "StartLifetimeRecoveryPass")

// The relay's writer lease and its converging reconcile

// (internal/console/relayops.go). Same placement rule as every pass

// above, and it returns immediately when CARLOS_RELAY_REGIONS is empty —

diff --git a/docs/superpowers/notes/census-proposal-self-serve-moves-1.md b/docs/superpowers/notes/census-proposal-self-serve-moves-1.md

index 9025d4d0f..abbc25665 100644

--- a/docs/superpowers/notes/census-proposal-self-serve-moves-1.md

+++ b/docs/superpowers/notes/census-proposal-self-serve-moves-1.md

@@ -11,3 +11,5 @@ Task 6a adds the provision lifetime judge and judged readers, listings and updat

The write census reported four changed sites and three added sites. `GetRecordRaw` remains a read-only key-site and shares its decoder with `ObserveWith`. `updateRecordSeeding` and `claimRawUpdateRecord` remain record-writers through `WriteRecord`: the former judges before each mutation, including its CAS retry; the latter judges before checking the caller's ETag. `IndexRecords` remains a read-only key-site. `ListRecordsCompleteJudged`, `ListRecordsStrictJudged` and `readAllRecordsJudged` are added as read-only key-sites, with separate reviewed dispositions for their completeness and error behavior. Every listed missing body remains a gap; only a present stale body is silently omitted. All Task 6a entries remain `PROPOSED-self-serve-moves-1`, awaiting Paul's census ruling. The removal loop and host and console consumers remain Task 6b work.

Task 0 adds the lifetimes cutover: the operator door `POST /api/cli/system/lifetimes/cutover`, the admission gate and its watcher in `internal/console/lifetimes_cutover.go`, and `carlos system lifetimes cutover`. The record-write census reported one unlisted and five changed sites. `Server.handleAPILifetimesCutover` is added as nonrecord: one create-only Put of `control/lifetimes-cutover.json` in the deployment bucket, with no record key and no record literal. `Server.runImportUpload` and `Server.runFrom` remain record-writers through `SetRecordSeed`; each now carries its request's admitted unit and ends it after its last write, with no write site added or moved. `Server.handleAPIInstanceList` remains a read-only key-site and lists through `ListRecordsCompleteJudged` with the request's judge, which is nil before the cutover. `Server.registerCLIRoutes` registers the door. `NewServer` adds the judge, the gate and its first marker read, and moves the claims-off app materialisation to the watch's first admitted iteration; it writes nothing. The root-read census reported four changed accepted-legacy console units (`serve`, `NewServer`, `Server.handleAPIInstanceList`, `Server.runFrom`), two re-pinned declarations (`runSystem`, `systemVerb`, the lifetimes verb at their existing authority) and three envelopes (`.`, `cmd/console`, `internal/console`). The four legacy units are re-pinned with reviewed bases, which adds four fingerprints and moves the seal; no root read, class or task is added and the registry fields stand. Everything here remains `PROPOSED-self-serve-moves-1`, awaiting Paul's census ruling.

+Task 3 adds the claims-off create protocol and its recovery (`internal/console/lifetimes.go`, `lifetimes_create.go`, `lifetimes_recovery.go`) and routes the create, seed, import and first-route attach doors through it when the request was admitted with lifetimes on. The record-write census reported two changed sites, two stale entries and five unlisted sites. `Server.handleAPIInstanceCreate` and `Server.attachDomainAs` remain record-writers through `CreateRecord` on the legacy path; admitted on, they leave for `createRecordWithLifetime` and `attachFirstRouteWithLifetime`. `Server.beginFrom` and `Server.beginImport` no longer write directly: both call `Server.createSeedRecord`, a record-writer that is either the lifetime create or today's create-only `CreateRecord`. `Server.allocRecordLocked` is a record-writer: one create-only `WriteRecord` of the allocation's persisted body from `{}` with the record sequence on, so the record lands with `record_seq: 1`. `Server.fenceRecord` is a record-writer: it rewrites the exact bytes it observed against their own ETag with the sequence on, and writes no other body. `Server.allocCleanLocked` and `Server.deleteUnderRetiredSlot` are key-sites: each removes a stale body only with `DeleteIfMatch` on an ETag it holds, never a plain delete, and neither Puts. The slot is written only through `provision.WriteHostIncarnation`, which is already nonrecord. The root-read census reported two changed accepted-legacy units, one unclassified unit and two envelopes. `serve` is re-pinned in Task 0's rows, since it now also starts the lifetime recovery pass. `Server.hostHolder` keeps its root-route disposition; its instance-record holder read is now judged under a request admitted on (a nil judge is `GetRecord`). `newLifetimeOpID` is classified like `mintLifetime`: a crypto/rand read for an operation id that grants no root authority. The `cmd/console` and `internal/console` envelopes are re-pinned. Three new fingerprints move the seal. No class or task is added, and the registry fields stand. Everything here remains `PROPOSED-self-serve-moves-1`, awaiting Paul's census ruling.

diff --git a/internal/console/apex.go b/internal/console/apex.go

index eb002fdc4..daea6a80f 100644

--- a/internal/console/apex.go

+++ b/internal/console/apex.go

@@ -147,7 +147,10 @@ func (s *Server) hostHolder(ctx context.Context, acct *Account, host, own string

if c.app == name {

continue

}

- rec, err := provision.GetRecord(ctx, s.cfg.Releases, acct.Sqid, c.app, host)

+ // Judged when the request was admitted with lifetimes on (a nil

+ // judge is GetRecord): a body its slot no longer vouches for holds

+ // nothing.

+ rec, _, err := provision.ReadRecordJudged(ctx, s.cfg.Releases, provision.JudgeFrom(ctx), acct.Sqid, c.app, host)

if errors.Is(err, os.ErrNotExist) {

continue

}

diff --git a/internal/console/apidomains.go b/internal/console/apidomains.go

index c97618b7d..d59a7edc1 100644

--- a/internal/console/apidomains.go

+++ b/internal/console/apidomains.go

@@ -976,6 +976,12 @@ func (s *Server) handleAPIDomainDetach(w http.ResponseWriter, r *http.Request) {

}

ctx, cancel := context.WithTimeout(r.Context(), instanceStoreTimeout)

defer cancel()

+ // Admitted on, the claim read, the record and the claim release run

+ // inside the app lock and the hostname exclusion that every attach of

+ // the hostname holds from its claim to its record (lifetimes.go).

+ if lifetimesUnit(ctx) {

+ defer s.lockAppHost(acct.Sqid, app, host)()

+ }

// The first-route half (#82): a claim with an empty Route minted the

// app's own primary route via a declared provision record, and that

// record must go BEFORE the claim — the mirror of attach's claim-then-

diff --git a/internal/console/apiinstanceimport.go b/internal/console/apiinstanceimport.go

index 109e236e7..0ac4b36b1 100644

--- a/internal/console/apiinstanceimport.go

+++ b/internal/console/apiinstanceimport.go

@@ -169,9 +169,17 @@ func (s *Server) beginImport(w http.ResponseWriter, r *http.Request, c seedCreat

// Not the request's cancellation: a client leaving now must not turn a

// write that landed into an error.

writeCtx, writeCancel := context.WithTimeout(context.WithoutCancel(ctx), instanceStoreTimeout)

- err = provision.CreateRecord(writeCtx, s.cfg.Releases, rec)

+ err = s.createSeedRecord(writeCtx, c, rec)

writeCancel()

if err != nil {

+ // A lifetime create refused, or left pending, wrote nothing this

+ // door may undo: a pending one's record, once recovery writes it,

+ // is an import awaiting its upload, which the seeds sweep owns.

+ if ref, ok := lifetimeCreateRefusal(host, err, importFailed); ok {

+ log.Printf("console: import %s: write the record: %v", host, err)

+ writeDoorRefusalJSON(w, r, ref)

+ return

+ }

if errors.Is(err, release.ErrCASConflict) {

jsonError(w, http.StatusConflict, "instance "+host+" already exists")

return

diff --git a/internal/console/apiinstances.go b/internal/console/apiinstances.go

index dcb75f76f..2b8228f95 100644

--- a/internal/console/apiinstances.go

+++ b/internal/console/apiinstances.go

@@ -1010,6 +1010,13 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)

if s.refuseServiceControlPlane(w, p, acct.ID, app, "create its instance records") {

return

}

+ // A create admitted with lifetimes on is bound to the app lifetime it

+ // was admitted for: the protocol compares it under the app lock, so a

+ // purge and a recreate under the same name meanwhile is refused.

+ lifetime, ok := s.doorLifetime(w, r, acct, app)

+ if !ok {

+ return

+ }

ctx, cancel := context.WithTimeout(r.Context(), instanceStoreTimeout)

defer cancel()

m, ok := s.instanceManifest(ctx, w, acct, app)

@@ -1565,7 +1572,9 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)

logTag: "api instances create " + acct.Sqid + "/" + app,

host: host, region: region, pool: rec.Pool,

exists: func(ctx context.Context) (bool, error) {

- _, err := provision.GetRecord(ctx, s.cfg.Releases, acct.Sqid, app, host)

+ // Judged when admitted on (a nil judge is GetRecord): a stale

+ // body is no record, and the create replaces it.

+ _, _, err := provision.ReadRecordJudged(ctx, s.cfg.Releases, provision.JudgeFrom(ctx), acct.Sqid, app, host)

if errors.Is(err, os.ErrNotExist) {

return false, nil

}

@@ -1584,12 +1593,12 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)

// writes its own record (apiinstanceseed.go).

if req.From != nil {

s.createFrom(w, r, seedCreate{acct: acct, app: app, principal: principal, message: message, m: m,

- rec: rec, placed: placed, regionFrom: regionFrom, siblingLabel: siblingLabel}, *req.From)

+ rec: rec, placed: placed, regionFrom: regionFrom, siblingLabel: siblingLabel, lifetime: lifetime}, *req.From)

return

}

if req.Import != nil {

s.createImport(w, r, seedCreate{acct: acct, app: app, principal: principal, message: message, m: m,

- rec: rec, placed: placed, regionFrom: regionFrom, siblingLabel: siblingLabel}, *req.Import)

+ rec: rec, placed: placed, regionFrom: regionFrom, siblingLabel: siblingLabel, lifetime: lifetime}, *req.Import)

return

}

// A fresh budget for the write and its rollback: validation and the

@@ -1660,14 +1669,29 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)

defer recancel()

}

var recordWriteErr error

- if claimRequest != nil {

+ switch {

+ case claimRequest != nil:

if !s.finishClaimCreate(w, claimRequest) {

return

}

- } else {

+ case lifetimesUnit(ctx):

+ // Dispatch by the admitted unit (lifetimes.go): the lifetime

+ // create, which answers the record as written.

+ var created *provision.Record

+ if created, recordWriteErr = s.createRecordWithLifetime(ctx, rec, lifetime); recordWriteErr == nil {

+ rec = *created

+ }

+ default:

recordWriteErr = provision.CreateRecord(ctx, s.cfg.Releases, rec)

}

if err := recordWriteErr; err != nil {

+ if ref, ok := lifetimeCreateRefusal(host, err, msgLifetimeCreatePending); ok {

+ if !errors.Is(err, errLifetimeSettling) && !errors.As(err, new(*hostHeldError)) {

+ log.Printf("console: api instances create %s/%s/%s: %v", acct.Sqid, app, host, err)

+ }

+ writeDoorRefusalJSON(w, r, ref)

+ return

+ }

if errors.Is(err, release.ErrCASConflict) {

// A lost race against a second signup for the same host, or a

// plain duplicate. Either way the winner's record stands.

@@ -1685,6 +1709,9 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)

// re-check a rename makes (appsettings.go).

if siblingLabel != "" {

if live, trashed, err := s.store.AppNameState(acct.ID, siblingLabel); err == nil && (live || trashed) {

+ // Today's chain in both modes. Task 4 binds it to the lifetime

+ // this create made (RetireExpectation{Incarnation:

+ // rec.Incarnation}) through retireRecordWithLifetime.

if derr := s.retireInstanceRecord(ctx, acct.Sqid, app, host, principal, mustCreateIncarnation(acct.Sqid, app, host, rec.CreatedAt)); derr != nil {

log.Printf("console: api instances create %s/%s/%s: %s was claimed mid-create and UNDO FAILED: %v", acct.Sqid, app, host, siblingLabel, derr)

}

@@ -2230,6 +2257,20 @@ func (s *Server) handleAPIInstanceDelete(w http.ResponseWriter, r *http.Request)

jsonError(w, http.StatusBadRequest, "invalid host")

return

}

+ // Admitted on, a hostname whose create, retirement or migration is

+ // still settling is not deleted under it (lifetimes.go). The delete

+ // itself is today's until Task 4 makes it a retirement.

+ if lifetimesUnit(ctx) {

+ switch err := s.hostSettling(acct.Sqid, app, host); {

+ case errors.Is(err, errLifetimeSettling):

+ jsonError(w, http.StatusConflict, msgLifetimeSettling)

+ return

+ case err != nil:

+ log.Printf("console: api instances delete %s/%s/%s: %v", acct.Sqid, app, host, err)

+ jsonError(w, http.StatusBadGateway, "could not delete the instance — try again")

+ return

+ }

// With CARLOS_MOVES_PHASE1 on, a host being moved is not deleted

// (M-C19, apiinstancemove.go); with it off this is main's door.

if s.refuseDeleteDuringMove(ctx, w, acct.Sqid, app, host) {

diff --git a/internal/console/apiinstanceseed.go b/internal/console/apiinstanceseed.go

index a5a91ffe0..c4f2a2d55 100644

--- a/internal/console/apiinstanceseed.go

+++ b/internal/console/apiinstanceseed.go

@@ -71,6 +71,20 @@ type seedCreate struct {

rec provision.Record

placed, regionFrom string

siblingLabel string

+ // lifetime is the apps.lifetime the create door read before any lock,

+ // for a create admitted with lifetimes on (doorLifetime).

+ lifetime string

+// createSeedRecord writes a seed door's first record: the lifetime create

+// when the request was admitted with lifetimes on (dispatch by the unit),

+// today's create-only write otherwise.

+func (s *Server) createSeedRecord(ctx context.Context, c seedCreate, rec provision.Record) error {

+ if lifetimesUnit(ctx) {

+ _, err := s.createRecordWithLifetime(ctx, rec, c.lifetime)

+ return err

+ }

+ return provision.CreateRecord(ctx, s.cfg.Releases, rec)

}

// seedCommonRefusal is both seed doors' shape rule: an ordinary instance.

@@ -343,7 +357,15 @@ func (s *Server) beginFrom(w http.ResponseWriter, r *http.Request, c seedCreate,

// copy while the reserving record held the host.

writeCtx, writeCancel := context.WithTimeout(context.WithoutCancel(ctx), instanceStoreTimeout)

defer writeCancel()

- if err := provision.CreateRecord(writeCtx, s.cfg.Releases, rec); err != nil {

+ if err := s.createSeedRecord(writeCtx, c, rec); err != nil {

+ // A pending lifetime create writes nothing further: the record

+ // recovery completes carries this reserving seed, which the seeds

+ // sweep then abandons.

+ if ref, ok := lifetimeCreateRefusal(host, err, failed); ok {

+ log.Printf("console: from %s: reserve %s: %v", source, host, err)

+ writeDoorRefusalJSON(w, r, ref)

+ return nil

+ }

if errors.Is(err, release.ErrCASConflict) {

jsonError(w, http.StatusConflict, "instance "+host+" already exists")

return nil

diff --git a/internal/console/applocks.go b/internal/console/applocks.go

index 0b0fff4e0..b374ad017 100644

--- a/internal/console/applocks.go

+++ b/internal/console/applocks.go

@@ -56,4 +56,16 @@ func (s *Server) lockAccount(sqid string) func() { return s.acctLocks.lock(sqid)

// lockApp is the per-app mutex every lifecycle transition takes: keep, ttl,

// restore, trash, the purge door and each step of the expiry pass (spec §5).

-func (s *Server) lockApp(sqid, app string) func() { return s.appLocks.lock(sqid + "/" + app) }

+//

+// LOCK ORDER: the account lock, then this one, then withHostname

+// (lifetimes.go's lock table). Never the other way round.

+func (s *Server) lockApp(sqid, app string) func() {

+ key := sqid + "/" + app

+ lockEvent("app", key, lockWaiting)

+ unlock := s.appLocks.lock(key)

+ lockEvent("app", key, lockHeld)

+ return func() {

+ lockEvent("app", key, lockReleased)

+ unlock()

+ }

diff --git a/internal/console/claim_invariant_fixtures_test.go b/internal/console/claim_invariant_fixtures_test.go

index 74883ad8b..9113a7443 100644

--- a/internal/console/claim_invariant_fixtures_test.go

+++ b/internal/console/claim_invariant_fixtures_test.go

@@ -1858,6 +1858,7 @@ var claimFixtures = map[string]claimFixture{

"worker StartRunSweep": claimRead("off on a host-claim console, where every run route refuses: it logs that and writes nothing"),

"worker StartSeedsPass": claimRead("off on a host-claim console, where every seed door and finishSeed refuse: it logs that and writes nothing"),

"worker StartLifetimesCutoverWatch": claimRead("off on a host-claim console, where lifetimes are not for it: it logs that and reads nothing"),

+ "worker StartLifetimeRecoveryPass": claimRead("off on a host-claim console, where lifetimes are not for it: it logs that and reads nothing"),

"worker StartAlertWatcher": claimNonApp("box alert state under alerts/boxes and operator mail"),

"worker StartEconomicsPass": claimNonApp("daily usage census, statements and bills: deployment and account economics"),

"worker StartGeoipJobs": claimNonApp("GeoIP orphan sweep, D1 expiry and feed status under control/geoip/: the platform database, no app"),

diff --git a/internal/console/claim_invariant_probes_worker_test.go b/internal/console/claim_invariant_probes_worker_test.go

index 605255021..97d974162 100644

--- a/internal/console/claim_invariant_probes_worker_test.go

+++ b/internal/console/claim_invariant_probes_worker_test.go

@@ -80,6 +80,7 @@ var claimWorkerProbes = map[string]claimProbe{

"worker StartRunSweep": claimLogWorkerProbe("StartRunSweep", nil, "run sweep", "host-claim console"),

"worker StartSeedsPass": claimLogWorkerProbe("StartSeedsPass", nil, "seeds pass", "host-claim console"),

"worker StartLifetimesCutoverWatch": claimLogWorkerProbe("StartLifetimesCutoverWatch", nil, "lifetimes cutover watch", "host-claim console"),

+ "worker StartLifetimeRecoveryPass": claimLogWorkerProbe("StartLifetimeRecoveryPass", nil, "lifetime recovery pass", "host-claim console"),

"worker LogPlacementDiscrepancies": claimLogWorkerProbe("LogPlacementDiscrepancies", claimRegionalApp,

msgPlacementDiscrepancy("drift.go.tito.io", "blog", "uk")),

diff --git a/internal/console/claim_workers.go b/internal/console/claim_workers.go

index 204edae8c..84cfef248 100644

--- a/internal/console/claim_workers.go

+++ b/internal/console/claim_workers.go

@@ -18,6 +18,7 @@ func (s *Server) registerClaimWorkers() {

"StartRunSweep": s.StartRunSweep,

"StartSeedsPass": s.StartSeedsPass,

"StartLifetimesCutoverWatch": s.StartLifetimesCutoverWatch,

+ "StartLifetimeRecoveryPass": s.StartLifetimeRecoveryPass,

"StartRelay": s.StartRelay,

"StartMailEventPush": s.StartMailEventPush,

"StartDoorKeyResign": s.StartDoorKeyResign,

diff --git a/internal/console/domainattach.go b/internal/console/domainattach.go

index 36769f333..57ce1c787 100644

--- a/internal/console/domainattach.go

+++ b/internal/console/domainattach.go

@@ -211,6 +211,9 @@ func (s *Server) attachDomainAs(ctx context.Context, acct *Account, app, host, k

if s.cfg.HostClaim {

return s.attachClaimFirstRoute(ctx, acct, app, host, channel, principal)

}

+ if lifetimesUnit(ctx) {

+ return s.attachFirstRouteWithLifetime(ctx, acct, app, host, channel, principal, message, claim, prove, verify)

+ }

created, changed, pending, err := s.attachDomainClaim(ctx, acct, app, claim, time.Now(), prove)

if err != nil {

return attachResult{}, claimRefusal(host, err)

@@ -330,6 +333,12 @@ func (s *Server) detachDomain(ctx context.Context, acct *Account, app, host, pri

if s.domainHostPolicy(host, acct.Sqid) == hostPlatform && !s.platformOperator(principal) {

return &doorRefusal{http.StatusForbidden, platformOperatorMsg}

}

+ if lifetimesUnit(ctx) {

+ // Admitted on: the claim read, the record and the claim release

+ // run inside the app lock and the hostname exclusion, the same two

+ // every attach of the hostname holds from its claim to its record.

+ defer s.lockAppHost(acct.Sqid, app, host)()

+ }

claim, err := domains.GetClaim(ctx, s.cfg.Releases, host)

if err != nil {

if errors.Is(err, os.ErrNotExist) {

@@ -566,6 +575,122 @@ func (s *Server) rollbackAttach(ctx context.Context, acct *Account, app, host st

}

+// attachFirstRouteWithLifetime is the claims-off first-route attach admitted

+// with lifetimes on: from the claim to the record it runs inside

+// s.lockApp(acct, app) and then the hostname exclusion (the app lock taken

+// outside the exclusion, never inside it). Every competing attach's claim

+// write or reuse and lifetime create, every rollback and every detach of

+// the hostname run under the same exclusion, so no attach reuses a claim

+// another is about to release, and no release happens while another attach

+// is between its claim and its record. lifetime is the apps.lifetime read

+// here, before any lock.

+func (s *Server) attachFirstRouteWithLifetime(ctx context.Context, acct *Account, app, host, channel, principal, message string, claim domains.Claim, prove func() bool, verify dnsRecord) (attachResult, *doorRefusal) {

+ life, err := s.store.AppLife(acct.ID, app)

+ if err != nil {

+ log.Printf("console: attach %s: read the app: %v", host, err)

+ return attachResult{}, &doorRefusal{http.StatusBadGateway, msgLifetimeAppUnreadable}

+ }

+ lifetime := ""

+ if life != nil {

+ lifetime = life.Lifetime

+ }

+ defer s.lockAppHost(acct.Sqid, app, host)()

+ return s.attachFirstRouteLocked(ctx, acct, app, host, channel, principal, message, lifetime, claim, prove, verify)

+// attachFirstRouteLocked is the first-route attach under both locks, which

+// the caller holds: the claim written or reused, then the record through

+// createRecordWithLifetimeLocked. It never retires a record: a failed

+// lifetime create is answered with its sentence and rollbackAttachLocked,

+// which can only release the claim.

+func (s *Server) attachFirstRouteLocked(ctx context.Context, acct *Account, app, host, channel, principal, message, lifetime string, claim domains.Claim, prove func() bool, verify dnsRecord) (attachResult, *doorRefusal) {

+ created, changed, pending, err := s.attachDomainClaim(ctx, acct, app, claim, time.Now(), prove)

+ if err != nil {

+ return attachResult{}, claimRefusal(host, err)

+ }

+ if pending {

+ // No record until the claim is proven, as on the legacy path.

+ if changed {

+ s.auditActivity(acct, app, principal, Activity{

+ Action: "domain-attach", TargetType: "instance", Target: host, Message: message,

+ Detail: host + " (first route, " + channel + ")" + pendingNote(true),

+ })

+ }

+ return attachResult{Claim: claim, Created: created, Channel: channel, Pending: true, Verify: verify}, nil

+ }

+ rec := provision.Record{

+ Schema: 1, Host: host, App: app, Account: acct.Sqid,

+ Channel: channel, Kind: "static",

+ CreatedAt: time.Now().UTC(), CreatedBy: principal,

+ }

+ ambiguous := &doorRefusal{http.StatusInternalServerError,

+ "could not declare the route for this domain — nothing was attached, try again"}

+ _, err = s.createRecordWithLifetimeLocked(ctx, rec, lifetime)

+ switch {

+ case err == nil:

+ s.auditActivity(acct, app, principal, Activity{

+ Action: "domain-attach", TargetType: "instance", Target: host, Message: message,

+ Detail: host + " (first route, " + channel + ")",

+ })

+ s.bumpSerial(ctx)

+ return attachResult{Claim: claim, Created: created, FirstRoute: true, Channel: channel, Verify: verify}, nil

+ case errors.Is(err, release.ErrCASConflict):

+ // A record of this lifetime is already at the key: adopt it as the

+ // legacy path does when this door could have written it.

+ existing, _, gerr := provision.ReadRecordJudged(ctx, s.cfg.Releases, s.judge, acct.Sqid, app, host)

+ if gerr == nil && attachWrote(*existing) {

+ if changed {

+ s.auditActivity(acct, app, principal, Activity{

+ Action: "domain-attach", TargetType: "instance", Target: host, Message: message,

+ Detail: host + " (first route, " + existing.Channel + ", record already present)",

+ })

+ s.bumpSerial(ctx)

+ }

+ return attachResult{Claim: claim, Created: created, FirstRoute: true, Channel: existing.Channel, Verify: verify}, nil

+ }

+ log.Printf("console: attach %s: existing record is not this door's to adopt (read: %v)", host, gerr)

+ s.rollbackAttachLocked(ctx, acct, app, host, created)

+ return attachResult{}, &doorRefusal{http.StatusUnprocessableEntity,

+ "a route record already exists for this host — detach it or pick another hostname"}

+ }

+ log.Printf("console: attach %s: record write: %v", host, err)

+ s.rollbackAttachLocked(ctx, acct, app, host, created)

+ if ref, ok := lifetimeCreateRefusal(host, err, ambiguous.Msg); ok {

+ if errors.Is(err, errLifetimePending) {

+ return attachResult{}, ambiguous

+ }

+ return attachResult{}, ref

+ }

+ return attachResult{}, ambiguous

+// rollbackAttachLocked is a failed lifetime attach's unwind, under both

+// locks. It never retires a record, and it releases the claim only when

+// this call minted it, no operation of the hostname is pending, and no

+// valid record stands under this key: a pending allocation keeps its claim,

+// so the record recovery writes is never left without one.

+func (s *Server) rollbackAttachLocked(ctx context.Context, acct *Account, app, host string, minted bool) {

+ if !minted {

+ return

+ }

+ if err := s.admit(host, ""); err != nil {

+ log.Printf("console: attach %s: the claim stays: %v", host, err)

+ return

+ }

+ obs, err := s.classifyLocked(ctx, acct.Sqid, app, host)

+ if err != nil {

+ log.Printf("console: attach %s: the claim stays: %v", host, err)

+ return

+ }

+ if validRecord(obs) {

+ log.Printf("console: attach %s: the claim stays: a record of this lifetime stands", host)

+ return

+ }

+ if _, err := s.detachDomainClaim(ctx, host, acct.Sqid, app); err != nil {

+ log.Printf("console: attach %s: claim rollback: %v (claim is stranded — detach clears it)", host, err)

+ }

// claimRefusal maps a lost claim write onto the door's answer. The 409 is

// ONE string whoever holds the host: domains.ErrHostUnavailable already

// refuses to name the holder, and the text here is its message verbatim —

diff --git a/internal/console/lifetimes.go b/internal/console/lifetimes.go

new file mode 100644

index 000000000..61852f8f0

--- /dev/null

+++ b/internal/console/lifetimes.go

@@ -0,0 +1,276 @@

+package console

+import (

+ "context"

+ "crypto/rand"

+ "encoding/hex"

+ "errors"

+ "fmt"

+ "log"

+ "net/http"

+ "sync"

+ "sync/atomic"

+ "time"

+ "github.com/carlosframework/platform/internal/provision"

+// Instance lifetimes on a claims-off console (self-serve moves plan 1,

+// Task 3): the per-hostname exclusion, the admission of an operation against

+// the intents already pending for its hostname, and the one classification

+// every lifetime step acts on.

+//

+// THE LOCK TABLE. Neither mutex is reentrant, and the order is always the

+// account lock (where a door already holds one), then s.lockApp, then

+// withHostname:

+//

+// - createRecordWithLifetime (the create, seed and import doors) takes

+// s.lockApp then withHostname; createRecordWithLifetimeLocked assumes

+// both.

+// - the attach door takes s.lockApp before it enters its hostname

+// exclusion; attachFirstRouteLocked and rollbackAttachLocked assume

+// both. A claims-off detach admitted on runs its claim read, record

+// removal and claim release inside the same two.

+// - the recovery pass takes s.lockApp then withHostname per row.

+// - retireSlotOrphanLocked assumes ONLY the hostname lock. It acts on the

+// slot it captured, by a CAS from that slot's ETag, and never takes the

+// slot owner's app lock: a create under key B retiring A's orphan slot

+// holds app(B) and the hostname; A's purge holds app(A) and waits for the

+// hostname; whichever CAS on the slot lands first wins and the other

+// re-observes under the hostname lock. Taking app(A) there would invert

+// the order.

+//

+// TestLockOrderIsRespected runs every caller under a checker fed by

+// lifetimeLockHook and fails on an inversion.

+const (

+ msgLifetimeSettling = "another change to this instance is still settling. try again in a minute"

+ msgLifetimeCreatePending = "could not create the instance. try again"

+ msgLifetimeAppUnreadable = "could not read the app. try again"

+ msgLifetimeAppTrashed = "the app is in the trash"

+ msgLifetimeAppChanged = "the app changed. try again"

+// errLifetimeSettling: another operation on this hostname is pending, or one

+// took the slot first. Nothing of this operation is left pending.

+var errLifetimeSettling = errors.New("console: another change to this hostname is still settling")

+// errLifetimePending: this operation's intent is durable and still pending;

+// the recovery pass finishes it. The caller writes nothing further.

+var errLifetimePending = errors.New("console: the lifetime operation is pending; recovery finishes it")

+// hostHeldError is a create refused because another record key holds the

+// hostname. Hold is what hostHeldSentence words it from.

+type hostHeldError struct {

+ Host string

+ Hold *apexHold

+func (e *hostHeldError) Error() string {

+ return "console: lifetimes: " + hostHeldSentence(e.Host, e.Hold)

+// lifetimeAppError is step 0's refusal: the app this create was admitted for

+// is unreadable, in the trash, gone, or a different lifetime under the same

+// name. Nothing is persisted.

+type lifetimeAppError struct {

+ Status int

+ Msg string

+ Err error

+func (e *lifetimeAppError) Error() string {

+ if e.Err != nil {

+ return "console: lifetimes: " + e.Msg + ": " + e.Err.Error()

+ }

+ return "console: lifetimes: " + e.Msg

+func (e *lifetimeAppError) Unwrap() error { return e.Err }

+// lifetimeCreateRefusal maps a lifetime create's error onto a door's answer.

+// pendingMsg is the door's sentence for errLifetimePending. ok is false for

+// release.ErrCASConflict and ordinary errors, which each door answers as it

+// always has. A 404 is the app gone: the doors answer it as their family's

+// own 404.

+func lifetimeCreateRefusal(host string, err error, pendingMsg string) (*doorRefusal, bool) {

+ var held *hostHeldError

+ var app *lifetimeAppError

+ switch {

+ case errors.As(err, &held):

+ return &doorRefusal{http.StatusConflict, hostHeldSentence(host, held.Hold)}, true

+ case errors.Is(err, errLifetimeSettling):

+ return &doorRefusal{http.StatusConflict, msgLifetimeSettling}, true

+ case errors.Is(err, errLifetimePending):

+ return &doorRefusal{http.StatusBadGateway, pendingMsg}, true

+ case errors.As(err, &app):

+ return &doorRefusal{app.Status, app.Msg}, true

+ }

+ return nil, false

+// lifetimesUnit reports whether ctx's work was admitted with lifetimes on:

+// the one switch every claims-off create, delete and detach site dispatches

+// by. A unit admitted before the cutover keeps today's legacy calls for its

+// whole life, and !cfg.HostClaim alone never selects the lifetime path.

+func lifetimesUnit(ctx context.Context) bool {

+ u := unitFrom(ctx)

+ return u != nil && u.mode == cutoverOn

+// Lock events for lifetimeLockHook.

+const (

+ lockWaiting = "waiting"

+ lockHeld = "held"

+ lockReleased = "released"

+// lifetimeLockHook, when set, sees every app and hostname lock as it is

+// waited for, held and released. A TEST SEAM: the lock-order checker.

+var lifetimeLockHook atomic.Pointer[func(kind, key, event string)]

+func lockEvent(kind, key, event string) {

+ if h := lifetimeLockHook.Load(); h != nil {

+ (*h)(kind, key, event)

+ }

+// withHostname runs fn holding host's exclusion: one mutex per hostname for

+// the life of the process. Not reentrant. Every create, retirement,

+// migration and recovery of a hostname in this console runs inside it.

+func (s *Server) withHostname(host string, fn func() error) error {

+ defer s.lockHostname(host)()

+ return fn()

+// lockHostname blocks until host's exclusion is held and returns the release.

+func (s *Server) lockHostname(host string) func() {

+ v, _ := s.hostLocks.LoadOrStore(host, &sync.Mutex{})

+ mu := v.(*sync.Mutex)

+ lockEvent("host", host, lockWaiting)

+ mu.Lock()

+ lockEvent("host", host, lockHeld)

+ return func() {

+ lockEvent("host", host, lockReleased)

+ mu.Unlock()

+ }

+// lockAppHost is s.lockApp then the hostname exclusion, the order of the

+// lock table, released in reverse:

+//

+// defer s.lockAppHost(acct.Sqid, app, host)()

+func (s *Server) lockAppHost(sqid, app, host string) func() {

+ unlockApp := s.lockApp(sqid, app)

+ unlockHost := s.lockHostname(host)

+ return func() {

+ unlockHost()

+ unlockApp()

+ }

+// withAppHost runs fn holding s.lockApp then the hostname exclusion.

+func (s *Server) withAppHost(sqid, app, host string, fn func() error) error {

+ defer s.lockAppHost(sqid, app, host)()

+ return fn()

+// admit refuses an operation on host while another operation's intent for it

+// is pending. ownOpID is the caller's own pending row, left out of the

+// count; "" for a new operation. A failed read is an ordinary error.

+func (s *Server) admit(host, ownOpID string) error {

+ n, err := s.store.PendingForHost(host, ownOpID)

+ if err != nil {

+ return fmt.Errorf("console: lifetimes: pending operations of %s: %w", host, err)

+ }

+ if n > 0 {

+ return errLifetimeSettling

+ }

+ return nil

+// classifyLocked observes acct/app/host's record and then the hostname's

+// slot (the judge's bounded rounds). The error is non-nil only for an

+// unreadable observation, which no step acts on.

+func (s *Server) classifyLocked(ctx context.Context, acct, app, host string) (provision.Observation, error) {

+ if s.judge == nil {

+ return provision.Observation{Verdict: provision.LifetimeUnreadable},

+ errors.New("console: lifetimes: no lifetime judge on this console")

+ }

+ obs := s.judge.Observe(ctx, s.cfg.Releases, acct, app, host)

+ if obs.Verdict == provision.LifetimeUnreadable {

+ err := obs.Err

+ if err == nil {

+ err = provision.ErrLifetimeUnreadable

+ }

+ return obs, fmt.Errorf("console: lifetimes: observe %s/%s/%s: %w", acct, app, host, err)

+ }

+ return obs, nil

+// validRecord reports whether obs is a record its slot vouches for.

+func validRecord(obs provision.Observation) bool {

+ return obs.Present && obs.Verdict == provision.LifetimeValid && obs.Rec != nil && !obs.Rec.Retired

+// slotOwnedBy reports whether slot belongs to the record key acct/app.

+func slotOwnedBy(slot *provision.HostIncarnation, acct, app string) bool {

+ return slot != nil && slot.Account == acct && slot.App == app

+// newLifetimeOpID is a fresh operation id: 16 random bytes in hex.

+func newLifetimeOpID() (string, error) {

+ var b [16]byte

+ if _, err := rand.Read(b[:]); err != nil {

+ return "", err

+ }

+ return hex.EncodeToString(b[:]), nil

+// lifetimeLogHourly logs once per hour per key: a row recovery leaves

+// pending on every pass would otherwise log every minute.

+func (s *Server) lifetimeLogHourly(key, format string, args ...any) {

+ now := time.Now()

+ if v, ok := s.lifetimeLogged.Load(key); ok && now.Sub(v.(time.Time)) < time.Hour {

+ return

+ }

+ s.lifetimeLogged.Store(key, now)

+ log.Printf(format, args...)

+// doorLifetime is the apps.lifetime a creation door admitted on reads before

+// it takes any lock, for step 0 to compare under the app lock. Off the

+// lifetime path it reads nothing. A row that is not there is "": step 0

+// answers that.

+func (s *Server) doorLifetime(w http.ResponseWriter, r *http.Request, acct *Account, app string) (string, bool) {

+ if !lifetimesUnit(r.Context()) {

+ return "", true

+ }

+ life, err := s.store.AppLife(acct.ID, app)

+ if err != nil {

+ log.Printf("console: lifetimes: read the app %s/%s: %v", acct.Sqid, app, err)

+ jsonError(w, http.StatusBadGateway, msgLifetimeAppUnreadable)

+ return "", false

+ }

+ if life == nil {

+ return "", true

+ }

+ return life.Lifetime, true

+// writeDoorRefusalJSON answers a refusal on a CLI door. A 404 is the

+// family's byte-identical http.NotFound.

+func writeDoorRefusalJSON(w http.ResponseWriter, r *http.Request, ref *doorRefusal) {

+ if ref.Status == http.StatusNotFound {

+ http.NotFound(w, r)

+ return

+ }

+ jsonError(w, ref.Status, ref.Msg)

+// hostSettling is the delete door's admission while lifetimes are on: a

+// hostname whose create, retirement or migration is still pending is not

+// deleted under it. Task 4 makes the delete itself a retirement.

+func (s *Server) hostSettling(sqid, app, host string) error {

+ return s.withAppHost(sqid, app, host, func() error { return s.admit(host, "") })

diff --git a/internal/console/lifetimes_create.go b/internal/console/lifetimes_create.go

new file mode 100644

index 000000000..ae2bbaf00

--- /dev/null

+++ b/internal/console/lifetimes_create.go

@@ -0,0 +1,537 @@

+package console

+import (

+ "context"

+ "encoding/json"

+ "errors"

+ "fmt"

+ "log"

+ "math"

+ "net/http"

+ "os"

+ "github.com/carlosframework/platform/internal/provision"

+ "github.com/carlosframework/platform/internal/release"

+// The create protocol (self-serve moves plan 1, Task 3; spec §6.4). A

+// claims-off create admitted on allocates the hostname's next incarnation

+// through a persisted intent (host_allocations), writes the slot naming this

+// record key and operation, then the record, then commits the intent. Once

+// the intent is durable no failure returns an ordinary error: the outcome is

+// errLifetimePending (the recovery pass owns it), a persisted terminal state,

+// or the record. The slot is never deleted and never advanced to abandon a

+// create, and every slot write names the exact prior state it replaces.

+// createPlan is what step 2 found: the slot the allocation replaces (nil for

+// none) and the stale bodies step 5 removes. owner* is a stale body under the

+// retired slot's previous owner key, a best-effort removal held in memory

+// only.

+type createPlan struct {

+ slot *provision.HostIncarnation

+ slotETag string

+ staleETag string

+ ownerAccount string

+ ownerApp string

+ ownerStaleETag string

+// createRecordWithLifetime is the one protocol every claims-off creation path

+// admitted on runs (the create, seed and import doors): s.lockApp, then

+// withHostname, then createRecordWithLifetimeLocked. lifetime is the

+// apps.lifetime the door read before taking any lock.

+func (s *Server) createRecordWithLifetime(ctx context.Context, rec provision.Record, lifetime string) (*provision.Record, error) {

+ var out *provision.Record

+ err := s.withAppHost(rec.Account, rec.App, rec.Host, func() error {

+ var err error

+ out, err = s.createRecordWithLifetimeLocked(ctx, rec, lifetime)

+ return err

+ })

+ return out, err

+// createRecordWithLifetimeLocked is the protocol under both locks, which the

+// caller holds (the attach door's form). It returns the record as written,

+// or: release.ErrCASConflict (a valid record under this key, the slot

+// untouched), *hostHeldError, *lifetimeAppError, errLifetimeSettling,

+// errLifetimePending, or a read error with nothing persisted.

+func (s *Server) createRecordWithLifetimeLocked(ctx context.Context, rec provision.Record, lifetime string) (*provision.Record, error) {

+ if len(rec.Maintenance) != 0 {

+ return nil, provision.ErrMaintenanceProjection

+ }

+ ctx = provision.WithLifetimes(ctx, s.judge)

+ host := rec.Host

+ // Step 0: the app's lifecycle, under the app lock.

+ acct, life, err := s.lifetimeAppCheck(ctx, rec.Account, rec.App, lifetime)

+ if err != nil {

+ return nil, err

+ }

+ // Step 1.

+ if err := s.admit(host, ""); err != nil {

+ return nil, err

+ }

+ // Step 2.

+ plan, err := s.planCreateLocked(ctx, acct, rec)

+ if err != nil {

+ return nil, err

+ }

+ // Step 3.

+ last, err := s.store.LastIncarnation(host)

+ if err != nil {

+ return nil, fmt.Errorf("console: lifetimes: last incarnation of %s: %w", host, err)

+ }

+ base := last

+ if plan.slot != nil && plan.slot.Incarnation > base {

+ base = plan.slot.Incarnation

+ }

+ if base >= math.MaxInt64 {

+ return nil, fmt.Errorf("console: lifetimes: %s: %w", host, provision.ErrLifetimeExhausted)

+ }

+ opID, err := newLifetimeOpID()

+ if err != nil {

+ return nil, err

+ }

+ rec.Incarnation, rec.RecordSeq, rec.Retired, rec.IncarnationMigrating = base+1, 0, false, ""

+ body, err := json.Marshal(rec)

+ if err != nil {

+ return nil, err

+ }

+ a := Allocation{OpID: opID, Account: rec.Account, App: rec.App, Host: host, Incarnation: rec.Incarnation,

+ State: lifetimePending, Body: body, PredecessorETag: plan.slotETag, StaleETag: plan.staleETag,

+ AppLifetime: life.Lifetime}

+ if err := s.store.BeginAllocation(a); err != nil {

+ return nil, fmt.Errorf("console: lifetimes: begin allocation of %s: %w", host, err)

+ }

+ // Durable from here: pending, terminal or the record, never an

+ // ordinary error.

+ if err := s.allocSlotLocked(ctx, a); err != nil {

+ return nil, err

+ }

+ if err := s.allocCleanLocked(ctx, a, a.StaleETag, plan); err != nil {

+ return nil, err

+ }

+ return s.allocRecordLocked(ctx, a)

+// lifetimeAppCheck is step 0: the account, then the apps row and the store's

+// trash marker, read under the app lock. lifetime is the apps.lifetime the

+// door admitted the request on; a different one is a purge and a recreate

+// under the same name since.

+func (s *Server) lifetimeAppCheck(ctx context.Context, sqid, app, lifetime string) (*Account, *AppLife, error) {

+ acct, err := s.store.AccountBySqid(sqid)

+ if err != nil {

+ return nil, nil, fmt.Errorf("console: lifetimes: account %s: %w", sqid, err)

+ }

+ if acct == nil {

+ return nil, nil, fmt.Errorf("console: lifetimes: account %s: %w", sqid, os.ErrNotExist)

+ }

+ life, err := s.store.AppLife(acct.ID, app)

+ if err != nil {

+ return nil, nil, &lifetimeAppError{http.StatusBadGateway, msgLifetimeAppUnreadable, err}

+ }

+ if life == nil {

+ return nil, nil, &lifetimeAppError{http.StatusNotFound, "404 page not found", os.ErrNotExist}

+ }

+ marker, err := release.ReadAppTombstone(ctx, s.cfg.Releases, sqid, app)

+ if err != nil {

+ return nil, nil, &lifetimeAppError{http.StatusBadGateway, msgLifetimeAppUnreadable, err}

+ }

+ if marker != nil || life.Trashed() {

+ return nil, nil, &lifetimeAppError{Status: http.StatusConflict, Msg: msgLifetimeAppTrashed}

+ }

+ if life.Lifetime != lifetime {

+ return nil, nil, &lifetimeAppError{Status: http.StatusConflict, Msg: msgLifetimeAppChanged}

+ }

+ return acct, life, nil

+// planCreateLocked is step 2. A valid record under this key is the create's

+// conflict. A live slot whose record key has no valid record is an orphan (a

+// purge deleted the record, or a durable allocation row was lost, which this

+// plan treats as outside its guarantees): it is retired first and the create

+// continues from the retired slot. Admission (step 1) ran under the same

+// hostname lock, so no operation is pending for the hostname here. A live

+// slot owned by another key with a valid record, or a holder hostHolder

+// finds, refuses the create.

+func (s *Server) planCreateLocked(ctx context.Context, acct *Account, rec provision.Record) (createPlan, error) {

+ host := rec.Host

+ for round := 0; ; round++ {

+ obs, err := s.classifyLocked(ctx, rec.Account, rec.App, host)

+ if err != nil {

+ return createPlan{}, err

+ }

+ if validRecord(obs) {

+ return createPlan{}, fmt.Errorf("console: lifetimes: %s: a record of this lifetime exists: %w", host, release.ErrCASConflict)

+ }

+ slot := obs.Slot

+ owned := slotOwnedBy(slot, rec.Account, rec.App)

+ if slot != nil && slot.State == provision.SlotLive {

+ if !owned {

+ other, err := s.classifyLocked(ctx, slot.Account, slot.App, host)

+ if err != nil {

+ return createPlan{}, err

+ }

+ if validRecord(other) {

+ hold := &apexHold{Holder: slot.App}

+ if slot.Account != rec.Account {

+ hold.OtherAccount = slot.Account

+ }

+ return createPlan{}, &hostHeldError{Host: host, Hold: hold}

+ }

+ if round > 0 {

+ // The orphan was retired a moment ago and the slot is live

+ // again: something outside this console wrote it.

+ return createPlan{}, errLifetimeSettling

+ }

+ if err := s.retireSlotOrphanLocked(ctx, host, slot, obs.SlotETag, ""); err != nil {

+ return createPlan{}, err

+ }

+ continue

+ }

+ plan := createPlan{slot: slot, slotETag: obs.SlotETag}

+ if obs.Present {

+ plan.staleETag = obs.ETag // a stale or corrupt body: never empty for a present one

+ }

+ if slot != nil && !owned {

+ prev, err := s.classifyLocked(ctx, slot.Account, slot.App, host)

+ if err != nil {

+ return createPlan{}, err

+ }

+ if prev.Present && prev.Verdict == provision.LifetimeStale {

+ plan.ownerAccount, plan.ownerApp, plan.ownerStaleETag = slot.Account, slot.App, prev.ETag

+ }

+ switch hold, err := s.hostHolder(ctx, acct, host, rec.App); {

+ case err != nil:

+ return createPlan{}, fmt.Errorf("console: lifetimes: host holder of %s: %w", host, err)

+ case hold != nil:

+ return createPlan{}, &hostHeldError{Host: host, Hold: hold}

+ }

+ return plan, nil

+ }

+// allocationIntent is the slot intent of an allocation.

+func allocationIntent(a Allocation) string { return "create " + a.OpID }

+// allocationSlot is the slot an allocation writes.

+func allocationSlot(a Allocation) provision.HostIncarnation {

+ return provision.HostIncarnation{Incarnation: a.Incarnation, State: provision.SlotLive,

+ Account: a.Account, App: a.App, Intent: allocationIntent(a)}

+// allocationOwnsSlot reports whether slot is exactly the one a wrote.

+func allocationOwnsSlot(slot *provision.HostIncarnation, a Allocation) bool {

+ return slot != nil && *slot == allocationSlot(a)

+// allocationSuperseded reports whether slot is a successor of a: a later

+// incarnation, or a's incarnation held by another key or another intent.

+func allocationSuperseded(slot *provision.HostIncarnation, a Allocation) bool {

+ if slot == nil {

+ return false

+ }

+ if slot.Incarnation > a.Incarnation {

+ return true

+ }

+ return slot.Incarnation == a.Incarnation &&

+ (!slotOwnedBy(slot, a.Account, a.App) || slot.Intent != allocationIntent(a))

+// settleAllocation moves a to a terminal state and answers what the caller

+// returns: done when the state is written, errLifetimePending when it could

+// not be (recovery re-derives it).

+func (s *Server) settleAllocation(a Allocation, state string, done error) error {

+ if err := s.store.SetAllocationState(a.OpID, state); err != nil {

+ log.Printf("console: lifetimes: allocation %s of %s: mark %s: %v", a.OpID, a.Host, state, err)

+ return errLifetimePending

+ }

+ return done

+// allocSlotLocked is step 4: the slot, by CAS from the predecessor's ETag.

+func (s *Server) allocSlotLocked(ctx context.Context, a Allocation) error {

+ _, err := provision.WriteHostIncarnation(ctx, s.cfg.Releases, a.Host, allocationSlot(a), a.PredecessorETag)

+ if err == nil {

+ return nil

+ }

+ if !errors.Is(err, release.ErrCASConflict) {

+ log.Printf("console: lifetimes: allocation %s of %s: write the slot: %v (pending)", a.OpID, a.Host, err)

+ return errLifetimePending

+ }

+ slot, _, rerr := provision.ReadHostIncarnation(ctx, s.cfg.Releases, a.Host)

+ switch {

+ case rerr != nil:

+ log.Printf("console: lifetimes: allocation %s of %s: re-read the slot: %v (pending)", a.OpID, a.Host, rerr)

+ return errLifetimePending

+ case allocationOwnsSlot(slot, a):

+ return nil

+ case allocationSuperseded(slot, a):

+ return s.settleAllocation(a, lifetimeSuperseded, errLifetimeSettling)

+ }

+ // The predecessor again, or anything else: leave it to recovery.

+ return errLifetimePending

+// allocCleanLocked is step 5: the stale body under this key, on its ETag,

+// and the best-effort stale body under the retired slot's previous owner.

+// Any failure, a conflict included, leaves the allocation pending.

+func (s *Server) allocCleanLocked(ctx context.Context, a Allocation, staleETag string, plan createPlan) error {

+ if staleETag != "" {

+ if err := release.DeleteIfMatch(ctx, s.cfg.Releases, provision.RecordKey(a.Account, a.App, a.Host), staleETag); err != nil {

+ log.Printf("console: lifetimes: allocation %s of %s: remove the stale record: %v (pending)", a.OpID, a.Host, err)

+ return errLifetimePending

+ }

+ if plan.ownerStaleETag != "" {

+ key := provision.RecordKey(plan.ownerAccount, plan.ownerApp, a.Host)

+ if err := release.DeleteIfMatch(ctx, s.cfg.Releases, key, plan.ownerStaleETag); err != nil {

+ log.Printf("console: lifetimes: allocation %s of %s: remove the stale record under %s/%s: %v (pending)",

+ a.OpID, a.Host, plan.ownerAccount, plan.ownerApp, err)

+ return errLifetimePending

+ }

+ return nil

+// allocRecordLocked is steps 6 and 7: the record create-only, with its

+// record_seq introduced, then the intent committed.

+func (s *Server) allocRecordLocked(ctx context.Context, a Allocation) (*provision.Record, error) {

+ key := provision.RecordKey(a.Account, a.App, a.Host)

+ _, seq, err := provision.WriteRecord(provision.WithRecordSequence(ctx), s.cfg.Releases, key, []byte(`{}`), a.Body, "")

+ if err != nil {

+ if !errors.Is(err, release.ErrCASConflict) {

+ log.Printf("console: lifetimes: allocation %s of %s: write the record: %v (pending)", a.OpID, a.Host, err)

+ return nil, errLifetimePending

+ }

+ landed, _, _, rerr := provision.GetRecordRaw(ctx, s.cfg.Releases, a.Account, a.App, a.Host)

+ if rerr != nil || landed.Retired || landed.Incarnation != a.Incarnation {

+ log.Printf("console: lifetimes: allocation %s of %s: the record write lost to another body (%v; pending)", a.OpID, a.Host, rerr)

+ return nil, errLifetimePending

+ }

+ seq = landed.RecordSeq // a replay of this allocation landed

+ }

+ var rec provision.Record

+ if err := json.Unmarshal(a.Body, &rec); err != nil {

+ log.Printf("console: lifetimes: allocation %s of %s: decode its body: %v (pending)", a.OpID, a.Host, err)

+ return nil, errLifetimePending

+ }

+ rec.RecordSeq = seq

+ if err := s.store.SetAllocationState(a.OpID, lifetimeCommitted); err != nil {

+ log.Printf("console: lifetimes: allocation %s of %s: commit: %v (pending; the record stands)", a.OpID, a.Host, err)

+ return nil, errLifetimePending

+ }

+ return &rec, nil

+// ---- the slot-bound orphan retirement ---------------------------------------

+// slotDisposition is what a slot retirement found.

+type slotDisposition int

+const (

+ slotRetired slotDisposition = iota + 1 // the write landed, or the slot is already {N, retired, this key}

+ slotSuperseded // a later incarnation, or another key

+ slotUnreadable // the write's outcome or the re-read is unknown

+// retireSlotOrphanLocked retires a live slot whose record key has no valid

+// record: a record_retirements row of kind orphan carrying the captured slot

+// ETag (cancelling the allocation named by cancels in the same SQLite

+// transaction, when set), then O1. It assumes only the hostname lock (the

+// lock table). nil: the slot is retired and the row committed;

+// errLifetimeSettling: a successor took the slot (the row is void);

+// errLifetimePending: the row is pending; anything else: nothing durable.

+func (s *Server) retireSlotOrphanLocked(ctx context.Context, host string, slot *provision.HostIncarnation, slotETag, cancels string) error {

+ opID, err := newLifetimeOpID()

+ if err != nil {

+ return err

+ }

+ r := Retirement{OpID: opID, Kind: retireOrphan, Account: slot.Account, App: slot.App, Host: host,

+ Incarnation: slot.Incarnation, SlotETag: slotETag, SlotIntent: slot.Intent, Cancels: cancels, State: lifetimePending}

+ if cancels == "" {

+ err = s.store.BeginRetirement(r)

+ } else {

+ err = s.store.BeginCancellingRetirement(r, cancels)

+ }

+ if err != nil {

+ return fmt.Errorf("console: lifetimes: begin the orphan retirement of %s: %w", host, err)

+ }

+ return s.orphanStepsLocked(ctx, r)

+// orphanStepsLocked is O1: the slot retired by CAS from the captured ETag,

+// then any body under the key (a late write, stale under the retired slot)

+// deleted on its ETag, then the row committed.

+func (s *Server) orphanStepsLocked(ctx context.Context, r Retirement) error {

+ switch disp, err := s.retireSlotLocked(ctx, r); disp {

+ case slotSuperseded:

+ return s.settleRetirement(r, lifetimeVoid, errLifetimeSettling)

+ case slotUnreadable:

+ log.Printf("console: lifetimes: orphan retirement %s of %s: retire the slot: %v (pending)", r.OpID, r.Host, err)

+ return errLifetimePending

+ }

+ if err := s.deleteUnderRetiredSlot(ctx, r, false); err != nil {

+ log.Printf("console: lifetimes: orphan retirement %s of %s: remove the body: %v (pending)", r.OpID, r.Host, err)

+ return errLifetimePending

+ }

+ return s.settleRetirement(r, lifetimeCommitted, nil)

+// settleRetirement is settleAllocation for a retirement row.

+func (s *Server) settleRetirement(r Retirement, state string, done error) error {

+ if err := s.store.SetRetirementState(r.OpID, state); err != nil {

+ log.Printf("console: lifetimes: retirement %s of %s: mark %s: %v", r.OpID, r.Host, state, err)

+ return errLifetimePending

+ }

+ return done

+// retireSlotLocked writes {N, retired, this key, "retire " + OpID} by CAS

+// from the retirement's captured slot ETag; on a conflict the slot is

+// re-read for the disposition.

+func (s *Server) retireSlotLocked(ctx context.Context, r Retirement) (slotDisposition, error) {

+ want := provision.HostIncarnation{Incarnation: r.Incarnation, State: provision.SlotRetired,

+ Account: r.Account, App: r.App, Intent: "retire " + r.OpID}

+ _, err := provision.WriteHostIncarnation(ctx, s.cfg.Releases, r.Host, want, r.SlotETag)

+ if err == nil {

+ return slotRetired, nil

+ }

+ if !errors.Is(err, release.ErrCASConflict) {

+ return slotUnreadable, err

+ }

+ slot, _, rerr := provision.ReadHostIncarnation(ctx, s.cfg.Releases, r.Host)

+ switch {

+ case rerr != nil:

+ return slotUnreadable, rerr

+ case slot == nil:

+ return slotUnreadable, fmt.Errorf("console: lifetimes: the slot of %s is gone", r.Host)

+ case slot.Incarnation > r.Incarnation || !slotOwnedBy(slot, r.Account, r.App):

+ return slotSuperseded, nil

+ case slot.Incarnation == r.Incarnation && slot.State == provision.SlotRetired:

+ return slotRetired, nil

+ }

+ return slotUnreadable, fmt.Errorf("console: lifetimes: the slot of %s is %+v", r.Host, *slot)

+// deleteUnderRetiredSlot removes any body under r's key on its ETag: every

+// body under a retired slot is stale. A conflict means another body landed

+// meanwhile, stale all the same: with again it is re-read and deleted on its

+// ETag once more (a further conflict is an error), without it the conflict

+// is logged and left.

+func (s *Server) deleteUnderRetiredSlot(ctx context.Context, r Retirement, again bool) error {

+ key := provision.RecordKey(r.Account, r.App, r.Host)

+ for round := 0; ; round++ {

+ _, etag, err := s.cfg.Releases.Get(ctx, key)

+ if errors.Is(err, os.ErrNotExist) {

+ return nil

+ }

+ if err != nil {

+ return err

+ }

+ err = release.DeleteIfMatch(ctx, s.cfg.Releases, key, etag)

+ switch {

+ case err == nil, !errors.Is(err, release.ErrCASConflict):

+ return err

+ case !again:

+ log.Printf("console: lifetimes: retirement %s of %s: a body landed under the retired slot; it is stale and the next create of the hostname removes it: %v", r.OpID, r.Host, err)

+ return nil

+ case round > 0:

+ return err

+ }

+// ---- the fence ---------------------------------------------------------------

+// fenceOwner is what this operation's own outstanding write would look like:

+// its tombstone, or a record marked with MarkerID.

+type fenceOwner struct {

+ Tombstone bool

+ MarkerID string

+// fenceResult is fenceRecord's answer.

+type fenceResult int

+const (

+ // fenced: a body that is neither this operation's tombstone nor its

+ // marker was rewritten with its sequence introduced or advanced, so no

+ // outstanding conditional write against an earlier ETag can land. The

+ // caller terminates on the observation.

+ fenced fenceResult = iota + 1

+ // ownWriteWon: the re-read found this operation's own write. The caller

+ // does not terminate; it dispatches again from the observation.

+ ownWriteWon

+ // fenceAbsent: no body to fence; the slot is the fence for an absent key.

+ fenceAbsent

+// fenceRounds bounds the fence's CAS retries.

+const fenceRounds = 3

+// lifetimesFenceDisabled turns fenceRecord's write off. A TEST SEAM, and only

+// one: it shows the defect the fence closes.

+var lifetimesFenceDisabled bool

+// fenceRecord makes every outstanding write against an earlier ETag of

+// acct/app/host's record harmless before a recovery branch terminates an

+// operation on evidence that the record is not what it captured: the current

+// body is rewritten against its own ETag with the record sequence on. The

+// caller holds the hostname lock. An error (an unreadable observation, a

+// body the write primitive refuses, or fenceRounds lost CAS rounds) leaves

+// the caller's operation pending.

+func (s *Server) fenceRecord(ctx context.Context, acct, app, host string, op fenceOwner) (fenceResult, provision.Observation, error) {

+ ctx = provision.WithRecordSequence(ctx)

+ key := provision.RecordKey(acct, app, host)

+ var obs provision.Observation

+ for round := 0; round < fenceRounds; round++ {

+ var err error

+ obs, err = s.classifyLocked(ctx, acct, app, host)

+ if err != nil {

+ return 0, obs, err

+ }

+ switch {

+ case !obs.Present:

+ return fenceAbsent, obs, nil

+ case obs.Rec != nil && op.Tombstone && obs.Rec.Retired:

+ return ownWriteWon, obs, nil

+ case obs.Rec != nil && op.MarkerID != "" && obs.Rec.IncarnationMigrating == op.MarkerID:

+ return ownWriteWon, obs, nil

+ }

+ if lifetimesFenceDisabled {

+ return fenced, obs, nil

+ }

+ etag, _, err := provision.WriteRecord(ctx, s.cfg.Releases, key, obs.Raw, obs.Raw, obs.ETag)

+ if err == nil {

+ obs.ETag = etag

+ return fenced, obs, nil

+ }

+ if !errors.Is(err, release.ErrCASConflict) {

+ return 0, obs, fmt.Errorf("console: lifetimes: fence %s: %w", key, err)

+ }

+ return 0, obs, fmt.Errorf("console: lifetimes: fence %s: lost %d CAS rounds", key, fenceRounds)

+// terminateFenced is how a recovery branch ends an operation on record

+// evidence: the record is fenced first and terminate runs only once it is.

+// ownWriteWon and fenceAbsent hand the observation back (redispatch true):

+// the caller dispatches again from it and never terminates on its earlier

+// classification. A fence or terminate that fails is errLifetimePending.

+func (s *Server) terminateFenced(ctx context.Context, acct, app, host string, op fenceOwner, terminate func() error) (obs provision.Observation, redispatch bool, err error) {

+ res, obs, err := s.fenceRecord(ctx, acct, app, host, op)

+ if err != nil {

+ log.Printf("console: lifetimes: %s/%s/%s: %v (pending)", acct, app, host, err)

+ return obs, false, errLifetimePending

+ }

+ if res != fenced {

+ return obs, true, nil

+ }

+ if err := terminate(); err != nil {

+ log.Printf("console: lifetimes: %s/%s/%s: terminate after the fence: %v (pending)", acct, app, host, err)

+ return obs, false, errLifetimePending

+ }

+ return obs, false, nil

diff --git a/internal/console/lifetimes_create_test.go b/internal/console/lifetimes_create_test.go

new file mode 100644

index 000000000..a0b04616c

--- /dev/null

+++ b/internal/console/lifetimes_create_test.go

@@ -0,0 +1,2355 @@

+package console

+import (

+ "bytes"

+ "context"

+ "encoding/json"

+ "errors"

+ "fmt"

+ "math"

+ "net/http"

+ "net/url"

+ "os"

+ "runtime"

+ "strings"

+ "sync"

+ "sync/atomic"

+ "testing"

+ "time"

+ "github.com/carlosframework/platform/internal/domains"

+ "github.com/carlosframework/platform/internal/provision"

+ "github.com/carlosframework/platform/internal/release"

+ "github.com/carlosframework/platform/internal/seeds"

+// The create protocol and its recovery (lifetimes_create.go,

+// lifetimes_recovery.go), driven through the real server, store and doors

+// of a claims-off console with lifetimes on, over a bucket that can fail,

+// pause or record any call.

+// ---- fixtures -------------------------------------------------------------

+// ltStore wraps the fixture's bucket: every call is recorded, and hook (when

+// set) runs before it; a non-nil error fails the call without running it.

+// after runs once a call returns.

+type ltStore struct {

+ release.Store

+ mu sync.Mutex

+ ops []string

+ hook func(op, key string) error

+ after func(op, key string, err error)

+func (s *ltStore) do(op, key string) error {

+ s.mu.Lock()

+ s.ops = append(s.ops, op+" "+key)

+ h := s.hook

+ s.mu.Unlock()

+ if h != nil {

+ return h(op, key)

+ }

+ return nil

+func (s *ltStore) done(op, key string, err error) {

+ s.mu.Lock()

+ a := s.after

+ s.mu.Unlock()

+ if a != nil {

+ a(op, key, err)

+ }

+func (s *ltStore) setHook(h func(op, key string) error) {

+ s.mu.Lock()

+ s.hook = h

+ s.mu.Unlock()

+func (s *ltStore) setAfter(a func(op, key string, err error)) {

+ s.mu.Lock()

+ s.after = a

+ s.mu.Unlock()

+func (s *ltStore) Get(ctx context.Context, key string) ([]byte, string, error) {

+ if err := s.do("get", key); err != nil {

+ return nil, "", err

+ }

+ b, e, err := s.Store.Get(ctx, key)

+ s.done("get", key, err)

+ return b, e, err

+func (s *ltStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {

+ if err := s.do("put", key); err != nil {

+ return "", err

+ }

+ e, err := s.Store.Put(ctx, key, body, ifMatch)

+ s.done("put", key, err)

+ return e, err

+func (s *ltStore) Delete(ctx context.Context, key string) error {

+ if err := s.do("delete", key); err != nil {

+ return err

+ }

+ err := s.Store.Delete(ctx, key)

+ s.done("delete", key, err)

+ return err

+func (s *ltStore) DeleteIfMatch(ctx context.Context, key, etag string) error {

+ if err := s.do("delete-if-match", key); err != nil {

+ return err

+ }

+ err := release.DeleteIfMatch(ctx, s.Store, key, etag)

+ s.done("delete-if-match", key, err)

+ return err

+// opsOf is every recorded call on key, in order.

+func (s *ltStore) opsOf(key string) []string {

+ s.mu.Lock()

+ defer s.mu.Unlock()

+ var out []string

+ for _, op := range s.ops {

+ if strings.HasSuffix(op, " "+key) {

+ out = append(out, strings.TrimSuffix(op, " "+key))

+ }

+ return out

+// index is where op first appears in the log, or -1.

+func (s *ltStore) index(op string) int {

+ s.mu.Lock()

+ defer s.mu.Unlock()

+ for i, o := range s.ops {

+ if o == op {

+ return i

+ }

+ return -1

+var errBucketDown = errors.New("the bucket did not answer")

+// failNth fails the nth (1-based) call of op on key, every later one too

+// when sticky.

+func failNth(op, key string, nth int32, sticky bool) func(string, string) error {

+ var n atomic.Int32

+ return func(o, k string) error {

+ if o != op || k != key {

+ return nil

+ }

+ got := n.Add(1)

+ if got == nth || (sticky && got > nth) {

+ return errBucketDown

+ }

+ return nil

+ }

+// pauseNth holds the nth call of op on key until released.

+type pauseGate struct {

+ entered chan struct{}

+ release chan struct{}

+ once sync.Once

+func newPauseGate() *pauseGate {

+ return &pauseGate{entered: make(chan struct{}), release: make(chan struct{})}

+func (p *pauseGate) open() { p.once.Do(func() { close(p.release) }) }

+func pauseNth(t *testing.T, op, key string, nth int32) (*pauseGate, func(string, string) error) {

+ t.Helper()

+ p := newPauseGate()

+ t.Cleanup(p.open)

+ var n atomic.Int32

+ return p, func(o, k string) error {

+ if o != op || k != key || n.Add(1) != nth {

+ return nil

+ }

+ close(p.entered)

+ <-p.release

+ return nil

+ }

+// chain runs hooks in order; the first error wins.

+func chain(hooks ...func(string, string) error) func(string, string) error {

+ return func(op, key string) error {

+ for _, h := range hooks {

+ if err := h(op, key); err != nil {

+ return err

+ }

+ return nil

+ }

+// ltOn is a claims-off console with lifetimes on: account bab, app shop on

+// shop.example.com, a member's token, and its bucket behind an ltStore.

+type ltOn struct {

+ lifetimesFx

+ bucket *ltStore

+func newLtOn(t *testing.T) ltOn {

+ t.Helper()

+ l := newLifetimesFx(t)

+ bucket := &ltStore{Store: l.fx.rel}

+ l.fx.srv.cfg.Releases = bucket

+ turnLifetimesOn(t, l.fx.srv, l.clock)

+ return ltOn{lifetimesFx: l, bucket: bucket}

+// turnLifetimesOn writes the cutover marker and lets the settle pass.

+func turnLifetimesOn(t *testing.T, srv *Server, clock *lifetimesClock) {

+ t.Helper()

+ ctx := context.Background()

+ body, _ := json.Marshal(cutoverMarker{At: clock.Now(), By: "paul@keymail.dev"})

+ if _, err := srv.cfg.Releases.Put(ctx, lifetimesCutoverKey, body, ""); err != nil && !errors.Is(err, release.ErrCASConflict) {

+ t.Fatal(err)

+ }

+ srv.lifetimesWatchOnce(ctx)

+ clock.Advance(lifetimeSettle)

+ srv.lifetimesWatchOnce(ctx)

+ if !srv.lifetimesOn() {

+ t.Fatalf("lifetimes are %s, want on", srv.cutover.status().State)

+ }

+// restart is a second console process over the same SQLite and bucket: it

+// shares no memory with the first (no locks, no in-flight state) and turns

+// lifetimes on from its own start.

+func (l ltOn) restart(t *testing.T) *Server {

+ t.Helper()

+ srv, err := NewServer(l.fx.srv.cfg, l.fx.st)

+ if err != nil {

+ t.Fatal(err)

+ }

+ ctx := context.Background()

+ srv.lifetimesWatchOnce(ctx)

+ l.clock.Advance(lifetimeSettle)

+ srv.lifetimesWatchOnce(ctx)

+ if !srv.lifetimesOn() {

+ t.Fatalf("the restarted console is %s, want on", srv.cutover.status().State)

+ }

+ return srv

+// recover runs one recovery pass on srv as its worker does.

+func recoverOn(t *testing.T, srv *Server) {

+ t.Helper()

+ if !srv.lifetimeRecoveryIteration(context.Background()) {

+ t.Fatal("the recovery pass did not run with lifetimes on")

+ }

+// onCtx is a unit of record work admitted on srv, for the cores the doors

+// call.

+func onCtx(t *testing.T, srv *Server) context.Context {

+ t.Helper()

+ u, ok, why := srv.lifetimesAdmit(context.Background(), true)

+ if !ok || u.mode != cutoverOn {

+ t.Fatalf("admitted %v %q, want a unit on", ok, why)

+ }

+ t.Cleanup(u.done)

+ return context.WithValue(srv.recordCtx(context.Background(), u), admittedUnitKey{}, u)

+func (l ltOn) create(t *testing.T, host string) (int, string) {

+ t.Helper()

+ code, _, body := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", ltApp), l.member, `{"host":"`+host+`"}`)

+ return code, body

+func recKey(app, host string) string { return provision.RecordKey("bab", app, host) }

+func slotOf(t *testing.T, st release.Store, host string) (*provision.HostIncarnation, string) {

+ t.Helper()

+ slot, etag, err := provision.ReadHostIncarnation(context.Background(), st, host)

+ if err != nil {

+ t.Fatalf("slot of %s: %v", host, err)

+ }

+ return slot, etag

+// rawOf is the record body under bab/app/host and its ETag; nil when absent.

+func rawOf(t *testing.T, st release.Store, app, host string) ([]byte, string) {

+ t.Helper()

+ b, e, err := st.Get(context.Background(), recKey(app, host))

+ if errors.Is(err, os.ErrNotExist) {

+ return nil, ""

+ }

+ if err != nil {

+ t.Fatal(err)

+ }

+ return b, e

+func decodeRec(t *testing.T, raw []byte) provision.Record {

+ t.Helper()

+ rec, err := provision.DecodeRecord(raw)

+ if err != nil {

+ t.Fatalf("decode %s: %v", raw, err)

+ }

+ return rec

+type allocRow struct {

+ OpID, App, State string

+ Incarnation int64

+ Pred, Stale string

+func allocRows(t *testing.T, st *Store, host string) []allocRow {

+ t.Helper()

+ rows, err := st.db.Query(`SELECT op_id, app, state, incarnation, predecessor_etag, stale_etag FROM host_allocations WHERE host = ? ORDER BY rowid`, host)

+ if err != nil {

+ t.Fatal(err)

+ }

+ defer rows.Close()

+ var out []allocRow

+ for rows.Next() {

+ var a allocRow

+ if err := rows.Scan(&a.OpID, &a.App, &a.State, &a.Incarnation, &a.Pred, &a.Stale); err != nil {

+ t.Fatal(err)

+ }

+ out = append(out, a)

+ }

+ return out

+type retireRow struct {

+ OpID, Kind, App, State, Cancels, SlotETag string

+ Incarnation int64

+func retireRows(t *testing.T, st *Store, host string) []retireRow {

+ t.Helper()

+ rows, err := st.db.Query(`SELECT op_id, kind, app, state, cancels, slot_etag, incarnation FROM record_retirements WHERE host = ? ORDER BY rowid`, host)

+ if err != nil {

+ t.Fatal(err)

+ }

+ defer rows.Close()

+ var out []retireRow

+ for rows.Next() {

+ var r retireRow

+ if err := rows.Scan(&r.OpID, &r.Kind, &r.App, &r.State, &r.Cancels, &r.SlotETag, &r.Incarnation); err != nil {

+ t.Fatal(err)

+ }

+ out = append(out, r)

+ }

+ return out

+// oneAlloc is the host's only allocation row, required in state.

+func oneAlloc(t *testing.T, st *Store, host, state string) allocRow {

+ t.Helper()

+ rows := allocRows(t, st, host)

+ if len(rows) != 1 || rows[0].State != state {

+ t.Fatalf("allocations of %s = %+v, want one %s", host, rows, state)

+ }

+ return rows[0]

+// requireServing: a record of shop at host at incarnation n, record_seq

+// present, under the live slot {n, this key, "create " + op}.

+func requireServing(t *testing.T, st release.Store, app, host string, n uint64, op string) provision.Record {

+ t.Helper()

+ raw, _ := rawOf(t, st, app, host)

+ if raw == nil {

+ t.Fatalf("no record of %s/%s", app, host)

+ }

+ rec := decodeRec(t, raw)

+ if rec.Incarnation != n || rec.RecordSeq == 0 || rec.Retired {

+ t.Fatalf("record = %s, want incarnation %d with a record_seq", raw, n)

+ }

+ slot, _ := slotOf(t, st, host)

+ want := provision.HostIncarnation{Incarnation: n, State: provision.SlotLive, Account: "bab", App: app, Intent: "create " + op}

+ if slot == nil || (op != "" && *slot != want) || slot.Incarnation != n || slot.State != provision.SlotLive || slot.App != app {

+ t.Fatalf("slot = %+v, want %+v", slot, want)

+ }

+ return rec

+// sqlFail makes every move of host_allocations to committed fail until the

+// returned func drops the trigger.

+func sqlFailCommit(t *testing.T, st *Store) func() {

+ t.Helper()

+ if _, err := st.db.Exec(`CREATE TRIGGER lt_fail_commit BEFORE UPDATE OF state ON host_allocations WHEN NEW.state = 'committed'

+BEGIN SELECT RAISE(ABORT, 'test: the commit is refused'); END`); err != nil {

+ t.Fatal(err)

+ }

+ var once sync.Once

+ drop := func() {

+ once.Do(func() {

+ if _, err := st.db.Exec(`DROP TRIGGER lt_fail_commit`); err != nil {

+ t.Fatal(err)

+ }

+ })

+ }

+ return drop

+// legacyBody writes a fieldless record of app at host straight to the

+// bucket, as main's create does.

+func legacyBody(t *testing.T, st release.Store, app, host, channel string) string {

+ t.Helper()

+ rec := provision.Record{Schema: 1, Host: host, Slug: strings.SplitN(host, ".", 2)[0], App: app,

+ Account: "bab", Channel: channel, CreatedAt: time.Now().UTC()}

+ body, _ := json.Marshal(rec)

+ etag, err := st.Put(context.Background(), recKey(app, host), body, "")

+ if err != nil {

+ t.Fatal(err)

+ }

+ return etag

+func writeSlot(t *testing.T, st release.Store, host string, slot provision.HostIncarnation) string {

+ t.Helper()

+ _, cur, err := provision.ReadHostIncarnation(context.Background(), st, host)

+ if err != nil {

+ t.Fatal(err)

+ }

+ etag, err := provision.WriteHostIncarnation(context.Background(), st, host, slot, cur)

+ if err != nil {

+ t.Fatal(err)

+ }

+ return etag

+// ---- the create -------------------------------------------------------------

+func TestCreateAllocatesSlotThenRecord(t *testing.T) {

+ l := newLtOn(t)

+ code, body := l.create(t, ltOne)

+ if code != http.StatusCreated {

+ t.Fatalf("create = %d %s", code, body)

+ }

+ a := oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ if a.Incarnation != 1 || a.Pred != "" || a.Stale != "" {

+ t.Fatalf("allocation = %+v, want incarnation 1 with no predecessor", a)

+ }

+ rec := requireServing(t, l.fx.rel, ltApp, ltOne, 1, a.OpID)

+ if rec.RecordSeq != 1 {

+ t.Fatalf("record_seq = %d, want 1", rec.RecordSeq)

+ }

+ slotPut := l.bucket.index("put " + provision.HostIncarnationKey(ltOne))

+ recPut := l.bucket.index("put " + recKey(ltApp, ltOne))

+ if slotPut < 0 || recPut < 0 || slotPut > recPut {

+ t.Fatalf("slot put at %d, record put at %d: want the slot first", slotPut, recPut)

+ }

+ var out map[string]any

+ _ = json.Unmarshal([]byte(body), &out)

+ if out["incarnation"] != float64(1) || out["record_seq"] != float64(1) {

+ t.Fatalf("201 body = %s, want the record as written", body)

+ }

+func TestCreatedRecordsCarryARecordSeq(t *testing.T) {

+ l := newLtOn(t)

+ for i, host := range []string{ltOne, ltOld} {

+ if code, body := l.create(t, host); code != http.StatusCreated {

+ t.Fatalf("create %s = %d %s", host, code, body)

+ }

+ raw, _ := rawOf(t, l.fx.rel, ltApp, host)

+ if !bytes.Contains(raw, []byte(`"record_seq":1`)) || !bytes.Contains(raw, []byte(`"incarnation":1`)) {

+ t.Fatalf("record %d = %s, want incarnation 1 and record_seq 1", i, raw)

+ }

+func TestEveryClaimsOffWriteAdvancesTheSequence(t *testing.T) {

+ l := newLtOn(t)

+ legacyBody(t, l.fx.rel, ltApp, ltOne, "stable")

+ _, e0 := rawOf(t, l.fx.rel, ltApp, ltOne)

+ ctx := onCtx(t, l.fx.srv)

+ etags := []string{e0}

+ for _, region := range []string{"uk", "", "uk"} {

+ if err := provision.SetRecordRegion(ctx, l.fx.srv.cfg.Releases, "bab", ltApp, ltOne, region); err != nil {

+ t.Fatal(err)

+ }

+ _, e := rawOf(t, l.fx.rel, ltApp, ltOne)

+ etags = append(etags, e)

+ }

+ raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne)

+ if rec := decodeRec(t, raw); rec.RecordSeq != 3 {

+ t.Fatalf("record_seq = %d after three judged writes of a legacy record, want 3 (introduced, then advanced)", rec.RecordSeq)

+ }

+ seen := map[string]bool{}

+ for _, e := range etags {

+ if seen[e] {

+ t.Fatalf("etags %v: the region A→B→A writes repeat an ETag", etags)

+ }

+ seen[e] = true

+ }

+func TestCreateAfterRetirementGetsNextIncarnation(t *testing.T) {

+ l := newLtOn(t)

+ pred := writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 1, State: provision.SlotRetired, Account: "bab", App: ltApp, Intent: "retire x"})

+ if code, body := l.create(t, ltOne); code != http.StatusCreated {

+ t.Fatalf("create = %d %s", code, body)

+ }

+ a := oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ if a.Incarnation != 2 || a.Pred != pred {

+ t.Fatalf("allocation = %+v, want incarnation 2 replacing the retired slot %s", a, pred)

+ }

+ requireServing(t, l.fx.rel, ltApp, ltOne, 2, a.OpID)

+ // The console's own counter wins over a lower slot: an earlier

+ // allocation of the hostname at 5 makes the next one 6.

+ if err := l.fx.st.BeginAllocation(Allocation{OpID: "earlier", Account: "bab", App: ltApp, Host: ltOld, Incarnation: 5, Body: []byte(`{}`)}); err != nil {

+ t.Fatal(err)

+ }

+ if err := l.fx.st.SetAllocationState("earlier", lifetimeCommitted); err != nil {

+ t.Fatal(err)

+ }

+ writeSlot(t, l.fx.rel, ltOld, provision.HostIncarnation{Incarnation: 1, State: provision.SlotRetired, Account: "bab", App: ltApp})

+ if code, body := l.create(t, ltOld); code != http.StatusCreated {

+ t.Fatalf("create %s = %d %s", ltOld, code, body)

+ }

+ requireServing(t, l.fx.rel, ltApp, ltOld, 6, "")

+func TestCreateReplacesStaleRecord(t *testing.T) {

+ l := newLtOn(t)

+ retiredSlot(t, l.fx.rel, ltOne)

+ stale := legacyBody(t, l.fx.rel, ltApp, ltOne, "edge") // an old console's late create

+ if code, body := l.create(t, ltOne); code != http.StatusCreated {

+ t.Fatalf("create = %d %s", code, body)

+ }

+ a := oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ if a.Stale != stale {

+ t.Fatalf("allocation stale etag = %q, want the legacy body's %q", a.Stale, stale)

+ }

+ rec := requireServing(t, l.fx.rel, ltApp, ltOne, 2, a.OpID)

+ if rec.Channel != "stable" {

+ t.Fatalf("record = %+v, want the new create's, not the legacy body", rec)

+ }

+ if ops := l.bucket.opsOf(recKey(ltApp, ltOne)); strings.Join(ops, ",") == "" || !contains(ops, "delete-if-match") {

+ t.Fatalf("record ops = %v, want the stale body deleted on its ETag", ops)

+ }

+func contains(xs []string, x string) bool {

+ for _, y := range xs {

+ if y == x {

+ return true

+ }

+ return false

+func TestCreateOfALiveInstanceIsTheExistingConflict(t *testing.T) {

+ l := newLtOn(t)

+ if code, body := l.create(t, ltOne); code != http.StatusCreated {

+ t.Fatalf("create = %d %s", code, body)

+ }

+ _, slotETag := slotOf(t, l.fx.rel, ltOne)

+ _, recETag := rawOf(t, l.fx.rel, ltApp, ltOne)

+ code, body := l.create(t, ltOne)

+ if code != http.StatusConflict || ltErr(body) != "instance "+ltOne+" already exists" {

+ t.Fatalf("second create = %d %s, want the existing 409", code, body)

+ }

+ if _, e := slotOf(t, l.fx.rel, ltOne); e != slotETag {

+ t.Fatal("the refused create touched the slot")

+ }

+ if _, e := rawOf(t, l.fx.rel, ltApp, ltOne); e != recETag {

+ t.Fatal("the refused create touched the record")

+ }

+ if rows := allocRows(t, l.fx.st, ltOne); len(rows) != 1 {

+ t.Fatalf("allocations = %+v, want only the first", rows)

+ }

+func TestCreateRefusesWhileAnOperationIsPending(t *testing.T) {

+ for name, begin := range map[string]func(st *Store) error{

+ "an allocation": func(st *Store) error {

+ return st.BeginAllocation(Allocation{OpID: "other", Account: "bab", App: ltApp, Host: ltOne, Incarnation: 1, Body: []byte(`{}`)})

+ },

+ "a retirement": func(st *Store) error {

+ return st.BeginRetirement(Retirement{OpID: "other", Kind: retireOrphan, Account: "bab", App: ltApp, Host: ltOne, Incarnation: 1})

+ },

+ } {

+ t.Run(name, func(t *testing.T) {

+ l := newLtOn(t)

+ if err := begin(l.fx.st); err != nil {

+ t.Fatal(err)

+ }

+ before := cutoverBucket(t, l.fx.rel)

+ code, body := l.create(t, ltOne)

+ if code != http.StatusConflict || ltErr(body) != msgLifetimeSettling {

+ t.Fatalf("create = %d %s, want 409 settling", code, body)

+ }

+ if after := cutoverBucket(t, l.fx.rel); len(after) != len(before) {

+ t.Fatalf("the refused create wrote to the bucket: %v", after)

+ }

+ })

+ }

+func TestCreateRefusesALiveSlotOfAnotherApp(t *testing.T) {

+ ctx := context.Background()

+ t.Run("another app of the account", func(t *testing.T) {

+ l := newLtOn(t)

+ writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 1, State: provision.SlotLive, Account: "bab", App: "web", Intent: "create x"})

+ body, _ := json.Marshal(provision.Record{Schema: 1, Host: ltOne, App: "web", Account: "bab", Channel: "stable", Incarnation: 1, CreatedAt: time.Now().UTC()})

+ if _, err := l.fx.rel.Put(ctx, recKey("web", ltOne), body, ""); err != nil {

+ t.Fatal(err)

+ }

+ _, slotETag := slotOf(t, l.fx.rel, ltOne)

+ // The door's own pre-lock check does not list web (no apps row);

+ // the slot is what refuses it.

+ code, out := l.create(t, ltOne)

+ if code != http.StatusConflict || ltErr(out) != hostHeldSentence(ltOne, &apexHold{Holder: "web"}) {

+ t.Fatalf("create = %d %s, want 409 held by web", code, out)

+ }

+ if _, e := slotOf(t, l.fx.rel, ltOne); e != slotETag || len(allocRows(t, l.fx.st, ltOne)) != 0 {

+ t.Fatal("the refused create wrote the slot or an allocation")

+ }

+ })

+ t.Run("an app of another account", func(t *testing.T) {

+ l := newLtOn(t)

+ writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 3, State: provision.SlotLive, Account: "acme", App: "web", Intent: "create x"})

+ body, _ := json.Marshal(provision.Record{Schema: 1, Host: ltOne, App: "web", Account: "acme", Channel: "stable", Incarnation: 3, CreatedAt: time.Now().UTC()})

+ if _, err := l.fx.rel.Put(ctx, provision.RecordKey("acme", "web", ltOne), body, ""); err != nil {

+ t.Fatal(err)

+ }

+ code, out := l.create(t, ltOne)

+ if code != http.StatusConflict || ltErr(out) != hostHeldSentence(ltOne, &apexHold{Holder: "web", OtherAccount: "acme"}) {

+ t.Fatalf("create = %d %s, want 409 routed for acme", code, out)

+ }

+ })

+func TestCreateReclaimsAnOrphanSlot(t *testing.T) {

+ ctx := context.Background()

+ l := newLtOn(t)

+ // A live slot of web with no record of web (a lost allocation row), and

+ // a stale body under shop from shop's earlier, retired lifetime.

+ orphan := writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 4, State: provision.SlotLive, Account: "bab", App: "web", Intent: "create lost"})

+ code, body := l.create(t, ltOne)

+ if code != http.StatusCreated {

+ t.Fatalf("create = %d %s", code, body)

+ }

+ rets := retireRows(t, l.fx.st, ltOne)

+ if len(rets) != 1 || rets[0].Kind != retireOrphan || rets[0].State != lifetimeCommitted || rets[0].SlotETag != orphan || rets[0].App != "web" {

+ t.Fatalf("retirements = %+v, want one committed orphan of web's slot", rets)

+ }

+ a := oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOne, 5, a.OpID)

+ // A stale body under the retired slot's previous owner is removed by the

+ // next create under another key.

+ m := newLtOn(t)

+ writeSlot(t, m.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 2, State: provision.SlotRetired, Account: "bab", App: "web", Intent: "retire y"})

+ webBody, _ := json.Marshal(provision.Record{Schema: 1, Host: ltOne, App: "web", Account: "bab", Channel: "stable", Incarnation: 2, CreatedAt: time.Now().UTC()})

+ if _, err := m.fx.rel.Put(ctx, recKey("web", ltOne), webBody, ""); err != nil {

+ t.Fatal(err)

+ }

+ if code, body := m.create(t, ltOne); code != http.StatusCreated {

+ t.Fatalf("create over a stale body of web = %d %s", code, body)

+ }

+ if raw, _ := rawOf(t, m.fx.rel, "web", ltOne); raw != nil {

+ t.Fatalf("web's stale body survived the create: %s", raw)

+ }

+ requireServing(t, m.fx.rel, ltApp, ltOne, 3, "")

+func TestCreateRefusesWhenTheAccountIsUnknown(t *testing.T) {

+ l := newLtOn(t)

+ before := cutoverBucket(t, l.fx.rel)

+ rec := provision.Record{Schema: 1, Host: ltOne, App: ltApp, Account: "nobody", Channel: "stable", CreatedAt: time.Now().UTC()}

+ _, err := l.fx.srv.createRecordWithLifetime(onCtx(t, l.fx.srv), rec, "")

+ if err == nil || errors.Is(err, release.ErrCASConflict) {

+ t.Fatalf("create for an unknown account = %v, want a read error", err)

+ }

+ if _, ok := lifetimeCreateRefusal(ltOne, err, msgLifetimeCreatePending); ok {

+ t.Fatalf("an unknown account is %v, want an ordinary read error (the door's 502)", err)

+ }

+ if after := cutoverBucket(t, l.fx.rel); len(after) != len(before) || len(allocRows(t, l.fx.st, ltOne)) != 0 {

+ t.Fatal("the refused create persisted something")

+ }

+func TestTwoAppsCreatingOneHostnameSerialise(t *testing.T) {

+ l := newLtOn(t)

+ if _, err := l.fx.st.CreateAppIn(OpsAccountID, "shop2", "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ enableInstances(t, l.fx, "shop2", provision.Manifest{Schema: 1, App: "shop2", Domain: "shop.example.com", Channel: "stable"})

+ gate, hook := pauseNth(t, "put", provision.HostIncarnationKey(ltOne), 1)

+ l.bucket.setHook(hook)

+ var waiting atomic.Int32

+ h := func(kind, key, event string) {

+ if kind == "host" && key == ltOne && event == lockWaiting {

+ waiting.Add(1)

+ }

+ lifetimeLockHook.Store(&h)

+ t.Cleanup(func() { lifetimeLockHook.Store(nil) })

+ first := make(chan int, 1)

+ go func() { code, _ := l.create(t, ltOne); first <- code }()

+ waitFor(t, gate.entered, "shop's slot write")

+ second := make(chan string, 1)

+ go func() {

+ code, _, body := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", "shop2"), l.member, `{"host":"`+ltOne+`"}`)

+ second <- fmt.Sprintf("%d %s", code, ltErr(body))

+ }()

+ deadline := time.Now().Add(10 * time.Second)

+ for waiting.Load() < 2 && time.Now().Before(deadline) {

+ time.Sleep(5 * time.Millisecond)

+ }

+ if waiting.Load() < 2 {

+ t.Fatal("shop2's create never waited on the hostname")

+ }

+ select {

+ case got := <-second:

+ t.Fatalf("shop2's create answered %s while shop held the hostname", got)

+ case <-time.After(50 * time.Millisecond):

+ }

+ gate.open()

+ if code := <-first; code != http.StatusCreated {

+ t.Fatalf("shop's create = %d", code)

+ }

+ if got, want := <-second, fmt.Sprintf("%d %s", http.StatusConflict, hostHeldSentence(ltOne, &apexHold{Holder: ltApp})); got != want {

+ t.Fatalf("shop2's create = %q, want %q", got, want)

+ }

+ if rows := allocRows(t, l.fx.st, ltOne); len(rows) != 1 || rows[0].App != ltApp {

+ t.Fatalf("allocations = %+v, want only shop's", rows)

+ }

+func TestExhaustionIsRefusedBeforeAnyWrite(t *testing.T) {

+ ctx := context.Background()

+ t.Run("an allocation at the top", func(t *testing.T) {

+ l := newLtOn(t)

+ if err := l.fx.st.BeginAllocation(Allocation{OpID: "top", Account: "bab", App: ltApp, Host: ltOne, Incarnation: math.MaxInt64, Body: []byte(`{}`)}); err != nil {

+ t.Fatal(err)

+ }

+ if err := l.fx.st.SetAllocationState("top", lifetimeVoid); err != nil {

+ t.Fatal(err)

+ }

+ // void does not count: a committed one does.

+ if err := l.fx.st.BeginAllocation(Allocation{OpID: "top2", Account: "bab", App: ltApp, Host: ltOne, Incarnation: math.MaxInt64, Body: []byte(`{}`)}); err != nil {

+ t.Fatal(err)

+ }

+ if err := l.fx.st.SetAllocationState("top2", lifetimeCommitted); err != nil {

+ t.Fatal(err)

+ }

+ rec := provision.Record{Schema: 1, Host: ltOne, App: ltApp, Account: "bab", Channel: "stable", CreatedAt: time.Now().UTC()}

+ life, _ := l.fx.st.AppLife(OpsAccountID, ltApp)

+ _, err := l.fx.srv.createRecordWithLifetime(onCtx(t, l.fx.srv), rec, life.Lifetime)

+ if !errors.Is(err, provision.ErrLifetimeExhausted) {

+ t.Fatalf("create = %v, want ErrLifetimeExhausted", err)

+ }

+ if rows := allocRows(t, l.fx.st, ltOne); len(rows) != 2 {

+ t.Fatalf("allocations = %+v, want no new row", rows)

+ }

+ if slot, _ := slotOf(t, l.fx.rel, ltOne); slot != nil {

+ t.Fatal("the refused create wrote a slot")

+ }

+ if code, _ := l.create(t, ltOne); code != http.StatusBadGateway {

+ t.Fatalf("the door = %d, want 502", code)

+ }

+ })

+ t.Run("a slot above the counter", func(t *testing.T) {

+ l := newLtOn(t)

+ if _, err := l.fx.rel.Put(ctx, provision.HostIncarnationKey(ltOne),

+ []byte(`{"incarnation":9223372036854775808,"state":"retired","account":"bab","app":"shop"}`), ""); err != nil {

+ t.Fatal(err)

+ }

+ // The door's pre-lock holder check meets the unreadable slot first

+ // and answers its own failure; either way nothing is written.

+ if code, body := l.create(t, ltOne); code < 500 {

+ t.Fatalf("create = %d %s, want a 5xx (unreadable)", code, body)

+ }

+ if rows := allocRows(t, l.fx.st, ltOne); len(rows) != 0 {

+ t.Fatalf("allocations = %+v, want none", rows)

+ }

+ // And under the locks, the protocol's own classification refuses it.

+ life, _ := l.fx.st.AppLife(OpsAccountID, ltApp)

+ rec := provision.Record{Schema: 1, Host: ltOne, App: ltApp, Account: "bab", Channel: "stable", CreatedAt: time.Now().UTC()}

+ if _, err := l.fx.srv.createRecordWithLifetime(onCtx(t, l.fx.srv), rec, life.Lifetime); !errors.Is(err, provision.ErrLifetimeExhausted) {

+ t.Fatalf("the protocol = %v, want the unreadable slot's ErrLifetimeExhausted", err)

+ }

+ if rows := allocRows(t, l.fx.st, ltOne); len(rows) != 0 {

+ t.Fatalf("allocations = %+v, want none", rows)

+ }

+ })

+ t.Run("a record above the counter", func(t *testing.T) {

+ for _, field := range []string{"incarnation", "move_seq"} {

+ l := newLtOn(t)

+ body := `{"schema":1,"host":"` + ltOne + `","app":"shop","account":"bab","channel":"stable","` + field + `":9223372036854775808}`

+ if _, err := l.fx.rel.Put(ctx, recKey(ltApp, ltOne), []byte(body), ""); err != nil {

+ t.Fatal(err)

+ }

+ if code, out := l.create(t, ltOne); code != http.StatusBadGateway {

+ t.Fatalf("%s: create = %d %s, want 502 (unreadable)", field, code, out)

+ }

+ code, _, _ := rawReq(t, http.MethodGet, instancesURL(l.fx, "bab", ltApp)+"/"+ltOne, l.member, "")

+ if code != http.StatusBadGateway {

+ t.Fatalf("%s: GET = %d, want 502 (unreadable)", field, code)

+ }

+ if raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne); string(raw) != body {

+ t.Fatalf("%s: the record changed: %s", field, raw)

+ }

+ })

+func ltErr(body string) string { return cutoverErrorOf(body) }

+// ---- failures after the allocation, and recovery ----------------------------

+// pendingCreate makes the create of host fail at the step named, through the

+// door, and requires the 502 and a pending allocation.

+func (l ltOn) pendingCreate(t *testing.T, host, step string) allocRow {

+ t.Helper()

+ slotKey, key := provision.HostIncarnationKey(host), recKey(ltApp, host)

+ var drop func()

+ switch step {

+ case "slot":

+ l.bucket.setHook(failNth("put", slotKey, 1, false))

+ case "stale":

+ l.bucket.setHook(failNth("delete-if-match", key, 1, false))

+ case "record":

+ l.bucket.setHook(failNth("put", key, 1, false))

+ case "commit":

+ drop = sqlFailCommit(t, l.fx.st)

+ default:

+ t.Fatalf("no step %q", step)

+ }

+ code, body := l.create(t, host)

+ l.bucket.setHook(nil)

+ if drop != nil {

+ drop()

+ }

+ if code != http.StatusBadGateway || ltErr(body) != msgLifetimeCreatePending {

+ t.Fatalf("create failing at %s = %d %s, want 502 %q", step, code, body, msgLifetimeCreatePending)

+ }

+ return oneAlloc(t, l.fx.st, host, lifetimePending)

+func TestEveryPostAllocationFailureIsPending(t *testing.T) {

+ for _, step := range []string{"slot", "stale", "record", "commit"} {

+ t.Run(step, func(t *testing.T) {

+ l := newLtOn(t)

+ if step == "stale" {

+ retiredSlot(t, l.fx.rel, ltOne)

+ legacyBody(t, l.fx.rel, ltApp, ltOne, "edge")

+ }

+ a := l.pendingCreate(t, ltOne, step)

+ // No retry from the door: a second create is refused while

+ // the first is settling.

+ if code, body := l.create(t, ltOne); code != http.StatusConflict || ltErr(body) != msgLifetimeSettling {

+ t.Fatalf("a create while pending = %d %s, want 409 settling", code, body)

+ }

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOne, uint64(a.Incarnation), a.OpID)

+ })

+ }

+func TestRecoveryWritesTheSlotWhenMissing(t *testing.T) {

+ l := newLtOn(t)

+ a := l.pendingCreate(t, ltOne, "slot")

+ if slot, _ := slotOf(t, l.fx.rel, ltOne); slot != nil {

+ t.Fatalf("slot = %+v after a failed slot write", slot)

+ }

+ recoverOn(t, l.restart(t))

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOne, 1, a.OpID)

+func TestRecoveryContinuesFromAPredecessorSlot(t *testing.T) {

+ l := newLtOn(t)

+ pred := writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 1, State: provision.SlotRetired, Account: "bab", App: ltApp, Intent: "retire x"})

+ a := l.pendingCreate(t, ltOne, "slot")

+ if a.Pred != pred || a.Incarnation != 2 {

+ t.Fatalf("allocation = %+v, want 2 over %s", a, pred)

+ }

+ recoverOn(t, l.restart(t))

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOne, 2, a.OpID)

+func TestRecoveryLeavesALandedCreateServing(t *testing.T) {

+ l := newLtOn(t)

+ a := l.pendingCreate(t, ltOne, "commit")

+ raw, etag := rawOf(t, l.fx.rel, ltApp, ltOne)

+ if raw == nil {

+ t.Fatal("the record did not land before the commit failed")

+ }

+ recoverOn(t, l.restart(t))

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOne, 1, a.OpID)

+ if _, now := rawOf(t, l.fx.rel, ltApp, ltOne); now != etag {

+ t.Fatal("recovery rewrote a record that had landed")

+ }

+func TestRecoveryRemovesAStaleRecordThenWrites(t *testing.T) {

+ l := newLtOn(t)

+ retiredSlot(t, l.fx.rel, ltOne)

+ first := legacyBody(t, l.fx.rel, ltApp, ltOne, "edge")

+ a := l.pendingCreate(t, ltOne, "stale")

+ if a.Stale != first {

+ t.Fatalf("allocation stale = %q, want %q", a.Stale, first)

+ }

+ // Another stale body replaces it before recovery (an old console's

+ // late write): recovery deletes what is there now, on its own ETag.

+ _, cur := rawOf(t, l.fx.rel, ltApp, ltOne)

+ body, _ := json.Marshal(provision.Record{Schema: 1, Host: ltOne, App: ltApp, Account: "bab", Channel: "beta", CreatedAt: time.Now().UTC()})

+ if _, err := l.fx.rel.Put(context.Background(), recKey(ltApp, ltOne), body, cur); err != nil {

+ t.Fatal(err)

+ }

+ recoverOn(t, l.restart(t))

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ rec := requireServing(t, l.fx.rel, ltApp, ltOne, 2, a.OpID)

+ if rec.Channel != "stable" {

+ t.Fatalf("record = %+v, want the allocation's body", rec)

+ }

+func TestRecoverySupersedesOnlyOnSuccessorEvidence(t *testing.T) {

+ cases := []struct {

+ name string

+ slot func(a allocRow) *provision.HostIncarnation

+ state string

+ }{

+ {"a later incarnation", func(a allocRow) *provision.HostIncarnation {

+ return &provision.HostIncarnation{Incarnation: 2, State: provision.SlotLive, Account: "bab", App: "web", Intent: "create y"}

+ }, lifetimeSuperseded},

+ {"its incarnation under another key", func(a allocRow) *provision.HostIncarnation {

+ return &provision.HostIncarnation{Incarnation: 1, State: provision.SlotLive, Account: "bab", App: "web", Intent: "create y"}

+ }, lifetimeSuperseded},

+ {"its incarnation and key with another intent", func(a allocRow) *provision.HostIncarnation {

+ return &provision.HostIncarnation{Incarnation: 1, State: provision.SlotLive, Account: "bab", App: ltApp, Intent: "create someone-else"}

+ }, lifetimeSuperseded},

+ {"no slot: the predecessor still", nil, lifetimeCommitted},

+ }

+ for _, c := range cases {

+ t.Run(c.name, func(t *testing.T) {

+ l := newLtOn(t)

+ a := l.pendingCreate(t, ltOne, "slot")

+ if c.slot != nil {

+ writeSlot(t, l.fx.rel, ltOne, *c.slot(a))

+ }

+ recoverOn(t, l.restart(t))

+ oneAlloc(t, l.fx.st, ltOne, c.state)

+ if c.state == lifetimeSuperseded {

+ if raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne); raw != nil {

+ t.Fatalf("a superseded allocation wrote its record: %s", raw)

+ }

+ })

+ }

+ t.Run("an unrelated earlier slot is no successor", func(t *testing.T) {

+ l := newLtOn(t)

+ pred := writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 1, State: provision.SlotRetired, Account: "bab", App: ltApp, Intent: "retire x"})

+ a := l.pendingCreate(t, ltOne, "slot")

+ if a.Pred != pred {

+ t.Fatal("no predecessor recorded")

+ }

+ // The slot moved on to another retired state below N: neither the

+ // predecessor nor a successor.

+ writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 1, State: provision.SlotRetired, Account: "bab", App: ltApp, Intent: "retire z"})

+ recoverOn(t, l.restart(t))

+ oneAlloc(t, l.fx.st, ltOne, lifetimePending)

+ })

+func TestRecoveryLeavesPendingWhenUnreadable(t *testing.T) {

+ t.Run("a failing slot read", func(t *testing.T) {

+ l := newLtOn(t)

+ l.pendingCreate(t, ltOne, "record")

+ l.bucket.setHook(failNth("get", provision.HostIncarnationKey(ltOne), 1, true))

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, ltOne, lifetimePending)

+ if raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne); raw != nil {

+ t.Fatal("recovery wrote the record on an unreadable observation")

+ }

+ l.bucket.setHook(nil)

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ })

+ t.Run("a malformed slot body", func(t *testing.T) {

+ l := newLtOn(t)

+ a := l.pendingCreate(t, ltOne, "record")

+ _, etag := slotOf(t, l.fx.rel, ltOne)

+ if _, err := l.fx.rel.Put(context.Background(), provision.HostIncarnationKey(ltOne), []byte(`{"incarnation":`), etag); err != nil {

+ t.Fatal(err)

+ }

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, ltOne, lifetimePending)

+ if raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne); raw != nil {

+ t.Fatal("recovery wrote the record beside a malformed slot")

+ }

+ _, bad, _ := l.fx.rel.Get(context.Background(), provision.HostIncarnationKey(ltOne))

+ good, _ := json.Marshal(allocationSlot(Allocation{OpID: a.OpID, Account: "bab", App: ltApp, Host: ltOne, Incarnation: 1}))

+ if _, err := l.fx.rel.Put(context.Background(), provision.HostIncarnationKey(ltOne), good, bad); err != nil {

+ t.Fatal(err)

+ }

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ })

+func TestRecoverySurvivesASecondCrash(t *testing.T) {

+ l := newLtOn(t)

+ a := l.pendingCreate(t, ltOne, "slot")

+ // The first recovery writes the slot and dies at the record.

+ second := l.restart(t)

+ l.bucket.setHook(failNth("put", recKey(ltApp, ltOne), 1, false))

+ recoverOn(t, second)

+ l.bucket.setHook(nil)

+ oneAlloc(t, l.fx.st, ltOne, lifetimePending)

+ if slot, _ := slotOf(t, l.fx.rel, ltOne); slot == nil {

+ t.Fatal("the first recovery did not write the slot")

+ }

+ recoverOn(t, l.restart(t))

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOne, 1, a.OpID)

+func TestRecoveryUsesOnlyPersistedState(t *testing.T) {

+ l := newLtOn(t)

+ a := l.pendingCreate(t, ltOne, "record")

+ var body []byte

+ if err := l.fx.st.db.QueryRow(`SELECT body FROM host_allocations WHERE op_id = ?`, a.OpID).Scan(&body); err != nil {

+ t.Fatal(err)

+ }

+ // A console that never saw the request: only SQLite and the bucket.

+ recoverOn(t, l.restart(t))

+ raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne)

+ want, err := provision.AdvanceRecordSequence([]byte(`{}`), body)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if !bytes.Equal(raw, want) {

+ t.Fatalf("record = %s, want the allocation's body with its record_seq: %s", raw, want)

+ }

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+// ---- late writes ------------------------------------------------------------

+func TestRecoveryReplaysCreateAndTheLateWriteIsHarmless(t *testing.T) {

+ l := newLtOn(t)

+ gate, hook := pauseNth(t, "put", recKey(ltApp, ltOne), 1)

+ l.bucket.setHook(hook)

+ answered := make(chan int, 1)

+ go func() { code, _ := l.create(t, ltOne); answered <- code }()

+ waitFor(t, gate.entered, "the first console's record write")

+ // The first console is gone; the second recovers from SQLite.

+ recoverOn(t, l.restart(t))

+ a := oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ _, etag := rawOf(t, l.fx.rel, ltApp, ltOne)

+ _, slotETag := slotOf(t, l.fx.rel, ltOne)

+ // Now the first console's create-only write arrives.

+ gate.open()

+ if code := <-answered; code != http.StatusBadGateway && code != http.StatusCreated {

+ t.Fatalf("the late request answered %d", code)

+ }

+ requireServing(t, l.fx.rel, ltApp, ltOne, 1, a.OpID)

+ if _, now := rawOf(t, l.fx.rel, ltApp, ltOne); now != etag {

+ t.Fatal("the late record write changed the record")

+ }

+ if _, now := slotOf(t, l.fx.rel, ltOne); now != slotETag {

+ t.Fatal("the late write changed the slot")

+ }

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+func TestLateSlotWriteDoesNotAbandonTheAllocation(t *testing.T) {

+ slotKey := provision.HostIncarnationKey(ltOne)

+ t.Run("the late write after recovery", func(t *testing.T) {

+ l := newLtOn(t)

+ gate, hook := pauseNth(t, "put", slotKey, 1)

+ l.bucket.setHook(hook)

+ answered := make(chan int, 1)

+ go func() { code, _ := l.create(t, ltOne); answered <- code }()

+ waitFor(t, gate.entered, "the first console's slot write")

+ recoverOn(t, l.restart(t))

+ a := oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ gate.open()

+ <-answered

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOne, 1, a.OpID)

+ })

+ t.Run("the late write lands after recovery read the predecessor", func(t *testing.T) {

+ l := newLtOn(t)

+ gate, hook := pauseNth(t, "put", slotKey, 1)

+ landed := make(chan struct{})

+ var once sync.Once

+ l.bucket.setAfter(func(op, key string, err error) {

+ if op == "put" && key == slotKey {

+ once.Do(func() { close(landed) })

+ }

+ })

+ // The first console, once its slot write lands, is held again

+ // before its record write: recovery alone must finish.

+ recGate, recHook := pauseNth(t, "put", recKey(ltApp, ltOne), 1)

+ var puts atomic.Int32

+ l.bucket.setHook(chain(func(op, key string) error {

+ if op == "put" && key == slotKey && puts.Add(1) == 2 {

+ // Recovery classified the predecessor (no slot) and is

+ // about to write: the original write lands first.

+ gate.open()

+ waitFor(t, landed, "the original slot write")

+ }

+ return nil

+ }, hook, recHook))

+ answered := make(chan int, 1)

+ go func() { code, _ := l.create(t, ltOne); answered <- code }()

+ waitFor(t, gate.entered, "the first console's slot write")

+ second := l.restart(t)

+ recoverOn(t, second)

+ waitFor(t, recGate.entered, "the first console's record write")

+ a := oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOne, 1, a.OpID)

+ _, etag := rawOf(t, l.fx.rel, ltApp, ltOne)

+ recGate.open() // the first console's create-only record write, late

+ <-answered

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ if _, now := rawOf(t, l.fx.rel, ltApp, ltOne); now != etag {

+ t.Fatal("the late record write changed the record")

+ }

+ })

+func TestDeleteDuringCreateRecoveryIsRefused(t *testing.T) {

+ l := newLtOn(t)

+ l.pendingCreate(t, ltOne, "record")

+ code, _, body := rawReq(t, http.MethodDelete, instancesURL(l.fx, "bab", ltApp)+"/"+ltOne, l.member, "")

+ if code != http.StatusConflict || ltErr(body) != msgLifetimeSettling {

+ t.Fatalf("delete while the create is pending = %d %s, want 409 settling", code, body)

+ }

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOne, 1, "")

+func TestSeedCreatePendingIsOwnedByTheSweep(t *testing.T) {

+ t.Run("copy from", testFromCreatePendingIsOwnedByTheSweep)

+ t.Run("import", func(t *testing.T) {

+ clock := useLifetimesClock(t)

+ f := newImportFixture(t)

+ bucket := &ltStore{Store: f.fx.rel}

+ f.fx.srv.cfg.Releases = bucket

+ turnLifetimesOn(t, f.fx.srv, clock)

+ bucket.setHook(failNth("put", provision.RecordKey("bab", fromApp, importHost), 1, false))

+ code, out := beginImport(t, f, importHost, importSHA, 4096)

+ bucket.setHook(nil)

+ if code != http.StatusBadGateway || out["error"] != importFailed {

+ t.Fatalf("an import whose record write failed = %d %v, want 502 %q", code, out, importFailed)

+ }

+ recoverOn(t, f.fx.srv)

+ rec, err := f.record(t, importHost)

+ if err != nil || rec.Seed == nil || rec.Seed.Kind != seeds.KindImport || rec.Seed.State != seeds.StateAwaitingUpload {

+ t.Fatalf("recovered record = %+v %v, want the import awaiting its upload", rec, err)

+ }

+ // Nobody holds its token; the sweep abandons it once it is stale.

+ f.fx.srv.seedsPass(onCtx(t, f.fx.srv), time.Now().Add(2*seedImportTTL))

+ rec, err = f.record(t, importHost)

+ if err != nil || rec.Seed == nil || rec.Seed.State == seeds.StateAwaitingUpload {

+ t.Fatalf("after the sweep = %+v %v, want the import abandoned", rec, err)

+ }

+ })

+func testFromCreatePendingIsOwnedByTheSweep(t *testing.T) {

+ clock := useLifetimesClock(t)

+ f := newFromFixture(t, 3)

+ bucket := &ltStore{Store: f.fx.rel}

+ f.fx.srv.cfg.Releases = bucket

+ turnLifetimesOn(t, f.fx.srv, clock)

+ const host = "copy.shop.example.com"

+ bucket.setHook(failNth("put", provision.RecordKey("bab", fromApp, host), 1, false))

+ code, out := f.create(t, `{"host":"`+host+`","from":{"host":"`+fromSource+`"}}`)

+ bucket.setHook(nil)

+ f.fx.srv.seedCopies.Wait()

+ if code != http.StatusBadGateway {

+ t.Fatalf("a --from create whose record write failed = %d %v, want 502", code, out)

+ }

+ if _, err := f.record(t, host); !errors.Is(err, os.ErrNotExist) {

+ t.Fatalf("record after the failed write: %v, want none yet", err)

+ }

+ recoverOn(t, f.fx.srv)

+ rec, err := f.record(t, host)

+ if err != nil || rec.Seed == nil || rec.Seed.State != seeds.StateReserving || rec.Incarnation != 1 {

+ t.Fatalf("recovered record = %+v %v, want the reserving seed the body carried", rec, err)

+ }

+ // No copy ever started; the seeds sweep abandons the reservation once it

+ // is stale.

+ f.fx.srv.seedsPass(onCtx(t, f.fx.srv), time.Now().Add(2*seedReserveTTL))

+ rec, err = f.record(t, host)

+ if err != nil || rec.Seed == nil || rec.Seed.State != seeds.StateAborted {

+ t.Fatalf("after the sweep = %+v %v, want the seed aborted", rec, err)

+ }

+// ---- an allocation is replayed only for the app lifetime that made it -------

+// trashShop puts shop in the trash, as the trash door does: the store's

+// marker, then the row.

+func trashShop(t *testing.T, l ltOn) {

+ t.Helper()

+ if err := release.TrashApp(context.Background(), l.fx.rel, "bab", ltApp, "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ if err := l.fx.st.TrashAppRow(OpsAccountID, ltApp, time.Now()); err != nil {

+ t.Fatal(err)

+ }

+// purgeShop is a trash and a purge: the row and the store's marker gone.

+func purgeShop(t *testing.T, l ltOn) {

+ t.Helper()

+ trashShop(t, l)

+ if err := l.fx.st.PurgeAppRow(OpsAccountID, ltApp); err != nil {

+ t.Fatal(err)

+ }

+ if err := l.fx.rel.Delete(context.Background(), release.AppTombstoneKey("bab", ltApp)); err != nil {

+ t.Fatal(err)

+ }

+// recreateShop is a purge and a create under the same name: a new lifetime.

+func recreateShop(t *testing.T, l ltOn) {

+ t.Helper()

+ before, _ := l.fx.st.AppLife(OpsAccountID, ltApp)

+ purgeShop(t, l)

+ if _, err := l.fx.st.CreateAppIn(OpsAccountID, ltApp, "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ if after, _ := l.fx.st.AppLife(OpsAccountID, ltApp); after == nil || after.Lifetime == before.Lifetime {

+ t.Fatalf("the recreated app has lifetime %+v, want a new one", after)

+ }

+func TestAllocationBoundToAppLifetime(t *testing.T) {

+ requireCancelled := func(t *testing.T, l ltOn, host string, wantSlot bool) {

+ t.Helper()

+ oneAlloc(t, l.fx.st, host, lifetimeVoid)

+ slot, _ := slotOf(t, l.fx.rel, host)

+ if !wantSlot {

+ if slot != nil {

+ t.Fatalf("slot = %+v, want none written", slot)

+ }

+ return

+ }

+ if slot == nil || slot.State != provision.SlotRetired || slot.Incarnation != 1 {

+ t.Fatalf("slot = %+v, want {1, retired}", slot)

+ }

+ rets := retireRows(t, l.fx.st, host)

+ a := allocRows(t, l.fx.st, host)[0]

+ if len(rets) != 1 || rets[0].Kind != retireOrphan || rets[0].Cancels != a.OpID || rets[0].State != lifetimeCommitted {

+ t.Fatalf("retirements = %+v, want one committed orphan cancelling %s", rets, a.OpID)

+ }

+ if raw, _ := rawOf(t, l.fx.rel, ltApp, host); raw != nil {

+ t.Fatalf("a body survived the cancellation: %s", raw)

+ }

+ t.Run("a live slot and no record, the app trashed", func(t *testing.T) {

+ l := newLtOn(t)

+ l.pendingCreate(t, ltOne, "record")

+ trashShop(t, l)

+ recoverOn(t, l.restart(t))

+ requireCancelled(t, l, ltOne, true)

+ })

+ t.Run("a slot and a record that landed late, the app trashed", func(t *testing.T) {

+ l := newLtOn(t)

+ l.pendingCreate(t, ltOne, "commit")

+ trashShop(t, l)

+ recoverOn(t, l.restart(t))

+ requireCancelled(t, l, ltOne, true)

+ })

+ t.Run("no slot", func(t *testing.T) {

+ l := newLtOn(t)

+ l.pendingCreate(t, ltOne, "slot")

+ trashShop(t, l)

+ recoverOn(t, l.restart(t))

+ requireCancelled(t, l, ltOne, false)

+ })

+ t.Run("the app purged and recreated under the same name", func(t *testing.T) {

+ l := newLtOn(t)

+ l.pendingCreate(t, ltOne, "record")

+ recreateShop(t, l)

+ recoverOn(t, l.restart(t))

+ requireCancelled(t, l, ltOne, true)

+ })

+ t.Run("a record write after the void is stale, and the next create removes it", func(t *testing.T) {

+ l := newLtOn(t)

+ gate, hook := pauseNth(t, "put", recKey(ltApp, ltOne), 1)

+ l.bucket.setHook(hook)

+ answered := make(chan int, 1)

+ go func() { code, _ := l.create(t, ltOne); answered <- code }()

+ waitFor(t, gate.entered, "the first console's record write")

+ second := l.restart(t)

+ recreateShop(t, l)

+ recoverOn(t, second)

+ requireCancelled(t, l, ltOne, true)

+ gate.open() // the outstanding record write lands under the retired slot

+ <-answered

+ raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne)

+ if raw == nil {

+ t.Fatal("the late write did not land")

+ }

+ if obs := l.fx.srv.judge.Observe(context.Background(), l.fx.rel, "bab", ltApp, ltOne); obs.Verdict != provision.LifetimeStale {

+ t.Fatalf("the late body is %v, want stale", obs.Verdict)

+ }

+ // The next create of the recreated app replaces it.

+ code, _, body := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", ltApp), l.member, `{"host":"`+ltOne+`"}`)

+ if code != http.StatusCreated {

+ t.Fatalf("the next create = %d %s", code, body)

+ }

+ requireServing(t, l.fx.rel, ltApp, ltOne, 2, "")

+ })

+func TestCreateChecksTheAppUnderTheLock(t *testing.T) {

+ cases := []struct {

+ name string

+ change func(t *testing.T, l ltOn)

+ code int

+ msg string

+ }{

+ {"trashed", trashShop, http.StatusConflict, msgLifetimeAppTrashed},

+ {"recreated", recreateShop, http.StatusConflict, msgLifetimeAppChanged},

+ {"purged", purgeShop, http.StatusNotFound, "404 page not found"},

+ }

+ for _, c := range cases {

+ t.Run(c.name, func(t *testing.T) {

+ l := newLtOn(t)

+ life, _ := l.fx.st.AppLife(OpsAccountID, ltApp)

+ // The door read the lifetime, then the app changed before it

+ // took the lock.

+ c.change(t, l)

+ rec := provision.Record{Schema: 1, Host: ltOne, App: ltApp, Account: "bab", Channel: "stable", CreatedAt: time.Now().UTC()}

+ _, err := l.fx.srv.createRecordWithLifetime(onCtx(t, l.fx.srv), rec, life.Lifetime)

+ ref, ok := lifetimeCreateRefusal(ltOne, err, msgLifetimeCreatePending)

+ if !ok || ref.Status != c.code || ref.Msg != c.msg {

+ t.Fatalf("create = %v (%+v), want %d %q", err, ref, c.code, c.msg)

+ }

+ if rows := allocRows(t, l.fx.st, ltOne); len(rows) != 0 {

+ t.Fatalf("allocations = %+v, want none", rows)

+ }

+ if slot, _ := slotOf(t, l.fx.rel, ltOne); slot != nil {

+ t.Fatal("a slot was written")

+ }

+ })

+ }

+ t.Run("an unreadable marker", func(t *testing.T) {

+ l := newLtOn(t)

+ if _, err := l.fx.rel.Put(context.Background(), release.AppTombstoneKey("bab", ltApp), []byte("{"), ""); err != nil {

+ t.Fatal(err)

+ }

+ life, _ := l.fx.st.AppLife(OpsAccountID, ltApp)

+ rec := provision.Record{Schema: 1, Host: ltOne, App: ltApp, Account: "bab", Channel: "stable", CreatedAt: time.Now().UTC()}

+ _, err := l.fx.srv.createRecordWithLifetime(onCtx(t, l.fx.srv), rec, life.Lifetime)

+ if ref, ok := lifetimeCreateRefusal(ltOne, err, ""); !ok || ref.Status != http.StatusBadGateway || ref.Msg != msgLifetimeAppUnreadable {

+ t.Fatalf("create = %v, want 502 %q", err, msgLifetimeAppUnreadable)

+ }

+ if rows := allocRows(t, l.fx.st, ltOne); len(rows) != 0 {

+ t.Fatalf("allocations = %+v, want none", rows)

+ }

+ })

+// ---- orphan retirements and their recovery ------------------------------------

+func TestOrphanRetirementRecoveryBranches(t *testing.T) {

+ ctx := context.Background()

+ // begin persists an orphan retirement of web's live slot at 1 and

+ // leaves it pending (O1 never ran).

+ begin := func(t *testing.T, l ltOn) Retirement {

+ t.Helper()

+ etag := writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 1, State: provision.SlotLive, Account: "bab", App: "web", Intent: "create lost"})

+ r := Retirement{OpID: "orphan-1", Kind: retireOrphan, Account: "bab", App: "web", Host: ltOne, Incarnation: 1, SlotETag: etag, SlotIntent: "create lost"}

+ if err := l.fx.st.BeginRetirement(r); err != nil {

+ t.Fatal(err)

+ }

+ return r

+ }

+ state := func(t *testing.T, l ltOn) string {

+ t.Helper()

+ rows := retireRows(t, l.fx.st, ltOne)

+ if len(rows) != 1 {

+ t.Fatalf("retirements = %+v", rows)

+ }

+ return rows[0].State

+ }

+ t.Run("pred: O1 then committed", func(t *testing.T) {

+ l := newLtOn(t)

+ begin(t, l)

+ recoverOn(t, l.restart(t))

+ if got := state(t, l); got != lifetimeCommitted {

+ t.Fatalf("state = %s", got)

+ }

+ if slot, _ := slotOf(t, l.fx.rel, ltOne); slot.State != provision.SlotRetired || slot.Incarnation != 1 || slot.Intent != "retire orphan-1" {

+ t.Fatalf("slot = %+v, want {1, retired, retire orphan-1}", slot)

+ }

+ })

+ t.Run("pred with a late body: the body goes too", func(t *testing.T) {

+ l := newLtOn(t)

+ begin(t, l)

+ body, _ := json.Marshal(provision.Record{Schema: 1, Host: ltOne, App: "web", Account: "bab", Channel: "stable", Incarnation: 1, CreatedAt: time.Now().UTC()})

+ if _, err := l.fx.rel.Put(ctx, recKey("web", ltOne), body, ""); err != nil {

+ t.Fatal(err)

+ }

+ recoverOn(t, l.restart(t))

+ if got := state(t, l); got != lifetimeCommitted {

+ t.Fatalf("state = %s", got)

+ }

+ if raw, _ := rawOf(t, l.fx.rel, "web", ltOne); raw != nil {

+ t.Fatalf("the body under the retired slot survived: %s", raw)

+ }

+ })

+ t.Run("succ: void", func(t *testing.T) {

+ l := newLtOn(t)

+ begin(t, l)

+ writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 2, State: provision.SlotLive, Account: "bab", App: ltApp, Intent: "create next"})

+ recoverOn(t, l.restart(t))

+ if got := state(t, l); got != lifetimeVoid {

+ t.Fatalf("state = %s, want void", got)

+ }

+ if slot, _ := slotOf(t, l.fx.rel, ltOne); slot.Incarnation != 2 || slot.State != provision.SlotLive {

+ t.Fatalf("the successor slot moved: %+v", slot)

+ }

+ })

+ t.Run("retired: the body deleted, committed", func(t *testing.T) {

+ l := newLtOn(t)

+ r := begin(t, l)

+ writeSlot(t, l.fx.rel, ltOne, provision.HostIncarnation{Incarnation: 1, State: provision.SlotRetired, Account: "bab", App: "web", Intent: "retire " + r.OpID})

+ legacyBody(t, l.fx.rel, "web", ltOne, "stable")

+ recoverOn(t, l.restart(t))

+ if got := state(t, l); got != lifetimeCommitted {

+ t.Fatalf("state = %s", got)

+ }

+ if raw, _ := rawOf(t, l.fx.rel, "web", ltOne); raw != nil {

+ t.Fatalf("the body survived: %s", raw)

+ }

+ })

+ t.Run("unreadable: pending, and a create stays refused", func(t *testing.T) {

+ l := newLtOn(t)

+ begin(t, l)

+ l.bucket.setHook(failNth("get", provision.HostIncarnationKey(ltOne), 1, true))

+ recoverOn(t, l.fx.srv)

+ l.bucket.setHook(nil)

+ if got := state(t, l); got != lifetimePending {

+ t.Fatalf("state = %s, want pending", got)

+ }

+ if code, body := l.create(t, ltOne); code != http.StatusConflict || ltErr(body) != msgLifetimeSettling {

+ t.Fatalf("create = %d %s, want 409 settling", code, body)

+ }

+ recoverOn(t, l.fx.srv)

+ if got := state(t, l); got != lifetimeCommitted {

+ t.Fatalf("state = %s once readable", got)

+ }

+ })

+ t.Run("a cancelling retirement is recovered before its host's allocations", func(t *testing.T) {

+ l := newLtOn(t)

+ l.pendingCreate(t, ltOne, "record")

+ a := allocRows(t, l.fx.st, ltOne)[0]

+ slot, etag := slotOf(t, l.fx.rel, ltOne)

+ r := Retirement{OpID: "cancel-1", Kind: retireOrphan, Account: "bab", App: ltApp, Host: ltOne, Incarnation: 1, SlotETag: etag, SlotIntent: slot.Intent}

+ if err := l.fx.st.BeginCancellingRetirement(r, a.OpID); err != nil {

+ t.Fatal(err)

+ }

+ recoverOn(t, l.restart(t))

+ oneAlloc(t, l.fx.st, ltOne, lifetimeVoid)

+ if rows := retireRows(t, l.fx.st, ltOne); len(rows) != 1 || rows[0].State != lifetimeCommitted {

+ t.Fatalf("retirements = %+v, want the cancellation committed", rows)

+ }

+ if raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne); raw != nil {

+ t.Fatal("the cancelled allocation's record was written")

+ }

+ })

+// ---- the claims-off first-route attach ----------------------------------------

+const ltDomain = "docs.example.com"

+func (l ltOn) acct(t *testing.T) *Account {

+ t.Helper()

+ acct, err := l.fx.st.AccountBySqid("bab")

+ if err != nil || acct == nil {

+ t.Fatalf("account bab: %v", err)

+ }

+ return acct

+func (l ltOn) attach(t *testing.T, ctx context.Context, app string) (attachResult, *doorRefusal) {

+ t.Helper()

+ return l.fx.srv.attachDomain(ctx, l.acct(t), app, ltDomain, "", "stable", "paul@keymail.dev")

+func claimHolder(t *testing.T, st release.Store, host string) string {

+ t.Helper()

+ c, err := domains.GetClaim(context.Background(), st, host)

+ if errors.Is(err, os.ErrNotExist) {

+ return ""

+ }

+ if err != nil {

+ t.Fatal(err)

+ }

+ return c.Account + "/" + c.App

+func TestAttachKeepsItsClaimOverAPendingAllocation(t *testing.T) {

+ for _, step := range []string{"record", "commit"} {

+ t.Run(step, func(t *testing.T) {

+ l := newLtOn(t)

+ var drop func()

+ if step == "record" {

+ l.bucket.setHook(failNth("put", recKey(ltApp, ltDomain), 1, false))

+ } else {

+ drop = sqlFailCommit(t, l.fx.st)

+ }

+ _, ref := l.attach(t, onCtx(t, l.fx.srv), ltApp)

+ l.bucket.setHook(nil)

+ if drop != nil {

+ drop()

+ }

+ if ref == nil || ref.Status != http.StatusInternalServerError {

+ t.Fatalf("attach = %+v, want the ambiguous refusal", ref)

+ }

+ oneAlloc(t, l.fx.st, ltDomain, lifetimePending)

+ if got := claimHolder(t, l.fx.rel, ltDomain); got != "bab/shop" {

+ t.Fatalf("claim = %q after a pending allocation, want it kept", got)

+ }

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, ltDomain, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltDomain, 1, "")

+ if got := claimHolder(t, l.fx.rel, ltDomain); got != "bab/shop" {

+ t.Fatalf("claim = %q once the record stands, want it", got)

+ }

+ })

+ }

+func TestAttachNeverRetiresOverALifetimeFailure(t *testing.T) {

+ slotKey, key := provision.HostIncarnationKey(ltDomain), recKey(ltApp, ltDomain)

+ cases := []struct {

+ name string

+ arrange func(t *testing.T, l ltOn) func()

+ status int

+ keepClaim bool

+ }{

+ {"the slot write", func(t *testing.T, l ltOn) func() {

+ l.bucket.setHook(failNth("put", slotKey, 1, false))

+ return func() { l.bucket.setHook(nil) }

+ }, http.StatusInternalServerError, true},

+ {"the record write", func(t *testing.T, l ltOn) func() {

+ l.bucket.setHook(failNth("put", key, 1, false))

+ return func() { l.bucket.setHook(nil) }

+ }, http.StatusInternalServerError, true},

+ {"the commit", func(t *testing.T, l ltOn) func() { return sqlFailCommit(t, l.fx.st) }, http.StatusInternalServerError, true},

+ {"another operation settling", func(t *testing.T, l ltOn) func() {

+ if err := l.fx.st.BeginRetirement(Retirement{OpID: "other", Kind: retireOrphan, Account: "bab", App: "web", Host: ltDomain, Incarnation: 1}); err != nil {

+ t.Fatal(err)

+ }

+ return func() {}

+ }, http.StatusConflict, true},

+ {"a live slot of another app", func(t *testing.T, l ltOn) func() {

+ writeSlot(t, l.fx.rel, ltDomain, provision.HostIncarnation{Incarnation: 1, State: provision.SlotLive, Account: "bab", App: "web", Intent: "create x"})

+ body, _ := json.Marshal(provision.Record{Schema: 1, Host: ltDomain, App: "web", Account: "bab", Channel: "stable", Incarnation: 1, CreatedAt: time.Now().UTC()})

+ if _, err := l.fx.rel.Put(context.Background(), recKey("web", ltDomain), body, ""); err != nil {

+ t.Fatal(err)

+ }

+ return func() {}

+ }, http.StatusConflict, false},

+ {"an unreadable slot", func(t *testing.T, l ltOn) func() {

+ l.bucket.setHook(failNth("get", slotKey, 1, false))

+ return func() { l.bucket.setHook(nil) }

+ }, http.StatusInternalServerError, false},

+ {"the app trashed under the lock", func(t *testing.T, l ltOn) func() {

+ if err := release.TrashApp(context.Background(), l.fx.rel, "bab", ltApp, "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ return func() {}

+ }, http.StatusConflict, false},

+ }

+ for _, c := range cases {

+ t.Run(c.name, func(t *testing.T) {

+ l := newLtOn(t)

+ done := c.arrange(t, l)

+ _, ref := l.attach(t, onCtx(t, l.fx.srv), ltApp)

+ done()

+ if ref == nil || ref.Status != c.status {

+ t.Fatalf("attach = %+v, want %d", ref, c.status)

+ }

+ for _, op := range l.bucket.opsOf(key) {

+ if op == "delete" || op == "delete-if-match" {

+ t.Fatalf("record ops %v: a failed attach removed a record", l.bucket.opsOf(key))

+ }

+ for _, r := range retireRows(t, l.fx.st, ltDomain) {

+ if r.OpID != "other" {

+ t.Fatalf("retirements = %+v: a failed attach began one", r)

+ }

+ if got := claimHolder(t, l.fx.rel, ltDomain); (got == "bab/shop") != c.keepClaim {

+ t.Fatalf("claim = %q, want kept %v", got, c.keepClaim)

+ }

+ })

+ }

+// hostWaiters counts the goroutines that have waited on host's exclusion.

+func hostWaiters(t *testing.T, host string) *atomic.Int32 {

+ t.Helper()

+ var n atomic.Int32

+ h := func(kind, key, event string) {

+ if kind == "host" && key == host && event == lockWaiting {

+ n.Add(1)

+ }

+ lifetimeLockHook.Store(&h)

+ t.Cleanup(func() { lifetimeLockHook.Store(nil) })

+ return &n

+func waitCount(t *testing.T, n *atomic.Int32, want int32, what string) {

+ t.Helper()

+ deadline := time.Now().Add(10 * time.Second)

+ for n.Load() < want && time.Now().Before(deadline) {

+ time.Sleep(5 * time.Millisecond)

+ }

+ if n.Load() < want {

+ t.Fatalf("timed out waiting for %s", what)

+ }

+func TestCompetingAttachCannotReleaseAnotherAttachsClaim(t *testing.T) {

+ claimKey := domains.ClaimKey(ltDomain)

+ setup := func(t *testing.T) ltOn {

+ l := newLtOn(t)

+ if _, err := l.fx.st.CreateAppIn(OpsAccountID, "shop2", "paul@keymail.dev", time.Now()); err != nil {

+ t.Fatal(err)

+ }

+ return l

+ }

+ // race runs shop's attach paused after its claim landed, starts shop2's

+ // attach behind it, then lets shop go on with fail armed.

+ race := func(t *testing.T, l ltOn, fail func(string, string) error) (*doorRefusal, *doorRefusal) {

+ t.Helper()

+ waiters := hostWaiters(t, ltDomain)

+ landed, cont := make(chan struct{}), make(chan struct{})

+ var once sync.Once

+ l.bucket.setAfter(func(op, key string, err error) {

+ if op == "put" && key == claimKey && err == nil {

+ once.Do(func() { close(landed); <-cont })

+ }

+ })

+ var failing atomic.Bool

+ l.bucket.setHook(func(op, key string) error {

+ if failing.Load() && fail != nil {

+ return fail(op, key)

+ }

+ return nil

+ })

+ ctxA, ctxB := onCtx(t, l.fx.srv), onCtx(t, l.fx.srv)

+ refA := make(chan *doorRefusal, 1)

+ go func() { _, ref := l.attach(t, ctxA, ltApp); refA <- ref }()

+ waitFor(t, landed, "shop's claim")

+ refB := make(chan *doorRefusal, 1)

+ go func() { _, ref := l.attach(t, ctxB, "shop2"); refB <- ref }()

+ waitCount(t, waiters, 2, "shop2's attach to wait on the hostname")

+ select {

+ case ref := <-refB:

+ t.Fatalf("shop2's attach answered %+v while shop held the hostname", ref)

+ case <-time.After(50 * time.Millisecond):

+ }

+ failing.Store(true)

+ close(cont)

+ a := <-refA

+ failing.Store(false)

+ return a, <-refB

+ }

+ t.Run("shop fails before its allocation", func(t *testing.T) {

+ l := setup(t)

+ a, b := race(t, l, failNth("get", provision.HostIncarnationKey(ltDomain), 1, false))

+ if a == nil || b != nil {

+ t.Fatalf("shop = %+v, shop2 = %+v: want shop refused and shop2 attached", a, b)

+ }

+ if got := claimHolder(t, l.fx.rel, ltDomain); got != "bab/shop2" {

+ t.Fatalf("claim = %q, want shop2's", got)

+ }

+ requireServing(t, l.fx.rel, "shop2", ltDomain, 1, "")

+ if raw, _ := rawOf(t, l.fx.rel, ltApp, ltDomain); raw != nil {

+ t.Fatal("shop left a record")

+ }

+ release := l.bucket.index("delete-if-match " + claimKey)

+ if release < 0 {

+ release = l.bucket.index("delete " + claimKey)

+ }

+ if release < 0 || release > l.bucket.index("put "+recKey("shop2", ltDomain)) {

+ t.Fatalf("shop's claim release at %d: want it before shop2's record", release)

+ }

+ })

+ t.Run("shop's allocation is pending", func(t *testing.T) {

+ l := setup(t)

+ a, b := race(t, l, failNth("put", recKey(ltApp, ltDomain), 1, false))

+ if a == nil || b == nil || b.Status != http.StatusConflict {

+ t.Fatalf("shop = %+v, shop2 = %+v: want both refused, shop2 by the claim", a, b)

+ }

+ if got := claimHolder(t, l.fx.rel, ltDomain); got != "bab/shop" {

+ t.Fatalf("claim = %q, want shop's kept over its pending record", got)

+ }

+ recoverOn(t, l.fx.srv)

+ requireServing(t, l.fx.rel, ltApp, ltDomain, 1, "")

+ })

+ t.Run("shop's record completes", func(t *testing.T) {

+ l := setup(t)

+ a, b := race(t, l, nil)

+ if a != nil || b == nil || b.Status != http.StatusConflict {

+ t.Fatalf("shop = %+v, shop2 = %+v: want shop attached and shop2 refused", a, b)

+ }

+ if got := claimHolder(t, l.fx.rel, ltDomain); got != "bab/shop" {

+ t.Fatalf("claim = %q", got)

+ }

+ })

+ t.Run("a detach in place of the rollback", func(t *testing.T) {

+ l := setup(t)

+ if _, ref := l.attach(t, onCtx(t, l.fx.srv), ltApp); ref != nil {

+ t.Fatalf("attach = %+v", ref)

+ }

+ waiters := hostWaiters(t, ltDomain)

+ gate := newPauseGate()

+ t.Cleanup(gate.open)

+ var once sync.Once

+ l.bucket.setHook(func(op, key string) error {

+ if key == claimKey && (op == "delete" || op == "delete-if-match") {

+ once.Do(func() { close(gate.entered); <-gate.release })

+ }

+ return nil

+ })

+ detached := make(chan *doorRefusal, 1)

+ ctxA, ctxB := onCtx(t, l.fx.srv), onCtx(t, l.fx.srv)

+ go func() { detached <- l.fx.srv.detachDomain(ctxA, l.acct(t), ltApp, ltDomain, "paul@keymail.dev") }()

+ waitFor(t, gate.entered, "shop's claim release")

+ refB := make(chan *doorRefusal, 1)

+ go func() { _, ref := l.attach(t, ctxB, "shop2"); refB <- ref }()

+ waitCount(t, waiters, 2, "shop2's attach to wait on the hostname")

+ gate.open()

+ if ref := <-detached; ref != nil {

+ t.Fatalf("detach = %+v", ref)

+ }

+ if ref := <-refB; ref != nil {

+ t.Fatalf("shop2's attach after the detach = %+v", ref)

+ }

+ if got := claimHolder(t, l.fx.rel, ltDomain); got != "bab/shop2" {

+ t.Fatalf("claim = %q, want shop2's", got)

+ }

+ recDel := l.bucket.index("delete " + recKey(ltApp, ltDomain))

+ claimPut := l.bucket.index("put " + recKey("shop2", ltDomain))

+ if recDel < 0 || claimPut < recDel {

+ t.Fatalf("shop's record removal at %d, shop2's record at %d: want the removal first", recDel, claimPut)

+ }

+ })

+func TestDetachCompletesThroughTheRealHandlers(t *testing.T) {

+ for _, door := range []string{"api", "browser"} {

+ t.Run(door, func(t *testing.T) {

+ l := newLtOn(t)

+ code, _, body := rawReq(t, http.MethodPost, domainsURL(l.fx, "bab", ltApp), l.member,

+ `{"host":"`+ltDomain+`","kind":"static","channel":"stable"}`)

+ if code != http.StatusOK {

+ t.Fatalf("attach = %d %s", code, body)

+ }

+ requireServing(t, l.fx.rel, ltApp, ltDomain, 1, "")

+ done := make(chan int, 1)

+ go func() {

+ if door == "api" {

+ code, _, _ := rawReq(t, http.MethodDelete, domainsURL(l.fx, "bab", ltApp)+"/"+ltDomain, l.member, "")

+ done <- code

+ return

+ }

+ c := signInPaul(t, l.fx)

+ csrf := sessionToken(t, l.fx, c)

+ code, _, _ := postForm(t, c, l.fx.u("/apps/shop/domains/detach"),

+ url.Values{"csrf": {csrf}, "host": {ltDomain}, "confirm": {ltDomain}})

+ done <- code

+ }()

+ select {

+ case code := <-done:

+ if want := map[string]int{"api": http.StatusOK, "browser": http.StatusSeeOther}[door]; code != want {

+ t.Fatalf("detach = %d, want %d", code, want)

+ }

+ case <-time.After(10 * time.Second):

+ t.Fatal("the detach deadlocked")

+ }

+ if got := claimHolder(t, l.fx.rel, ltDomain); got != "" {

+ t.Fatalf("claim = %q after the detach", got)

+ }

+ if raw, _ := rawOf(t, l.fx.rel, ltApp, ltDomain); raw != nil {

+ t.Fatal("the record outlived its claim")

+ }

+ recDel := l.bucket.index("delete " + recKey(ltApp, ltDomain))

+ claimDel := l.bucket.index("delete-if-match " + domains.ClaimKey(ltDomain))

+ if claimDel < 0 {

+ claimDel = l.bucket.index("delete " + domains.ClaimKey(ltDomain))

+ }

+ if recDel < 0 || claimDel < 0 || recDel > claimDel {

+ t.Fatalf("record removal at %d, claim release at %d: want the record first", recDel, claimDel)

+ }

+ })

+ }

+// ---- the fence ----------------------------------------------------------------

+// outstanding starts a conditional record write of body against etag (an

+// operation's tombstone or marker PUT) held before it reaches the bucket;

+// the returned channel answers its outcome once released.

+func outstanding(t *testing.T, l ltOn, gate *pauseGate, before, body []byte, etag string) <-chan error {

+ t.Helper()

+ out := make(chan error, 1)

+ go func() {

+ _, _, err := provision.WriteRecord(context.Background(), l.bucket, recKey(ltApp, ltOne), before, body, etag)

+ out <- err

+ }()

+ waitFor(t, gate.entered, "the outstanding write")

+ return out

+func TestFenceMakesAnOutstandingWriteHarmless(t *testing.T) {

+ ctx := context.Background()

+ // setup: a legacy body A without record_seq at ETag E, an outstanding

+ // tombstone PUT against E, and a classification that saw B.

+ setup := func(t *testing.T) (ltOn, []byte, string, <-chan error, *pauseGate) {

+ l := newLtOn(t)

+ e := legacyBody(t, l.fx.rel, ltApp, ltOne, "stable")

+ a, _ := rawOf(t, l.fx.rel, ltApp, ltOne)

+ tomb, err := provision.TombstoneBody(a)

+ if err != nil {

+ t.Fatal(err)

+ }

+ gate, hook := pauseNth(t, "put", recKey(ltApp, ltOne), 1)

+ l.bucket.setHook(hook)

+ late := outstanding(t, l, gate, a, tomb, e)

+ l.bucket.setHook(nil)

+ b := bytes.Replace(a, []byte(`"channel":"stable"`), []byte(`"channel":"beta"`), 1)

+ eb, err := l.fx.rel.Put(ctx, recKey(ltApp, ltOne), b, e)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if obs, err := l.fx.srv.classifyLocked(ctx, "bab", ltApp, ltOne); err != nil || obs.ETag != eb {

+ t.Fatalf("classification = %+v %v, want the record at B", obs, err)

+ }

+ // A comes back: the bytes and the ETag recur.

+ if again, err := l.fx.rel.Put(ctx, recKey(ltApp, ltOne), a, eb); err != nil || again != e {

+ t.Fatalf("restoring A = %q %v, want the ETag %q again", again, err, e)

+ }

+ return l, a, e, late, gate

+ }

+ t.Run("the fence closes it", func(t *testing.T) {

+ l, _, e, late, gate := setup(t)

+ terminated := 0

+ obs, redispatch, err := l.fx.srv.terminateFenced(ctx, "bab", ltApp, ltOne, fenceOwner{Tombstone: true}, func() error { terminated++; return nil })

+ if err != nil || redispatch || terminated != 1 {

+ t.Fatalf("terminateFenced = %v %v, terminated %d: want terminated once", redispatch, err, terminated)

+ }

+ if obs.ETag == e {

+ t.Fatal("the fence left the ETag the outstanding write names")

+ }

+ gate.open()

+ if err := <-late; !errors.Is(err, release.ErrCASConflict) {

+ t.Fatalf("the late tombstone PUT = %v, want a CAS conflict", err)

+ }

+ raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne)

+ if rec := decodeRec(t, raw); rec.Retired || rec.RecordSeq != 1 {

+ t.Fatalf("record = %s, want A fenced with record_seq 1", raw)

+ }

+ })

+ t.Run("without the fence the late write lands", func(t *testing.T) {

+ l, _, _, late, gate := setup(t)

+ lifetimesFenceDisabled = true

+ t.Cleanup(func() { lifetimesFenceDisabled = false })

+ if _, _, err := l.fx.srv.terminateFenced(ctx, "bab", ltApp, ltOne, fenceOwner{Tombstone: true}, func() error { return nil }); err != nil {

+ t.Fatal(err)

+ }

+ gate.open()

+ if err := <-late; err != nil {

+ t.Fatalf("the late tombstone PUT = %v; with the fence off it lands (the defect)", err)

+ }

+ raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne)

+ if rec := decodeRec(t, raw); !rec.Retired {

+ t.Fatalf("record = %s, want the late tombstone (the defect the fence closes)", raw)

+ }

+ })

+ t.Run("released during the fence: the fence wins", func(t *testing.T) {

+ l, _, _, late, gate := setup(t)

+ var once sync.Once

+ l.bucket.setAfter(func(op, key string, err error) {

+ if op == "put" && key == recKey(ltApp, ltOne) && err == nil {

+ once.Do(func() {

+ gate.open()

+ if err := <-late; !errors.Is(err, release.ErrCASConflict) {

+ t.Errorf("the late PUT after the fence = %v, want a CAS conflict", err)

+ }

+ })

+ }

+ })

+ terminated := 0

+ if _, redispatch, err := l.fx.srv.terminateFenced(ctx, "bab", ltApp, ltOne, fenceOwner{Tombstone: true}, func() error { terminated++; return nil }); err != nil || redispatch || terminated != 1 {

+ t.Fatalf("terminateFenced = %v %v, terminated %d", redispatch, err, terminated)

+ }

+ })

+ for _, own := range []string{"tombstone", "marker"} {

+ t.Run("released during the fence: the operation's "+own+" wins", func(t *testing.T) {

+ l := newLtOn(t)

+ e := legacyBody(t, l.fx.rel, ltApp, ltOne, "stable")

+ a, _ := rawOf(t, l.fx.rel, ltApp, ltOne)

+ body, op := a, fenceOwner{}

+ if own == "tombstone" {

+ body, _ = provision.TombstoneBody(a)

+ op.Tombstone = true

+ } else {

+ body = bytes.Replace(a, []byte(`{`), []byte(`{"incarnation_migrating":"m1",`), 1)

+ op.MarkerID = "m1"

+ }

+ gate, hook := pauseNth(t, "put", recKey(ltApp, ltOne), 1)

+ l.bucket.setHook(hook)

+ late := outstanding(t, l, gate, a, body, e)

+ // The fence is about to write: the operation's own write is

+ // released and lands first.

+ var once sync.Once

+ l.bucket.setHook(func(o, k string) error {

+ if o == "put" && k == recKey(ltApp, ltOne) {

+ once.Do(func() {

+ gate.open()

+ if err := <-late; err != nil {

+ t.Errorf("the operation's write = %v, want it to land", err)

+ }

+ })

+ }

+ return nil

+ })

+ terminated := 0

+ obs, redispatch, err := l.fx.srv.terminateFenced(ctx, "bab", ltApp, ltOne, op, func() error { terminated++; return nil })

+ if err != nil || !redispatch || terminated != 0 {

+ t.Fatalf("terminateFenced = %v %v, terminated %d: want a redispatch, never a termination", redispatch, err, terminated)

+ }

+ if own == "tombstone" && (obs.Rec == nil || !obs.Rec.Retired) {

+ t.Fatalf("observation = %+v, want the tombstone", obs)

+ }

+ if own == "marker" && (obs.Rec == nil || obs.Rec.IncarnationMigrating != "m1") {

+ t.Fatalf("observation = %+v, want the marked record", obs)

+ }

+ })

+ }

+func TestFenceResultsDriveTheCaller(t *testing.T) {

+ ctx := context.Background()

+ cases := []struct {

+ name string

+ arrange func(t *testing.T, l ltOn)

+ op fenceOwner

+ redispatch bool

+ terminated int

+ pending bool

+ }{

+ {"fenced", func(t *testing.T, l ltOn) { legacyBody(t, l.fx.rel, ltApp, ltOne, "stable") }, fenceOwner{Tombstone: true}, false, 1, false},

+ {"own tombstone", func(t *testing.T, l ltOn) {

+ legacyBody(t, l.fx.rel, ltApp, ltOne, "stable")

+ raw, e := rawOf(t, l.fx.rel, ltApp, ltOne)

+ tomb, _ := provision.TombstoneBody(raw)

+ if _, err := l.fx.rel.Put(ctx, recKey(ltApp, ltOne), tomb, e); err != nil {

+ t.Fatal(err)

+ }

+ }, fenceOwner{Tombstone: true}, true, 0, false},

+ {"own marker", func(t *testing.T, l ltOn) {

+ body, _ := json.Marshal(provision.Record{Schema: 1, Host: ltOne, App: ltApp, Account: "bab", Channel: "stable", IncarnationMigrating: "m1", CreatedAt: time.Now().UTC()})

+ if _, err := l.fx.rel.Put(ctx, recKey(ltApp, ltOne), body, ""); err != nil {

+ t.Fatal(err)

+ }

+ }, fenceOwner{MarkerID: "m1"}, true, 0, false},

+ {"absent", func(t *testing.T, l ltOn) {}, fenceOwner{Tombstone: true}, true, 0, false},

+ {"a fence that cannot be written", func(t *testing.T, l ltOn) {

+ legacyBody(t, l.fx.rel, ltApp, ltOne, "stable")

+ l.bucket.setHook(failNth("put", recKey(ltApp, ltOne), 1, true))

+ }, fenceOwner{Tombstone: true}, false, 0, true},

+ }

+ for _, c := range cases {

+ t.Run(c.name, func(t *testing.T) {

+ l := newLtOn(t)

+ c.arrange(t, l)

+ _, beforeETag := rawOf(t, l.fx.rel, ltApp, ltOne)

+ // A real pending row the termination ends.

+ if err := l.fx.st.BeginRetirement(Retirement{OpID: "r1", Kind: retireNormal, Account: "bab", App: ltApp, Host: ltOne, Incarnation: 1}); err != nil {

+ t.Fatal(err)

+ }

+ terminated := 0

+ obs, redispatch, err := l.fx.srv.terminateFenced(ctx, "bab", ltApp, ltOne, c.op, func() error {

+ terminated++

+ return l.fx.st.SetRetirementState("r1", lifetimeVoid)

+ })

+ l.bucket.setHook(nil)

+ if redispatch != c.redispatch || terminated != c.terminated || errors.Is(err, errLifetimePending) != c.pending {

+ t.Fatalf("terminateFenced = redispatch %v, terminated %d, %v; want %v, %d, pending %v", redispatch, terminated, err, c.redispatch, c.terminated, c.pending)

+ }

+ want := lifetimePending

+ if c.terminated == 1 {

+ want = lifetimeVoid

+ raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne)

+ if rec := decodeRec(t, raw); rec.RecordSeq != 1 || obs.ETag == beforeETag {

+ t.Fatalf("fenced record = %s, want record_seq introduced and a new ETag", raw)

+ }

+ if rows := retireRows(t, l.fx.st, ltOne); rows[0].State != want {

+ t.Fatalf("row = %s, want %s", rows[0].State, want)

+ }

+ })

+ }

+// ---- the cutover gates the protocol -------------------------------------------

+func TestCutoverGatesLifetimes(t *testing.T) {

+ ctx := context.Background()

+ l := newLifetimesFx(t)

+ s := l.fx.srv

+ // No marker: the create door writes a plain record, as main; the

+ // recovery pass idles.

+ if code, _, body := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", ltApp), l.member, `{"host":"`+ltOne+`"}`); code != http.StatusCreated {

+ t.Fatalf("create before the marker = %d %s", code, body)

+ }

+ if raw := rawRecord(t, l.fx.rel, ltOne); bytes.Contains(raw, []byte("record_seq")) || bytes.Contains(raw, []byte("incarnation")) {

+ t.Fatalf("record before the marker = %s, want main's plain record", raw)

+ }

+ if slot, _ := slotOf(t, l.fx.rel, ltOne); slot != nil || len(allocRows(t, l.fx.st, ltOne)) != 0 {

+ t.Fatal("a create before the marker wrote a slot or an allocation")

+ }

+ if s.lifetimeRecoveryIteration(ctx) {

+ t.Fatal("the recovery pass ran before the marker")

+ }

+ // A marker younger than the settle: every mutation 503, reads judged,

+ // the pass idle.

+ if code, _, body := l.cutover(t, l.operator); code != http.StatusOK {

+ t.Fatalf("cutover = %d %s", code, body)

+ }

+ if code, _, body := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", ltApp), l.member, `{"host":"`+ltOld+`"}`); code != http.StatusServiceUnavailable || ltErr(body) != msgLifetimesDraining {

+ t.Fatalf("create while draining = %d %s", code, body)

+ }

+ retiredSlot(t, l.fx.rel, ltFresh)

+ legacyRecord(t, l.fx.rel, ltFresh, "stable")

+ if code, _, _ := rawReq(t, http.MethodGet, instancesURL(l.fx, "bab", ltApp)+"/"+ltFresh, l.member, ""); code != http.StatusNotFound {

+ t.Fatalf("a judged GET of an obsolete body while draining = %d, want 404", code)

+ }

+ if s.lifetimeRecoveryIteration(ctx) {

+ t.Fatal("the recovery pass ran while draining")

+ }

+ // Drained and settled: the lifetime protocol.

+ l.clock.Advance(lifetimeSettle)

+ s.lifetimesWatchOnce(ctx)

+ if !s.lifetimesOn() {

+ t.Fatal("not on after the settle")

+ }

+ if code, _, body := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", ltApp), l.member, `{"host":"`+ltOld+`"}`); code != http.StatusCreated {

+ t.Fatalf("create once on = %d %s", code, body)

+ }

+ a := oneAlloc(t, l.fx.st, ltOld, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltOld, 1, a.OpID)

+ if !s.lifetimeRecoveryIteration(ctx) {

+ t.Fatal("the recovery pass idled once on")

+ }

+// ---- dispatch by the admitted unit --------------------------------------------

+// testActivationWaitsForTheCreateDoor is the create door's half of

+// TestActivationWaitsForAdmittedWork: a real create admitted before the

+// marker, held before its record write across it; its legacy write lands,

+// and lifetimes turn on only after it ends and the settle passes.

+func testActivationWaitsForTheCreateDoor(t *testing.T) {

+ l := newLifetimesFx(t)

+ pauses := map[string]*putPause{recKey(ltApp, ltOne): {nth: 1}}

+ pausePuts(t, l.fx, pauses)

+ created := make(chan int, 1)

+ go func() {

+ code, _, _ := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", ltApp), l.member, `{"host":"`+ltOne+`"}`)

+ created <- code

+ }()

+ waitFor(t, pauses[recKey(ltApp, ltOne)].entered, "the create's record write")

+ if code, st, body := l.cutover(t, l.operator); code != http.StatusOK || st.State != "draining" || st.Pending != 1 {

+ t.Fatalf("cutover = %d %s, want draining with the create pending", code, body)

+ }

+ l.clock.Advance(lifetimeSettle + time.Second)

+ l.fx.srv.lifetimesWatchOnce(context.Background())

+ if l.fx.srv.lifetimesOn() {

+ t.Fatal("on while a create admitted before the marker runs")

+ }

+ close(pauses[recKey(ltApp, ltOne)].release)

+ if code := <-created; code != http.StatusCreated {

+ t.Fatalf("the held create = %d", code)

+ }

+ raw := rawRecord(t, l.fx.rel, ltOne)

+ if bytes.Contains(raw, []byte("record_seq")) || bytes.Contains(raw, []byte("incarnation")) {

+ t.Fatalf("the before create wrote %s, want its legacy record", raw)

+ }

+ if slot, _ := slotOf(t, l.fx.rel, ltOne); slot != nil || len(allocRows(t, l.fx.st, ltOne)) != 0 {

+ t.Fatal("the before create allocated a lifetime")

+ }

+ last := l.clock.Now()

+ l.clock.Advance(lifetimeSettle - time.Second)

+ l.fx.srv.lifetimesWatchOnce(context.Background())

+ if l.fx.srv.lifetimesOn() {

+ t.Fatal("on before the settle after the create ended")

+ }

+ l.clock.Advance(time.Second)

+ l.fx.srv.lifetimesWatchOnce(context.Background())

+ if st := l.fx.srv.cutover.status(); st.State != "on" || !st.At.Equal(last.Add(lifetimeSettle)) {

+ t.Fatalf("after the settle = %+v, want on", st)

+ }

+func TestDispatchFollowsTheUnit(t *testing.T) {

+ ctx := context.Background()

+ l := newLifetimesFx(t)

+ acct, _ := l.fx.st.AccountBySqid("bab")

+ legacyRecord(t, l.fx.rel, ltOld, "stable")

+ if _, ref := l.fx.srv.attachDomain(ctx, acct, ltApp, ltDomain, "", "stable", "paul@keymail.dev"); ref != nil {

+ t.Fatalf("attach = %+v", ref)

+ }

+ bucket := &ltStore{Store: l.fx.rel}

+ l.fx.srv.cfg.Releases = bucket

+ createGate, createHook := pauseNth(t, "put", recKey(ltApp, ltOne), 1)

+ deleteGate, deleteHook := pauseNth(t, "delete-if-match", recKey(ltApp, ltOld), 1) // deleteRead's conditional delete

+ detachGate, detachHook := pauseNth(t, "delete", recKey(ltApp, ltDomain), 1)

+ bucket.setHook(chain(createHook, deleteHook, detachHook))

+ // Three units admitted before the marker, each held inside its door.

+ answers := make(chan string, 3)

+ go func() {

+ code, _, _ := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", ltApp), l.member, `{"host":"`+ltOne+`"}`)

+ answers <- fmt.Sprintf("create %d", code)

+ }()

+ go func() {

+ code, _, _ := rawReq(t, http.MethodDelete, instancesURL(l.fx, "bab", ltApp)+"/"+ltOld, l.member, "")

+ answers <- fmt.Sprintf("delete %d", code)

+ }()

+ go func() {

+ code, _, _ := rawReq(t, http.MethodDelete, domainsURL(l.fx, "bab", ltApp)+"/"+ltDomain, l.member, "")

+ answers <- fmt.Sprintf("detach %d", code)

+ }()

+ for what, g := range map[string]*pauseGate{"create": createGate, "delete": deleteGate, "detach": detachGate} {

+ waitFor(t, g.entered, "the "+what)

+ }

+ if code, st, body := l.cutover(t, l.operator); code != http.StatusOK || st.Pending != 3 {

+ t.Fatalf("cutover = %d %s, want draining with three units", code, body)

+ }

+ createGate.open()

+ deleteGate.open()

+ detachGate.open()

+ got := map[string]bool{}

+ for i := 0; i < 3; i++ {

+ got[<-answers] = true

+ }

+ for _, want := range []string{"create 201", "delete 200", "detach 200"} {

+ if !got[want] {

+ t.Fatalf("answers = %v, want %s", got, want)

+ }

+ // Each took the legacy path: a plain record, no slot, no intent, and

+ // deletes with no retirement.

+ if raw := rawRecord(t, l.fx.rel, ltOne); bytes.Contains(raw, []byte("incarnation")) || bytes.Contains(raw, []byte("record_seq")) {

+ t.Fatalf("the before create wrote %s", raw)

+ }

+ for _, host := range []string{ltOne, ltOld, ltDomain} {

+ if slot, _ := slotOf(t, l.fx.rel, host); slot != nil || len(allocRows(t, l.fx.st, host)) != 0 || len(retireRows(t, l.fx.st, host)) != 0 {

+ t.Fatalf("%s: a before unit wrote a slot or an intent", host)

+ }

+ bucket.setHook(nil)

+ l.clock.Advance(lifetimeSettle)

+ l.fx.srv.lifetimesWatchOnce(ctx)

+ if !l.fx.srv.lifetimesOn() {

+ t.Fatal("not on")

+ }

+ // An on unit's create takes createRecordWithLifetime.

+ if code, _, body := rawReq(t, http.MethodPost, instancesURL(l.fx, "bab", ltApp), l.member, `{"host":"`+ltFresh+`"}`); code != http.StatusCreated {

+ t.Fatalf("create once on = %d %s", code, body)

+ }

+ a := oneAlloc(t, l.fx.st, ltFresh, lifetimeCommitted)

+ requireServing(t, l.fx.rel, ltApp, ltFresh, 1, a.OpID)

+ // An on unit's delete and readDeletedRecord are still today's (Task 4

+ // retires through a tombstone and judges the read): the record goes

+ // unconditionally and the live slot stays, an orphan the next create

+ // reclaims.

+ if code, _, body := rawReq(t, http.MethodDelete, instancesURL(l.fx, "bab", ltApp)+"/"+ltFresh, l.member, ""); code != http.StatusOK {

+ t.Fatalf("delete once on = %d %s", code, body)

+ }

+ if slot, _ := slotOf(t, l.fx.rel, ltFresh); slot == nil || slot.State != provision.SlotLive || len(retireRows(t, l.fx.st, ltFresh)) != 0 {

+ t.Fatalf("slot = %+v: the on delete is still the legacy one until Task 4", slot)

+ }

+ retiredSlot(t, l.fx.rel, ltOld)

+ legacyRecord(t, l.fx.rel, ltOld, "stable")

+ if rec, _, err := l.fx.srv.readDeletedRecord(onCtx(t, l.fx.srv), "bab", ltApp, ltOld); err != nil || rec == nil {

+ t.Fatalf("readDeletedRecord of a stale body = %+v %v: still the raw read until Task 4", rec, err)

+ }

+// ---- the lock order -------------------------------------------------------------

+// lockChecker sees every app and hostname lock through lifetimeLockHook and

+// records an inversion (an app lock waited for under a hostname lock) or a

+// hostname lock waited for under another.

+type lockChecker struct {

+ mu sync.Mutex

+ held map[string][]string // goroutine -> held locks, in order

+ seq []string // every lock as it is held: "<goroutine> app|host <key>"

+ errs []string

+func ltGoroutine() string {

+ var buf [64]byte

+ n := runtime.Stack(buf[:], false)

+ return strings.Fields(string(buf[:n]))[1]

+func installLockChecker(t *testing.T) *lockChecker {

+ t.Helper()

+ c := &lockChecker{held: map[string][]string{}}

+ h := func(kind, key, event string) {

+ g := ltGoroutine()

+ c.mu.Lock()

+ defer c.mu.Unlock()

+ lock := kind + " " + key

+ switch event {

+ case lockWaiting:

+ for _, h := range c.held[g] {

+ if strings.HasPrefix(h, "host ") {

+ c.errs = append(c.errs, fmt.Sprintf("%s waited for while %s is held", lock, h))

+ }

+ case lockHeld:

+ c.held[g] = append(c.held[g], lock)

+ c.seq = append(c.seq, g+" "+lock)

+ case lockReleased:

+ hs := c.held[g]

+ for i := len(hs) - 1; i >= 0; i-- {

+ if hs[i] == lock {

+ c.held[g] = append(hs[:i], hs[i+1:]...)

+ break

+ }

+ lifetimeLockHook.Store(&h)

+ t.Cleanup(func() { lifetimeLockHook.Store(nil) })

+ return c

+// order is the locks taken by the goroutine that took first, in order.

+func (c *lockChecker) taken(prefix string) []string {

+ c.mu.Lock()

+ defer c.mu.Unlock()

+ var out []string

+ for _, s := range c.seq {

+ f := strings.SplitN(s, " ", 2)

+ if strings.HasPrefix(f[1], prefix) {

+ out = append(out, f[1])

+ }

+ return out

+func (c *lockChecker) check(t *testing.T) {

+ t.Helper()

+ c.mu.Lock()

+ defer c.mu.Unlock()

+ if len(c.errs) > 0 {

+ t.Fatalf("lock order violated:\n%s", strings.Join(c.errs, "\n"))

+ }

+func TestLockOrderIsRespected(t *testing.T) {

+ l := newLtOn(t)

+ c := installLockChecker(t)

+ // The create door: the app lock, then the hostname.

+ if code, body := l.create(t, ltOne); code != http.StatusCreated {

+ t.Fatalf("create = %d %s", code, body)

+ }

+ if got := c.taken(""); len(got) < 2 || got[0] != "app bab/shop" || got[1] != "host "+ltOne {

+ t.Fatalf("the create took %v, want the app lock then the hostname", got)

+ }

+ // A create under shop retiring web's orphan slot takes no lock of web.

+ writeSlot(t, l.fx.rel, ltOld, provision.HostIncarnation{Incarnation: 1, State: provision.SlotLive, Account: "bab", App: "web", Intent: "create lost"})

+ if code, body := l.create(t, ltOld); code != http.StatusCreated {

+ t.Fatalf("create over an orphan = %d %s", code, body)

+ }

+ if got := c.taken("app bab/web"); len(got) != 0 {

+ t.Fatalf("the orphan retirement took %v", got)

+ }

+ // Attach and detach, through the API and the browser.

+ if code, _, body := rawReq(t, http.MethodPost, domainsURL(l.fx, "bab", ltApp), l.member, `{"host":"`+ltDomain+`","kind":"static","channel":"stable"}`); code != http.StatusOK {

+ t.Fatalf("attach = %d %s", code, body)

+ }

+ if code, _, body := rawReq(t, http.MethodDelete, domainsURL(l.fx, "bab", ltApp)+"/"+ltDomain, l.member, ""); code != http.StatusOK {

+ t.Fatalf("detach = %d %s", code, body)

+ }

+ if _, ref := l.attach(t, onCtx(t, l.fx.srv), ltApp); ref != nil {

+ t.Fatalf("attach = %+v", ref)

+ }

+ b := signInPaul(t, l.fx)

+ if code, _, body := postForm(t, b, l.fx.u("/apps/shop/domains/detach"),

+ url.Values{"csrf": {sessionToken(t, l.fx, b)}, "host": {ltDomain}, "confirm": {ltDomain}}); code != http.StatusSeeOther {

+ t.Fatalf("browser detach = %d %s", code, body)

+ }

+ // The delete door's admission.

+ if code, _, body := rawReq(t, http.MethodDelete, instancesURL(l.fx, "bab", ltApp)+"/"+ltOne, l.member, ""); code != http.StatusOK {

+ t.Fatalf("delete = %d %s", code, body)

+ }

+ // Recovery: an allocation, an allocation cancelled with its slot, and an

+ // orphan retirement.

+ l.pendingCreate(t, ltFresh, "record")

+ recoverOn(t, l.fx.srv)

+ l.pendingCreate(t, "two.shop.example.com", "record")

+ etag := writeSlot(t, l.fx.rel, "three.shop.example.com", provision.HostIncarnation{Incarnation: 1, State: provision.SlotLive, Account: "bab", App: "web", Intent: "create lost"})

+ if err := l.fx.st.BeginRetirement(Retirement{OpID: "orphan-3", Kind: retireOrphan, Account: "bab", App: "web", Host: "three.shop.example.com", Incarnation: 1, SlotETag: etag, SlotIntent: "create lost"}); err != nil {

+ t.Fatal(err)

+ }

+ trashShop(t, l)

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, "two.shop.example.com", lifetimeVoid)

+ if rows := retireRows(t, l.fx.st, "three.shop.example.com"); len(rows) != 1 || rows[0].State != lifetimeCommitted {

+ t.Fatalf("orphan retirement = %+v", rows)

+ }

+ c.check(t)

+func TestRecoveryWaitsForAnotherPendingOperation(t *testing.T) {

+ l := newLtOn(t)

+ l.pendingCreate(t, ltOne, "record")

+ // Another operation's intent for the hostname (a kind this pass does

+ // not recover yet): the allocation is not touched while it is pending.

+ if err := l.fx.st.BeginRetirement(Retirement{OpID: "other", Kind: retireNormal, Account: "bab", App: ltApp, Host: ltOne, Incarnation: 1}); err != nil {

+ t.Fatal(err)

+ }

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, ltOne, lifetimePending)

+ if raw, _ := rawOf(t, l.fx.rel, ltApp, ltOne); raw != nil {

+ t.Fatal("recovery wrote the record while another operation was pending")

+ }

+ if err := l.fx.st.SetRetirementState("other", lifetimeVoid); err != nil {

+ t.Fatal(err)

+ }

+ recoverOn(t, l.fx.srv)

+ oneAlloc(t, l.fx.st, ltOne, lifetimeCommitted)

diff --git a/internal/console/lifetimes_cutover_test.go b/internal/console/lifetimes_cutover_test.go

index 373ff190b..9281c25c0 100644

--- a/internal/console/lifetimes_cutover_test.go

+++ b/internal/console/lifetimes_cutover_test.go

@@ -540,6 +540,11 @@ func requireListing(t *testing.T, l lifetimesFx, hosts ...string) {

}

func TestActivationWaitsForAdmittedWork(t *testing.T) {

+ t.Run("the channel door and the import continuation", testActivationWaitsForChannelAndImport)

+ t.Run("the create door", testActivationWaitsForTheCreateDoor) // lifetimes_create_test.go

+func testActivationWaitsForChannelAndImport(t *testing.T) {

clock := useLifetimesClock(t)

f := newImportFixture(t)

if err := f.fx.st.SetOperator("paul@keymail.dev", true); err != nil {

diff --git a/internal/console/lifetimes_recovery.go b/internal/console/lifetimes_recovery.go

new file mode 100644

index 000000000..72c076941

--- /dev/null

+++ b/internal/console/lifetimes_recovery.go

@@ -0,0 +1,216 @@

+package console

+import (

+ "context"

+ "errors"

+ "fmt"

+ "log"

+ "time"

+ "github.com/carlosframework/platform/internal/provision"

+ "github.com/carlosframework/platform/internal/release"

+// The lifetime recovery pass (self-serve moves plan 1, Task 3): every intent

+// a crash, a failed write or an ambiguous answer left pending is finished

+// or ended here, from the console's SQLite rows and fresh reads alone. It

+// never relies on memory of the operation that wrote the row. Each row is

+// recovered under s.lockApp then withHostname, admitted against every other

+// pending operation of its hostname, after a fresh classification; a re-read

+// that fails leaves the row pending and selects no terminal branch.

+const lifetimeRecoveryInterval = 60 * time.Second

+// StartLifetimeRecoveryPass runs the recovery pass now and every 60 seconds

+// for the life of the process. Each iteration is a unit of record work

+// admitted as a writer, and it does work only when admitted on. A host-claim

+// console has no lifetimes: it logs that and returns.

+func (s *Server) StartLifetimeRecoveryPass(ctx context.Context) {

+ if s.cfg.HostClaim {

+ log.Printf("console: lifetime recovery pass: off on a host-claim console, where lifetimes are not for it")

+ return

+ }

+ if !s.lifetimesGoverned() {

+ return

+ }

+ lifetimesWorker(ctx, lifetimeRecoveryInterval, func(ctx context.Context) { s.lifetimeRecoveryIteration(ctx) })

+// lifetimeRecoveryIteration is one pass as a unit: skipped unless admitted on.

+// It reports whether the pass ran.

+func (s *Server) lifetimeRecoveryIteration(ctx context.Context) bool {

+ return s.lifetimesIteration(ctx, "lifetime recovery", true, true, func(ctx context.Context) {

+ if err := s.recoverLifetimes(ctx); err != nil {

+ log.Printf("console: lifetime recovery pass: %v", err)

+ }

+ })

+// recoverLifetimes is one pass: the orphan retirements (those cancelling an

+// allocation first), then the allocations. Task 4 adds the other retirement

+// kinds and Task 5 the migrations, in that order. An error is a listing that

+// failed; a row that could not be settled stays pending for the next pass.

+func (s *Server) recoverLifetimes(ctx context.Context) error {

+ ctx = provision.WithLifetimes(ctx, s.judge)

+ rets, err := s.store.PendingRetirementsCancellingFirst()

+ if err != nil {

+ return fmt.Errorf("pending retirements: %w", err)

+ }

+ for _, r := range rets {

+ if r.Kind != retireOrphan {

+ s.lifetimeLogHourly("retirement-kind "+r.OpID, "console: lifetime recovery: retirement %s of %s is of kind %s, which this console does not recover yet; it stays pending", r.OpID, r.Host, r.Kind)

+ continue

+ }

+ r := r

+ _ = s.withAppHost(r.Account, r.App, r.Host, func() error {

+ s.recoverOrphanLocked(ctx, r)

+ return nil

+ })

+ }

+ allocs, err := s.store.PendingAllocations()

+ if err != nil {

+ return fmt.Errorf("pending allocations: %w", err)

+ }

+ for _, a := range allocs {

+ a := a

+ _ = s.withAppHost(a.Account, a.App, a.Host, func() error {

+ s.recoverAllocationLocked(ctx, a)

+ return nil

+ })

+ }

+ return nil

+// recoverOrphanLocked is the orphan branch of the retirement table: a

+// successor slot voids it; the slot already retired by it commits it once

+// any body under the key is gone; the captured live slot (pred) runs O1;

+// anything else stays pending.

+func (s *Server) recoverOrphanLocked(ctx context.Context, r Retirement) {

+ if err := s.admit(r.Host, r.OpID); err != nil {

+ s.lifetimeLogHourly("retirement "+r.OpID, "console: lifetime recovery: retirement %s of %s waits: %v", r.OpID, r.Host, err)

+ return

+ }

+ obs, err := s.classifyLocked(ctx, r.Account, r.App, r.Host)

+ if err != nil {

+ s.lifetimeLogHourly("retirement "+r.OpID, "console: lifetime recovery: retirement %s of %s stays pending: %v", r.OpID, r.Host, err)

+ return

+ }

+ slot := obs.Slot

+ owned := slotOwnedBy(slot, r.Account, r.App)

+ succ := slot != nil && (slot.Incarnation > r.Incarnation || !owned)

+ pred := owned && slot.Incarnation == r.Incarnation && slot.State == provision.SlotLive && slot.Intent == r.SlotIntent

+ retired := owned && slot.Incarnation == r.Incarnation && slot.State == provision.SlotRetired

+ switch {

+ case succ:

+ _ = s.settleRetirement(r, lifetimeVoid, nil)

+ case retired:

+ if err := s.deleteUnderRetiredSlot(ctx, r, true); err != nil {

+ s.lifetimeLogHourly("retirement "+r.OpID, "console: lifetime recovery: retirement %s of %s: remove the body under the retired slot: %v (pending)", r.OpID, r.Host, err)

+ return

+ }

+ _ = s.settleRetirement(r, lifetimeCommitted, nil)

+ case pred:

+ _ = s.orphanStepsLocked(ctx, r)

+ default:

+ s.lifetimeLogHourly("retirement "+r.OpID, "console: lifetime recovery: retirement %s of %s: the slot is neither its own nor a successor (%+v); it stays pending", r.OpID, r.Host, slot)

+ }

+// recoverAllocationLocked is the allocation table, first matching branch:

+//

+// - unreadable: pending;

+// - a successor slot (a later incarnation, or N held by another key or

+// another intent): superseded;

+// - the app lifetime that authorised the allocation is gone (no apps row,

+// another lifetime, trashed, or the store's trash marker): the

+// allocation's own slot is retired as an orphan cancelling it, in one

+// transaction; with no slot of its own it is void;

+// - its slot and a record at N under this key: committed;

+// - its slot and a stale or corrupt body: step 5 on the current ETag, then

+// 6 and 7;

+// - its slot and no record: 6 and 7;

+// - no slot and no predecessor, or the predecessor still there: step 4

+// onward;

+// - anything else: pending, logged once an hour.

+//

+// A branch that wrote a step classifies again (bounded), so the next step is

+// chosen from what is there now, never from memory.

+func (s *Server) recoverAllocationLocked(ctx context.Context, a Allocation) {

+ logKey := "allocation " + a.OpID

+ if err := s.admit(a.Host, a.OpID); err != nil {

+ s.lifetimeLogHourly(logKey, "console: lifetime recovery: allocation %s of %s waits: %v", a.OpID, a.Host, err)

+ return

+ }

+ for round := 0; round < 3; round++ {

+ obs, err := s.classifyLocked(ctx, a.Account, a.App, a.Host)

+ if err != nil {

+ s.lifetimeLogHourly(logKey, "console: lifetime recovery: allocation %s of %s stays pending: %v", a.OpID, a.Host, err)

+ return

+ }

+ slot := obs.Slot

+ if allocationSuperseded(slot, a) {

+ _ = s.settleAllocation(a, lifetimeSuperseded, nil)

+ return

+ }

+ alive, err := s.allocationAppAlive(ctx, a)

+ if err != nil {

+ s.lifetimeLogHourly(logKey, "console: lifetime recovery: allocation %s of %s: read the app: %v (pending)", a.OpID, a.Host, err)

+ return

+ }

+ ours := allocationOwnsSlot(slot, a)

+ if !alive {

+ if ours {

+ if err := s.retireSlotOrphanLocked(ctx, a.Host, slot, obs.SlotETag, a.OpID); err != nil && !errors.Is(err, errLifetimeSettling) {

+ s.lifetimeLogHourly(logKey, "console: lifetime recovery: allocation %s of %s: cancel it: %v", a.OpID, a.Host, err)

+ }

+ return

+ }

+ _ = s.settleAllocation(a, lifetimeVoid, nil)

+ return

+ }

+ switch {

+ case ours && obs.Present && obs.Rec != nil && !obs.Rec.Retired && obs.Rec.Incarnation == a.Incarnation:

+ _ = s.settleAllocation(a, lifetimeCommitted, nil)

+ return

+ case ours && obs.Present:

+ if err := s.allocCleanLocked(ctx, a, obs.ETag, createPlan{}); err != nil {

+ return

+ }

+ _, _ = s.allocRecordLocked(ctx, a)

+ return

+ case ours:

+ _, _ = s.allocRecordLocked(ctx, a)

+ return

+ case (slot == nil && a.PredecessorETag == "") || (slot != nil && obs.SlotETag == a.PredecessorETag):

+ if err := s.allocSlotLocked(ctx, a); err != nil {

+ return

+ }

+ continue // the slot is ours now: on from what is there

+ }

+ s.lifetimeLogHourly(logKey, "console: lifetime recovery: allocation %s of %s: the slot is %+v and the record %v; it stays pending", a.OpID, a.Host, slot, obs.Present)

+ return

+ }

+// allocationAppAlive reports whether the app lifetime that authorised a is

+// still the app's: its apps row exists under the same lifetime, untrashed,

+// and the store has no trash marker for it. A read that fails is an error,

+// never an answer.

+func (s *Server) allocationAppAlive(ctx context.Context, a Allocation) (bool, error) {

+ acct, err := s.store.AccountBySqid(a.Account)

+ if err != nil {

+ return false, err

+ }

+ if acct == nil {

+ return false, nil

+ }

+ life, err := s.store.AppLife(acct.ID, a.App)

+ if err != nil {

+ return false, err

+ }

+ marker, err := release.ReadAppTombstone(ctx, s.cfg.Releases, a.Account, a.App)

+ if err != nil {

+ return false, err

+ }

+ return life != nil && life.Lifetime == a.AppLifetime && !life.Trashed() && marker == nil, nil

diff --git a/internal/console/server.go b/internal/console/server.go

index cb45d66f7..f8e94f6b3 100644

--- a/internal/console/server.go

+++ b/internal/console/server.go

@@ -1144,6 +1144,11 @@ type Server struct {

// (applocks.go). Zero values are ready.

appLocks keyedLocks

acctLocks keyedLocks

+ // hostLocks is withHostname's mutex per hostname (lifetimes.go), and

+ // lifetimeLogged the last time each hourly lifetime log line was

+ // written. Zero values are ready.

+ hostLocks sync.Map

+ lifetimeLogged sync.Map

// relayNow is the relay paths' clock seam: nil means time.Now. Tests

// drive lease expiry with it; nothing in production sets it.

relayNow func() time.Time

diff --git a/internal/console/testdata/claim-routes.txt b/internal/console/testdata/claim-routes.txt

index 722422057..65b2dc9cb 100644

--- a/internal/console/testdata/claim-routes.txt

+++ b/internal/console/testdata/claim-routes.txt

@@ -293,6 +293,7 @@ worker StartExpiryPass

worker StartGeoipJobs

worker StartHostReports

worker StartInfraPass

+worker StartLifetimeRecoveryPass

worker StartLifetimesCutoverWatch

worker StartMailEventPush

worker StartPlanLimitsPass

diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go

index 1c15e4713..f4b46b3e6 100644

--- a/internal/provision/plan14_root_reads_test.go

+++ b/internal/provision/plan14_root_reads_test.go

@@ -1268,7 +1268,9 @@ func TestPlan14RootReadCensus(t *testing.T) {

}

// Task 0 (the lifetimes cutover) re-pins four accepted-legacy console

// units, which moves the seal; still rev3=PROPOSED-self-serve-moves-1.

- seal = "d77a71da099f1c0477430fb08a9c6f505319bfd9e72d929938e712754c693f1f"

+ // Task 3 (the create protocol) re-pins serve and Server.hostHolder and

+ // adds newLifetimeOpID, which moves it again.

+ seal = "a11aea6699d2f113c315e8cacb6d2a6708395db567db08f74f215191f8958cd0"

errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)

errs = append(errs, checkRootReadDeclarations(got, inv)...)

if len(errs) > 0 {

diff --git a/internal/provision/self_serve_moves1_census_test.go b/internal/provision/self_serve_moves1_census_test.go

index 9e90d1dcb..40f29ff70 100644

--- a/internal/provision/self_serve_moves1_census_test.go

+++ b/internal/provision/self_serve_moves1_census_test.go

@@ -45,6 +45,35 @@ package provision

// - Server.registerCLIRoutes: key-site as before; registers the cutover door.

// - NewServer: key-site as before; the judge, the gate and its first marker

// read; claims-off app materialisation moves to the cutover watch. No write.

+//

+// Task 3 (the create protocol and its recovery), reviewed dispositions (all

+// PROPOSED):

+// - apiinstances.go:Server.handleAPIInstanceCreate: record-writer as before

+// through CreateRecord for a unit admitted before the cutover; a unit

+// admitted on dispatches to createRecordWithLifetime, and the leftover

+// rail's existence read is judged. No new write site.

+// - domainattach.go:Server.attachDomainAs: record-writer as before through

+// CreateRecord on the legacy path; a unit admitted on leaves for

+// attachFirstRouteWithLifetime. No new write site.

+// - apiinstanceseed.go:Server.beginFrom, apiinstanceimport.go:

+// Server.beginImport: no write site any more; each calls

+// createSeedRecord.

+// - apiinstanceseed.go:Server.createSeedRecord: record-writer; the seed

+// doors' first record: createRecordWithLifetime admitted on, else

+// today's CreateRecord, create-only.

+// - lifetimes_create.go:Server.allocRecordLocked: record-writer; steps 6

+// and 7: one create-only WriteRecord of the allocation's persisted body

+// from {} with the record sequence on (record_seq introduced), then the

+// SQLite commit. No raw Put.

+// - lifetimes_create.go:Server.fenceRecord: record-writer; WriteRecord of

+// the exact bytes observed against their own ETag with the sequence on,

+// so record_seq is introduced or advanced; never another body.

+// - lifetimes_create.go:Server.allocCleanLocked: key-site; DeleteIfMatch of

+// a stale body on the ETag the allocation recorded or observed, never a

+// plain delete; no Put.

+// - lifetimes_create.go:Server.deleteUnderRetiredSlot: key-site; reads a

+// body under a retired slot and DeleteIfMatch-es it on that ETag, never

+// a plain delete; no Put.

func plan14SelfServeMoves1WriteCensusInventory() []censusEntry {

removed := []struct{ file, unit string }{

{"internal/provision/provision.go", "GetRecordETag"},

@@ -56,6 +85,10 @@ func plan14SelfServeMoves1WriteCensusInventory() []censusEntry {

{"internal/console/apiinstanceseed.go", "Server.runFrom"},

{"internal/console/authcli.go", "Server.registerCLIRoutes"},

{"internal/console/server.go", "NewServer"},

+ {"internal/console/apiinstances.go", "Server.handleAPIInstanceCreate"},

+ {"internal/console/domainattach.go", "Server.attachDomainAs"},

+ {"internal/console/apiinstanceimport.go", "Server.beginImport"},

+ {"internal/console/apiinstanceseed.go", "Server.beginFrom"},

}

added := []censusEntry{

{"internal/provision/provision.go", "updateRecordSeeding", "record-writer", "key:RecordKey,key:WriteRecord", "20e3b0498fb8d410c3d10bc7c788b0eec137cf58aa4dfa2afb42da837ebd71ea"},

@@ -72,6 +105,13 @@ func plan14SelfServeMoves1WriteCensusInventory() []censusEntry {

{"internal/console/apiinstanceseed.go", "Server.runFrom", "record-writer", "key:SetRecordSeed", "0f0ed34b5a5c99cc7f8245f2e8d6284c300b4c7314b312af69f96d6d49f0d4c0"},

{"internal/console/authcli.go", "Server.registerCLIRoutes", "key-site", "lit:instances,lit:sidecars", "59882a91226ccba287af507482bea1aba49d6cab94fd32de0f2d586276ae2f92"},

{"internal/console/server.go", "NewServer", "key-site", "lit:instances", "737b2f91242d1a6750de07b43ea0c72bc0bc24365211e8a6abf7b3307126f5e8"},

+ {"internal/console/apiinstances.go", "Server.handleAPIInstanceCreate", "record-writer", "key:CreateRecord", "328f0b07e8b52250c5c2a58aa67d52c53cc2a51b2fd891836c061c6cff8ce2a7"},

+ {"internal/console/domainattach.go", "Server.attachDomainAs", "record-writer", "key:CreateRecord", "205df195585b26df17f51b5d9ac5006a308104612df794d91208a6a040bfaf59"},

+ {"internal/console/apiinstanceseed.go", "Server.createSeedRecord", "record-writer", "key:CreateRecord", "a6fabf99b13e7e2302dbf2c30fb7c782f76f787141a5cc12580fde59805dfd4a"},

+ {"internal/console/lifetimes_create.go", "Server.allocRecordLocked", "record-writer", "key:RecordKey,key:WriteRecord", "e04927e2e52a0362da469199c04735fa3bfd926c1c898e63b4370a8d79f46daa"},

+ {"internal/console/lifetimes_create.go", "Server.fenceRecord", "record-writer", "key:RecordKey,key:WriteRecord", "33d320aac193a0878b668054ea3b0d6872665c86851efa01d9e0d2532a902162"},

+ {"internal/console/lifetimes_create.go", "Server.allocCleanLocked", "key-site", "key:RecordKey", "035cf56d8f69517b66dc8ea6f9121d463e482d7dd05daf2b0c494aef1255f522"},

+ {"internal/console/lifetimes_create.go", "Server.deleteUnderRetiredSlot", "key-site", "key:RecordKey", "d0c8412a40e6f00d5db66e301fd108a7d5cddbfbe6dd80a93f02ac31f7cd24a0"},

}

entries := plan14ClaimRosterBindingWriteCensusInventory()

for _, r := range removed {

diff --git a/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv b/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv

index b14faf9f5..cc7abecd4 100644

--- a/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv

+++ b/internal/provision/testdata/plan14-root-reads-self-serve-moves-1.tsv

@@ -45,13 +45,13 @@ internal/console/store_gate.go claimSQLCensus legacy accepted-legacy accepted-le

@remove internal/console/tempstore.go Store.CreateTemporaryAppInRegion

@removed internal/console/tempstore.go Store.CreateTemporaryAppInRegion H3 census-ruling-PROPOSED-self-serve-moves-1;rev3=PROPOSED-self-serve-moves-1;basis=constructor remains declared but has no direct read; app insertion and lifetime minting moved to insertAppRow, classified above; proposed removal only

# Task 2 envelopes, rev3=PROPOSED-self-serve-moves-1: internal/console covers the lifetime store, app constructors, lifetime projection and table-scoped gate labels; internal/provision (above) adds only ErrLifetimeExhausted beyond Task 1. Registry fields are unchanged.

-@coverage internal/console 91dfd69fe16f100052635575cac99818d9bd89c8b477c481ef44663524ea6311

+@coverage internal/console 6f7b6ec4fede81a3a6b4bdc64d6d6028400bccca3d6f4b3850053dfd25497738

# Task 6a: rev3=PROPOSED-self-serve-moves-1; reviewed provision envelope now includes bounded record-then-slot observations, judged read/list/update gates and signed-counter exhaustion at slot read. No removal-loop, host or console change; registry fields and the legacy seal stand.

internal/provision/rootrecord.go IndexRecords authority B3,C rev3=PROPOSED-self-serve-moves-1;basis=retain every discovered candidate owner key per host for fresh lifetime judgement at Find; partial discovery still retries and no verdict or body is cached e44ac431865bf38baa4105f66c9cb23014cc3c7ffcd41d5385244cc76fc4c0b9

internal/provision/rootrecord.go RecordIndex.Find authority B3,C rev3=PROPOSED-self-serve-moves-1;basis=observe each candidate through the context judge, refuse corruption and read failures, require one valid owner and validate its schema and shape; nil-judge duplicates retain permission refusal 760eed59cbd6a8ddab1098ce3a5f7a4272ecda81c17cc079dc135aba3a85a8f8

# Task 0 (the lifetimes cutover), rev3=PROPOSED-self-serve-moves-1: four changed accepted-legacy console units re-pinned with reviewed bases (serve starts the cutover watch; NewServer adds the judge, the gate and its first marker read and moves claims-off app materialisation to the watch; the instance listing reads judged under an admitted-on unit; runFrom ends its handed-over unit); system.go runSystem and systemVerb re-pinned for the lifetimes verb at their existing authority; envelopes for ., cmd/console and internal/console (refreshed above) re-pinned. Four new legacy fingerprints need a new seal, checked after the preceding one. No root read, class or task is added; registry fields stand.

-@legacy d5a330a59973d7a46729154b5b2a1ee79d06622f9d7d9fe08f31bec85ab7c935 cmd/console/main.go serve legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=the console (cmd/console serve under console.service, user carlos-console): Task 0 starts the lifetimes cutover watch beside the other claim workers; the same three data path literals move; no privileged reader is added data-literal@4077,data-literal@5178,data-literal@5415 9cc6c0cbc9da08f9d4466069251d8028710ca2fa85a01be81dae7ce47bd2e4d9

-cmd/console/main.go serve legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=the console (cmd/console serve under console.service, user carlos-console): Task 0 starts the lifetimes cutover watch beside the other claim workers; the same three data path literals move; no privileged reader is added data-literal@4077,data-literal@5178,data-literal@5415 9cc6c0cbc9da08f9d4466069251d8028710ca2fa85a01be81dae7ce47bd2e4d9

+@legacy db1053faf4844420750a6ae9ada80386f31cb1065f0960175212941bbd908229 cmd/console/main.go serve legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=the console (cmd/console serve under console.service, user carlos-console): Task 0 starts the lifetimes cutover watch and Task 3 the lifetime recovery pass beside the other claim workers; the same three data path literals move; no privileged reader is added data-literal@4077,data-literal@5178,data-literal@5415 115729fb838b72ec095cdbdb0ceedacb8f7021ac1f0bbca3f89f32ff5092a887

+cmd/console/main.go serve legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=the console (cmd/console serve under console.service, user carlos-console): Task 0 starts the lifetimes cutover watch and Task 3 the lifetime recovery pass beside the other claim workers; the same three data path literals move; no privileged reader is added data-literal@4077,data-literal@5178,data-literal@5415 115729fb838b72ec095cdbdb0ceedacb8f7021ac1f0bbca3f89f32ff5092a887

@legacy 0b23119c935fb39171bb2787d8725ca0eb7736c067eab08b07236df0a23c6596 internal/console/apiinstances.go Server.handleAPIInstanceList legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=API listing reads through ListRecordsCompleteJudged with the request judge, which is nil before the lifetimes cutover; the same SQLite query moves; no root settings decision occurs here file:Query@1830 ac702aaa6da4c5a6ed7bb69427ba03ec3f2ee7753cce57c58349a978bbdcfadf

internal/console/apiinstances.go Server.handleAPIInstanceList legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=API listing reads through ListRecordsCompleteJudged with the request judge, which is nil before the lifetimes cutover; the same SQLite query moves; no root settings decision occurs here file:Query@1830 ac702aaa6da4c5a6ed7bb69427ba03ec3f2ee7753cce57c58349a978bbdcfadf

@legacy 992541f409e55b7643b6d49ab5e583b87d7bb4ab23c7d86c60e89ebfa07287d6 internal/console/apiinstanceseed.go Server.runFrom legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=the detached half of the --from door, unchanged, now ends the request unit it was handed after its last write; the same scratch Stat moves; every failure still moves the record to aborted file:Stat@3142 0f0ed34b5a5c99cc7f8245f2e8d6284c300b4c7314b312af69f96d6d49f0d4c0

@@ -61,4 +61,9 @@ internal/console/server.go NewServer legacy accepted-legacy accepted-legacy;rev3

system.go runSystem authority B,C,D,F,G,J4 rev3=PROPOSED-self-serve-moves-1;identity=explicit operator CLI uid 0;basis=adds the lifetimes family to the system dispatch (carlos system lifetimes cutover, a console door call as the operator); every existing command branch unchanged 8276403c56bc08639e62dee3a348573707c6f5b6909c8b1423bdc21e900390c1

system.go systemVerb authority B,C,D,F,G,J4 rev3=PROPOSED-self-serve-moves-1;identity=explicit operator CLI uid 0;basis=adds the lifetimes family to the system dispatch (carlos system lifetimes cutover, a console door call as the operator); every existing command branch unchanged 5952f7a80803ed5279dea42b3b2166e2d2d93ddd9fbe2e13ac8b1c00044bdb70

@coverage . fc604d5a74c58e761759747a2a4b8f166c51576eb4c457c55370fa2d678f4bd1

-@coverage cmd/console 3d40f88dab5d2cb0eff35c526ef7c2a19fd3fb700dad33678e630e9e0ced10f5

+@coverage cmd/console dc88a1b2acc2da4c812cb9cc748409ae89798931bc59f5391edcb693b5ed9a52

+# Task 3 (the create protocol and its recovery), rev3=PROPOSED-self-serve-moves-1: serve re-pinned in Task 0's rows above (it now also starts the lifetime recovery pass); Server.hostHolder re-pinned at its root-route disposition (the holder read is judged under a unit admitted on); newLifetimeOpID classified like mintLifetime (crypto/rand for an operation id). The cmd/console and internal/console envelopes (above) are re-pinned over the create protocol, its recovery, the door routing and the attach/detach exclusion. Three new legacy fingerprints move the seal. Registry fields stand.

+@legacy 2230ae1fd590a6e3d76cf38fe3e8c308ed9b9ec8bf5c0875ca13db23d1ac6776 internal/console/apex.go Server.hostHolder legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=root-route: the registry route read is unchanged; the instance-record holder read under each other app is judged when the request was admitted with lifetimes on (a nil judge is GetRecord), so a body its slot no longer vouches for holds nothing registry:Account@363,registry:Account@383,registry:Account@454,registry:Account@474,registry:Account@524,registry:App@344,registry:App@440,whole:Route@330 5fb4c31472674d0f1d4232877d6dd860980381c9842ec0fea9084517034037f7

+internal/console/apex.go Server.hostHolder legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=root-route: the registry route read is unchanged; the instance-record holder read under each other app is judged when the request was admitted with lifetimes on (a nil judge is GetRecord), so a body its slot no longer vouches for holds nothing registry:Account@363,registry:Account@383,registry:Account@454,registry:Account@474,registry:Account@524,registry:App@344,registry:App@440,whole:Route@330 5fb4c31472674d0f1d4232877d6dd860980381c9842ec0fea9084517034037f7

+@legacy 44be6376317c5819fc902e851b039330aff78a8f2ddc5338e95054d863151c02 internal/console/lifetimes.go newLifetimeOpID legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=console.service user carlos-console, J4 root-input: reads crypto/rand for an independent 32-hex lifetime operation id (allocations and retirements); randomness grants no root authority file:Read@71 34364098aef13cf16819336874c65fe5aa38350ec6557cb0ab67fbc3596a0df3

+internal/console/lifetimes.go newLifetimeOpID legacy accepted-legacy accepted-legacy;rev3=PROPOSED-self-serve-moves-1;basis=console.service user carlos-console, J4 root-input: reads crypto/rand for an independent 32-hex lifetime operation id (allocations and retirements); randomness grants no root authority file:Read@71 34364098aef13cf16819336874c65fe5aa38350ec6557cb0ab67fbc3596a0df3