amadan
plan: self-serve moves 4, rev 3 (Astra plan 4 review 2: F1 to F13; Paul's cleanup-exclusion ruling: every residence change holds the hostname key in the reservation table)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+87
−81
docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md+87 −81
CI failed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok amadan.net/carlos/platform.git/launchctx 0.014s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (1 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (143 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 4.040s
ok github.com/carlosframework/platform/cmd/moveprobe 12.005s
ok github.com/carlosframework/platform/cmd/zddprobe 0.768s
ok github.com/carlosframework/platform/internal/activator 59.009s
ok github.com/carlosframework/platform/internal/agentsock 0.036s
ok github.com/carlosframework/platform/internal/alerting 0.005s
ok github.com/carlosframework/platform/internal/analytics 0.032s
ok github.com/carlosframework/platform/internal/appconfig 3.030s
ok github.com/carlosframework/platform/internal/apperr 0.188s
ok github.com/carlosframework/platform/internal/applog 0.314s
ok github.com/carlosframework/platform/internal/awsinv 0.006s
ok github.com/carlosframework/platform/internal/boxchoice 0.006s
ok github.com/carlosframework/platform/internal/boxid 0.955s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.011s
ok github.com/carlosframework/platform/internal/caps 0.024s
ok github.com/carlosframework/platform/internal/certwarm 0.003s
ok github.com/carlosframework/platform/internal/cgroup 0.778s
ok github.com/carlosframework/platform/internal/changelog 0.591s
ok github.com/carlosframework/platform/internal/claimactivation 1.226s
ok github.com/carlosframework/platform/internal/clamd 0.059s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.871s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.012s
ok github.com/carlosframework/platform/internal/dbverify 0.005s
ok github.com/carlosframework/platform/internal/dnsrec 0.005s
--- FAIL: TestFenceLanguagesAreDeclared (0.00s)
fences_test.go:32: cli.md:2345: fence declares no language
fences_test.go:32: cli.md:2351: fence declares no language
FAIL
FAIL github.com/carlosframework/platform/internal/docsite 0.015s
ok github.com/carlosframework/platform/internal/domains 2.910s
ok github.com/carlosframework/platform/internal/doorkey 0.007s
ok github.com/carlosframework/platform/internal/economics 0.006s
ok github.com/carlosframework/platform/internal/edge 45.637s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.009s
ok github.com/carlosframework/platform/internal/fleetapi 0.184s
ok github.com/carlosframework/platform/internal/gate 1.868s
ok github.com/carlosframework/platform/internal/genselect 0.021s
ok github.com/carlosframework/platform/internal/geo 0.018s
ok github.com/carlosframework/platform/internal/geoip 0.043s
ok github.com/carlosframework/platform/internal/historyreadback 0.269s
ok github.com/carlosframework/platform/internal/host 139.888s
ok github.com/carlosframework/platform/internal/hostactors 6.413s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.009s
ok github.com/carlosframework/platform/internal/hostbox 3.301s
ok github.com/carlosframework/platform/internal/hostclaim 0.960s
ok github.com/carlosframework/platform/internal/hostenv 0.009s
ok github.com/carlosframework/platform/internal/hostevidence 0.011s
ok github.com/carlosframework/platform/internal/hostexec 5.827s
ok github.com/carlosframework/platform/internal/hostfence 0.047s
ok github.com/carlosframework/platform/internal/hostfiles 10.384s
ok github.com/carlosframework/platform/internal/hostfloor 0.028s
ok github.com/carlosframework/platform/internal/hostguard 1.043s
ok github.com/carlosframework/platform/internal/hostops 2.303s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.006s
ok github.com/carlosframework/platform/internal/hostquarantine 0.007s
ok github.com/carlosframework/platform/internal/hostrelocate 0.717s
ok github.com/carlosframework/platform/internal/hostresolve 0.260s
ok github.com/carlosframework/platform/internal/hostwrite 0.015s
ok github.com/carlosframework/platform/internal/infraview 0.006s
ok github.com/carlosframework/platform/internal/instancectl 5.267s
ok github.com/carlosframework/platform/internal/instanceexec 2.873s
ok github.com/carlosframework/platform/internal/instancefs 0.021s
ok github.com/carlosframework/platform/internal/instancekey 0.005s
ok github.com/carlosframework/platform/internal/instanceupdates 0.020s
ok github.com/carlosframework/platform/internal/layout 2.072s
ok github.com/carlosframework/platform/internal/lbhealth 0.005s
ok github.com/carlosframework/platform/internal/lease 10.154s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.578s
ok github.com/carlosframework/platform/internal/ledger 0.016s
ok github.com/carlosframework/platform/internal/mail 0.269s
ok github.com/carlosframework/platform/internal/maintenance 0.094s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.584s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.095s
ok github.com/carlosframework/platform/internal/metrics 0.007s
ok github.com/carlosframework/platform/internal/moves 21.812s
ok github.com/carlosframework/platform/internal/moves/archive 1.140s
ok github.com/carlosframework/platform/internal/objstore 0.312s
ok github.com/carlosframework/platform/internal/ops 0.199s
ok github.com/carlosframework/platform/internal/park 0.348s
ok github.com/carlosframework/platform/internal/plans 0.005s
ok github.com/carlosframework/platform/internal/privatefile 0.003s
ok github.com/carlosframework/platform/internal/provision 13.791s
ok github.com/carlosframework/platform/internal/publicrelease 0.007s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.755s
ok github.com/carlosframework/platform/internal/relay 7.255s
ok github.com/carlosframework/platform/internal/release 25.437s
ok github.com/carlosframework/platform/internal/rollout 3.883s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.008s
ok github.com/carlosframework/platform/internal/rootparticipation 0.059s
ok github.com/carlosframework/platform/internal/rootstate 0.031s
ok github.com/carlosframework/platform/internal/runs 0.574s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.801s
ok github.com/carlosframework/platform/internal/seeds 0.015s
ok github.com/carlosframework/platform/internal/sim/hostlife 11.705s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.917s
ok github.com/carlosframework/platform/internal/status 0.069s
ok github.com/carlosframework/platform/internal/steering 0.011s
ok github.com/carlosframework/platform/internal/stopevidence 0.214s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.010s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.006s
ok github.com/carlosframework/platform/internal/usage 0.016s
ok github.com/carlosframework/platform/internal/version 0.004s
ok github.com/carlosframework/platform/internal/vet 1.687s
FAIL
=== internal_console@0 (90 s)
ok github.com/carlosframework/platform/internal/console 88.112s
=== internal_console@1 (37 s)
ok github.com/carlosframework/platform/internal/console 34.876s
=== internal_console@2 (36 s)
2026/10/22 21:05:44 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:44 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:44 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:44 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:44 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:44 console: alert watch: no keymail grant for ops-b@keymail.dev — not notified (they must sign in to the console once)
2026/10/22 21:05:44 console: alert watch: unparsable alerts/boxes/box-junk.json: invalid character 'n' looking for beginning of object key string
2026/10/22 21:05:44 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:44 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: could not notify alice@keymail.dev of a CLI token issue: Post "http://127.0.0.1:46103/api/app/send": dial tcp 127.0.0.1:46103: connect: connection refused
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: app region bab/piped: (default)
2026/10/22 21:05:45 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: could not notify alice@keymail.dev of a CLI token issue: Post "http://127.0.0.1:36431/api/app/send": dial tcp 127.0.0.1:36431: connect: connection refused
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:46 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:46 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:05:46 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:46 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:05:46 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:46 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:46 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:46 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:46 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: door keys bab/titogo: overwriting an unverifiable object: invalid character 'o' in literal null (expecting 'u')
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: door keys bab/titogo: overwriting an unverifiable object: doorkey: bad signature
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: api features block bab/titogo: features at apps/bab/titogo/features.json: unknown schema 9
2026/10/22 21:05:47 console: api features get bab/titogo: features at apps/bab/titogo/features.json: unknown schema 9
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: import old.shop.example.com: panic in the upload of seed sd_arxjemblc67zivwrly4pqerf64: a fault in the check
goroutine 2323 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:26 +0x64
github.com/carlosframework/platform/internal/console.(*Server).runImportUpload.func2()
/var/lib/amadan-runner/w2/repos/carlos__platform/wt/design-self-serve-offline-moves/internal/console/apiinstanceimport.go:561 +0x48
panic({0x18a7e40?, 0x1db1f90?})
/usr/local/go/src/runtime/panic.go:860 +0x12c
github.com/carlosframework/platform/internal/console.TestImportUploadPanicAborts.func2({0x1dd6ce0?, 0x24651770a020?}, {0x1a3185c5000?, 0x6de68?})
/var/lib/amadan-runner/w2/repos/carlos__platform/wt/design-self-serve-offline-moves/internal/console/apiinstanceimport_test.go:976 +0x2c
github.com/carlosframework/platform/internal/console.(*Server).runImportUpload(0x2465170e8a88, 0x24651770d3b0)
/var/lib/amadan-runner/w2/repos/carlos__platform/wt/design-self-serve-offline-moves/internal/console/apiinstanceimport.go:568 +0x198
created by github.com/carlosframework/platform/internal/console.(*Server).handleAPIImportUpload in goroutine 2267
/var/lib/amadan-runner/w2/repos/carlos__platform/wt/design-self-serve-offline-moves/internal/console/apiinstanceimport.go:442 +0x8ec
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:47 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:48 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:49 console: api instance move bab/titogo/acme.go.tito.io: typed move write refused: hostops: identity mismatch
2026/10/22 21:05:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:49 console: api instance move bab/titogo/acme.go.tito.io: typed move write refused: hostops: identity mismatch
2026/10/22 21:05:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:49 console: api instance move bab/titogo/acme.go.tito.io: typed move write refused: hostops: identity mismatch
2026/10/22 21:05:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:50 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: offline move acme.shop.example.com: ignoring control/offline-moves/acme.shop.example.com/mv_zfffh2jqpvlrawt7onlwodkvvu/source/ie-x.json: not a report of this move
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:51 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 edges: box "legacy" has an edges file under control/edges/ but no status snapshot at all, so no region outside the home is automatically eligible until it reports; said once
2026/10/22 21:05:52 edges: 1 edges file(s) under control/edges/ but no fresh box snapshot matched any of them, so automatic region placement sees the home alone (are edges and status in different buckets? see CARLOS_STATUS_BUCKET/CARLOS_STATUS_PREFIX)
2026/10/22 21:05:52 console: an incomplete read of the edges files or box snapshots (edges complete true, snapshots complete false), so no region outside the home is automatically eligible this request; said once
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:05:52 console: from main.shop.example.com: plan: dbdoor: the last planned file's header does not describe it
2026/10/22 21:05:52 console: from main.shop.example.com: plan: file does not exist
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:05:52 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:05:52 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: could not notify paul@keymail.dev of a CLI token issue: Post "http://127.0.0.1:35321/api/app/send": dial tcp 127.0.0.1:35321: connect: connection refused
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:52 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: relay enabled bab/titogo in eu-west-1 by alice@keymail.dev (incarnation 199e9bbcd0154842, op 1)
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: api run bab/shop/acme.example.com run_oi5p3onou3da3z7apun2a2yiwq: delete after ack: delete refused
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:54 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: could not notify eve@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: could not notify owner@example.teſt of a CLI token issue: keymail: lookup owner@example.teſt: keymail: GET /api/federation/lookup?addr=owner%40example.te%C5%BFt&from=console%40127.0.0.1%3A45625: could not determine target mailbox
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: api store approvals: read store/entitlements/bab.json: objstore: store/entitlements/bab.json: invalid character 'o' in literal null (expecting 'u')
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: could not notify owner@example.teſt of a CLI token issue: keymail: lookup owner@example.teſt: keymail: GET /api/federation/lookup?addr=owner%40example.te%C5%BFt&from=console%40127.0.0.1%3A38345: could not determine target mailbox
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:55 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: api system antivirus: read: release: system/antivirus.json: schema 99 is not 1
2026/10/22 21:05:56 console: api system antivirus: replacing an unreadable record on disable: release: system/antivirus.json: schema 99 is not 1
2026/10/22 21:05:56 console: object scanning disabled deployment-wide by alice@keymail.dev
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: geoip upload: unexpected EOF
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: could not notify paul@keymail.dev of a CLI token issue: Post "http://127.0.0.1:46499/api/app/send": dial tcp 127.0.0.1:46499: connect: connection refused
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: bab/tmp-a is in the trash in the bucket; marking it trashed here
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: bab/tmp-b is in the trash in the bucket; marking it trashed here
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:05:56 console: api updates install bab/km/a.example.com: AccessDenied: s3://carlos-prod-secret-bucket/ledgers request id ABC123
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: api releases list bab/km: tags: AccessDenied: s3://carlos-prod-secret-bucket/tags request id DEF456
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: app region bab/console: (default)
2026/10/22 21:05:56 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:56 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:57 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:57 console: architecture refresh: console: infra inventory disabled for this process
2026/10/22 21:05:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:57 console: app region bab/hub-a-0: (default)
2026/10/22 21:05:57 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:57 console: app region bab/hub-a-1: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-2: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-3: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-4: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-5: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-6: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-7: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-8: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-9: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-10: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-11: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-12: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-13: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-14: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-15: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-16: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-17: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-18: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-19: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-20: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-21: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-22: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-23: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-24: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-25: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-26: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-27: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-28: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-29: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-30: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-31: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-32: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-33: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-34: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-35: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-36: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-37: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-38: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-39: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-40: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-41: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-42: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-43: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-44: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-45: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-46: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-47: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-48: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-49: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-50: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-51: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-52: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-53: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-54: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-55: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-56: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-57: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-58: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-59: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-60: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-61: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-62: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-63: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-64: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-65: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-66: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-67: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-68: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-69: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-70: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-71: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-72: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-73: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-74: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-75: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-76: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-77: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-78: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-79: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-80: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-81: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-82: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-83: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-84: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-85: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-86: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-87: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-88: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-89: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-90: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-91: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-92: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-93: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-94: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-95: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-96: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-97: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-98: (default)
2026/10/22 21:05:57 console: app region bab/hub-a-99: (default)
2026/10/22 21:05:57 console: app region bab/hub-b-fresh: (default)
2026/10/22 21:05:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:57 console: materialized account bac (Kass) from the bucket
2026/10/22 21:05:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:57 edges: skipping control/edges/uk-box.json in the placement join: not a versioned edges file
2026/10/22 21:05:57 edges: box "uk-box" has a fresh status snapshot but no edges file under control/edges/, so no region outside the home is automatically eligible until it publishes one; said once
2026/10/22 21:05:57 console: claim request refused: hostops: the placement read was incomplete, so no box could be judged
2026/10/22 21:05:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:57 console: claim request refused: hostops: the placement read was incomplete, so no box could be judged
2026/10/22 21:05:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:58 console: claim request refused: move room check: edges store down
2026/10/22 21:05:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:58 console: claim request refused: move room check: hostops: the placement read was incomplete, so no box could be judged
2026/10/22 21:05:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:58 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:58 console: backfilled account manifest for bab (ops)
2026/10/22 21:05:58 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:05:59 console: claim recovery: context canceled
2026/10/22 21:05:59 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:00 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:00 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:00 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:00 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 console: could not notify paul@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:01 console: api instances create bab/titogo: refused: acme.go.tito.io still holds replica data from a deleted record (#199); purge it to free the host
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:01 console: api sidecars create bab/titogo: refused: bot.titogo.bab.sidecar.internal still holds replica data from a deleted record (#199); purge it to free the host
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:01 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:01 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:01 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: materialized account bac (Testco) from the bucket
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: deploy signer for bab/hello: mint key grant: invalid name "deploy.bad name!" (letters, digits, . _ - ; up to 64)
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: attach story.carlosframework.com: existing record is not this door's to adopt (read: provision: record bab/story/story.carlosframework.com: provision: record is corrupt: provision: invalid maintenance projection or protected fields)
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: attach story.carlosframework.com: record write: opStore: timed out after the object landed
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: attach story.carlosframework.com: record write: opStore: bucket unavailable
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: could not notify alice@keymail.dev of a CLI token issue: keymail: grant revoked or invalid
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: edge proof: first signed request from the edge — X-Forwarded-For from loopback now needs a proof
2026/10/22 21:06:02 console: a loopback request carried X-Forwarded-For with no valid edge proof; counting it against the peer address. Expected now and then from a process on this box; on every request it means the edge no longer signs (rolled back?) and every client shares one rate limit
2026/10/22 21:06:02 console: a loopback request carried X-Forwarded-For with no valid edge proof; counting it against the peer address. Expected now and then from a process on this box; on every request it means the edge no longer signs (rolled back?) and every client shares one rate limit
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: expiry pass: bab/tmp-a is marked as the control plane; not purging it
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:02 console: expiry pass: bab/tmp-a has email estate; not purging it
2026/10/22 21:06:02 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: geoip feed test-feed: edition 2026-10 rejected: wrong type
2026/10/22 21:06:03 console: geoip feed status unreadable; this check's outcome was not recorded
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: instance store bab/shop/canary: seed sd_aaaaaaaaaaaaaaaaaaaaaaaaaa declaration not rolled back: the record was written since (its grant block or another writer), left as it is
2026/10/22 21:06:03 console: instance store provision bab/shop/canary: objstore: ensure user carlos-store-bab-shop-canary: iam is down
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: could not notify alice@keymail.dev of a CLI token issue: Post "http://127.0.0.1:44033/api/app/send": dial tcp 127.0.0.1:44033: connect: connection refused
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:03 console: materialized account bac (Kass) from the bucket
2026/10/22 21:06:03 console: materialized account bac (Testco) from the bucket
2026/10/22 21:06:03 console: reaped account bac (Testco) — its manifest is tombstoned
2026/10/22 21:06:03 console: materialized account bac (Testco) from the bucket
2026/10/22 21:06:03 console: bac/alpha is in the trash in the bucket — marking it trashed here
2026/10/22 21:06:03 console: bac/beta has no trash marker and still has objects — restoring it here
2026/10/22 21:06:03 console: bac/gamma is gone from the bucket — releasing the name here
2026/10/22 21:06:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:04 console: moves: warning: acme.go.tito.io: the log of move mv_retired at done is held: the writer refuses it (hostops: identity mismatch); its entry is kept until its incarnation is purged
2026/10/22 21:06:04 console: moves: warning: acme.go.tito.io: the log of move mv_earlier at done is held: the writer refuses it (hostops: identity mismatch); its entry is kept until its incarnation is purged
2026/10/22 21:06:04 console: audit instance-delete titogo: hostops: pending
2026/10/22 21:06:13 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: execute promote 1f4b488d53a1baeb985a83a3f23aca90: 1 of 2 confirmations counted (1 rows, quorum stored 2)
2026/10/22 21:06:14 console: execute promote 5f5f018d834ba9bc26647cca7875b080: change belongs to 1/"kass", not 1/"widget"
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: app region bab/bakeapp: (default)
2026/10/22 21:06:14 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:14 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:15 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:15 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:15 console: claim SQL receipt database moved from instance i-original to i-replacement; it has a new database ID, and effects bound to the old one stay unsettled
2026/10/22 21:06:15 console: claim SQL receipt database moved from instance i-replacement to i-original; it has a new database ID, and effects bound to the old one stay unsettled
2026/10/22 21:06:15 console: alert watch: no keymail grant for ops-b@keymail.dev — not notified (they must sign in to the console once)
2026/10/22 21:06:15 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:15 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:15 console: plan limits: bab: bucket unavailable
2026/10/22 21:06:15 console: plan limits: byv: bucket unavailable
2026/10/22 21:06:15 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:15 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:15 console: api instances create bab/titogo: incomplete edges or snapshot read, region uk unconfirmed
2026/10/22 21:06:15 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:15 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: relay enabled bab/titogo in eu-west-1 by alice@keymail.dev (incarnation 5847ed29766efc2e, op 1)
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: relay enabled bab/titogo in eu-west-1 by alice@keymail.dev (incarnation 7cfd3fe5a2e53112, op 1)
2026/10/22 21:06:16 console: relay rotated eu-west-1 to kid k2, re-issued 1 app(s)
2026/10/22 21:06:16 console: relay rotated eu-west-1 to kid k3, re-issued 1 app(s)
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: api instances list bab/kass: 1 record(s) unreadable [pi-host.example.test]; every fleet counts toward the rollout join
2026/10/22 21:06:16 status: fleet: skipping status/boxes/broken.json: corrupt snapshot: invalid character 'n' looking for beginning of object key string
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 status: fleet: skipping status/boxes/broken.json: corrupt snapshot: invalid character 'n' looking for beginning of object key string
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: scan drain: bab/kass: queue dropped — this store may no longer be scanned
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: a clamd address is configured but object scanning is not enabled — run `carlos system antivirus enable`
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:06:16 console: instance bab/shop/other.shop.example.com binds environment "canary": the store seed sd_im5r7fqdtmukevqrtynztutzju made (bucket carlos-bab-shop-canary) is no longer that seed's, and its finish leaves it standing
2026/10/22 21:06:16 console: finished seed sd_im5r7fqdtmukevqrtynztutzju of bab/shop/canary.shop.example.com
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:06:16 console: instance bab/shop/other.shop.example.com binds environment "canary": the store seed sd_25ip3l3l6s2m63wlypmdzvlosy made (bucket carlos-bab-shop-canary) is no longer that seed's, and its finish leaves it standing
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:06:16 console: finish seed sd_nt24qfylmsm4p4ks44yns37qtq of canary.shop.example.com: its replica is outside this console's data bucket; not deleting it
2026/10/22 21:06:16 console: finished seed sd_nt24qfylmsm4p4ks44yns37qtq of bab/shop/canary.shop.example.com
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:06:16 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:06:16 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: seeds pass: off on a host-claim console, where every seed door refuses
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/22 21:06:17 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/22 21:06:17 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/22 21:06:17 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/22 21:06:17 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/22 21:06:17 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/22 21:06:17 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/22 21:06:17 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/22 21:06:17 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/22 21:06:17 console: seeds pass: deleted seeds/canary.shop.example.com/sd_5omoeurtisrkrmmzyq66lvi67y/snapshot.db
2026/10/22 21:06:17 console: seeds pass: finish seed sd_5omoeurtisrkrmmzyq66lvi67y of canary.shop.example.com: finish seed sd_5omoeurtisrkrmmzyq66lvi67y of canary.shop.example.com: this console has no storage backend to tear its store down with
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:17 console: steering hostname "*.oncarlos.com" is not a valid domain — custom-domain guidance keeps the A-record form
2026/10/22 21:06:17 console: steering hostname "not a host" is not a valid domain — custom-domain guidance keeps the A-record form
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: fleet hosts: bucket list refused
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: ses events: signature verification failed: fake: signature does not verify
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: store decision bab: keymail to paul@keymail.dev: delivery failed
2026/10/22 21:06:17 console: store application bab: keymail to paul@keymail.dev: delivery failed
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:17 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: account telemetry bab: dirstore: invalid key "apps/bab//telemetry.json"
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
--- FAIL: TestEveryClosedDoorRefusesAServiceCredential (0.05s)
svcroutes_test.go:1011: walked 64 closed doors, want all 64 (and 63 today: instance-token, three door-key doors, the routes door, instances-manifest, the app update policy, sidecars-manifest, the route-channel move, three instance update settings doors, two db replica doors, six fleets, four services, two email credentials, email pause, the build probe, the swap policy, the live tenancy read, the layout read, the activation-cancel door, nine system doors, three deployment-wide antivirus, three platform geoip doors, two pointer digest doors, the channel resign, two release authorize doors, four store-entitlement doors, two account plan doors, and the five run doors)
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: could not notify paul@keymail.dev of a CLI token issue: Post "http://127.0.0.1:45101/api/app/send": dial tcp 127.0.0.1:45101: connect: connection refused
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: app region bab/four: (default)
2026/10/22 21:06:18 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: backfilled account manifest for bab (ops)
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/22 21:06:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
FAIL
FAIL github.com/carlosframework/platform/internal/console 34.249s
FAIL
=== internal_console@3 (40 s)
ok github.com/carlosframework/platform/internal/console 37.243s
=== internal_console@4 (44 s)
ok github.com/carlosframework/platform/internal/console 42.146s
=== internal_console@5 (37 s)
ok github.com/carlosframework/platform/internal/console 35.230s
=== internal_console@6 (42 s)
ok github.com/carlosframework/platform/internal/console 39.631s
=== internal_console@7 (41 s)
ok github.com/carlosframework/platform/internal/console 39.001s
=== root@0 (30 s)
ok github.com/carlosframework/platform 27.504s
=== root@1 (45 s)
ok github.com/carlosframework/platform 42.349s
=== root@2 (46 s)
ok github.com/carlosframework/platform 44.030s
FAILED: rest internal_console@2
make: *** [Makefile:19: test] Error 1
amadan: step gate failed: exit status 2
amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md| index ffd482dc8..46c41f140 100644 |
| --- a/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| +++ b/docs/superpowers/plans/2026-10-22-self-serve-moves-4-cleanup.md |
| @@ -2,145 +2,151 @@ |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| -Rev 2, 2026-10-22. Rev 1 was reviewed by Astra (`after236/selfserve-plan4-review.md`, NOT READY, 18 findings). Rev 2 answers them: the cleanup is its own operation with its own id, keyed to the move it follows; the four conditions are checked at **every** deletion and the exclusion that makes the delete-after-listing window safe is argued from the move's one-box-per-region rule, the cleanup reservation and durable admission; the obligation is bound to the retirement's incarnation and the move it followed; the 24-hour clock starts from the finish the box observed, never from the `gone` step; every key reaches a terminal outcome (deleted, marked, superseded) and the bucket step ends when all do; one restartable lifecycle with a dispatcher covers every retired obligation whatever its state; the status report publishes a complete, identified, lifecycle-aware summary that plan 2's reclaim rule can trust; the shared reservation helper has a typed signature with mode, clock and per-kind reclaim callbacks; root checks identity and cancellation at the unlink boundary under its own exclusion, derives the two canonical paths itself, persists progress and never drops a request silently; cleanup joins the hostname admission that creates and allocation recovery go through; production wiring is named; the build-reference predicate is corrected; every interface is typed with its wire names; a per-box reaper with completion evidence; one dispatch per pass with budgets; a durable cleanup report and persisted paging state. |
| +Rev 3, 2026-10-22. Rev 1 was reviewed by Astra (`after236/selfserve-plan4-review.md`, NOT READY, 18 findings); rev 2 (f1681c63) answered them and was reviewed (`selfserve-plan4-review2.md`, NOT READY, F1 to F13). Rev 3 answers F1 to F13 on the strength of **Paul's ruling of 2026-10-22** (amadan #343): every change of a hostname's residence goes through the per-box reservation table of plan 2. Concretely: the table gains **hostname keys** beside box keys, and the console's lifetime create (plan 1 Task 3), `carlos apps place` and the legacy move door, the self-serve and operator moves (plan 2) and this plan's cleanup all hold an entry on `host/<hostname>` for the duration of their durable work, so the exclusion is **durable and cross-process**, not a console mutex (F1, F2); condition (1) regains the region check (F1); a root request is preceded by a persisted `publishing` state and reconciled on restart (F3); a corrupt `retired.json` stays a **blocking obligation** that the status report names and never an absence, and the status contract is reconciled with plan 2's reclaim rule (F4); root's unlink has an intent/result journal, re-checks identity and cancellation immediately before `unlinkat` under its own exclusion, and derives its authority from the bucket (the `cleanup` reservation on `host/<hostname>` and the judged record), not from the request body (F5, F6); one ordered transition table with the original finishing move waiting and the fleet check before acquisition (F7); §7.6's process and reference checks are kept with the candidate link excluded (F8); the shared wire types live in a new leaf package `internal/offlinecleanup` and the root filesystem code is platform-tagged (F9); every park class persists an end observation and the reaper measures seven days from it, legacy parks kept (F10); a deadline-bearing cleanup context with per-operation bounds and a lock argument from the adopt entry point (F11); the remaining contract gaps are decided (F12); the report is published under the move id as well as the cleanup id and pending pages are durable per key (F13). |
| -Builds on plans 1 to 3: `withHostname`/`admit` (plan 1), `moves.Reserve`/`Release` with `(Kind, ID)` identity and the legacy projection (plan 2 Task 3), the self-serve finaliser (plan 2 Task 8), the box's `cfg.Versioned` arming (plan 2 Task 6), the box's offline steps. |
| +Builds on plans 1 to 3: `withHostname`/`admit`, the create protocol and allocation recovery (plan 1 Task 3), judged reads (plan 1 Task 6), `moves.Reserve`/`Release` with `(Kind, ID)` identity and the legacy projection (plan 2 Task 3, extended here with hostname keys), the self-serve finaliser (plan 2 Task 8), the box's `cfg.Versioned` arming (plan 2 Task 6), the box's offline steps. |
| -**Goal:** Twenty-four hours after a move the box observed as finished, the source box retires the old region's replica objects from evidence it recorded at finish, under a `cleanup` reservation, key by key with the four conditions re-read at each deletion, never deleting a version it did not inventory; root removes its own leftovers only through a request file it answers with `.done` or `.cancelled`; parked local state goes seven days after the obligation ends; nothing times out into a release. |
| +**Goal:** Twenty-four hours after a move the box observed as finished, the source box retires the old region's replica objects from evidence it recorded at finish, under a `cleanup` reservation on its own box and on the hostname, key by key with the four conditions re-read at each deletion, never deleting a version it did not inventory; root removes its own leftovers only through a request it answers with `.done` or `.cancelled` after checking the bucket's authority; parked local state goes seven days after its end observation; nothing times out into a release. |
| -**Architecture:** At `gone` the source box writes `retired.json` under the move's park directory with the exact bucket inventory (keys and current version ids under `instances/<host>/`, `lock.json` excluded) and the root-owned local paths with their identities. The box's moves pass runs a dispatcher over every `retired.json` on the box: it observes the finish, waits 24 hours, takes a `cleanup` reservation under the cleanup's own id, and deletes key by key, re-reading the record, the lifted tombstone, the registry and the object's current version before each plain `Delete`; it records a terminal outcome per key; then it writes `root-cleanup.json`, root answers `.done` or `.cancelled`, and the reservation is released. Supersession (a later move, a recreation, a deletion) ends the obligation and cancels any outstanding root request. The create door's admission refuses a hostname under cleanup. |
| +**Architecture:** At `gone` the source box writes `retired.json` under the move's op directory with the exact bucket inventory (keys and current version ids under the host's hibernate prefix, `lock.json` excluded) and the root-owned local paths with their identities. The box's moves pass runs a dispatcher over every `retired.json` on the box: it observes the finish, waits 24 hours, takes a `cleanup` reservation on `box/<own box>` and `host/<hostname>` under the cleanup's own id, and deletes key by key, re-reading the record (with its region), the slot, the lifted tombstone, the registry and the object's current version before each plain `Delete`; it records a terminal outcome per key; then it records `publishing`, writes `root-cleanup.json`, root verifies the reservation and the record in the bucket and answers `.done` or `.cancelled`, and the reservation is released. Supersession (a later move, a recreation, a deletion, a residence change) ends the obligation and cancels any outstanding root request. Because every residence change of the hostname holds `host/<hostname>` in the same table, none can run while cleanup holds it, and cleanup cannot begin while one does. |
| -**Tech Stack:** Go. `internal/moves` (`Retired`, the reservation helper), `internal/host` (the moves pass, the status report, the reaper, root's adopt pass), `internal/release` (root's request handler), `internal/lease` (`VersionedStore`, `MirrorStore`, `LockState`, `IsLiftedTombstone`), `internal/status`, `internal/console` (admission), `internal/alerting`. |
| +**Tech Stack:** Go. `internal/offlinecleanup` (new leaf package: the wire types), `internal/moves` (`Retired`, the reservation helper with hostname keys), `internal/host` (the moves pass, the status report, the reaper, root's adopt pass), `internal/release` (root's request handler), `internal/lease` (`VersionedStore`, `MirrorStore`, `LockState`, `IsLiftedTombstone`), `internal/status`, `internal/console` (admission and the residence-change doors), `internal/provision` (`SetRecordRegion` callers), `internal/alerting`. |
| -**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §11. |
| +**Spec:** `docs/superpowers/specs/2026-10-21-self-serve-offline-moves-design.md` §7.6, §7.1 (the `cleanup` reservation), §8 (the cleanup pages), §11. Deviation stated and ruled (Paul, 2026-10-22): placement and the legacy move door take a reservation, so §7.1's table serialises every residence change, not only moves. |
| ## Global Constraints |
| - Every data deletion is a plain `DeleteObject` (a delete marker); `VersionId` deletes are never used for cleanup; `lock.json` is never deleted. |
| -- **The four conditions at every deletion**, re-read immediately before each `Delete`: (1) the judged record of the retirement's `(Account, App, Host)` is present, at `Incarnation == Retired.Incarnation`, carries no move, and its `LastMove.ID == Retired.MoveID` with `Outcome == done`; (2) the source bucket's `instances/<host>/lock.json` is the lifted tombstone of `Retired.MoveID` (`lease.LockState` owner `== moves.TombstoneOwner(MoveID)` and `lease.IsLiftedTombstone`); (3) the box's registry has no row for the host; (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the inventory's `VersionID` as `IsLatest` and not a delete marker. A read error in any → stop this pass (the obligation stays `cleaning`). (1) failing because the record moved on → `superseded`. (4) failing because another version is current → that key `superseded`. |
| -- **The exclusion argument** (why the window between (4) and `Delete` is safe): the only writers to `instances/<host>/` in the source bucket are the source box's activator on a wake (it needs a registry row, condition (3), and a row needs a record naming this region, condition (1)); litestream in a running container (none: `Quiesced` and `Procs == 0` were reported at the move and no row exists); a move's stager (writes `moves/`, not `instances/`, and no move can begin while the `cleanup` reservation holds this box); and a create of the hostname into this region (refused by admission while the reservation holds, Task 5). A second box in the same region could wake the host without this box's row: cleanup therefore **refuses to run** while the fleet shows more than one box in the source region (the move's own one-box rule, re-checked each pass). Under these, no writer can produce a new current version between (4) and the delete. |
| +- **Hostname keys in the reservation table (the ruling).** `active.json`'s `Reservations` map holds two key forms: `box/<box id>` (plan 2's per-box entries, today's keys renamed by the normaliser) and `host/<hostname>`. An acquisition names a set of keys; it conflicts when any key is held by another `(Kind, ID)`. Kinds: `move` (plan 2), `cleanup` (this plan), `create` (plan 1's lifetime create: `{create, OpID}` on `host/<hostname>`, acquired in step 1 of the create protocol **before** `BeginAllocation` and released by the allocation's terminal state, recovery holding it meanwhile), `place` (`carlos apps place`, the browser placement, and the legacy move door's `MoveComplete`: `{place, opID}` on `host/<hostname>` and on the boxes of the regions involved for the duration of the record write, released after it). The sentinel `"*"` of plan 2 conflicts with every key. **No code path changes a record's `Region`, creates a record, or deletes a record's replica without holding `host/<hostname>`**; `TestEveryResidenceChangeHoldsTheHostnameKey` (an AST audit over `SetRecordRegion`, `OfflineFlip`, `MoveComplete`, `createRecordWithLifetime`, `cleanupDispatch` callers) pins it. |
| +- **The four conditions at every deletion**, re-read immediately before each `Delete`: (1) the judged observation of the retirement's `(Account, App, Host)` is present and valid, at `Incarnation == Retired.Incarnation`, with `Region == Retired.To` **or a region the record has since moved on to that is not `Retired.From`** (the spec's "names the destination or a later region"), carries no move, and `LastMove.ID == Retired.MoveID` with `Outcome == done`; **and** the hostname's slot is `{Retired.Incarnation, live, owned by the record}`; (2) the source bucket's `<hibernate prefix>/lock.json` is the lifted tombstone of `Retired.MoveID` (`lease.LockState` owner `== moves.TombstoneOwner(MoveID)` and `lease.IsLiftedTombstone`); (3) the box's registry has no row for the host; (4) `ListVersions(bucket, key)` filtered to `Key == key` shows the inventory's `VersionID` as `IsLatest` and not a delete marker. A read error in any → stop this pass (the obligation stays `cleaning`). (1) failing because the record moved on, was recreated or deleted → `superseded`. (1) failing because `Region == Retired.From` (a placement back) → `superseded` (the data is live again). (4) failing because another version is current → that key `superseded`; (4) with no latest object and no marker → that key `absent` (terminal). |
| +- **The exclusion argument** (why the window between (4) and `Delete` is safe), now a **lock argument**: every writer to the hibernate prefix needs the host to reside in this region, and every residence change of the hostname holds `host/<hostname>` in the same table cleanup holds it in: the lifetime create (`create`), placement and the legacy move (`place`), the moves (`move`). The activator's wake needs a registry row, which reconcile mints only from a record naming this region, which no path can produce while cleanup holds the key; litestream runs only in a woken container; the stager writes `moves/`, not the hibernate prefix, and cannot start (its move holds the box). Already-admitted work: a residence change that acquired `host/<hostname>` before cleanup finishes first (cleanup's acquisition is refused `busy` until the key is free), and a create admitted before cleanup acquired is refused at its durable step because its `create` acquisition is attempted **before** `BeginAllocation`, inside `withHostname`, so there is no admitted-but-unreserved create. A second box in the same region cannot wake the host without a row either; cleanup still refuses to run while the fleet shows more than one box in the source region (belt and braces; re-checked before acquisition and under it). |
| - The `cleanup` reservation is held from before the first deletion through root's `.done` or `.cancelled`; a request root never answers keeps it held and pages every 10 minutes; nothing releases on a timer. |
| -- Root removes a recorded path only when, at the unlink boundary and under its adopt exclusion, the entry's identity matches (device, inode, and for a link its target), the registry has no row for the host, and `.cancel` is absent. |
| -- The reservation helper lives in `internal/moves` (`Reserve`, `Release`) with typed options and errors; the console's `reserveBoxes`/`releaseBoxes` (plan 2) are thin wrappers. |
| -- Every deletion, skip and refusal is durable in `retired.json` (`Keys`) and published in the cleanup report under `control/offline-moves/<host>/<cleanup id>/cleanup/<box>.json`. |
| -- Census: the box's cleanup units and root's handler take rows in `plan14-root-reads-self-serve-moves-4.tsv`, loaded last by a new block at the end of `TestPlan14RootReadCensus`, `rev3=PROPOSED-self-serve-moves-4`, with rows for the actual readers (the record, the lock state, the registry, the versions listing, the request file) at the `authority` class with the evidence basis each needs, not the wrapper's label; both census tests green. |
| -- Copy rows (approved by Paul 2026-10-22; later additions pending): "the previous instance at this hostname is still being cleaned up. try again later". No tenant names in tests. |
| +- Root removes a recorded path only when, at the unlink boundary and under its adopt exclusion, the bucket shows `{cleanup, CleanupID}` holding `host/<hostname>` and the judged record away from this region, the entry's identity matches (device, inode, and for a link its target), the registry has no row for the host, and `.cancel` is absent; the identity is re-read immediately before `unlinkat` on the same directory handle. |
| +- The reservation helper lives in `internal/moves` (`Reserve`, `Release`) with typed options and results; the console's `reserveBoxes`/`releaseBoxes` (plan 2) are thin wrappers. |
| +- Every deletion, skip and refusal is durable in `retired.json` (`Keys` and `Refusals`) and published in the cleanup report under **both** `control/offline-moves/<host>/<move id>/cleanup/<box>.json` (so the move's log reader finds it) and `.../<cleanup id>/cleanup/<box>.json`. |
| +- Census: the box's cleanup units and root's handler take rows in `plan14-root-reads-self-serve-moves-4.tsv`, loaded by a new `loadRootReadInventory(..., true)` block in `internal/provision/plan14_root_reads_test.go` after every existing overlay and before `compareRootReads`, `rev3=PROPOSED-self-serve-moves-4`, with rows for the actual readers (the record, the slot, the lock state, the registry, the versions listing, the request file, root's reservation and record reads) at the `authority` class with the evidence basis each needs (`E2,J3,J4` where the destructive reader's obligations apply), not the wrapper's label; both census tests green. |
| +- Copy rows (pending Paul's review): "the previous instance at this hostname is still being cleaned up. try again later"; "cleanup of <host> waits for root"; "cleanup of <host> skipped <key>: another version is current". No tenant names in tests. |
| - Local tests: touched packages with `-run`; CI runs the full suite. |
| ## Review Focus |
| -1. The record changes **after** one deletion (a second move B→C, a recreation in the same region, a deletion): the next key's condition (1) fails, nothing more is deleted, the obligation is `superseded`, and an outstanding root request is cancelled (Task 2, `TestCleanupRefusesWhenTheRecordMovedAgainMidPass`). |
| -2. A new object version written after the listing and before the delete: not possible under the exclusion argument, and the test proves the argument's premises are checked: with a fake writer injected between (4) and `Delete`, the deleted version is the one listed and the fake's version becomes current again under the marker; and when a premise fails (a second box in the region, a row present) cleanup stops before any delete (Task 2, `TestExclusionPremisesAreCheckedEveryKey`). |
| -3. A delayed finaliser release of the move's reservation cannot touch the cleanup's (distinct `(Kind, ID)`); a cleanup reservation refuses a create of the hostname at durable admission, including a recovered pending allocation, and a move of it (Tasks 2 and 5, `TestCleanupExclusionIsMutualAndDurable`). |
| -4. Root never answers: the reservation stays, a page every 10 minutes with persisted cadence across restarts, no release; root answers after an hour → `.done`, release; a failed release retries; a failed page delivery is logged and retried (Task 4, `TestUnansweredRootRequestHoldsAndPages`). |
| -5. A cancel landing during root's checks, a root crash after one unlink, and `.done` winning the cancellation race: root deletes nothing after seeing `.cancel` at the boundary, `.cancelled` or `.done` names exactly what was removed (from the persisted progress), and the agent releases on either (Task 4, `TestCancelIsAcknowledgedBeforeRelease`). |
| +1. The record changes **after** one deletion (a second move B→C, a recreation in the same region, a deletion, a placement back to the source): the next key's condition (1) fails, nothing more is deleted, the obligation is `superseded`, and an outstanding root request is cancelled (Task 2, `TestCleanupRefusesWhenTheRecordMovedAgainMidPass`). |
| +2. Writer exclusion is durable: a lifetime create, a placement and a legacy move each attempted while cleanup holds `host/<hostname>` are refused at their durable step (409 / busy) and leave nothing behind; cleanup attempted while any of them holds the key is `busy`; a create paused **after** its pre-lock checks and before its `create` acquisition, with cleanup acquiring meanwhile, is refused; the fake-writer witness proves the writer is excluded, not that its version is hidden (Tasks 2 and 5, `TestResidenceChangesAndCleanupExclude`). |
| +3. A delayed finaliser release of the move's reservation cannot touch the cleanup's (distinct `(Kind, ID)`); a `create` entry held by a pending allocation blocks cleanup until recovery ends the allocation (Tasks 2 and 5, `TestCleanupExclusionIsMutualAndDurable`). |
| +4. Root never answers: the reservation stays, a page every 10 minutes with persisted cadence across restarts, no release; root answers after an hour → `.done`, release; a failed release retries; a failed page delivery is logged and retried; a request published but not recorded (crash) is reconciled on restart and cancelled under supersession (Tasks 2 and 4, `TestUnansweredRootRequestHoldsAndPages`, `TestPublishedRequestSurvivesTheCrash`). |
| +5. A cancel landing during root's checks, a root crash after one unlink, and `.done` winning the cancellation race: root deletes nothing after seeing `.cancel` at the boundary, `.cancelled` or `.done` names exactly what was **removed** (from the journal's `removed` lines, never from intents), an intent with no result is reported `unknown`, and the agent releases on either (Task 4, `TestCancelIsAcknowledgedBeforeRelease`, `TestRootCrashAfterOneUnlinkIsReportedFromProgress`). |
| --- |
| ### Task 1: `retired.json` at finish, the finish observation, and the status report's summary |
| **Files:** |
| -- Modify: `internal/moves/offline.go` (`Retired`, `OfflineOpDir`, `RetiredFile`), `internal/host/movesoffline.go` (`offlineSourceGone`: when the row is gone, before the `gone` report, write `retired.json` once, state `open`; a present file is never rewritten), `internal/host/movescleanup.go` (new: the dispatcher's first step, the **finish observation**: on each moves pass, for a `retired.json` with `FinishedAt` zero, the judged record cleared with `LastMove.ID == MoveID && Outcome == done` → `FinishedAt = LastMove.At` persisted; a record still in `offline-finishing` → wait; a record that moved on → `superseded` (Task 2)), `internal/host/agent.go` (the status tick fills `p.Retirements`, `p.RetirementsComplete`, `p.RetirementsErr`, `p.BoxID`), `internal/status/snapshot.go` (`Probes.Retirements []RetiredSummary`, `Probes.RetirementsComplete bool`, `Probes.RetirementsErr string`; `CollectBox`'s attachment condition includes `RetirementsComplete || len(Retirements) > 0 || RetirementsErr != ""`; the copies into `BoxMetrics`), `internal/host/fileid_linux.go` and `fileid_other.go` (`fileIdentity(path string) (moves.FileIdentity, error)` from `syscall.Stat_t` on Linux; `errors.ErrUnsupported` elsewhere) |
| -- Test: `internal/host/movesoffline_retired_test.go`, `internal/status/snapshot_test.go` |
| +- Create: `internal/offlinecleanup/wire.go` (the leaf package: `Retired`, `InventoryObject`, `KeyOutcome`, `Refusal`, `FileIdentity`, `LocalPath`, `RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `RetiredSummary`; imports only the standard library), `internal/host/fileid_linux.go`, `internal/host/fileid_other.go` |
| +- Modify: `internal/moves/offline.go` (`OfflineOpDir`, `RetiredFile`), `internal/host/movesoffline.go` (`offlineSourceGone`: when the row is gone, before the `gone` report, write `retired.json` once, state `open`; a present file is never rewritten), `internal/host/movescleanup.go` (new: the dispatcher's first step, the **finish observation**), `internal/host/agent.go` (the status tick fills `p.Retirements`, `p.RetirementsComplete`, `p.RetirementsErr`), `internal/status/snapshot.go` (`Probes.Retirements []offlinecleanup.RetiredSummary`, `Probes.RetirementsComplete bool`, `Probes.RetirementsErr string`; `CollectBox`'s attachment condition includes them; the copies into `BoxMetrics`, private) |
| +- Test: `internal/host/movesoffline_retired_test.go`, `internal/status/snapshot_test.go`, `internal/offlinecleanup/wire_test.go` |
| **Interfaces:** |
| -- `type moves.Retired struct { ID string \`json:"id"\` (a new id, `"cu_" + random`, the cleanup operation's identity); MoveID string \`json:"move_id"\`; Host string \`json:"host"\`; Account string \`json:"account"\`; App string \`json:"app"\`; Incarnation uint64 \`json:"incarnation"\`; From string \`json:"from"\`; To string \`json:"to"\`; OpenAt time.Time \`json:"open_at"\`; FinalTXID uint64 \`json:"final_txid"\`; TombstoneOwner string \`json:"tombstone_owner"\`; Bucket string \`json:"bucket"\`; BucketRegion string \`json:"bucket_region"\`; BucketInventory []InventoryObject \`json:"bucket_inventory"\`; LocalInventory []LocalPath \`json:"local_inventory"\`; WrittenAt time.Time \`json:"written_at"\`; FinishedAt time.Time \`json:"finished_at,omitzero"\`; State string \`json:"state"\`; CleaningSince time.Time \`json:"cleaning_since,omitzero"\`; ReservedAt time.Time \`json:"reserved_at,omitzero"\`; Keys map[string]KeyOutcome \`json:"keys,omitempty"\`; RootRequest string \`json:"root_request,omitempty"\`; EndedAt time.Time \`json:"ended_at,omitzero"\`; LastPagedAt time.Time \`json:"last_paged_at,omitzero"\`; Reason string \`json:"reason,omitempty"\` }`. |
| -- `State` ∈ `open` (written at `gone`), `finished` (the finish observed, waiting 24 h), `cleaning` (reserved, deleting), `bucket-done` (every key terminal), `root-requested`, `cancel-pending` (superseded while a root request was outstanding), `ended` (terminal; `Reason` is `cleaned` or `superseded`). `RootRequest` ∈ "", `pending`, `done`, `cancelled`. |
| -- `type InventoryObject struct { Key string \`json:"key"\`; VersionID string \`json:"version_id"\`; ETag string \`json:"etag"\`; Size int64 \`json:"size"\` }`; `type KeyOutcome struct { State string \`json:"state"\` (pending, deleted, marked, superseded); VersionID string \`json:"version_id,omitempty"\`; At time.Time \`json:"at,omitzero"\`; Detail string \`json:"detail,omitempty"\` }`; `type FileIdentity struct { Dev uint64 \`json:"dev"\`; Inode uint64 \`json:"inode"\`; LinkTarget string \`json:"link_target,omitempty"\` }`; `type LocalPath struct { Path string \`json:"path"\`; Identity FileIdentity \`json:"identity"\` }`. |
| -- `func OfflineOpDir(dataDir, host, moveID string) string` (`/data/.offline-moves/<host>/<moveID>/`, the parent of the role directories `OfflineParkDir` returns); `func RetiredFile(dataDir, host, moveID string) string` (`OfflineOpDir(...) + "retired.json"`). |
| -- Evidence at `gone`: the record is still in `offline-finishing` with `Move` present: `OpenAt = *Move.OpenedAt`, `Incarnation = rec.Incarnation`, `From/To` from the move, `FinalTXID` from the sealer's `sealed` report, `TombstoneOwner = moves.TombstoneOwner(MoveID)`, `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`; `BucketInventory` from `cfg.Versioned(ctx, bucket, region).ListVersions(ctx, bucket, "instances/<host>/")` keeping `IsLatest && !DeleteMarker`, `Key != ".../lock.json"`; `LocalInventory` from `fileIdentity` on `release.LiveLink(opt, host)` and `/data/programs/<host>.json` (a missing path omitted; an identity error → the `gone` step fails this pass, nothing written). A `retired.json` that does not decode is renamed `retired.json.corrupt` and paged once. |
| -- `type status.RetiredSummary struct { ID string \`json:"id"\`; MoveID string \`json:"move_id"\`; Host string \`json:"host"\`; State string \`json:"state"\`; RootRequest string \`json:"root_request,omitempty"\`; UpdatedAt int64 \`json:"updated_at"\` }`; `BoxMetrics.Retirements []RetiredSummary \`json:"retirements,omitempty"\``, `BoxMetrics.RetirementsComplete bool \`json:"retirements_complete,omitempty"\`` (every `retired.json` under `/data/.offline-moves/*/*/` was read), `BoxMetrics.RetirementsErr string \`json:"retirements_err,omitempty"\``. Plan 2's reclaim rule reads these: a `cleanup` reservation is reclaimable only when the owning box's fresh report (`Metrics.BoxID` equal to the entry's key) shows `RetirementsComplete` and either no entry with that id or an entry with `State == ended`; `ended` is written only after the release attempt (Task 2's lifecycle), so a report can never show a terminal state while a root request can still appear. |
| - |
| -- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGone` (fields and inventory as above; `lock.json` excluded; a second gone pass does not rewrite; an identity error writes nothing); `TestFinishObservedFromTheClearedRecord` (`FinishedAt` set from `LastMove.At` once the record clears; a record still finishing → unchanged; another `LastMove.ID` → superseded); `TestCorruptRetiredIsRenamedAndPaged`; `TestStatusRetirementsThroughTheRealPath` (`CollectBox → WriteBox → LoadBoxesComplete → BuildFleet → PublishFleet`: private only; `RetirementsComplete` false with `RetirementsErr` when a directory is unreadable). |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'RetiredJSON|FinishObserved|CorruptRetired|StatusRetirements' -count=1`, `go test ./internal/status -run 'Retirements' -count=1`. |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run to verify they pass.** |
| -- [ ] **Step 5: Commit** `moves: the source box records retired.json at gone and observes the finish; the status report summarises retirements with completeness and identity`. |
| - |
| -### Task 2: The shared reservation helper and the cleanup lifecycle |
| +- `type offlinecleanup.Retired struct { ID string \`json:"id"\` ("cu_" + 16 random hex); MoveID, Host, Account, App string; Incarnation uint64; From, To string; OpenAt time.Time; FinalTXID uint64 (from the immutable seal at `moves.OfflineSealKey(host, moveID)`, validated as the landing validates it; never from the mutable sealer report); TombstoneOwner string; Bucket, BucketRegion string; Prefix string (`activator.HibernatePrefix(cfg.OwnPrefix, host)`, persisted: the actual location, never recomputed); BucketInventory []InventoryObject; LocalInventory []LocalPath; WrittenAt, FinishedAt time.Time; State string; CleaningSince, ReservedAt time.Time; Keys map[string]KeyOutcome; Refusals []Refusal; RootRequest string; EndedAt, LastPagedAt time.Time; Reason string; PendingPages []PendingPage }` (every field tagged snake_case; times `omitzero`). |
| +- `State` ∈ `open`, `finished`, `cleaning`, `bucket-done`, `publishing`, `root-requested`, `cancel-pending`, `ended`, `corrupt` (a file that does not decode or fails validation: renamed `retired.json.corrupt` **and** replaced by a stub `retired.json` `{id (from the directory when the original cannot be read), state: corrupt, reason}`, so the obligation stays listed, blocks the hostname key through its reservation if one is held (the stub keeps `ReservedAt` when recoverable) and pages once per hour until an operator repairs it; never an absence). `RootRequest` ∈ "", `pending`, `done`, `cancelled`; it is **cleared to ""** when `State` becomes `ended`. |
| +- **Validation** (`offlinecleanup.ValidateRetired`): `ID` matches `^cu_[0-9a-f]{16}$`; `MoveID` non-empty; `Host` passes the host shape; `Account`, `App` non-empty; `Incarnation > 0`; `From != To`; `State` in the set; `WrittenAt` non-zero; `Prefix` non-empty and ends with `/`; inventory keys non-empty, unique, under `Prefix`, none `Prefix + "lock.json"`; `LocalInventory` paths among the two canonical ones; a failing validation is `corrupt`. |
| +- `type InventoryObject struct { Key, VersionID, ETag string; Size int64 }`; `type KeyOutcome struct { State string (pending, deleted, marked, superseded, absent); VersionID string; At time.Time; Detail string }`; `type Refusal struct { At time.Time; Condition int (1 to 4); Detail string }` (one entry per stop-the-pass refusal; capped at the last 50); `type FileIdentity struct { Dev, Inode uint64; Mode string (symlink or regular); LinkTarget string }`; `type LocalPath struct { Which string (live-link, programs-file); Path string; Identity FileIdentity }`; `type PendingPage struct { Condition string; Key string; Since time.Time; Attempts int }`. |
| +- `func OfflineOpDir(dataDir, host, moveID string) string`, `func RetiredFile(dataDir, host, moveID string) string`; `fileIdentity(path) (FileIdentity, error)` uses `Lstat` (never follows), records `Mode`, and `Readlink` for a symlink; a path that is absent is omitted; any other error fails the `gone` step this pass (nothing written). |
| +- Evidence at `gone`: the record is still in `offline-finishing` with `Move` present: `OpenAt = *Move.OpenedAt`, `Incarnation = rec.Incarnation`, `From/To` from the move, `FinalTXID` from the seal, `TombstoneOwner = moves.TombstoneOwner(MoveID)`, `Bucket/BucketRegion = cfg.OwnBucket/OwnBucketRegion`, `Prefix` as above; `BucketInventory` from `cfg.Versioned(ctx, bucket, region).ListVersions(ctx, bucket, Prefix)` keeping `IsLatest && !DeleteMarker` and `Key != Prefix + "lock.json"`; `LocalInventory` from `fileIdentity` on `release.LiveLink(opt, host)` and `filepath.Join(dataDir, "programs", host + ".json")`. The write is temp-file, `fsync`, rename, directory `fsync` (every state write below the same; `writeRetired` is the one writer). |
| +- **The finish observation** (`open` → `finished`): the judged record cleared with `LastMove.ID == MoveID && Outcome == done` → `FinishedAt = LastMove.At`; the record still carrying **this** move (`Move != nil && Move.ID == MoveID`, in `offline-finishing`) → wait (never supersession: the original move's finishing has precedence over the supersession rule); any other record state → `superseded` (Task 2's rule). |
| +- `type RetiredSummary struct { ID, MoveID, Host, State, RootRequest string; ReleasePending bool (a reservation is held: `ReservedAt` non-zero and `State != ended`); UpdatedAt int64 (the `retired.json`'s mtime at collection, unix seconds) }`; `BoxMetrics.Retirements`, `RetirementsComplete` (every `*/*/retired.json` and `*.corrupt` under the op root was read), `RetirementsErr`. **The reclaim contract with plan 2** (one statement, both plans): a `cleanup` reservation is reclaimable only when the owning box's fresh report (`Metrics.BoxID` equal to the entry's box key, `GeneratedAt` later than the reservation's `ReservedAt` plus one status interval, so the observation post-dates the reservation) shows `RetirementsComplete` and an entry with that id in `State == ended` with `ReleasePending == false`; an absent id is **never** evidence (a corrupt file is listed as `corrupt`); plan 2's Task 3 reclaim text is amended to this wording. |
| + |
| +- [ ] **Step 1: Write the failing tests**: `TestRetiredJSONWrittenOnceAtGone` (fields, `Prefix`, `FinalTXID` from the seal; `lock.json` excluded; a second gone pass does not rewrite; an identity error writes nothing; `Lstat` semantics for a dangling link); `TestFinishObservedFromTheClearedRecord` (`FinishedAt` from `LastMove.At`; the original move still finishing → unchanged; another `LastMove.ID` → superseded); `TestCorruptRetiredStaysAnObligation` (renamed, the stub listed as `corrupt`, paged hourly, never absent from the summary); `TestValidateRetired` (each rule); `TestStatusRetirementsThroughTheRealPath` (`CollectBox → WriteBox → LoadBoxesComplete → BuildFleet → PublishFleet`: private only; `RetirementsComplete` false with `RetirementsErr` when a directory is unreadable; `ReleasePending` and `UpdatedAt` as defined). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/host -run 'RetiredJSON|FinishObserved|CorruptRetired|StatusRetirements' -count=1`, `go test ./internal/offlinecleanup -count=1`, `go test ./internal/status -run 'Retirements' -count=1`. |
| +- [ ] **Step 3: Implement.** **Step 4: Run to verify they pass.** |
| +- [ ] **Step 5: Commit** `moves: the source box records retired.json at gone and observes the finish; the status report summarises retirements with completeness, identity and release state`. |
| + |
| +### Task 2: Hostname keys in the reservation table, the shared helper, and the cleanup lifecycle |
| **Files:** |
| - Create: `internal/moves/reserve.go`, `internal/host/movescleanup.go` (the dispatcher and the bucket step) |
| -- Modify: `internal/console/offlinereserve.go` (wrappers), `internal/host/movespass.go` (one `w.cleanupDispatch(ctx, s)` per moves pass, after the record loop and only when this box has no offline record in flight; budget 200 keys or 60 s per pass; `armOffline` arms `cfg.Versioned` (plan 2) and `cfg.PlainMirror func(ctx, bucket, region string) (lease.MirrorStore, error)` (the plain-delete mirror, `lease.NewMirrorS3`-shaped as `offlineMirror` builds it)) |
| +- Modify: `internal/moves/offline.go` (`OfflineReservation` unchanged; the key forms `box/<id>`, `host/<hostname>`, `"*"`; the normaliser renames legacy bare box ids to `box/<id>` on the first write), `internal/console/offlinereserve.go` (wrappers over `moves.Reserve`/`Release`; plan 2's selections add `host/<hostname>`), `internal/host/movespass.go` (one `w.cleanupDispatch(ctx, s)` per moves pass, after the record loop and only when this box has no offline record in flight, under a context with a 60 s deadline and per-operation 10 s bounds (`context.WithTimeout` around each read, list, delete and report write); fairness: obligations are visited round-robin from a persisted cursor (`cleanup-cursor` file), each getting at most 50 keys per pass; the offline report refresh is **not** blocked: the dispatcher releases `w.mu` between keys), `armOffline` arms `cfg.Versioned` (plan 2) and `cfg.PlainMirror func(ctx context.Context, bucket, region string) (lease.MirrorStore, error)` (built as `offlineMirror` builds it, through `lease.NewS3Mirror(ctx, srcRegion, dstRegion)` with both regions the source bucket's) |
| - Test: `internal/moves/reserve_test.go`, `internal/host/movescleanup_test.go` |
| **Interfaces:** |
| -- `type moves.ReserveOptions struct { Exclusive bool (refuse when any other operation holds anything: plan 2's switch-off mode); Now func() time.Time; Reclaim func(ctx context.Context, boxID string, held OfflineReservation) (bool, error) }`; `type ErrReserved struct { Holder OfflineReservation }`; `var ErrUnreadable = errors.New("moves: the reservation table cannot be read")`; `func Reserve(ctx context.Context, s release.Store, boxes []string, r OfflineReservation, opt ReserveOptions) error` (an own `(Kind, ID)` entry is a retry and keeps its `ReservedAt`; a legacy body is normalised; a malformed entry → `ErrUnreadable`; the projection; one conditional write; one CAS retry); `func Release(ctx context.Context, s release.Store, kind, id string) error` (its entries only; the projection rewritten; the object deleted when empty). |
| -- The console's reclaim callback is plan 2's rule. **The box's reclaim callback** refuses every `move` entry (it has no console state) and reclaims a `cleanup` entry only when `boxID == cfg.BoxID` and its own `retired.json` with that id is `ended` or absent. |
| -- `func (w *movesWorker) cleanupDispatch(ctx context.Context, s release.Store) error`: lists `/data/.offline-moves/*/*/retired.json`; for each, by `State`: |
| - - `open`: Task 1's finish observation → `finished`. |
| - - `finished`: `now - FinishedAt >= 24 h` → `Reserve([cfg.BoxID], {cleanup, r.ID, host, acct, app, now}, opts)` (busy → log once per hour); the fleet check (exactly one box publishes the source region, else stay with a page every hour); `ReservedAt`, `CleaningSince = now`, `State = cleaning`; every inventory key `Keys[key] = pending`. |
| - - `cleaning`: for each `pending` key in order, within the budget: the four conditions (a read error → stop the pass); (1) failing → `superseded` for the obligation (below); (3) failing → stop (a row appeared: an operator or a bug; page); (2) failing (a live lease, another owner, or no tombstone) → stop and page; (4): the listed current version equals the inventory's → `PlainMirror.Delete(bucket, key)` → `Keys[key] = {deleted, VersionID, now}`; a delete marker current → `marked`; another version current → `superseded` with its version id in `Detail`, paged once; every key terminal → `bucket-done`. |
| - - `bucket-done`: `LocalInventory` empty → `ended`, `Reason: cleaned` after the release; else write `root-cleanup.json` (Task 4), touch the stage marker, `RootRequest = pending`, `State = root-requested`. |
| - - `root-requested`: `.done` → `RootRequest = done` → release → `ended` (`cleaned`); `.cancelled` → `RootRequest = cancelled` → release → `ended`; neither and `now - LastPagedAt >= 10 min` → page ("cleanup of <host> waits for root"), `LastPagedAt = now`. |
| +- `type moves.ReserveOptions struct { Exclusive bool; Now func() time.Time; Reclaim func(ctx context.Context, key string, held OfflineReservation) (bool, error) }`; `type ErrReserved struct { Key string; Holder OfflineReservation }`; `var ErrUnreadable`; `func Reserve(ctx context.Context, s release.Store, keys []string, r OfflineReservation, opt ReserveOptions) (OfflineReservation, error)` (returns the entry as written or as already held by this `(Kind, ID)`, so a caller that crashed after acquiring recovers `ReservedAt` by calling again; an own retry must cover exactly the expected keys, as plan 2 says; a legacy body is normalised; a malformed entry → `ErrUnreadable`; the projection; one conditional write; one CAS retry); `func Release(ctx context.Context, s release.Store, kind, id string) error`. |
| +- The box's reclaim callback refuses every `move`, `create` and `place` entry (it has no console state) and reclaims a `cleanup` entry only when `key == "box/" + cfg.BoxID` (or the paired `host/` key of the same id) and its own `retired.json` with that id is `ended`; absent or `corrupt` → not reclaimable. `Exclusive` on the box comes from `cfg.SelfServeMoves != "offline"` read from the deployment manifest's console switch published in `control/console.json` (plan 2 writes it) and defaults to true when unreadable. |
| +- `func (w *movesWorker) cleanupDispatch(ctx context.Context, s release.Store) error`, the **ordered transition table** (the first matching row of each state applies; "the record" is the judged observation; `unreadable` anywhere → wait and log once per hour): |
| + - any non-terminal state, the record carries **this** move (`Move.ID == MoveID`) → wait (the original move is finishing; never supersession). |
| + - any non-terminal state, the record moved on (`Move != nil` with another id, `LastMove.ID != MoveID`, `Incarnation` differs, absent, stale, or `Region == From`) → **supersession**: `RootRequest ∈ {publishing, pending}` → write `.cancel`, `State = cancel-pending`; else release (if `ReservedAt` set) → `ended`, `Reason: superseded`, `RootRequest = ""`. |
| + - `open` → the finish observation (Task 1) → `finished`. |
| + - `finished`, `now - FinishedAt >= 24 h` → **the fleet check first** (exactly one box publishes the source region: else stay, page hourly) → `Reserve(["box/" + cfg.BoxID, "host/" + host], {cleanup, r.ID, host, acct, app, now}, opts)` (busy → stay, log once per hour) → `ReservedAt` from the returned entry, `CleaningSince = now`, `State = cleaning`, every inventory key `pending` (one write) → the fleet check **again under the reservation** (a change → release, stay `finished`, page). |
| + - `cleaning`: for each `pending` key in order within the budget: the four conditions (a read error → `Refusals` entry, stop the pass); (1) failing by supersession → the supersession row; (3) failing (a row appeared) → `Refusals`, stop, page (an intended indefinite refusal until an operator or supersession resolves it; the reservation stays); (2) failing (a live lease, another owner, no tombstone) → `Refusals`, stop, page (likewise); (4): the inventory's version is latest → `PlainMirror.Delete(bucket, key)` → `Keys[key] = {deleted, VersionID, now}` (persisted **before** the next key; a crash between the delete and the write → the next pass finds a marker current → `marked`, which is the same end state); a marker current → `marked`; another version current → `superseded` + a `PendingPage{skipped, key}`; no latest and no marker → `absent`; every key terminal → `bucket-done`. |
| + - `bucket-done`: `LocalInventory` empty → release → `ended` (`cleaned`); else `RootRequest = publishing` (persisted **first**) → write `root-cleanup.json` create-only (an existing file for this id → fine) → touch the stage marker → `RootRequest = pending`, `State = root-requested`. |
| + - `publishing` (found on restart): the request file exists → `pending`, `root-requested`; absent → write it, continue; supersession from `publishing` writes `.cancel` regardless (the request may exist). |
| + - `root-requested`: `.done` → validate the ack (`CleanupID`, `Host`, `MoveID` match; else `corrupt` ack → page hourly, stay) → release → `ended` (`cleaned`), `RootRequest = ""`; `.cancelled` → release → `ended`; neither and `now - LastPagedAt >= 10 min` → page ("cleanup of <host> waits for root"), `LastPagedAt = now` (persisted before the page is sent). |
| - `cancel-pending`: `.done` or `.cancelled` → release → `ended` (`superseded`); neither → page every 10 minutes. |
| - - **supersession** (from any non-terminal state): the record moved on (`Move != nil`, or `LastMove.ID != MoveID`, or `Incarnation` differs, or the record is absent or stale) → `RootRequest == pending` → write `.cancel`, `State = cancel-pending`; else release (if reserved) → `ended` (`superseded`). |
| - - `ended`: Task 3's reaper. |
| - Every state write is a temp-file rename of `retired.json`; `ReservedAt` and `CleaningSince` are never overwritten once set; a crash between any two steps resumes at the state found. The release is retried each pass until it succeeds (the state stays before `ended`). |
| -- The cleanup report: role `cleanup` added to `offlineRoles`, phases `cleaning`, `bucket-done`, `root-requested`, `ended`, `superseded`; `OfflineReport.Cleanup *CleanupReport{Keys map[string]KeyOutcome; RootRequest string}` rewritten each pass at `control/offline-moves/<host>/<cleanup id>/cleanup/<box>.json`. |
| + - `ended`: Task 3's reaper. `corrupt`: page hourly, nothing else. |
| + - **Pending pages** (`PendingPages`): a page is persisted before it is sent and removed after a successful send; a failed send increments `Attempts` and is retried next pass; pages survive terminal transitions and are drained before the reaper may remove the directory. |
| +- The cleanup report: role `cleanup` added to `offlineRoles`, phases `cleaning`, `bucket-done`, `root-requested`, `ended`, `superseded`, `corrupt`; `OfflineReport.Cleanup *offlinecleanup.CleanupReport{CleanupID string; MoveID string; Keys map[string]KeyOutcome; Refusals []Refusal; RootRequest string}` (tagged), validated by plan 2's `ValidateOfflineReportFor` matrix (role `cleanup` ↔ those phases with `Cleanup != nil`), rewritten each pass at both paths of the Global Constraint; the move's log reader (`carlos move offline status --log`) lists it under the move. |
| -- [ ] **Step 1: Write the failing tests** against a `DirStore` (deployment), a versioned fake (data bucket) recording every `Delete` with no version id, a real registry, an injected clock: `TestReserveSharedByConsoleAndBox` (both wrappers; distinct kinds with equal ids are distinct operations; a move's release never touches a cleanup's entries); `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish` (the `gone` step at T; the record clears at T+23 h; cleanup eligible at T+47 h, not T+24 h); `TestFourConditionsAtEveryKey` (each condition failing between key 1 and key 2: stop or supersede as specified, nothing more deleted); `TestCleanupRefusesWhenTheRecordMovedAgainMidPass` (B→C after key 1; a recreation; a deletion; each → superseded, root cancelled when requested); `TestExclusionPremisesAreCheckedEveryKey` (a second box in the region → no delete; a row appearing → stop; the fake writer case of Review Focus 2); `TestCleanupSkipsAChangedVersionAndFinishes` (one key superseded and paged; the obligation reaches `bucket-done`); `TestLockJsonNeverDeletedAcrossRetries`; `TestNoVersionIdOnDeletes`; `TestLifecycleResumesAtEveryState` (a crash after each state write, restart, completion; `ReservedAt`/`CleaningSince` preserved); `TestBoxReclaimRefusesMoves`; `TestCleanupReportCarriesKeys`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'Reserve' -count=1`, `go test ./internal/host -run 'Cleanup|FourConditions|Exclusion|Lifecycle|LockJson|NoVersionId|BoxReclaim' -count=1`. |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Reserve|Reclaim' -count=1`. |
| -- [ ] **Step 5: Commit** `moves: the shared reservation helper; the source box retires the old replica through one restartable lifecycle, re-reading the four conditions at every key`. |
| +- [ ] **Step 1: Write the failing tests** against a `DirStore` (deployment), a versioned fake (data bucket) recording every `Delete` with no version id, a real registry, an injected clock: `TestReserveHostnameKeys` (two key forms; legacy bare ids normalised to `box/`; `host/` conflicts across kinds; `Reserve` returns the held entry on a retry); `TestReserveSharedByConsoleAndBox`; `TestCleanupRunsTwentyFourHoursAfterTheObservedFinish`; `TestFleetCheckBeforeAndUnderTheReservation`; `TestFourConditionsAtEveryKey` (each condition failing between key 1 and key 2: stop, supersede or the per-key outcome as specified; the region check both ways); `TestCleanupRefusesWhenTheRecordMovedAgainMidPass` (B→C; a recreation; a deletion; a placement back to the source → each superseded, root cancelled when requested or publishing); `TestResidenceChangesAndCleanupExclude` (Review Focus 2 with the real `createRecordWithLifetime`, the real placement door and the legacy move door); `TestCleanupSkipsAChangedVersionAndFinishes` (the key superseded, a pending page persisted, sent, removed; a failed send retried); `TestLockJsonNeverDeletedAcrossRetries`; `TestNoVersionIdOnDeletes`; `TestLifecycleResumesAtEveryState` (a crash after each state write and **after each external side effect before its state write**: after the request file, after the marker, after a delete, after a page; restart; completion; `ReservedAt`/`CleaningSince` preserved; `publishing` reconciled); `TestOriginalFinishingMoveIsNotSupersession`; `TestBoxReclaimRefusesOtherKinds`; `TestCleanupReportUnderBothIds`; `TestBudgetAndFairness` (two obligations, slow I/O: the pass ends at its deadline, the cursor advances, the offline report refresh is not blocked). |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/moves -run 'Reserve' -count=1`, `go test ./internal/host -run 'Cleanup|FourConditions|Residence|Lifecycle|LockJson|NoVersionId|BoxReclaim|FleetCheck|Budget' -count=1`. |
| +- [ ] **Step 3: Implement.** **Step 4: Run to verify they pass**, plus `go test ./internal/console -run 'Reserve|Reclaim' -count=1`. |
| +- [ ] **Step 5: Commit** `moves: hostname keys in the reservation table; the source box retires the old replica through one restartable lifecycle, re-reading the four conditions at every key`. |
| -### Task 3: The per-box reaper |
| +### Task 3: The per-box reaper with end observations |
| **Files:** |
| - Create: `internal/host/movesreap.go` |
| -- Modify: `internal/host/movespass.go` (`w.reapParked(ctx, now)` once per moves pass after the dispatcher) |
| +- Modify: `internal/host/movespass.go` (`w.observeParkEnds(ctx)` and `w.reapParked(ctx, now)` once per moves pass after the dispatcher), `internal/host/movesoffline.go` (every role writes nothing new; the observer below writes `park-ended.json`) |
| - Test: `internal/host/movesreap_test.go` |
| **Interfaces:** |
| -- `func (w *movesWorker) reapParked(ctx context.Context, now time.Time) error`: for each `/data/.offline-moves/<host>/<id>/`: with a `retired.json` → reap when `State == ended && now - EndedAt >= 7 d`; without one (the target's, the sealer's, `target-before`, `verify` directories of a move) → reap when the judged record of the host shows `LastMove.ID == id` with `LastMove.At` at least 7 d ago, or the record is absent and the directory's newest mtime is at least 7 d ago **and** no `retired.json` for that id exists on this box; a directory with an outstanding `root-cleanup.json` and no `.done`/`.cancelled` is never reaped; legacy directories that match neither rule are left and logged once per day. `os.RemoveAll` of the op directory only; the host directory when empty. |
| +- `park-ended.json` (`{move_id, host, incarnation, at, reason}`) in an op directory **without** a `retired.json`: written by `observeParkEnds` when the judged record shows `LastMove.ID == id` (`at = LastMove.At`, `reason = LastMove.Outcome`) or `Move.ID == id` has gone from the record and `LastMove.ID` is a later move (`at = now`, `reason = superseded`), or the record is absent and the directory's op id is not referenced by any `retired.json` on this box (`at = now`, `reason = record gone`); written once; so a later move overwriting `LastMove` never loses the evidence, because the observation was taken while it was available (each pass observes every unended park). |
| +- `func (w *movesWorker) reapParked(ctx, now) error`: an op directory is reaped when (a) it has `retired.json` in `ended` with `now - EndedAt >= 7 d` and no `PendingPages`, or (b) it has `park-ended.json` with `now - at >= 7 d`; a directory with neither, or with an outstanding `root-cleanup.json` lacking `.done`/`.cancelled`, or a `retired.json` in any other state, is never reaped; legacy directories (no `retired.json`, no `park-ended.json`, and no record evidence this pass) are left and logged once per day. `os.RemoveAll` of the op directory only; the host directory when empty. |
| -- [ ] **Step 1: Write the failing tests**: `TestParkedStateReapedSevenDaysAfterTheObligationEnds`; `TestTargetAndSealerParksReapedOnRecordEvidence`; `TestOutstandingRootRequestIsNeverReaped`; `TestLegacyDirectoriesAreLeft`. |
| -- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/host -run 'Reap' -count=1`. |
| -- [ ] **Step 5: Commit** `agent: parked offline-move state is reaped on completion evidence seven days after its obligation ends`. |
| +- [ ] **Step 1: Write the failing tests**: `TestParkedStateReapedSevenDaysAfterTheObligationEnds`; `TestParkEndObservedOnceAndSurvivesALaterMove` (a second move inside seven days overwrites `LastMove`; the park is still reaped seven days after its own end); `TestOldParkWithoutEvidenceIsLeft` (an old directory, record absent today → `park-ended.json` today, reaped seven days later, not today); `TestOutstandingRootRequestIsNeverReaped`; `TestPendingPagesBlockTheReaper`; `TestLegacyDirectoriesAreLeft`. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/host -run 'Reap|ParkEnd' -count=1`. |
| +- [ ] **Step 5: Commit** `agent: parked offline-move state records its end observation and is reaped seven days after it`. |
| -### Task 4: Root's leftovers through a request file |
| +### Task 4: Root's leftovers through a request it verifies in the bucket |
| **Files:** |
| -- Create: `internal/release/rootcleanup.go`, `internal/host/movescleanup_root.go` |
| -- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: `rootCleanupPass(ctx, dataDir, release.OptDir(), reg)` beside `heldStagePass`, in the stage-requested and hourly modes, nonfatal to adoption, refusing destructive work unless the adopt lock is held, as `sweepStagedVersions` does) |
| +- Create: `internal/release/rootcleanup.go` (the protocol, portable), `internal/release/rootcleanup_linux.go` (`fstatat`, `readlinkat`, `unlinkat`), `internal/release/rootcleanup_other.go` (an implementation that answers every request `.cancelled` with `Reason: "unsupported platform"`), `internal/host/movescleanup_root.go` |
| +- Modify: `internal/host/agent.go` (`runAdoptUnderRootRelease`: `rootCleanupPass(ctx, dataDir, release.OptDir(), store, reg, locked bool, now func() time.Time)` beside `heldStagePass`, in the stage-requested and hourly modes, nonfatal to adoption, `locked` from the pass's own lock result (`passLockErr == nil`)) |
| - Test: `internal/release/rootcleanup_test.go`, `internal/host/movescleanup_root_test.go` |
| **Interfaces:** |
| -- Files under `OfflineOpDir`: `root-cleanup.json` (`type release.RootCleanupRequest struct { Host string \`json:"host"\`; MoveID string \`json:"move_id"\`; CleanupID string \`json:"cleanup_id"\`; Paths []RootCleanupPath \`json:"paths"\`; WrittenAt time.Time \`json:"written_at"\` }`, `type RootCleanupPath struct { Which string \`json:"which"\` (live-link or programs-file); Identity moves.FileIdentity \`json:"identity"\` }`), `root-cleanup.progress` (one path per line, appended before each unlink), `root-cleanup.cancel` (the agent's), `root-cleanup.done` and `root-cleanup.cancelled` (`type RootCleanupAck struct { Removed []string \`json:"removed"\`; Kept []string \`json:"kept"\`; Reason string \`json:"reason,omitempty"\`; At time.Time \`json:"at"\` }`). |
| -- `func release.ServeRootCleanupRequests(ctx context.Context, dataDir, optDir string, locked bool, rowAbsent func(host string) (bool, error)) error`: `locked == false` → return (no destructive work without the adopt lock); lists `/data/.offline-moves/*/*/root-cleanup.json` through `safefile`; for each without `.done`/`.cancelled`: the host is the enclosing host directory's name (must pass `ValidHostShape`) and the move id its parent; the request is read bounded (4096 bytes); a malformed request, a `Host` or `MoveID` disagreeing with the directory, or an unknown `Which` → `.cancelled` with `Reason: "malformed request"` (never a silent drop); **root derives the two paths itself**: `release.LiveLink(optDir, host)` and `/data/programs/<host>.json`; for each requested `Which` in order: open the parent directory (`safefile.OpenDir`), `fstatat` the entry by name (no following), compare `Dev`, `Inode` and (for the link) `readlinkat` with the recorded identity, `rowAbsent(host)`, then check `.cancel` **after** those checks and immediately before `unlinkat` within the same directory handle; a mismatch → `Kept` with the reason; `.cancel` present → stop, `.cancelled` listing the progress file's entries as `Removed`; before each `unlinkat` append the path to `root-cleanup.progress`; after all → `.done` with `Removed` from the progress file and `Kept`. A request older than 30 days without an answer is answered `.cancelled` with `Reason: "stale request"` (the agent's own supersession path handles the rest). The build directory is never touched; the "live link names the build" rule of rev 1 is dropped: the candidate link **is** the live link, and other hosts' links to the same build are theirs. |
| -- The agent: `writeRootCleanupRequest(dataDir string, r moves.Retired) error` (create-only); the dispatcher (Task 2) handles `.done`/`.cancelled`/paging/cancel. |
| +- Files under `OfflineOpDir`: `root-cleanup.json` (`offlinecleanup.RootCleanupRequest{Host, MoveID, CleanupID string; Paths []RootCleanupPath{Which string; Identity FileIdentity}; WrittenAt time.Time}`), `root-cleanup.journal` (lines `intent <which>` before each unlink and `removed <which>` after), `root-cleanup.cancel` (the agent's), `root-cleanup.done` and `root-cleanup.cancelled` (`RootCleanupAck{CleanupID, Host, MoveID string; Removed, Kept, Unknown []string; Reason string; At time.Time}`; `Unknown` = intents with no `removed` line). |
| +- **Authority** (`release.ServeRootCleanupRequests(ctx, dataDir, optDir string, s release.Store, reg rowAbsent func(host string) (bool, error), locked bool, now func() time.Time) error`): `locked == false` → return; for each request without an ack (listed through `safefile`, the host from the directory name passing `ValidHostShape`, the move id its parent): read bounded (4096 bytes), validate (`Host`, `MoveID` agree with the directory; `CleanupID` matches `^cu_`; each `Which` known and unique; a malformed request → `.cancelled` `Reason: "malformed request"`); **then the bucket**: `active.json` must hold `{cleanup, CleanupID}` on `host/<Host>` (else `.cancelled` `Reason: "no cleanup reservation"`), and the judged record of the hostname (found through the request's account and app, which the request carries: add `Account, App` to the request) must be valid with `Region != this box's region` and `LastMove.ID == MoveID` (else `.cancelled` `Reason: "the instance is not away"`); a bucket read error → leave the request for the next pass. Root derives the two paths itself (`release.LiveLink(optDir, host)`, `filepath.Join(dataDir, "programs", host + ".json")`). **The spec's process and reference checks** (§7.6), with the candidate link excluded: for the live link, no other host's live link under `optDir` targets the same build **and** no running process (procfs scan of `/proc/*/exe` and `/proc/*/cwd`) lives under the build the link targets → else `Kept` ("the build is still referenced"); for the programs file, no other programs file references the host → else `Kept`. |
| +- **The unlink boundary**, per `Which`, inside `release.WithRootCleanupLock(host)` (a flock on `<OfflineOpDir>/root-cleanup.lock`, root-only, serialising root's cleanup against root's own adopt writes to the same paths, which are the only other writers of the live link and programs file: the agent never writes them): open the parent (`safefile.OpenDir`); `fstatat` by name (no follow) and `readlinkat` for a link; compare with the recorded identity (`Dev`, `Inode`, `Mode`, `LinkTarget`) → mismatch → `Kept` ("identity changed"); `rowAbsent(host)` → a row → `Kept`; append `intent <which>` to the journal and `fsync`; **re-check `.cancel` and re-`fstatat` the entry** (identity unchanged, else `Kept`); `unlinkat`; append `removed <which>`, `fsync`. `.cancel` present at the re-check → stop, write `.cancelled` from the journal. After all paths → `.done` from the journal (`Removed` = `removed` lines, `Unknown` = intents without one, `Kept` with reasons). A request older than 30 days without an answer → `.cancelled` `Reason: "stale request"`. Read, append and unlink failures: the pass stops for this request and retries next time (the journal keeps the truth); an ack write failure → retried next pass. |
| +- The agent: `writeRootCleanupRequest(dataDir string, r offlinecleanup.Retired) error` (create-only); the dispatcher (Task 2) handles `.done`/`.cancelled`/paging/cancel; an ack whose identity does not match the request → the obligation `corrupt` (paged). |
| -- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyRecordedIdentities` (matching link and programs file → removed and in `.done`; a link whose target changed → kept; a changed inode → kept; a row present → kept; a path outside the two canonical ones in the request → `.cancelled` malformed); `TestCancelIsAcknowledgedBeforeRelease` (`.cancel` written while root is between its identity checks and the unlink (a hook) → nothing unlinked, `.cancelled`; after one unlink → `.cancelled` names exactly that path from the progress file; `.done` landing before the agent's `.cancel` → the agent releases on `.done`); `TestRootCrashAfterOneUnlinkIsReportedFromProgress`; `TestUnansweredRootRequestHoldsAndPages` (no root pass for an hour: pages at 10, 20, ... minutes with `LastPagedAt` persisted across an agent restart; the reservation held; then root answers → release; a failing release retries next pass; a failing page is logged and retried); `TestNoLocalLeftoversEndsAtOnce`; `TestRootCleanupNeedsTheAdoptLock`; `TestRootCleanupIsServedInBothModes`. |
| -- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootCleanup|RootRemoves|CancelIs|RootCrash' -count=1`, `go test ./internal/host -run 'RootCleanup|Unanswered|NoLocalLeftovers' -count=1`. |
| -- [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run to verify they pass**, then the root-read census. |
| -- [ ] **Step 5: Commit** `root: removes an old instance's live link and programs file only through a cleanup request it answers with done or cancelled, checking identity and cancellation at the unlink`. |
| +- [ ] **Step 1: Write the failing tests**: `TestRootRemovesOnlyRecordedIdentities`; `TestRootAuthorityComesFromTheBucket` (no reservation → cancelled; the record home again → cancelled; a forged request for a host that is live → cancelled, nothing unlinked); `TestProcessAndReferenceChecksKeep` (another host's link to the same build; a process under the build; another programs file naming the host); `TestCancelIsAcknowledgedBeforeRelease` (`.cancel` landing between the intent line and the re-check → nothing unlinked, `.cancelled` with `Unknown` empty; after one unlink → `.cancelled` names exactly that path from `removed`); `TestRootCrashAfterOneUnlinkIsReportedFromProgress` (a crash after `intent` and before `unlinkat` → `Unknown`, not `Removed`; after `unlinkat` and before `removed` → the restart re-checks the entry: absent → `removed`); `TestEntryReplacedBeforeUnlinkIsKept` (the entry replaced between the first `fstatat` and the re-check → `Kept`); `TestUnansweredRootRequestHoldsAndPages`; `TestPublishedRequestSurvivesTheCrash`; `TestNoLocalLeftoversEndsAtOnce`; `TestRootCleanupNeedsTheAdoptLock`; `TestRootCleanupIsServedInBothModes`; `TestUnsupportedPlatformCancels`. |
| +- [ ] **Step 2: Run to verify they fail**: `go test ./internal/release -run 'RootCleanup|RootRemoves|RootAuthority|ProcessAndReference|CancelIs|RootCrash|EntryReplaced' -count=1`, `go test ./internal/host -run 'RootCleanup|Unanswered|PublishedRequest|NoLocalLeftovers' -count=1`. |
| +- [ ] **Step 3: Implement.** **Step 4: Run to verify they pass**, then the root-read census and `GOOS=darwin go build ./internal/release/ ./internal/host/`, `GOOS=windows go vet ./internal/release/ ./internal/host/`. |
| +- [ ] **Step 5: Commit** `root: removes an old instance's live link and programs file only through a cleanup request it verifies in the bucket, journalling intent and result at the unlink`. |
| -### Task 5: Cleanup joins hostname admission |
| +### Task 5: Residence changes hold the hostname key |
| **Files:** |
| -- Modify: `internal/console/lifetimes.go` (plan 1's `admit(host, ownOpID)` also reads `active.json`: a `cleanup` entry whose `Host == host` → `errHostUnderCleanup`; a read failure → `errLifetimeUnreadable` (502); the create door maps `errHostUnderCleanup` to 409 "the previous instance at this hostname is still being cleaned up. try again later"; allocation recovery leaves a pending allocation pending while the host is under cleanup), `internal/console/apiinstances.go` (the existing `hostHolder` and `hostClaimHolder` checks and their failure answers stay), `internal/console/offlinereserve.go` (a move's reservation is refused by a `cleanup` holder as plan 2 states) |
| -- Test: `internal/console/lifetimes_cleanup_test.go` |
| +- Modify: `internal/console/lifetimes_create.go` (plan 1's create protocol step 1: after `admit(host, "")` and before `BeginAllocation`, `moves.Reserve(["host/" + host], {create, OpID, host, acct, app, now}, consoleOpts)`; `ErrReserved` by a `cleanup` holder → `errHostUnderCleanup` (409 "the previous instance at this hostname is still being cleaned up. try again later"); by any other holder → `errLifetimeSettling` (409 settling); `ErrUnreadable` → 502; the entry is released by the allocation's terminal state (`committed`, `superseded`, `void`, `refused`) in the same code path that sets it, and by recovery when it terminalises a row; a pending allocation keeps the key, so cleanup is `busy` until recovery ends it), `internal/console/apiinstancemove.go` (the legacy move door's `MoveComplete`: `{place, opID}` on `host/<host>` and the two regions' boxes around the record write), `internal/console/appsettings.go` and `fleets.go` (`placeAppTarget`'s callers, `carlos apps place`: `{place, opID}` on `host/<host>` per instance around each `SetRecordRegion`), `internal/console/offlinereserve.go` (plan 2's selections include `host/<hostname>`) |
| +- Test: `internal/console/lifetimes_cleanup_test.go`, `internal/console/residence_reservation_test.go` |
| -- [ ] **Step 1: Write the failing tests**: `TestCreateRefusesUnderACleanupReservationAtAdmission` (the check is inside `withHostname`: a create paused after its pre-lock checks, a cleanup reservation taken, the create resumes → 409, no slot, no row); `TestPendingAllocationWaitsUnderCleanup`; `TestCreateProceedsAfterRelease`; `TestMoveWaitsUnderACleanupReservation`; `TestHolderChecksPreserved`. |
| -- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Cleanup|Reserve|HolderChecks' -count=1`. |
| -- [ ] **Step 5: Commit** `console: a hostname under cleanup is refused at durable admission until the cleanup ends`. |
| +- [ ] **Step 1: Write the failing tests**: `TestCreateReservesTheHostnameKeyBeforeAllocating` (paused after `admit` and before `Reserve`, cleanup acquires → the create is refused 409 cleanup, no slot, no row; paused after `Reserve` → cleanup is `busy`; the key released at every terminal state); `TestPendingAllocationHoldsTheKeyUntilRecoveryEnds`; `TestPlacementAndLegacyMoveReserveTheHostname` (each refused `busy` under a cleanup holder; each blocks cleanup while held; released after the write); `TestEveryResidenceChangeHoldsTheHostnameKey` (the AST audit); `TestHolderChecksPreserved`. |
| +- [ ] **Step 2: Run to verify they fail; Step 3: implement; Step 4: run** `go test ./internal/console -run 'Cleanup|Reserve|Residence|HolderChecks' -count=1`. |
| +- [ ] **Step 5: Commit** `console: every residence change of a hostname holds its key in the reservation table; a hostname under cleanup is refused at durable admission`. |
| ### Task 6: End to end |
| **Files:** |
| -- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's harness, a clock the test advances, a root pass driven by the test) |
| +- Test: `internal/e2e/selfserve/cleanup_test.go` (plan 2's bridge, a clock the test advances, a root pass driven by the test with `locked = true`) |
| -- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish` (a forward move done; at +24 h from the finish the source bucket's replica keys carry delete markers written without version ids, `lock.json` stands, `retired.json` is `root-requested`; root's pass answers `.done` naming the live link and programs file; the reservation is released; the state `ended`; at +7 d the park directory is gone); `TestCleanupRefusalsMidPass` (the record moved again after one key; a current version differs; a move holds the box; two boxes in the region); `TestCleanupSupersededCancelsRoot` (a recreation during the root step: `.cancel`, `.cancelled`, release, `ended` superseded); `TestCreateUnderCleanupRefusedThenAllowed`; `TestCleanupStatusSupportsReclaim` (plan 2's reclaim rule against the real status report through its lifecycle: not reclaimable until `ended`). |
| +- [ ] **Step 1: Write the tests**: `TestCleanupTwentyFourHoursAfterTheObservedFinish`; `TestCleanupRefusalsMidPass` (the record moved again after one key; a placement back; a current version differs; a move holds the box; two boxes in the region); `TestCleanupSupersededCancelsRoot`; `TestCreateUnderCleanupRefusedThenAllowed`; `TestCleanupStatusSupportsReclaim` (plan 2's reclaim rule against the real status report through its lifecycle, including a corrupt file and a report older than the reservation: not reclaimable until `ended` in a report that post-dates the reservation); `TestCleanupReportInTheMoveLog`. |
| - [ ] **Step 2: Run them; fix what they find within this plan's scope.** |
| -- [ ] **Step 3: Run both census tests and the cross-builds** (`GOOS=darwin go build ./internal/host/`, `GOOS=windows go vet ./internal/host/`: the identity code is behind build tags); write `docs/superpowers/notes/census-proposal-self-serve-moves-4.md`. |
| +- [ ] **Step 3: Run both census tests and the cross-builds**; write `docs/superpowers/notes/census-proposal-self-serve-moves-4.md`. |
| - [ ] **Step 4: Commit** `self-serve moves: cleanup end to end`; push. |
| --- |
| ## Self-review notes |
| -- Spec coverage: §7.6 the source replica (T1, T2: inventory at finish, the 24-hour wait from the observed finish, the four conditions at every deletion, plain deletes, per-key logging, retry and supersession, completion when every key is terminal), parked local state (T3), root-owned leftovers (T4), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the admission check (T2, T5); §8's cleanup pages (T2, T4); §11's cleanup witnesses (T6). One stated strengthening: cleanup refuses while the source region has more than one box, because the exclusion argument needs the move's one-box rule. |
| -- Names used across tasks: `moves.Retired` (`State`, `Keys`, `RootRequest`, `FinishedAt`, `LastPagedAt`), `InventoryObject`, `KeyOutcome`, `FileIdentity`, `LocalPath`, `OfflineOpDir`, `RetiredFile`, `fileIdentity`, `status.RetiredSummary`, `RetirementsComplete`, `RetirementsErr`, `moves.Reserve`, `moves.Release`, `ReserveOptions`, `ErrReserved`, `ErrUnreadable`, `cleanupDispatch`, `reapParked`, `release.RootCleanupRequest`, `RootCleanupPath`, `RootCleanupAck`, `ServeRootCleanupRequests`, `writeRootCleanupRequest`, `errHostUnderCleanup`, `cfg.PlainMirror`. |
| -- Review Focus 1 to 5 are pinned by T2, T2, T2+T5, T4, T4. |
| +- Spec coverage: §7.6 the source replica (T1, T2), parked local state (T3), root-owned leftovers with the process and reference checks (T4), the staged objects of an aborted move (unchanged); §7.1's `cleanup` reservation and the hostname keys (T2, T5); §8's cleanup pages (T2, T4); §11's cleanup witnesses (T6). Stated strengthenings: cleanup refuses while the source region has more than one box; every residence change holds the hostname key (Paul's ruling 2026-10-22). |
| +- Plan 2 amendments this plan requires (small, listed for the plan 2 build): the reservation keys become `box/<id>` and `host/<hostname>`; selections include `host/<hostname>`; the reclaim rule's cleanup clause reads as Task 1 states it. |
| +- Names used across tasks: `offlinecleanup.Retired` (`State`, `Keys`, `Refusals`, `RootRequest`, `FinishedAt`, `LastPagedAt`, `PendingPages`, `Prefix`), `InventoryObject`, `KeyOutcome`, `Refusal`, `PendingPage`, `FileIdentity`, `LocalPath`, `RetiredSummary` (`ReleasePending`, `UpdatedAt`), `ValidateRetired`, `OfflineOpDir`, `RetiredFile`, `fileIdentity`, `writeRetired`, `moves.Reserve` (returns the entry), `moves.Release`, `ReserveOptions`, `ErrReserved`, `ErrUnreadable`, `cleanupDispatch`, `observeParkEnds`, `reapParked`, `park-ended.json`, `RootCleanupRequest` (`Account`, `App`), `RootCleanupPath`, `RootCleanupAck` (`Unknown`), `ServeRootCleanupRequests`, `WithRootCleanupLock`, `writeRootCleanupRequest`, `errHostUnderCleanup`, `cfg.PlainMirror`, the key forms `box/`, `host/`, `"*"`. |
| +- Review Focus 1 to 5 are pinned by T2, T2+T5, T2+T5, T2+T4, T4. |