roll notes: Gate F needs a fresh probe app; revert removes the leftover g<n>/; Gate F passed (#272)
Gate F and "The probe": the flagship probe runs against a fresh throwaway app with no program history, not bab/adopt-probe. adopt-probe's recorded program is adoptprobe, and root refuses a zddprobe build as the wrong program unless it is shipped with `carlos ship --program-change`; that refusal, not a slow root pass, stopped the first Gate F's zdd-1. The steps are the 2026-10-14 retry's: an unregioned app in bab, one instance, curl --resolve to the box while DNS lags (load() takes RESOLVE), and deleting the app afterwards. Revert: after unsetting CARLOS_SWAP and restarting the edge, remove the last generation's leftover g<n>/ under the instance run dir (dev round 5). Gate F's status: passed on annie-welding on 2026-10-14; annie stays armed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+64
−20
docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md+64 −20
CI failed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.013s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 365.337s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 4.676s
ok github.com/carlosframework/platform/cmd/moveprobe 93.242s
ok github.com/carlosframework/platform/cmd/zddprobe 0.743s
ok github.com/carlosframework/platform/internal/activator 235.253s
ok github.com/carlosframework/platform/internal/agentsock 0.032s
ok github.com/carlosframework/platform/internal/alerting 0.015s
ok github.com/carlosframework/platform/internal/analytics 0.023s
ok github.com/carlosframework/platform/internal/appconfig 2.960s
ok github.com/carlosframework/platform/internal/apperr 0.246s
ok github.com/carlosframework/platform/internal/applog 0.241s
ok github.com/carlosframework/platform/internal/awsinv 0.032s
ok github.com/carlosframework/platform/internal/boxchoice 0.016s
ok github.com/carlosframework/platform/internal/boxid 1.033s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.017s
ok github.com/carlosframework/platform/internal/caps 0.037s
ok github.com/carlosframework/platform/internal/certwarm 0.006s
ok github.com/carlosframework/platform/internal/cgroup 0.804s
ok github.com/carlosframework/platform/internal/changelog 0.417s
ok github.com/carlosframework/platform/internal/clamd 0.067s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 373.342s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.218s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.011s
ok github.com/carlosframework/platform/internal/dbverify 0.008s
ok github.com/carlosframework/platform/internal/dnsrec 0.005s
ok github.com/carlosframework/platform/internal/docsite 0.006s
ok github.com/carlosframework/platform/internal/domains 2.357s
ok github.com/carlosframework/platform/internal/doorkey 0.008s
ok github.com/carlosframework/platform/internal/economics 0.007s
ok github.com/carlosframework/platform/internal/edge 28.558s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.140s
ok github.com/carlosframework/platform/internal/gate 1.661s
ok github.com/carlosframework/platform/internal/genselect 0.033s
ok github.com/carlosframework/platform/internal/geo 0.078s
ok github.com/carlosframework/platform/internal/geoip 0.056s
ok github.com/carlosframework/platform/internal/host 134.901s
ok github.com/carlosframework/platform/internal/hostactors 0.844s
ok github.com/carlosframework/platform/internal/hostbox 3.264s
ok github.com/carlosframework/platform/internal/hostclaim 1.233s
ok github.com/carlosframework/platform/internal/hostevidence 0.005s
ok github.com/carlosframework/platform/internal/hostexec 8.445s
ok github.com/carlosframework/platform/internal/hostfence 0.078s
ok github.com/carlosframework/platform/internal/hostfiles 11.833s
ok github.com/carlosframework/platform/internal/hostfloor 0.059s
ok github.com/carlosframework/platform/internal/hostguard 1.200s
ok github.com/carlosframework/platform/internal/hostops 2.404s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.015s
ok github.com/carlosframework/platform/internal/hostquarantine 0.005s
ok github.com/carlosframework/platform/internal/hostrelocate 0.952s
ok github.com/carlosframework/platform/internal/hostresolve 0.471s
ok github.com/carlosframework/platform/internal/hostwrite 0.001s
ok github.com/carlosframework/platform/internal/infraview 0.005s
ok github.com/carlosframework/platform/internal/instancectl 1.858s
ok github.com/carlosframework/platform/internal/instanceexec 2.911s
ok github.com/carlosframework/platform/internal/instancefs 0.007s
ok github.com/carlosframework/platform/internal/instancekey 0.004s
ok github.com/carlosframework/platform/internal/instanceupdates 0.010s
ok github.com/carlosframework/platform/internal/layout 2.064s
ok github.com/carlosframework/platform/internal/lease 9.600s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.012s
ok github.com/carlosframework/platform/internal/ledger 0.043s
ok github.com/carlosframework/platform/internal/mail 0.325s
ok github.com/carlosframework/platform/internal/maintenance 0.115s
ok github.com/carlosframework/platform/internal/maintenanceexec 6.119s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.204s
ok github.com/carlosframework/platform/internal/metrics 0.006s
ok github.com/carlosframework/platform/internal/moves 24.632s
ok github.com/carlosframework/platform/internal/moves/archive 1.370s
ok github.com/carlosframework/platform/internal/objstore 0.445s
ok github.com/carlosframework/platform/internal/ops 0.290s
ok github.com/carlosframework/platform/internal/park 0.466s
ok github.com/carlosframework/platform/internal/privatefile 0.015s
2026/10/14 14:45:25 provision: listing apps/bab/blog/instances/: list refused — its records are unseen this pass
2026/10/14 14:45:25 provision: listing apps/bab/: listdirs refused — its records are unseen this pass
2026/10/14 14:45:25 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:25 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:25 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:25 provision: listing apps/bac/tito/instances/: list refused — its records are unseen this pass
2026/10/14 14:45:25 provision: listing apps/bac/tito/instances/: list refused — its records are unseen this pass
2026/10/14 14:45:25 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:25 provision: remove retired instance route shop.example: provision: claim admission unavailable
2026/10/14 14:45:25 provision: remove retired instance route shop.example: provision: claim admission unavailable
2026/10/14 14:45:25 provision: remove retired sidecar route worker.sidecar.internal: provision: claim admission unavailable
2026/10/14 14:45:25 provision: remove retired sidecar route worker.sidecar.internal: provision: claim admission unavailable
2026/10/14 14:45:25 provision: claim install acme.go.tito.io: provision: claim admission unavailable
2026/10/14 14:45:25 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:25 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/14 14:45:25 provision: claim install bot.eleven.bac.sidecar.internal: provision: claim admission unavailable
2026/10/14 14:45:25 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/14 14:45:25 provision: claim install bot.eleven.bac.sidecar.internal: registry: claim row ownership changed or unknown
2026/10/14 14:45:25 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/14 14:45:25 provision: acme.go.tito.io: not minting this pass: the box's layout intent could not be read: no sealed config
2026/10/14 14:45:25 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:25 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:25 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/14 14:45:25 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/14 14:45:25 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/14 14:45:25 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/14 14:45:25 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:25 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/14 14:45:25 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/14 14:45:25 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/14 14:45:25 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:25 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/14 14:45:25 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:25 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/14 14:45:25 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/14 14:45:25 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/14 14:45:25 provision: claim install bot.eleven.bac.sidecar.internal: registry: claim row ownership changed or unknown
2026/10/14 14:45:25 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/14 14:45:26 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/14 14:45:26 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/14 14:45:26 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:26 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/14 14:45:26 provision: skipping record apps/bac/tito/instances/new.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/14 14:45:26 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/14 14:45:26 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/14 14:45:26 provision: skipping record apps/bac/tito/instances/new.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/14 14:45:26 provision: minted instance route rj.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rb.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rd.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rg.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rh.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route ri.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rk.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rl.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route ra.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rc.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route re.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rf.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rb.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rc.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route rd.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route ra.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: skipping record apps/bac/tito/instances/rb.go.tito.io.json: read failed: read refused
2026/10/14 14:45:26 provision: skipping record apps/bac/tito/instances/rb.go.tito.io.json: read failed: read refused
2026/10/14 14:45:26 provision: minted instance route re.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.appe.example (app "appe" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.appf.example (app "appf" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.appi.example (app "appi" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.appd.example (app "appd" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.appg.example (app "appg" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.apph.example (app "apph" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.appj.example (app "appj" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.appa.example (app "appa" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.appb.example (app "appb" acct "bac")
2026/10/14 14:45:26 provision: minted instance route x.appc.example (app "appc" acct "bac")
2026/10/14 14:45:26 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/14 14:45:26 provision: minted instance route r0098.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0103.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0008.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0011.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0012.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0047.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0118.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0127.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0090.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0101.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0000.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0036.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0056.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0072.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0073.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0079.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0020.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0043.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0045.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0046.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0094.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0095.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0114.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0055.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0085.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0005.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0037.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0050.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0115.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0125.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0111.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0052.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0016.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0044.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0074.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0088.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0051.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0080.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0122.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0035.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0070.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0006.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0029.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0039.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0066.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0089.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0123.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0054.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0069.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0119.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0076.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0097.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0003.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0019.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0083.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0017.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0023.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0048.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0068.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0099.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0007.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0015.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0021.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0081.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0129.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0033.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0105.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0113.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0027.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0028.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0030.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0092.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0093.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0107.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0108.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0040.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0053.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0058.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0071.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0077.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0078.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0084.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0031.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0061.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0065.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0041.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0042.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0063.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0013.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0034.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0062.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0009.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0087.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0102.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0010.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0014.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0024.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0032.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0086.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0100.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0124.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0126.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0049.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0059.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0104.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0067.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0121.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0026.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0022.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0057.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0075.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0112.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0018.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0064.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0082.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0120.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0004.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0106.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0110.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0117.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0128.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0025.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0038.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0091.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0109.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0116.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0001.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0002.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0060.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:26 provision: minted instance route r0096.go.tito.io (app "tito" acct "bac")
--- FAIL: TestPlan14RootReadCensus (15.86s)
plan14_root_reads_test.go:674: root read census: 9 discrepancies
changed root read internal/activator/activator.go:Service.BeginSidecarRequest; review class, task, sites and digest
changed root read internal/activator/activator.go:Service.begin; review class, task, sites and digest
changed root read internal/activator/activator.go:Service.sweepOnce; review class, task, sites and digest
changed root read internal/activator/activator.go:Service.wake; review class, task, sites and digest
changed root source envelope internal/activator
changed legacy declaration internal/activator/activator.go:Service.BeginSidecarRequest
changed legacy declaration internal/activator/activator.go:Service.begin
changed legacy declaration internal/activator/activator.go:Service.sweepOnce
changed legacy declaration internal/activator/activator.go:Service.wake
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted instance route console.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted service route ha.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted service route svc.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted instance route whole.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted instance route shared.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: converged acme.go.tito.io channel stable -> edge
2026/10/14 14:45:43 provision: converged acme.go.tito.io channel edge -> canary/rehearsal
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: empty channel
2026/10/14 14:45:43 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: converged acme.go.tito.io environment "" -> "staging"
2026/10/14 14:45:43 provision: converged acme.go.tito.io environment "staging" -> ""
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: converged acme.go.tito.io steering "" -> "latency"
2026/10/14 14:45:43 provision: converged acme.go.tito.io steering "latency" -> ""
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: converged acme.go.tito.io hibernate false -> true
2026/10/14 14:45:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:43 provision: converged acme.go.tito.io hibernate true -> false (replication "daily" cleared: only a hibernating route's rung is read)
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: converged acme.go.tito.io status-named false -> true
2026/10/14 14:45:44 provision: converged acme.go.tito.io status-named true -> false
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted service route ha.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7001,127.0.0.1:7002" -> "127.0.0.1:7001,127.0.0.1:7003" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/14 14:45:44 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7001,127.0.0.1:7003" -> "127.0.0.1:7003,127.0.0.1:7001" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/14 14:45:44 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7003,127.0.0.1:7001" -> "127.0.0.1:7003,127.0.0.1:7001" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/healthz\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/14 14:45:44 provision: minted service route whole.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: converged whole.go.tito.io upstreams addr "127.0.0.1:9443" -> "", addrs "" -> "127.0.0.1:9443,127.0.0.1:9444" (probe "" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/14 14:45:44 provision: converged whole.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:9443,127.0.0.1:9444" -> "" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "")
2026/10/14 14:45:44 provision: whole.go.tito.io: record describes a different route SHAPE than the row (kind "instance"/"service" unit ""/"" socket "/run/carlos/whole.go.tito.io.sock"/"" db "/data/whole.go.tito.io.db"/"") — not converged in place; `carlos remove whole.go.tito.io` purges this route's capability grants (registry.Delete's contract, not a bug) before the next pass re-mints it, so treat that as a last resort, not a routine fix
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted service route whole.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: skipping accounts/bac/account.json: malformed account manifest: invalid character 'o' in literal null (expecting 'u')
2026/10/14 14:45:44 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: manifest bac/tito: read failed: provision: manifest bac/tito: invalid character 'o' in literal null (expecting 'u')
2026/10/14 14:45:44 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: disagrees with its own path (host="wrong-host.go.tito.io" account="bac" app="tito")
2026/10/14 14:45:44 provision: skipping record apps/bac/tito/instances/acme$evil.go.tito.io.json: host "acme$evil.go.tito.io" is not a hostname
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: skipping record apps/bac/tito/instances/acme$evil.go.tito.io.json: host "acme$evil.go.tito.io" is not a hostname
2026/10/14 14:45:44 provision: not removing acme$evil.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/14 14:45:44 provision: skipping record apps/bac/tito/instances/bad-service.go.tito.io.json: invalid shape: kind service with no addr: the platform only proxies to a service, so the record must name the upstream it proxies to
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: skipping record apps/bac/tito/instances/bad-service.go.tito.io.json: invalid shape: kind service with no addr: the platform only proxies to a service, so the record must name the upstream it proxies to
2026/10/14 14:45:44 provision: not removing bad-service.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/14 14:45:44 provision: skipping record apps/bac/tito/instances/no-channel.go.tito.io.json: empty channel
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: skipping record apps/bac/tito/instances/no-channel.go.tito.io.json: empty channel
2026/10/14 14:45:44 provision: not removing no-channel.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/14 14:45:44 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/14 14:45:44 provision: removed retired instance route ghost.go.tito.io (record gone)
2026/10/14 14:45:44 provision: duplicate host dup.go.tito.io: keeping apps/bac/tito/instances/dup.go.tito.io.json, skipping apps/bad/tito/instances/dup.go.tito.io.json
2026/10/14 14:45:44 provision: minted instance route dup.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/14 14:45:44 provision: removed retired instance route acme.go.tito.io (app in the trash)
2026/10/14 14:45:44 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/14 14:45:44 provision: removed retired static route site.go.tito.io (app in the trash)
2026/10/14 14:45:44 provision: trash marker for bac/tito unreadable: release: malformed trash marker for bac/tito: invalid character 'n' looking for beginning of object key string — not minting for this app
2026/10/14 14:45:44 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/14 14:45:44 provision: removed retired instance route acme.go.tito.io (app in the trash)
2026/10/14 14:45:44 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/14 14:45:44 provision: trash marker for bac/tito unreadable: release: malformed trash marker for bac/tito: invalid character 'n' looking for beginning of object key string — not minting for this app
2026/10/14 14:45:44 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/14 14:45:44 provision: converged acme.go.tito.io channel stable -> edge
2026/10/14 14:45:44 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/14 14:45:44 provision: converged acme.go.tito.io channel canary/rehearsal -> edge
2026/10/14 14:45:44 provision: converged acme.go.tito.io environment "staging" -> ""
2026/10/14 14:45:44 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/14 14:45:44 provision: acme.go.tito.io exists but is not the route this record describes (backing "unit" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/14 14:45:44 provision: acme.go.tito.io exists but is not the route this record describes (tunnel "pi-fleet" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/14 14:45:44 provision: acme.go.tito.io exists but is not the route this record describes (caps "port:25" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/14 14:45:44 provision: acme.go.tito.io exists but is not the route this record describes (platform (owned by the domains reconcile pass)) — leaving it alone
2026/10/14 14:45:44 provision: acme.go.tito.io exists but is not the route this record describes (canary (owned by the domains reconcile pass)) — leaving it alone
2026/10/14 14:45:44 provision: acme.go.tito.io exists but is not the route this record describes (db "/data/other.db"/"/data/acme.go.tito.io.db") — leaving it alone
2026/10/14 14:45:44 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/14 14:45:44 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/14 14:45:44 provision: dry-run: would mint instance route acme.go.tito.io (app "tito" acct "bac" channel "stable")
2026/10/14 14:45:44 provision: dry-run: would adopt existing route adopt.go.tito.io (app "tito" acct "bac") — this record would own its channel and environment
2026/10/14 14:45:44 provision: dry-run: would converge adopt.go.tito.io channel stable -> edge
2026/10/14 14:45:44 provision: dry-run: would converge adopt.go.tito.io environment "" -> "staging"
2026/10/14 14:45:44 provision: dry-run: would converge owned.go.tito.io channel stable -> edge
2026/10/14 14:45:44 provision: dry-run: would converge owned.go.tito.io environment "" -> "staging"
2026/10/14 14:45:44 provision: dry-run: would converge owned.go.tito.io hibernate false -> true
2026/10/14 14:45:44 provision: dry-run: would converge owned.go.tito.io seed
2026/10/14 14:45:44 provision: dry-run: would remove retired instance route gone.go.tito.io (record gone)
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: adopted existing route adopt.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/14 14:45:44 provision: converged adopt.go.tito.io channel stable -> edge
2026/10/14 14:45:44 provision: converged adopt.go.tito.io environment "" -> "staging"
2026/10/14 14:45:44 provision: converged owned.go.tito.io channel stable -> edge
2026/10/14 14:45:44 provision: converged owned.go.tito.io environment "" -> "staging"
2026/10/14 14:45:44 provision: converged owned.go.tito.io hibernate false -> true
2026/10/14 14:45:44 provision: removed retired instance route gone.go.tito.io (record gone)
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: adopted existing route story.carlosframework.com (app "story" acct "bad") — this record now owns its channel and environment
2026/10/14 14:45:44 provision: story.carlosframework.com exists but is not the route this record describes (kind "instance"/"static") — leaving it alone
2026/10/14 14:45:44 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted service route pub.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted service route pubs.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted service route priv.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route unit.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route exec.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted service route loop.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted service route named.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted static route pisite.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: removed retired static route site.go.tito.io (record gone)
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme-uk.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: converged acme.go.tito.io replicas "ie,uk" -> "uk"
2026/10/14 14:45:44 provision: converged acme.go.tito.io replicas "uk" -> ""
2026/10/14 14:45:44 provision: minted instance route acme-uk.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: converged acme-uk.go.tito.io replica mark true -> false
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: acme.go.tito.io: row is on the nested layout (db "/data/instances/acme.go.tito.io/acme.go.tito.io.db" socket "/run/carlos/instances/acme.go.tito.io/app.sock"); the record's shared paths are not drift
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:44 provision: acme.go.tito.io: not minting this pass: the box's layout intent could not be read: store down
2026/10/14 14:45:44 provision: router.go.tito.io: not minting this pass: the box's layout intent could not be read: store down
2026/10/14 14:45:44 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/14 14:45:44 provision: converged acme.go.tito.io channel stable -> edge
2026/10/14 14:45:44 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
2026/10/14 14:45:45 provision: not removing acme.go.tito.io (record gone): participant of a running operation
2026/10/14 14:45:45 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/14 14:45:45 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
2026/10/14 14:45:45 provision: refusing to mint held route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/14 14:45:45 provision: held acme.go.tito.io landing by mv_acme (was ready)
2026/10/14 14:45:45 provision: acme.go.tito.io: held restoring by rb_other, not by mv_acme; leaving it
2026/10/14 14:45:45 provision: acme.go.tito.io: the row is not this pass's (app "other" acct ""); not holding it for mv_acme
2026/10/14 14:45:45 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/14 14:45:45 provision: dry-run: would mint instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac" channel "stable")
2026/10/14 14:45:45 provision: acme.go.tito.io: not held landing by mv_acme: the row's hold changed under this pass; the next pass judges it afresh
2026/10/14 14:45:45 provision: held acme.go.tito.io landing by mv_acme (was ready)
2026/10/14 14:45:45 provision: removed retired sidecar route gone.eleven.bac.sidecar.internal (record gone)
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: converged sidecar bot.eleven.bac.sidecar.internal args ["sidecar","run"] -> ["worker","run"]
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: converged sidecar bot.eleven.bac.sidecar.internal ceilings 300/3600 -> 60/120
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: converged sidecar bot.eleven.bac.sidecar.internal channel stable -> canary/eleven
2026/10/14 14:45:45 provision: converged sidecar bot.eleven.bac.sidecar.internal environment "other" -> "relay"
2026/10/14 14:45:45 provision: converged sidecar bot.eleven.bac.sidecar.internal hibernate true -> false
2026/10/14 14:45:45 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: read failed: failStore: reads disabled under apps/bac/eleven/sidecars/
2026/10/14 14:45:45 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: malformed: invalid character 'n' looking for beginning of object key string
2026/10/14 14:45:45 provision: not removing sidecar bot.eleven.bac.sidecar.internal: its record exists but could not be read cleanly this pass
2026/10/14 14:45:45 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: empty channel
2026/10/14 14:45:45 provision: skipping sidecar record apps/bac/eleven/sidecars/held.json: empty channel
2026/10/14 14:45:45 provision: not removing sidecar held.eleven.bac.sidecar.internal: its record exists but could not be read cleanly this pass
2026/10/14 14:45:45 provision: removed retired sidecar route old.eleven.bac.sidecar.internal (record gone)
2026/10/14 14:45:45 provision: bac/eleven has sidecar records but no sidecars manifest — not minting
2026/10/14 14:45:45 provision: bac/eleven is in the trash — not minting sidecar routes for it
2026/10/14 14:45:45 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/14 14:45:45 provision: sidecar hub.relay.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: minted sidecar route hub.relay.bac.sidecar.internal (app "relay" acct "bac" args ["relay"])
2026/10/14 14:45:45 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/14 14:45:45 provision: removed retired sidecar route bot.eleven.bac.sidecar.internal (record gone)
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/14 14:45:45 provision: skipping sidecar record apps/bac/relay/sidecars/hub.json: read failed: failStore: reads disabled under apps/bac/relay/sidecars/
2026/10/14 14:45:45 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/14 14:45:45 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/14 14:45:45 provision: dry-run: would mint sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" channel "stable" args ["sidecar","run"])
2026/10/14 14:45:45 provision: dry-run: would remove retired sidecar route gone.eleven.bac.sidecar.internal (record gone)
2026/10/14 14:45:45 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/14 14:45:45 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:45 provision: converged acme.go.tito.io update settings "{\"updates\":\"manual\",\"channel\":\"stable\"}" -> ""
2026/10/14 14:45:45 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:45 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:45 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/14 14:45:45 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/14 14:45:45 provision: acme.go.tito.io: not minted this pass: the app's update policy could not be read
2026/10/14 14:45:45 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/14 14:45:45 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/14 14:45:45 provision: acme.go.tito.io: not minted this pass: the app's update policy could not be read
2026/10/14 14:45:45 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
FAIL
FAIL github.com/carlosframework/platform/internal/provision 20.232s
ok github.com/carlosframework/platform/internal/publicrelease 0.010s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.627s
ok github.com/carlosframework/platform/internal/relay 7.332s
ok github.com/carlosframework/platform/internal/release 24.404s
ok github.com/carlosframework/platform/internal/rollout 4.938s
ok github.com/carlosframework/platform/internal/rolloutwords 0.007s
ok github.com/carlosframework/platform/internal/rootaws 0.014s
ok github.com/carlosframework/platform/internal/rootparticipation 0.017s
ok github.com/carlosframework/platform/internal/rootstate 0.035s
ok github.com/carlosframework/platform/internal/s3compat 0.006s
ok github.com/carlosframework/platform/internal/safefile 0.008s
ok github.com/carlosframework/platform/internal/schedule 3.060s
ok github.com/carlosframework/platform/internal/sim/hostlife 22.206s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.633s
ok github.com/carlosframework/platform/internal/status 0.056s
ok github.com/carlosframework/platform/internal/steering 0.009s
ok github.com/carlosframework/platform/internal/stopevidence 0.289s
ok github.com/carlosframework/platform/internal/svcrole 0.005s
ok github.com/carlosframework/platform/internal/telemetry 0.035s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.008s
ok github.com/carlosframework/platform/internal/unixdial 0.004s
ok github.com/carlosframework/platform/internal/updatewords 0.005s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.004s
ok github.com/carlosframework/platform/internal/vet 1.897s
FAIL
make: *** [Makefile:18: test] Error 1
amadan: step gate failed: exit status 2
amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md| index 996bfe31c..48548fc37 100644 |
| --- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| +++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-roll-notes.md |
| @@ -18,8 +18,8 @@ plans 1 to 6, in `platform-design-zero-downtime-deploys`. |
| - **AWS profiles.** Assumed-role profiles need the Bash sandbox off for that |
| command. Dev is profile `kass`. The flagship is profile `keymail`. |
| - **Order.** Boxes first, then the console's swap-policy settings, then the |
| - flag: dev, then `bab/adopt-probe` on the flagship, then the rest of the |
| - flagship. The deploy docs and release notes carry the migrations rule |
| + flag: dev, then one flagship box with a fresh probe app (Gate F), then the |
| + rest of the flagship. The deploy docs and release notes carry the migrations rule |
| before the flag goes on across the fleet. Plan 5 keeps the flag off |
| outside dev until this plan has merged. |
| - **The flag.** `CARLOS_SWAP=overlap` in a box's `/etc/carlos/host.env`. Only |
| @@ -1049,6 +1049,15 @@ first. The unit is `carlos-edge` and it loads `/etc/carlos/host.env` |
| Restarting `carlos-edge` stops every instance on the box (spec section 10). |
| Each wakes on its next request. |
| +To revert a box, remove the `CARLOS_SWAP=` line from `/etc/carlos/host.env` |
| +and restart `carlos-edge`. Then remove the last generation's leftover `g<n>/` |
| +directory under each instance's run directory, |
| +`/run/carlos/instances/<host>/g<n>/`. Dev round 5 (#272) found that the |
| +flag-off restart leaves it behind with a stale `app.sock`. Nothing uses a |
| +`g<n>/` directory with the flag off, so it stays until a reboot. List what is |
| +left with `ls -d /run/carlos/instances/*/g[0-9]*/` after the restart, and |
| +remove each one. |
| + |
| Then run the probe below against a dev app. Create one with: |
| ```bash |
| @@ -1071,19 +1080,45 @@ dash means the box did not arm (see the dash note above). |
| ## Gate F: the flagship probe box |
| -Needs Paul's approval, naming the box and its region. The approval also |
| -names the probe's app actions: flipping `bab/adopt-probe`'s swap policy to |
| -`hold` and back, and four deploys of it. **Do not start this gate while the |
| -"Before Gate E: owed" list has an open item that Paul has not waived.** |
| - |
| -Find the probe's region with: |
| +**Status: passed on annie-welding (`ie`) on 2026-10-14 (#272). annie stays |
| +armed.** The first attempt, on 2026-10-13, was reverted. The retry ran the |
| +probe against a fresh app, as below. Its arm restart left one tenant's lease |
| +held for 15 minutes; that is the edge restart, not the flag (#292). |
| -```bash |
| -carlos instances list --app adopt-probe --account bab --console https://console.carloku.com |
| -``` |
| - |
| -(REGION column; the instance is `probe1.bab.oncarlos.com`.) The flagship |
| -boxes, profile `keymail`: |
| +Needs Paul's approval, naming the box and its region. The approval also |
| +names the probe's app actions: creating a throwaway app in `bab` with one |
| +instance on that box, flipping its swap policy to `hold` and back, four |
| +deploys of it, and deleting it afterwards. **Do not start this gate while |
| +the "Before Gate E: owed" list has an open item that Paul has not waived.** |
| + |
| +**Use a fresh app with no program history, not `bab/adopt-probe`.** |
| +adopt-probe's recorded program is `adoptprobe`, so root refuses a zddprobe |
| +build for it as the wrong program unless that build is shipped with |
| +`carlos ship --program-change`, which `carlos deploy` cannot do. That |
| +refusal, not a slow root pass, is what stopped zdd-1 in the first Gate F: |
| +root logged `wrong program` and restored the old build, and the watch timed |
| +out waiting for an install that would never come. |
| + |
| +The 2026-10-14 retry set the app up like this: |
| + |
| +1. Create the app in `bab` through the console's New app form, with no |
| + region. It was `bab/zdd-probe`. |
| +2. An operator runs `carlos instances enable --app zdd-probe --account bab`, |
| + then creates one instance: |
| + |
| + ```bash |
| + carlos instances create --app zdd-probe --account bab --host zddprobe1.bab.oncarlos.com --console https://console.carloku.com |
| + ``` |
| + |
| +3. Check it landed on the gate's box: `carlos instances list --app zdd-probe |
| + --account bab --console https://console.carloku.com` shows that box's |
| + region (the retry's was "Ireland · Dublin"), and only that box's edges |
| + file lists the host. |
| +4. A new host may not resolve yet. While DNS lags, point the load at the |
| + box's public IPv4 with `RESOLVE` (see "The probe"): |
| + `RESOLVE=zddprobe1.bab.oncarlos.com:443:<box IP>`. |
| + |
| +The flagship boxes, profile `keymail`: |
| | box | instance id | region | |
| | --- | --- | --- | |
| @@ -1099,9 +1134,16 @@ Run Gate E's flag command against that one box with `--profile keymail` and |
| its region. Then run the probe with: |
| ```bash |
| -APP=adopt-probe ACCOUNT=bab CONSOLE=https://console.carloku.com HOST=probe1.bab.oncarlos.com |
| +APP=zdd-probe ACCOUNT=bab CONSOLE=https://console.carloku.com HOST=zddprobe1.bab.oncarlos.com |
| +RESOLVE=zddprobe1.bab.oncarlos.com:443:<box IP> # only while DNS lags |
| ``` |
| +Afterwards, delete the app: `carlos apps delete zdd-probe --account bab |
| +--console https://console.carloku.com`. On 2026-10-14 the retry then removed |
| +the instance's leftovers on annie by hand (its instance, builds, links, data |
| +and `/data/programs/` file) and its bucket object, and checked that no |
| +`g<n>/` directory was left. An empty `carlos-instances-zddprobe1…slice` stayed. |
| + |
| ## Gate G: the rest of the flagship |
| Needs Paul's approval, naming each box before it is done. **Do not start |
| @@ -1116,8 +1158,8 @@ is Tito's operator's call; do not touch Tito's boxes from here. |
| ## The probe |
| Spec section 12: zero failed requests under continuous load, and no request |
| -held longer than a normal response. Run it at Gate E (dev) and Gate F |
| -(`adopt-probe`). |
| +held longer than a normal response. Run it at Gate E (dev) and Gate F (a |
| +fresh app on the flagship box, never `adopt-probe`; see Gate F for why). |
| ### 1. Build the probe app, in scratch |
| @@ -1142,7 +1184,8 @@ done |
| ### 2. A load phase |
| Four request loops for the given seconds, cache-busted, logging status and |
| -time: |
| +time. With `RESOLVE` set (`<host>:443:<box IP>`), curl skips DNS and goes |
| +straight to that box: |
| ```bash |
| load() { # usage: load <phase> <secs> |
| @@ -1150,7 +1193,7 @@ load() { # usage: load <phase> <secs> |
| for i in 1 2 3 4; do |
| ( end=$((SECONDS+$2)); n=0 |
| while [ $SECONDS -lt $end ]; do n=$((n+1)) |
| - curl -s -o /dev/null -m 35 -w '%{http_code} %{time_total}\n' "https://$HOST/?zdd=$1-$i-$n" >> "$TMPDIR/zdd-probe/$1/load-$i.log" |
| + curl -s ${RESOLVE:+--resolve "$RESOLVE"} -o /dev/null -m 35 -w '%{http_code} %{time_total}\n' "https://$HOST/?zdd=$1-$i-$n" >> "$TMPDIR/zdd-probe/$1/load-$i.log" |
| done ) & |
| done |
| wait |
| @@ -1199,7 +1242,8 @@ rerun the hold phase once with the policy already set. |
| rm -rf "$TMPDIR/zdd-probe" |
| ``` |
| -On dev, run `carlos apps delete zdd-probe` once Paul is done with it. |
| +On dev, run `carlos apps delete zdd-probe` once Paul is done with it. On |
| +the flagship, delete the Gate F app as Gate F says. |
| ## Names checked against this branch |