amadan
instance-control: a misplaced rootfs unit's files go only once it is proven gone; overlapping starts counted (rootfs apps plan 2, task 5 fix 1)
The C5/C10 refusal now proves the stop: systemctl stop must succeed, the unit must read inactive or failed, and the control group the post-start show reported (a misplaced unit's own, never one under the host slice) must be absent or empty. Otherwise the files stay and a generation or worker answers statusGenUnproven, which the client reads as ErrGenUnproven. The 503 names the first check that is not ok, so the probe-less stub says rootfs-probe. The activator's unitName answers an error for a worker and newUnitProcFor refuses it. genStarting counts the starts in flight, so a reused worker name's overlapping starts both hold the sweep off, and the exe layout's temp files are unique.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9 files changed,
+347
−51
internal/activator/backend_exec.go+11 −8internal/activator/rootfs_worker_unit_test.go+31 −0internal/activator/unitproc.go+26 −9internal/activator/unitproc_gen_test.go+4 −4internal/instancectl/client.go+3 −1internal/instancectl/gen.go+6 −4internal/instancectl/rootfs.go+48 −22internal/instancectl/rootfs_test.go+207 −0internal/instancectl/server.go+11 −3
CI failed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (2 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (21 s)
=== gofmt (1 s)
gofmt -l .: clean
=== rest (71 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 6.315s
ok github.com/carlosframework/platform/cmd/moveprobe 12.251s
ok github.com/carlosframework/platform/cmd/zddprobe 0.749s
ok github.com/carlosframework/platform/internal/activator 58.791s
ok github.com/carlosframework/platform/internal/agentsock 0.038s
ok github.com/carlosframework/platform/internal/alerting 0.005s
ok github.com/carlosframework/platform/internal/analytics 0.064s
ok github.com/carlosframework/platform/internal/appconfig 4.298s
ok github.com/carlosframework/platform/internal/apperr 0.151s
ok github.com/carlosframework/platform/internal/applog 0.315s
ok github.com/carlosframework/platform/internal/assetcache 0.109s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.007s
ok github.com/carlosframework/platform/internal/boxid 1.082s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.009s
ok github.com/carlosframework/platform/internal/caps 0.032s
ok github.com/carlosframework/platform/internal/certwarm 0.003s
ok github.com/carlosframework/platform/internal/cgroup 0.791s
ok github.com/carlosframework/platform/internal/changelog 0.492s
ok github.com/carlosframework/platform/internal/claimactivation 1.340s
ok github.com/carlosframework/platform/internal/clamd 0.066s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 8.309s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.019s
ok github.com/carlosframework/platform/internal/dbquery 0.018s
ok github.com/carlosframework/platform/internal/dbreads 2.230s
? github.com/carlosframework/platform/internal/dbreads/dbreadstest [no test files]
ok github.com/carlosframework/platform/internal/dbverify 0.014s
ok github.com/carlosframework/platform/internal/dnsrec 0.005s
ok github.com/carlosframework/platform/internal/docsite 0.007s
ok github.com/carlosframework/platform/internal/domains 4.204s
ok github.com/carlosframework/platform/internal/doorkey 0.008s
ok github.com/carlosframework/platform/internal/economics 0.004s
ok github.com/carlosframework/platform/internal/edge 47.584s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.028s
ok github.com/carlosframework/platform/internal/fleetapi 0.483s
ok github.com/carlosframework/platform/internal/gate 3.154s
ok github.com/carlosframework/platform/internal/genselect 0.370s
ok github.com/carlosframework/platform/internal/geo 0.015s
ok github.com/carlosframework/platform/internal/geoip 0.046s
ok github.com/carlosframework/platform/internal/historyreadback 0.563s
ok github.com/carlosframework/platform/internal/hostactors 7.640s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.026s
ok github.com/carlosframework/platform/internal/hostbox 5.057s
ok github.com/carlosframework/platform/internal/hostclaim 1.512s
ok github.com/carlosframework/platform/internal/hostenv 0.153s
ok github.com/carlosframework/platform/internal/hostevidence 0.301s
ok github.com/carlosframework/platform/internal/hostexec 7.759s
ok github.com/carlosframework/platform/internal/hostfence 0.088s
ok github.com/carlosframework/platform/internal/hostfiles 11.325s
ok github.com/carlosframework/platform/internal/hostfloor 0.034s
ok github.com/carlosframework/platform/internal/hostguard 1.230s
ok github.com/carlosframework/platform/internal/hostops 4.057s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.007s
ok github.com/carlosframework/platform/internal/hostquarantine 0.012s
ok github.com/carlosframework/platform/internal/hostrelocate 1.151s
ok github.com/carlosframework/platform/internal/hostresolve 0.230s
ok github.com/carlosframework/platform/internal/hostwrite 0.025s
ok github.com/carlosframework/platform/internal/infraview 0.003s
ok github.com/carlosframework/platform/internal/instancectl 5.671s
ok github.com/carlosframework/platform/internal/instanceexec 3.188s
ok github.com/carlosframework/platform/internal/instancefs 0.010s
ok github.com/carlosframework/platform/internal/instancekey 0.005s
ok github.com/carlosframework/platform/internal/instanceupdates 0.020s
ok github.com/carlosframework/platform/internal/layout 2.813s
ok github.com/carlosframework/platform/internal/lbhealth 0.007s
ok github.com/carlosframework/platform/internal/lease 10.397s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.730s
ok github.com/carlosframework/platform/internal/ledger 0.288s
ok github.com/carlosframework/platform/internal/mail 0.620s
ok github.com/carlosframework/platform/internal/maintenance 0.096s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.862s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.172s
ok github.com/carlosframework/platform/internal/metrics 0.006s
ok github.com/carlosframework/platform/internal/moves 24.750s
ok github.com/carlosframework/platform/internal/moves/archive 1.285s
ok github.com/carlosframework/platform/internal/objstore 0.667s
ok github.com/carlosframework/platform/internal/ops 0.481s
ok github.com/carlosframework/platform/internal/park 0.546s
ok github.com/carlosframework/platform/internal/plans 0.011s
ok github.com/carlosframework/platform/internal/privatefile 0.015s
2026/10/25 13:05:43 provision: listing apps/bab/blog/instances/: list refused — its records are unseen this pass
2026/10/25 13:05:43 provision: listing apps/bab/: listdirs refused — its records are unseen this pass
2026/10/25 13:05:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:43 provision: listing apps/bac/tito/instances/: list refused — its records are unseen this pass
2026/10/25 13:05:43 provision: listing apps/bac/tito/instances/: list refused — its records are unseen this pass
2026/10/25 13:05:43 provision: skipping record apps/bac/tito/instances/one.example.com.json: unreadable: read refused
2026/10/25 13:05:43 provision: listing apps/bac/tito/instances/: list refused — its records are unseen this pass
2026/10/25 13:05:43 provision: an app's records could not be listed: they are skipped
2026/10/25 13:05:43 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:43 provision: remove retired instance route shop.example: provision: claim admission unavailable
2026/10/25 13:05:43 provision: remove retired instance route shop.example: provision: claim admission unavailable
2026/10/25 13:05:43 provision: remove retired sidecar route worker.sidecar.internal: provision: claim admission unavailable
2026/10/25 13:05:43 provision: remove retired sidecar route worker.sidecar.internal: provision: claim admission unavailable
2026/10/25 13:05:43 provision: claim install acme.go.tito.io: provision: claim admission unavailable
2026/10/25 13:05:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:43 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/25 13:05:43 provision: claim install bot.eleven.bac.sidecar.internal: provision: claim admission unavailable
2026/10/25 13:05:43 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/25 13:05:43 provision: claim install bot.eleven.bac.sidecar.internal: registry: claim row ownership changed or unknown
2026/10/25 13:05:43 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/25 13:05:43 provision: acme.go.tito.io: not minting this pass: the box's layout intent could not be read: no sealed config
2026/10/25 13:05:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:43 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/25 13:05:43 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/25 13:05:43 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/25 13:05:43 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/25 13:05:43 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/25 13:05:44 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/25 13:05:44 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/25 13:05:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/25 13:05:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/25 13:05:44 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/25 13:05:44 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/25 13:05:44 provision: claim install bot.eleven.bac.sidecar.internal: registry: claim row ownership changed or unknown
2026/10/25 13:05:44 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/25 13:05:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/25 13:05:44 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/25 13:05:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:44 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:44 provision: skipping record apps/bac/tito/instances/new.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:44 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/25 13:05:44 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:44 provision: skipping record apps/bac/tito/instances/new.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:44 provision: minted instance route rl.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route rc.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route rf.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route rh.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route ri.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route rj.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route ra.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route rb.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route rd.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route re.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route rg.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:44 provision: minted instance route rk.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route ra.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route rb.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route rc.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route rd.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: skipping record apps/bac/tito/instances/rb.go.tito.io.json: read failed: read refused
2026/10/25 13:05:45 provision: skipping record apps/bac/tito/instances/rb.go.tito.io.json: read failed: read refused
2026/10/25 13:05:45 provision: minted instance route re.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.appd.example (app "appd" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.appe.example (app "appe" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.appg.example (app "appg" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.appc.example (app "appc" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.appf.example (app "appf" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.apph.example (app "apph" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.appi.example (app "appi" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.appj.example (app "appj" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.appa.example (app "appa" acct "bac")
2026/10/25 13:05:45 provision: minted instance route x.appb.example (app "appb" acct "bac")
2026/10/25 13:05:45 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/25 13:05:45 provision: minted instance route r0120.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0087.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0125.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0061.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0113.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0035.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0057.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0068.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0076.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0004.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0039.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0108.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0013.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0019.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0033.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0046.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0103.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0116.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0036.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0047.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0066.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0086.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0102.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0115.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0038.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0077.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0085.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0105.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0049.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0069.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0001.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0037.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0041.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0048.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0050.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0053.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0062.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0124.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0005.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0060.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0078.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0104.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0122.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0006.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0024.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0027.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0067.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0089.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0098.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0112.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0044.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0010.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0014.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0018.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0020.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0032.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0040.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0099.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0101.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0110.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0000.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0071.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0096.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0015.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0034.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0092.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0097.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0107.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0054.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0129.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0065.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0094.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0119.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0008.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0009.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0017.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0022.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0025.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0059.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0106.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0121.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0055.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0074.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0091.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0117.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0118.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0002.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0111.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0051.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0063.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0064.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0080.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0081.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0109.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0128.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0021.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0029.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0030.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0026.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0042.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0073.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0082.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0090.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0126.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0007.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0043.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0095.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0093.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0058.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0079.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0088.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0127.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0023.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0031.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0072.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0123.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0012.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0045.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0084.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0114.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0052.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0003.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0016.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0083.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0011.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0028.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0056.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0070.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0075.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:45 provision: minted instance route r0100.go.tito.io (app "tito" acct "bac")
--- FAIL: TestPlan9B8RecordWriteCensus (1.20s)
plan9_b8_census_test.go:372: record-write census rejected 2 site(s); update the explicit inventory only after reviewing the source:
stale inventory entry internal/instancectl/gen.go:GenDropIn: no such unit with a write site any more
unlisted site internal/instancectl/gen.go:hostMountLines [lit:instances]: review it, classify it, and update its explicit inventory entry
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted service route svc.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route whole.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route console.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted service route ha.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route shared.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: converged acme.go.tito.io channel stable -> edge
2026/10/25 13:05:48 provision: converged acme.go.tito.io channel edge -> canary/rehearsal
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: empty channel
2026/10/25 13:05:48 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: converged acme.go.tito.io environment "" -> "staging"
2026/10/25 13:05:48 provision: converged acme.go.tito.io environment "staging" -> ""
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: converged acme.go.tito.io steering "" -> "latency"
2026/10/25 13:05:48 provision: converged acme.go.tito.io steering "latency" -> ""
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: converged acme.go.tito.io hibernate false -> true
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: converged acme.go.tito.io hibernate true -> false (replication "daily" cleared: only a hibernating route's rung is read)
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: converged acme.go.tito.io status-named false -> true
2026/10/25 13:05:48 provision: converged acme.go.tito.io status-named true -> false
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted service route ha.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7001,127.0.0.1:7002" -> "127.0.0.1:7001,127.0.0.1:7003" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/25 13:05:48 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7001,127.0.0.1:7003" -> "127.0.0.1:7003,127.0.0.1:7001" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/25 13:05:48 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7003,127.0.0.1:7001" -> "127.0.0.1:7003,127.0.0.1:7001" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/healthz\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/25 13:05:48 provision: minted service route whole.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: converged whole.go.tito.io upstreams addr "127.0.0.1:9443" -> "", addrs "" -> "127.0.0.1:9443,127.0.0.1:9444" (probe "" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/25 13:05:48 provision: converged whole.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:9443,127.0.0.1:9444" -> "" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "")
2026/10/25 13:05:48 provision: whole.go.tito.io: record describes a different route SHAPE than the row (kind "instance"/"service" unit ""/"" socket "/run/carlos/whole.go.tito.io.sock"/"" db "/data/whole.go.tito.io.db"/"") — not converged in place; `carlos remove whole.go.tito.io` purges this route's capability grants (registry.Delete's contract, not a bug) before the next pass re-mints it, so treat that as a last resort, not a routine fix
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: minted service route whole.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: skipping accounts/bac/account.json: malformed account manifest: invalid character 'o' in literal null (expecting 'u')
2026/10/25 13:05:48 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: manifest bac/tito: read failed: provision: manifest bac/tito: invalid character 'o' in literal null (expecting 'u')
2026/10/25 13:05:48 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: disagrees with its own path (host="wrong-host.go.tito.io" account="bac" app="tito")
2026/10/25 13:05:48 provision: skipping record apps/bac/tito/instances/acme$evil.go.tito.io.json: host "acme$evil.go.tito.io" is not a hostname
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: skipping record apps/bac/tito/instances/acme$evil.go.tito.io.json: host "acme$evil.go.tito.io" is not a hostname
2026/10/25 13:05:48 provision: not removing acme$evil.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/25 13:05:48 provision: skipping record apps/bac/tito/instances/bad-service.go.tito.io.json: invalid shape: kind service with no addr: the platform only proxies to a service, so the record must name the upstream it proxies to
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: skipping record apps/bac/tito/instances/bad-service.go.tito.io.json: invalid shape: kind service with no addr: the platform only proxies to a service, so the record must name the upstream it proxies to
2026/10/25 13:05:48 provision: not removing bad-service.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/25 13:05:48 provision: skipping record apps/bac/tito/instances/no-channel.go.tito.io.json: empty channel
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: skipping record apps/bac/tito/instances/no-channel.go.tito.io.json: empty channel
2026/10/25 13:05:48 provision: not removing no-channel.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:48 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/25 13:05:48 provision: removed retired instance route ghost.go.tito.io (record gone)
2026/10/25 13:05:48 provision: duplicate host dup.go.tito.io: keeping apps/bac/tito/instances/dup.go.tito.io.json, skipping apps/bad/tito/instances/dup.go.tito.io.json
2026/10/25 13:05:48 provision: minted instance route dup.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/25 13:05:48 provision: removed retired instance route acme.go.tito.io (app in the trash)
2026/10/25 13:05:48 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/25 13:05:48 provision: removed retired static route site.go.tito.io (app in the trash)
2026/10/25 13:05:48 provision: trash marker for bac/tito unreadable: release: malformed trash marker for bac/tito: invalid character 'n' looking for beginning of object key string — not minting for this app
2026/10/25 13:05:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/25 13:05:48 provision: removed retired instance route acme.go.tito.io (app in the trash)
2026/10/25 13:05:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/25 13:05:48 provision: trash marker for bac/tito unreadable: release: malformed trash marker for bac/tito: invalid character 'n' looking for beginning of object key string — not minting for this app
2026/10/25 13:05:48 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/25 13:05:48 provision: converged acme.go.tito.io channel stable -> edge
2026/10/25 13:05:48 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/25 13:05:48 provision: converged acme.go.tito.io channel canary/rehearsal -> edge
2026/10/25 13:05:48 provision: converged acme.go.tito.io environment "staging" -> ""
2026/10/25 13:05:48 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/25 13:05:48 provision: acme.go.tito.io exists but is not the route this record describes (backing "unit" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/25 13:05:48 provision: acme.go.tito.io exists but is not the route this record describes (tunnel "pi-fleet" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/25 13:05:48 provision: acme.go.tito.io exists but is not the route this record describes (caps "port:25" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/25 13:05:48 provision: acme.go.tito.io exists but is not the route this record describes (platform (owned by the domains reconcile pass)) — leaving it alone
2026/10/25 13:05:48 provision: acme.go.tito.io exists but is not the route this record describes (canary (owned by the domains reconcile pass)) — leaving it alone
2026/10/25 13:05:48 provision: acme.go.tito.io exists but is not the route this record describes (db "/data/other.db"/"/data/acme.go.tito.io.db") — leaving it alone
2026/10/25 13:05:48 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/25 13:05:48 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/25 13:05:48 provision: dry-run: would mint instance route acme.go.tito.io (app "tito" acct "bac" channel "stable")
2026/10/25 13:05:48 provision: dry-run: would adopt existing route adopt.go.tito.io (app "tito" acct "bac") — this record would own its channel and environment
2026/10/25 13:05:48 provision: dry-run: would converge adopt.go.tito.io channel stable -> edge
2026/10/25 13:05:48 provision: dry-run: would converge adopt.go.tito.io environment "" -> "staging"
2026/10/25 13:05:48 provision: dry-run: would converge owned.go.tito.io channel stable -> edge
2026/10/25 13:05:48 provision: dry-run: would converge owned.go.tito.io environment "" -> "staging"
2026/10/25 13:05:48 provision: dry-run: would converge owned.go.tito.io hibernate false -> true
2026/10/25 13:05:48 provision: dry-run: would converge owned.go.tito.io seed
2026/10/25 13:05:48 provision: dry-run: would remove retired instance route gone.go.tito.io (record gone)
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:48 provision: adopted existing route adopt.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/25 13:05:48 provision: converged adopt.go.tito.io channel stable -> edge
2026/10/25 13:05:48 provision: converged adopt.go.tito.io environment "" -> "staging"
2026/10/25 13:05:48 provision: converged owned.go.tito.io channel stable -> edge
2026/10/25 13:05:48 provision: converged owned.go.tito.io environment "" -> "staging"
2026/10/25 13:05:48 provision: converged owned.go.tito.io hibernate false -> true
2026/10/25 13:05:48 provision: removed retired instance route gone.go.tito.io (record gone)
2026/10/25 13:05:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: adopted existing route story.carlosframework.com (app "story" acct "bad") — this record now owns its channel and environment
2026/10/25 13:05:49 provision: story.carlosframework.com exists but is not the route this record describes (kind "instance"/"static") — leaving it alone
2026/10/25 13:05:49 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted service route pub.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted service route pubs.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted service route loop.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted service route named.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted service route priv.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route unit.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route exec.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted static route pisite.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: removed retired static route site.go.tito.io (record gone)
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route acme-uk.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: converged acme.go.tito.io replicas "ie,uk" -> "uk"
2026/10/25 13:05:49 provision: converged acme.go.tito.io replicas "uk" -> ""
2026/10/25 13:05:49 provision: minted instance route acme-uk.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: converged acme-uk.go.tito.io replica mark true -> false
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: acme.go.tito.io: row is on the nested layout (db "/data/instances/acme.go.tito.io/acme.go.tito.io.db" socket "/run/carlos/instances/acme.go.tito.io/app.sock"); the record's shared paths are not drift
2026/10/25 13:05:49 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:49 provision: acme.go.tito.io: not minting this pass: the box's layout intent could not be read: store down
2026/10/25 13:05:49 provision: router.go.tito.io: not minting this pass: the box's layout intent could not be read: store down
2026/10/25 13:05:49 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/25 13:05:49 provision: converged acme.go.tito.io channel stable -> edge
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
2026/10/25 13:05:49 provision: not removing acme.go.tito.io (record gone): participant of a running operation
2026/10/25 13:05:49 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/25 13:05:49 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
2026/10/25 13:05:49 provision: refusing to mint held route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/25 13:05:49 provision: held acme.go.tito.io landing by mv_acme (was ready)
2026/10/25 13:05:49 provision: acme.go.tito.io: held restoring by rb_other, not by mv_acme; leaving it
2026/10/25 13:05:49 provision: acme.go.tito.io: the row is not this pass's (app "other" acct ""); not holding it for mv_acme
2026/10/25 13:05:49 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/25 13:05:49 provision: dry-run: would mint instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac" channel "stable")
2026/10/25 13:05:49 provision: acme.go.tito.io: not held landing by mv_acme: the row's hold changed under this pass; the next pass judges it afresh
2026/10/25 13:05:49 provision: held acme.go.tito.io landing by mv_acme (was ready)
2026/10/25 13:05:49 provision: minted instance route acme.example.com (app "shop" acct "bac")
2026/10/25 13:05:49 provision: minted instance route plain.example.com (app "shop" acct "bac")
2026/10/25 13:05:49 provision: skipping record apps/bac/shop/instances/garbled.shop.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:49 provision: skipping record apps/bac/shop/instances/liar.shop.example.com.json: disagrees with its own path (host="other.shop.example.com" account="bac" app="shop")
2026/10/25 13:05:49 provision: skipping record apps/bac/shop/instances/garbled.shop.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:49 provision: skipping record apps/bac/shop/instances/liar.shop.example.com.json: disagrees with its own path (host="other.shop.example.com" account="bac" app="shop")
2026/10/25 13:05:49 provision: skipping record apps/bac/shop/instances/garbled.shop.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:49 provision: skipping record apps/bac/shop/instances/liar.shop.example.com.json: disagrees with its own path (host="other.shop.example.com" account="bac" app="shop")
2026/10/25 13:05:49 provision: skipping record apps/bac/shop/instances/garbled.shop.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:49 provision: listing apps/bac/shop/instances/: list refused — its records are unseen this pass
2026/10/25 13:05:50 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:50 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:50 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/25 13:05:50 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:50 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/25 13:05:50 provision: minted sidecar route bot.shop.bac.sidecar.internal (app "shop" acct "bac" args ["run"])
2026/10/25 13:05:50 provision: removed retired sidecar route bot.shop.bac.sidecar.internal (record gone)
2026/10/25 13:05:50 provision: minted sidecar route bot.shop.bum.sidecar.internal (app "shop" acct "bum" args ["run"])
2026/10/25 13:05:50 provision: sidecar bot.shop.bum.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:50 provision: sidecar bot.shop.bum.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:50 provision: removed retired sidecar route bot.shop.bum.sidecar.internal (record gone)
2026/10/25 13:05:50 provision: minted sidecar route bot.shop.bum.sidecar.internal (app "shop" acct "bum" args ["run"])
2026/10/25 13:05:50 provision: sidecar bot.shop.bum.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:50 provision: sidecar bot.shop.bum.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:50 provision: removed retired sidecar route bot.shop.bum.sidecar.internal (record gone)
2026/10/25 13:05:51 provision: a.shop.example.com: swapped for import sd_eoky42j3xqd4g3c4nthb4632bm (the original route is journaled)
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_eoky42j3xqd4g3c4nthb4632bm committed: its gate is cleared and its journal row dropped
2026/10/25 13:05:51 provision: a.shop.example.com: row is on the nested layout (db "/data/instances/a.shop.example.com/a.shop.example.com.db" socket "/run/carlos/instances/a.shop.example.com/app.sock"); the record's shared paths are not drift
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_tqsa4wur6th6hz5eqk5fh4b7q4 rolled back to its original route
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_tqsa4wur6th6hz5eqk5fh4b7q4: the host's replica is left alone: the record apps/bac/shop/instances/a.shop.example.com.json holds the host
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_tqsa4wur6th6hz5eqk5fh4b7q4 is cleaned up and its journal row retired
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_rda2b3n2b7mseernde6csstfea rolled back to its original route
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_rda2b3n2b7mseernde6csstfea: the host's replica is left alone: the record apps/bac/shop/instances/a.shop.example.com.json holds the host
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_rda2b3n2b7mseernde6csstfea is cleaned up and its journal row retired
2026/10/25 13:05:51 provision: dry-run: would redo a.shop.example.com's swap record CAS for import sd_2ipqi2k5qhcp4xizrdslxvrdnq
2026/10/25 13:05:51 provision: dry-run: would clear a.shop.example.com's import gate
2026/10/25 13:05:51 provision: dry-run: would converge a.shop.example.com seed source
2026/10/25 13:05:51 provision: a.shop.example.com: row is on the nested layout (db "/data/instances/a.shop.example.com/a.shop.example.com.db" socket "/run/carlos/instances/a.shop.example.com/app.sock"); the record's shared paths are not drift
2026/10/25 13:05:51 provision: a.shop.example.com: the import's record apps/bac/shop/instances/a.shop.example.com.json is not one this pass can use (connection reset)
2026/10/25 13:05:51 provision: a.shop.example.com: the import's record apps/bac/shop/instances/a.shop.example.com.json is not one this pass can use (provision: invalid maintenance projection or protected fields)
2026/10/25 13:05:51 provision: a.shop.example.com: the import's record apps/bac/shop/instances/a.shop.example.com.json is not one this pass can use (the record under apps/bac/shop/instances/a.shop.example.com.json is apps/bac/shop/instances/b.shop.example.com.json's)
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_f4jmuwbjzrlhl77eeh236mfpby cannot roll back on this box: the activator is not wired; it stays journaled and gated
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_h5lqfdzizlv4y5ia5b76rd2h4y cannot roll back on this box: the activator is not wired; it stays journaled and gated
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_u5rzgj7pz45gqdsjiehxztmifi is journaled, but this box cannot act on imports; its row is left alone
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_5pndg37fb7jgpp6iofg6xn3f5a is journaled, but this box cannot act on imports; its row is left alone
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_tdn7lukbm3gg25w5ui7gtdrp7y is journaled, but this box cannot act on imports; its row is left alone
2026/10/25 13:05:51 provision: the import journal could not be read: registry: import journal a.shop.example.com: original row: unexpected end of JSON input (no import is acted on and no route is removed this pass)
2026/10/25 13:05:51 provision: b.shop.example.com: import sd_fyifulq7am57xpm2ihdt3a74g4 left alone this pass: the import journal could not be read
2026/10/25 13:05:51 provision: c.shop.example.com: import sd_mo2dkju4rxdgd7sg7chhahozji left alone this pass: the import journal could not be read
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_3y2fpn6ncozifkaefm6vvwzadu committed: its gate is cleared and its journal row dropped
2026/10/25 13:05:51 provision: a.shop.example.com: not converging import sd_q4fm7a5iagvaritekajbbzcasu: it is swapped but not committed, and the row on this box has no import gate
2026/10/25 13:05:51 provision: a.shop.example.com: not converging import sd_otm7cjk7rx2buii76b4je3jzta: it is swapped but not committed, and the row on this box has no import gate
2026/10/25 13:05:51 provision: a.shop.example.com: not converging import sd_plwvdzmn3vhaxocqeoaitewqqu: it is placed but not committed, and the row on this box has no import gate
2026/10/25 13:05:51 provision: a.shop.example.com: not converging import sd_e7th2rrfiks4gbp7nj5xckxl3u: it is placed but not committed, and the row on this box has no import gate
2026/10/25 13:05:51 provision: a.shop.example.com: row is on the nested layout (db "/data/instances/a.shop.example.com/a.shop.example.com.db" socket "/run/carlos/instances/a.shop.example.com/app.sock"); the record's shared paths are not drift
2026/10/25 13:05:51 provision: converge seed source on a.shop.example.com: constraint failed: seed_source is read-only here (1811)
2026/10/25 13:05:51 provision: a.shop.example.com: row is on the nested layout (db "/data/instances/a.shop.example.com/a.shop.example.com.db" socket "/run/carlos/instances/a.shop.example.com/app.sock"); the record's shared paths are not drift
2026/10/25 13:05:51 provision: a.shop.example.com: import sd_bzi6ewivdjj6jpyx6z4hfgavke committed: its gate is cleared and its journal row dropped
2026/10/25 13:05:51 provision: a.shop.example.com: row is on the nested layout (db "/data/instances/a.shop.example.com/a.shop.example.com.db" socket "/run/carlos/instances/a.shop.example.com/app.sock"); the record's shared paths are not drift
2026/10/25 13:05:51 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:51 provision: acme.go.tito.io: row is on the nested layout (db "/data/instances/acme.go.tito.io/acme.go.tito.io.db" socket "/run/carlos/instances/acme.go.tito.io/app.sock"); the record's shared paths are not drift
2026/10/25 13:05:51 provision: acme.go.tito.io: not minting a seeded instance: claim control is on for this box
2026/10/25 13:05:51 provision: acme.go.tito.io: not minting a seeded instance: this box does not mint nested (<nil>)
2026/10/25 13:05:52 provision: acme.go.tito.io: aborted seed sd_llaxqfaracnjil7x2yvb2jzxqe: this box cannot read seed markers, and places no seed; nothing local is removed
2026/10/25 13:05:52 provision: acme.go.tito.io: aborted seed sd_llaxqfaracnjil7x2yvb2jzxqe rolled back (no box had swapped it)
2026/10/25 13:05:52 provision: acme.go.tito.io: not minting a seeded instance: its record does not describe a hibernating instance
2026/10/25 13:05:52 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:52 provision: removed instance route acme.go.tito.io (its seed sd_hdi2fgyus4mbo2o27odbggzen4 was aborted); the next pass rolls it back
2026/10/25 13:05:52 provision: acme.go.tito.io: aborted seed sd_hdi2fgyus4mbo2o27odbggzen4: this box cannot read seed markers, and places no seed; nothing local is removed
2026/10/25 13:05:52 provision: acme.go.tito.io: aborted seed sd_hdi2fgyus4mbo2o27odbggzen4 rolled back (no box had swapped it)
2026/10/25 13:05:52 provision: acme.go.tito.io: aborted seed sd_53pn4jpri3h75tqhg6zi7si6p4 has a row this pass may not remove; leaving the seed aborted
2026/10/25 13:05:52 provision: acme.go.tito.io: aborted seed sd_ynnjt7on4hqygnl6j7lspzop44 rolled back (no box had swapped it)
2026/10/25 13:05:52 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:52 provision: removed instance route acme.go.tito.io (its seed sd_sav6qery63wmshu7iiyncruiy4 was aborted); the next pass rolls it back
2026/10/25 13:05:52 provision: acme.go.tito.io: aborted seed sd_sav6qery63wmshu7iiyncruiy4 has no route on this box: its local data is removed
2026/10/25 13:05:52 provision: acme.go.tito.io: aborted seed sd_sav6qery63wmshu7iiyncruiy4 rolled back (no box had swapped it)
2026/10/25 13:05:52 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_wnuhnf6rqvwnrykmm2iz3fneim (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_wnuhnf6rqvwnrykmm2iz3fneim rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_wnuhnf6rqvwnrykmm2iz3fneim is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_cjvqxe3ohkn5tq36bxj2ivyjty (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_cjvqxe3ohkn5tq36bxj2ivyjty rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_cjvqxe3ohkn5tq36bxj2ivyjty is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_ukastlc75naxm5rtke3uzzuaiy (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_ukastlc75naxm5rtke3uzzuaiy rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_ukastlc75naxm5rtke3uzzuaiy: the host's replica is left alone: its record no longer names this seed
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_ukastlc75naxm5rtke3uzzuaiy is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_wre2mbg44oxi4ybuph5i3v4m4e (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_wre2mbg44oxi4ybuph5i3v4m4e rollback deferred: the host is not fenced and stopped yet (<nil>)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_wre2mbg44oxi4ybuph5i3v4m4e rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_wre2mbg44oxi4ybuph5i3v4m4e is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_g4wh7klgssnfsna6eqpcebtw2e (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_g4wh7klgssnfsna6eqpcebtw2e rollback deferred: the host is not fenced and stopped yet (stop unproven)
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_bejvn5wcik7f2q5dgsstynwfyq (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_bejvn5wcik7f2q5dgsstynwfyq is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_uj5y2hjf5hx5yby3wybwaocrqi (the original route is journaled)
2026/10/25 13:05:52 provision: skipping record apps/bac/shop/instances/a.shop.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_uj5y2hjf5hx5yby3wybwaocrqi left alone this pass: its record could not be read
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_lnw6gjnqzqavvplfowjwh2ndya (the original route is journaled)
2026/10/25 13:05:52 provision: skipping record apps/bac/shop/instances/a.shop.example.com.json: read failed: connection reset
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_lnw6gjnqzqavvplfowjwh2ndya left alone this pass: its record could not be read
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_imhgkdcubcsv73wmw2agb266em (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_imhgkdcubcsv73wmw2agb266em: close the import gate: constraint failed: injected (1811)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_imhgkdcubcsv73wmw2agb266em rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_imhgkdcubcsv73wmw2agb266em is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_k7dleq7ifhao56wfb5xhtgzg4e (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_k7dleq7ifhao56wfb5xhtgzg4e: close the import gate: constraint failed: injected (1811)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_k7dleq7ifhao56wfb5xhtgzg4e rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_k7dleq7ifhao56wfb5xhtgzg4e is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_rh7ip74cxjtl2v74fjjzx6hslu (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_rh7ip74cxjtl2v74fjjzx6hslu rollback deferred: the host is not fenced and stopped yet (injected failure at fence)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_rh7ip74cxjtl2v74fjjzx6hslu rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_rh7ip74cxjtl2v74fjjzx6hslu is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_uz7jjfcminstonianbdcjdub2i (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_uz7jjfcminstonianbdcjdub2i rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_uz7jjfcminstonianbdcjdub2i is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_yzizgdgz3dtcbpsipg2aoitga4 (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_yzizgdgz3dtcbpsipg2aoitga4: retire its instance key: injected failure at retire-key
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_yzizgdgz3dtcbpsipg2aoitga4 rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_yzizgdgz3dtcbpsipg2aoitga4 is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_rcpzxctcpv4ejwz7parugtvikq (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_rcpzxctcpv4ejwz7parugtvikq rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_rcpzxctcpv4ejwz7parugtvikq is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_imzy72jnsywjmqgtkvhk2ekjky (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_imzy72jnsywjmqgtkvhk2ekjky: remove its local data: injected failure at remove-local
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_imzy72jnsywjmqgtkvhk2ekjky rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_imzy72jnsywjmqgtkvhk2ekjky is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_djndz5pweqkktwvvu4yv2ud44a (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_djndz5pweqkktwvvu4yv2ud44a rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_djndz5pweqkktwvvu4yv2ud44a is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_hxx6ihagmqje6w3ft35upjlyr4 (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_hxx6ihagmqje6w3ft35upjlyr4: remove its local data: injected failure at remove-local-done
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_hxx6ihagmqje6w3ft35upjlyr4 rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_hxx6ihagmqje6w3ft35upjlyr4 is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_mw7sti66upx4qixzmuf3nefccu (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_mw7sti66upx4qixzmuf3nefccu rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_mw7sti66upx4qixzmuf3nefccu is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_vauknv5ihmks4k72wm2p2yiuqe (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_vauknv5ihmks4k72wm2p2yiuqe: restore the original route: constraint failed: injected (1811)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_vauknv5ihmks4k72wm2p2yiuqe rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_vauknv5ihmks4k72wm2p2yiuqe is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_cmpommckf2tvat7wyvyru7plwi (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_cmpommckf2tvat7wyvyru7plwi: restore the original route: constraint failed: injected (1811)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_cmpommckf2tvat7wyvyru7plwi rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_cmpommckf2tvat7wyvyru7plwi is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_wzr6arifoo6nihkdg252t532me (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_wzr6arifoo6nihkdg252t532me is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_666yxmwdtchtk3tkkl5n4l7y6e (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_666yxmwdtchtk3tkkl5n4l7y6e rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_666yxmwdtchtk3tkkl5n4l7y6e: delete its snapshot: injected failure at delete-snapshot
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_666yxmwdtchtk3tkkl5n4l7y6e is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_a43tylpox67wlhpgqc6qy2dx7i (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_a43tylpox67wlhpgqc6qy2dx7i rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_a43tylpox67wlhpgqc6qy2dx7i is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_gfzl7npsvdenhso3ldlch3mdpa (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_gfzl7npsvdenhso3ldlch3mdpa rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_gfzl7npsvdenhso3ldlch3mdpa: delete the host's replica: injected failure at delete-replica
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_gfzl7npsvdenhso3ldlch3mdpa is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_5lggdfzoq324dcdsqqhgkjeqdu (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_5lggdfzoq324dcdsqqhgkjeqdu rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_5lggdfzoq324dcdsqqhgkjeqdu is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_yl3drv4yscbq5ogsgdbmitgy4a (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_yl3drv4yscbq5ogsgdbmitgy4a rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_yl3drv4yscbq5ogsgdbmitgy4a: retire its journal row: constraint failed: injected (1811)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_yl3drv4yscbq5ogsgdbmitgy4a is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_mhmbn2r2a5hswsfoutitx7fube (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_mhmbn2r2a5hswsfoutitx7fube rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_mhmbn2r2a5hswsfoutitx7fube: retire its journal row: constraint failed: injected (1811)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_mhmbn2r2a5hswsfoutitx7fube is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_oswlp35pkpgpd4fkjj3z2ib7iu (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_oswlp35pkpgpd4fkjj3z2ib7iu rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_oswlp35pkpgpd4fkjj3z2ib7iu is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: mark import sd_oswlp35pkpgpd4fkjj3z2ib7iu rolled back: injected failure at cas
2026/10/25 13:05:52 provision: a.shop.example.com: aborted seed sd_oswlp35pkpgpd4fkjj3z2ib7iu rolled back (no box had swapped it)
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_73ylst6nzgm4gxsqfpgm5sqwwm (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_73ylst6nzgm4gxsqfpgm5sqwwm rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_73ylst6nzgm4gxsqfpgm5sqwwm is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: aborted seed sd_73ylst6nzgm4gxsqfpgm5sqwwm rolled back (no box had swapped it)
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_dpqatqqfyxijszdxjx5zmpfqma (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_dpqatqqfyxijszdxjx5zmpfqma rollback deferred: the host is not fenced and stopped yet (<nil>)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_dpqatqqfyxijszdxjx5zmpfqma rollback deferred: the host is not fenced and stopped yet (<nil>)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_dpqatqqfyxijszdxjx5zmpfqma rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_dpqatqqfyxijszdxjx5zmpfqma is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_y72thd4gvtm27jwzcz4ibveizq (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_y72thd4gvtm27jwzcz4ibveizq rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_y72thd4gvtm27jwzcz4ibveizq is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_3xjladqc2uxj2lscrebcu4sfsq (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_3xjladqc2uxj2lscrebcu4sfsq rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_3xjladqc2uxj2lscrebcu4sfsq is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: minted instance route a.shop.example.com (app "shop" acct "bac")
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_2rdykklp76cuuchbtxnxbytp2m (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_2rdykklp76cuuchbtxnxbytp2m rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_2rdykklp76cuuchbtxnxbytp2m is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: minted instance route a.shop.example.com (app "shop" acct "bac")
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_dgkywfulyx4d7ma3qtwuyhxcae (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_dgkywfulyx4d7ma3qtwuyhxcae rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_dgkywfulyx4d7ma3qtwuyhxcae is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: minted instance route a.shop.example.com (app "shop" acct "bac")
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_mqhkv3vhfalwnob4khdfdcpbw4 (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_mqhkv3vhfalwnob4khdfdcpbw4 rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_mqhkv3vhfalwnob4khdfdcpbw4 is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: minted instance route a.shop.example.com (app "other" acct "bac")
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_ffofrm25h7f3ki6g4mjylca76a (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_ffofrm25h7f3ki6g4mjylca76a rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_ffofrm25h7f3ki6g4mjylca76a is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: minted instance route a.shop.example.com (app "other" acct "bac")
2026/10/25 13:05:52 provision: a.shop.example.com: swapped for import sd_hys3co65njjbg24tmx6ltwd5lq (the original route is journaled)
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_hys3co65njjbg24tmx6ltwd5lq rolled back to its original route
2026/10/25 13:05:52 provision: a.shop.example.com: import sd_hys3co65njjbg24tmx6ltwd5lq is cleaned up and its journal row retired
2026/10/25 13:05:52 provision: minted instance route a.shop.example.com (app "other" acct "bac")
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_dxhcalcksexhac6benpmirqzke (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_dxhcalcksexhac6benpmirqzke has no route on this box: its local data is removed and no route is put back
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_dxhcalcksexhac6benpmirqzke is cleaned up and its journal row retired
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_7xf4rjqdzvfyut6daxzgxxpumq (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_omuiuqanwuvgsc2ldmjdu4rwoa (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_lakvlshyuup67aeklkabspdqae (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_lakvlshyuup67aeklkabspdqae cannot roll back on this box: the activator is not wired; it stays journaled and gated
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_l4iupwgvvv3cpu43qlwikgv5f4 (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_l4iupwgvvv3cpu43qlwikgv5f4 is not rolled back: the route changed under the fence, so nothing there is touched
2026/10/25 13:05:53 provision: a.shop.example.com: import "../not-a-seed" names no snapshot and no replica; only its journal row is cleaned up
2026/10/25 13:05:53 provision: a.shop.example.com: import ../not-a-seed is cleaned up and its journal row retired
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_dgku245st3s7baq7skgwrc6tp4 (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_bayd7cfcxydrfmtqdqaq6jhn24 (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_tbfsrqxej7v3qyldjauakn4qhy (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_t5ioierbvh7wxpq5cmseisgpnu (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_45ucg2agqu4e2xjl5z2nzrg664 (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_45ucg2agqu4e2xjl5z2nzrg664 rollback deferred: the host is not fenced and stopped yet (stop evidence unreadable)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_mtfhdtveutwi3cyuyrjkzf3ama (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_mtfhdtveutwi3cyuyrjkzf3ama has no route on this box: its local data is removed and no route is put back
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_mtfhdtveutwi3cyuyrjkzf3ama is cleaned up and its journal row retired
2026/10/25 13:05:53 provision: minted instance route a.shop.example.com (app "shop" acct "bac")
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_fom2girohpkzqsb22tmpflsx74 (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_fom2girohpkzqsb22tmpflsx74: remove its local data: injected failure at remove-local
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_fom2girohpkzqsb22tmpflsx74 is not rolled back: the route at the host is not the import's, so nothing there is touched
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_2lzmimbw6pboobp63p7mg62jre (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_2lzmimbw6pboobp63p7mg62jre: a route appeared at the host, so its local database is left and the rollback waits
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_6udemfgoimden4nkqapb3jcusm (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_6udemfgoimden4nkqapb3jcusm rollback deferred: no fence generation: registry busy
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_3ovv6oo5fflfnaoz37ihihwyri (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_3ovv6oo5fflfnaoz37ihihwyri rollback deferred: no fence generation: the generation source answered 0
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_izdvhjgut53qbrow76pw5qfojy (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_p474ew4py627s3t2gjarvqo5sq (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_kvr5qovdakq5dwppzq3byawzkq (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import deferred: this box cannot read the instance's env file
2026/10/25 13:05:53 provision: a.shop.example.com: import deferred: the instance's env file could not be read: a symlink
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_66cqztpgfnosfze4s6yyeowqmq (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_66cqztpgfnosfze4s6yyeowqmq rolled back to its original route
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_66cqztpgfnosfze4s6yyeowqmq is cleaned up and its journal row retired
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_mlkpguu3x7qjkvqdhxig6opyle (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_mlkpguu3x7qjkvqdhxig6opyle rolled back to its original route
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_mlkpguu3x7qjkvqdhxig6opyle is cleaned up and its journal row retired
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_djghhh5ridhqnws74wj73zplqy (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_djghhh5ridhqnws74wj73zplqy rolled back to its original route
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_djghhh5ridhqnws74wj73zplqy is cleaned up and its journal row retired
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_hxgbe5f2r7jsivybbxlxwhbwny (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_hxgbe5f2r7jsivybbxlxwhbwny: retire its instance key: the env file is a symlink
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_hxgbe5f2r7jsivybbxlxwhbwny rolled back to its original route
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_hxgbe5f2r7jsivybbxlxwhbwny is cleaned up and its journal row retired
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_r6ha5sbgm4qxoydat4wldzhise (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: row is on the nested layout (db "/data/instances/a.shop.example.com/a.shop.example.com.db" socket "/run/carlos/instances/a.shop.example.com/app.sock"); the record's shared paths are not drift
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_lwpngz53jmsrq5w4bo2rc5hwne (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_xzhy77nxonkx6bowmtchpb3wie refused: there is no route at a.shop.example.com to replace
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_rkjv4h2cdyuqgsxj3n2eywl7oa refused: the route's address is 127.0.0.1:9, not 127.0.0.1:8401
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_cappe5zwlkbzbciygfvwqt2vwe refused: the route's address is 127.0.0.1:8401,127.0.0.1:8402, not 127.0.0.1:8401
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_wzytqxzrsrughcijirli4kjxmm refused: the route's address is 127.0.0.1:9, not 127.0.0.1:8401
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_mhjijt7r5bw4lz3grngu2dztpe refused: the route belongs to app other
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_rixraj3v3yajoky46fw4r5pjku refused: the route is filed under another account
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_fuylfafp5p2edu66n4rwfomlom refused: the route is filed under the operator account, and no assignment maps shop to this app
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_fsnae2rp7yjwatcenzqbgiwupe refused: the route is filed under the operator account, and no assignment maps shop to this app
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_jshznx7r2vkzy3mth3h6hzcdda refused: the route is filed under the operator account, and no assignment maps shop to this app
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_gszxkvzihupyrp7gnuek5xnr4e refused: the route is filed under the operator account, and no assignment maps shop to this app
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_onf2nwuwq27iv7gvxkwsccg2dm refused: the route is not an address route this import can take over (socket)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_m4rqnzah6dcniwnt33m526u6cm refused: the route is not an address route this import can take over (no address)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_hf3oepcqepbgznpihewrqt35g4 refused: the route is not an address route this import can take over (provisioned)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_ktvm26kuvuqvzl4ekicf4lryvq refused: the route is not an address route this import can take over (kind static)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_ht5m4hvol5mcrh6ymk36fn6pua refused: the route is not an address route this import can take over (backing)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_4yxctxy5apd3fhelxqrs54hmiq refused: the route is not an address route this import can take over (tunnel)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_ity5inihouzjys2joquf56esli refused: the route is not an address route this import can take over (caps)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_zdf74tm6vo343p3pcmf3bponom refused: the route is not an address route this import can take over (platform)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_pq6sww44tbjdkwbhvwsamzs2qy refused: the route is not an address route this import can take over (canary)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_gyvtkpudjouavumexo47ihk2ei refused: the route is not an address route this import can take over (kind sidecar)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_e55fm7bp5vzl5loh5srfsfdfya refused: the route is not an address route this import can take over (region)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_v7ch6opywhwipyhbs335wuybxa refused: the route is not an address route this import can take over (fleet_box)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_44esnt4d47m6nuzki5qbyjpi5m refused: the route is not an address route this import can take over (hibernate)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_di7p5ut5xnddkhqha2piwazpfq refused: the route is not an address route this import can take over (claim_owner)
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_x5ufn2xnm2nd5ybf5dhlikgczq refused: this box does not run instances nested
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_a2ojo3vcmi6vdlpr4z32omlp5y refused: this box does not run instances nested
2026/10/25 13:05:53 provision: a.shop.example.com: import sd_tvfzouykyk23jox7wlhyglhiby refused: claim control is on for this box
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_jbzg6xsfukrlybtygcoi5ifhnu (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: swapped for import sd_wlb5wnb6nweexjy4j2u4bjkuae (the original route is journaled)
2026/10/25 13:05:53 provision: a.shop.example.com: import deferred: this box cannot read the legacy route assignments
2026/10/25 13:05:53 provision: a.shop.example.com: import deferred: the legacy route assignments could not be read: store down
2026/10/25 13:05:53 provision: a.shop.example.com: not swapping for import sd_e42mt5mjsue2avmr7ycdamjebi: its verify hash is not a sha256
2026/10/25 13:05:53 provision: a.shop.example.com: not swapping for import sd_arrhvofki4hw2rsvmarlhwcz4e: its verify hash is not a sha256
2026/10/25 13:05:54 provision: a.shop.example.com: not swapping for import sd_mrhtsw2yttak6husqu6phqjnjy: its verify hash is not a sha256
2026/10/25 13:05:54 provision: a.shop.example.com: not swapping for import sd_../../x: its host or seed id is malformed (seeds: not a seed id)
2026/10/25 13:05:54 provision: A.shop.example.com/..: not swapping for import sd_3gcl67wb3dvun27qez3n3thm24: its host or seed id is malformed (seeds: not a host)
2026/10/25 13:05:54 provision: a.shop.example.com: not swapping for import sd_a2wb44znjn2wkkf5msvbuczuam: its record does not describe a nested, hibernating instance
2026/10/25 13:05:54 provision: a.shop.example.com: not swapping for import sd_qvbfzcfpyc55j2vm6pe6jyfcsy: its record does not describe a nested, hibernating instance
2026/10/25 13:05:54 provision: a.shop.example.com: not swapping for import sd_ewicd55y6f7jxlocb7bzvpikoq: its record does not describe a nested, hibernating instance
2026/10/25 13:05:54 provision: a.shop.example.com: not swapping for import sd_jel2yix2xzkkjwh2ig5qkc3ntq: its record does not describe a nested, hibernating instance
2026/10/25 13:05:54 provision: a.shop.example.com: import deferred: the box's layout intent could not be read: store down
2026/10/25 13:05:54 provision: a.shop.example.com: swapped for import sd_q4qy4uxnjzm2sicms56gnxcngu (the original route is journaled)
2026/10/25 13:05:54 provision: a.shop.example.com: import deferred: the box's nested probe did not answer: ready probe timed out
2026/10/25 13:05:54 provision: a.shop.example.com: swapped for import sd_pyfzgn3aqkdjfmqjmmsatvtuxy (the original route is journaled)
2026/10/25 13:05:54 provision: a.shop.example.com: swapped for import sd_oqx7f2olbbxtog6orz2n2vxcs4 (the original route is journaled)
2026/10/25 13:05:54 provision: a.shop.example.com: import deferred: the route is held by operation op_1
2026/10/25 13:05:54 provision: dry-run: would consider swapping a.shop.example.com for import sd_62vebwxhsogiiacaat5uxvzgei
2026/10/25 13:05:54 provision: a.shop.example.com: import sd_eo4zli3tlh26bsj4odlumg6pcq rolled back to its original route
2026/10/25 13:05:54 provision: a.shop.example.com: import sd_eo4zli3tlh26bsj4odlumg6pcq: the host's replica is left alone: the record apps/bac/shop/instances/a.shop.example.com.json holds the host
2026/10/25 13:05:54 provision: a.shop.example.com: import sd_eo4zli3tlh26bsj4odlumg6pcq is cleaned up and its journal row retired
2026/10/25 13:05:54 provision: a.shop.example.com: import sd_s5hgxtaspj5wj77n4nfhnyjdje left alone: the import journal holds seed sd_2eu67mef3fpm7kjtkab6y6hshy at swapped
2026/10/25 13:05:54 provision: a.shop.example.com: not minting import sd_beiwqn3mzj72tebd4m7qhdi2he: it is swapped but not committed, and only its swap makes its row
2026/10/25 13:05:54 provision: a.shop.example.com: not minting import sd_p2xjllabsga2mdpewwdtsm2v6i: it is placed but not committed, and only its swap makes its row
2026/10/25 13:05:54 provision: a.shop.example.com: swapped for import sd_vim2rwkynwp5z7utopl6bdt7ya (the original route is journaled)
2026/10/25 13:05:54 provision: a.shop.example.com: not minting import sd_vim2rwkynwp5z7utopl6bdt7ya: it is swapped but not committed, and only its swap makes its row
2026/10/25 13:05:54 provision: a.shop.example.com: not minting import sd_vim2rwkynwp5z7utopl6bdt7ya: it is placed but not committed, and only its swap makes its row
2026/10/25 13:05:54 provision: minted instance route a.shop.example.com (app "shop" acct "bac")
2026/10/25 13:05:54 provision: removed retired sidecar route gone.eleven.bac.sidecar.internal (record gone)
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: converged sidecar bot.eleven.bac.sidecar.internal args ["sidecar","run"] -> ["worker","run"]
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: converged sidecar bot.eleven.bac.sidecar.internal ceilings 300/3600 -> 60/120
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: converged sidecar bot.eleven.bac.sidecar.internal channel stable -> canary/eleven
2026/10/25 13:05:54 provision: converged sidecar bot.eleven.bac.sidecar.internal environment "other" -> "relay"
2026/10/25 13:05:54 provision: converged sidecar bot.eleven.bac.sidecar.internal hibernate true -> false
2026/10/25 13:05:54 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: read failed: failStore: reads disabled under apps/bac/eleven/sidecars/
2026/10/25 13:05:54 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: malformed: invalid character 'n' looking for beginning of object key string
2026/10/25 13:05:54 provision: not removing sidecar bot.eleven.bac.sidecar.internal: its record exists but could not be read cleanly this pass
2026/10/25 13:05:54 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: empty channel
2026/10/25 13:05:54 provision: skipping sidecar record apps/bac/eleven/sidecars/held.json: empty channel
2026/10/25 13:05:54 provision: not removing sidecar held.eleven.bac.sidecar.internal: its record exists but could not be read cleanly this pass
2026/10/25 13:05:54 provision: removed retired sidecar route old.eleven.bac.sidecar.internal (record gone)
2026/10/25 13:05:54 provision: bac/eleven has sidecar records but no sidecars manifest — not minting
2026/10/25 13:05:54 provision: bac/eleven is in the trash — not minting sidecar routes for it
2026/10/25 13:05:54 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:54 provision: sidecar hub.relay.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: minted sidecar route hub.relay.bac.sidecar.internal (app "relay" acct "bac" args ["relay"])
2026/10/25 13:05:54 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:54 provision: removed retired sidecar route bot.eleven.bac.sidecar.internal (record gone)
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/25 13:05:54 provision: skipping sidecar record apps/bac/relay/sidecars/hub.json: read failed: failStore: reads disabled under apps/bac/relay/sidecars/
2026/10/25 13:05:54 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/25 13:05:54 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/25 13:05:54 provision: dry-run: would mint sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" channel "stable" args ["sidecar","run"])
2026/10/25 13:05:54 provision: dry-run: would remove retired sidecar route gone.eleven.bac.sidecar.internal (record gone)
2026/10/25 13:05:54 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/25 13:05:54 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:54 provision: converged acme.go.tito.io update settings "{\"updates\":\"manual\",\"channel\":\"stable\"}" -> ""
2026/10/25 13:05:54 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:54 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:54 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/25 13:05:54 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/25 13:05:54 provision: acme.go.tito.io: not minted this pass: the app's update policy could not be read
2026/10/25 13:05:54 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/25 13:05:54 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/25 13:05:54 provision: acme.go.tito.io: not minted this pass: the app's update policy could not be read
2026/10/25 13:05:54 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
2026/10/25 13:05:54 provision: listing apps/bqd/: listdirs refused — its records are unseen this pass
2026/10/25 13:05:54 provision: listing apps/bqd/: listdirs refused — its records are unseen this pass
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: record view: change index unusable (release: change index unreadable: store has no conditional read: unsupported operation): the walk records no generations
2026/10/25 13:05:54 provision: record view: change index unusable (get refused): walking
2026/10/25 13:05:54 provision: record view: change index unusable (release: change index malformed: invalid character 'n' looking for beginning of object key string): walking
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: listing apps/bqd/: listdirs refused — its records are unseen this pass
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: listing apps/bqd/: listdirs refused — its records are unseen this pass
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: listing apps/bqd/: listdirs refused — its records are unseen this pass
2026/10/25 13:05:54 provision: record view: re-reading apps/bqc/shop failed: retrying on the next refresh
2026/10/25 13:05:54 provision: listing apps/bqc/shop/instances/: context canceled — its records are unseen this pass
2026/10/25 13:05:54 provision: record view: re-reading apps/bqc/shop failed: retrying on the next refresh
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: listing apps/bqc/shop/instances/: list refused — its records are unseen this pass
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: listing apps/bqc/shop/instances/: list refused — its records are unseen this pass
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: an app's records could not be listed: they are skipped
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping accounts/bqd/account.json: malformed account manifest: invalid character 'n' looking for beginning of object key string
2026/10/25 13:05:54 provision: skipping accounts/bqd/account.json: malformed account manifest: invalid character 'n' looking for beginning of object key string
2026/10/25 13:05:54 provision: skipping accounts/bqd/account.json: malformed account manifest: invalid character 'n' looking for beginning of object key string
2026/10/25 13:05:54 provision: record view: re-reading accounts/bqd failed: retrying on the next refresh
2026/10/25 13:05:54 provision: skipping accounts/bqd/account.json: read failed: list refused
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: minted instance route one.example.com (app "shop" acct "bum")
2026/10/25 13:05:54 provision: skipping record apps/bum/shop/instances/one.example.com.json: empty channel
2026/10/25 13:05:54 provision: not removing one.example.com: its record exists but could not be read cleanly this pass
2026/10/25 13:05:54 provision: skipping record apps/bum/shop/instances/one.example.com.json: empty channel
2026/10/25 13:05:54 provision: not removing one.example.com: its record exists but could not be read cleanly this pass
2026/10/25 13:05:54 provision: minted instance route one.example.com (app "shop" acct "bum")
2026/10/25 13:05:54 provision: removed retired instance route one.example.com (record gone)
2026/10/25 13:05:54 provision: minted instance route one.example.com (app "shop" acct "bum")
2026/10/25 13:05:54 provision: removed retired instance route one.example.com (record gone)
2026/10/25 13:05:54 provision: record view: change index unusable (context deadline exceeded): walking
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/25 13:05:54 provision: record view: change index unusable (context deadline exceeded): the walk records no generations
2026/10/25 13:05:54 provision: skipping record apps/bqe/store/instances/bad.store.example.com.json: malformed: provision: invalid maintenance projection or protected fields
--- FAIL: TestPlan14RootReadCensus (16.18s)
plan14_root_reads_test.go:600: root read census: 66 discrepancies
changed registry schema
changed root read internal/activator/backend_exec.go:ExecBackend.LegacyRunGone; review class, task, sites and digest
changed root read internal/activator/backend_exec.go:ExecBackend.killFailedGen; review class, task, sites and digest
changed root read internal/activator/backend_exec.go:ExecBackend.spawnUnitFrom; review class, task, sites and digest
changed root read internal/activator/backend_exec.go:ExecBackend.startSidecarUnitFrom; review class, task, sites and digest
changed root read internal/activator/backend_exec.go:ExecBackend.startUnitAt; review class, task, sites and digest
changed root read internal/activator/backend_exec.go:ExecBackend.waitRoleCgroupEmpty; review class, task, sites and digest
changed root read internal/activator/unitproc.go:ExecBackend.newUnitProc; review class, task, sites and digest
changed root read internal/activator/unitproc.go:ExecBackend.newUnitProcFor; review class, task, sites and digest
changed root read internal/host/buildstate.go:buildStates.retained; review class, task, sites and digest
changed root read internal/host/claimunit_dropin_linux.go:openClaimUnitDropin; review class, task, sites and digest
changed root read internal/instancectl/cmd.go:Server.handleCmdStatus; review class, task, sites and digest
changed root read internal/instancectl/server.go:Server.RunProbe; review class, task, sites and digest
changed root read internal/instancectl/server.go:parseRequest; review class, task, sites and digest
changed root read internal/instancectl/server.go:runWithClaim; review class, task, sites and digest
changed root read internal/instancectl/server.go:type Server; review class, task, sites and digest
changed root read internal/registry/registry.go:Registry.Upsert; review class, task, sites and digest
changed root read internal/registry/registry.go:Registry.migrate; review class, task, sites and digest
changed root read internal/registry/registry.go:routeArgs; review class, task, sites and digest
changed root read internal/registry/registry.go:scanRoute; review class, task, sites and digest
changed root read internal/release/adopt.go:adoptHost; review class, task, sites and digest
changed root source envelope internal/activator
changed root source envelope internal/host
changed root source envelope internal/host/units
changed root source envelope internal/instancectl
changed root source envelope internal/registry
changed root source envelope internal/release
stale root read internal/host/claimstop_run.go:RunInstanceControl
stale root read internal/instancectl/client.go:Client.do
stale root read internal/instancectl/gen.go:GenDropIn
stale root read internal/instancectl/server.go:WaitCgroupEmpty
stale root read internal/release/adopt.go:stageFrom
stale root read internal/release/adopt.go:type stageCheck
unclassified root read internal/host/claimstop_run.go:instanceRegistryHostRuntime registry:Runtime@322,whole:Route@270
unclassified root read internal/host/units/carlos-instance-rootfs-cmd@.service:external file:external@0
unclassified root read internal/host/units/carlos-instance-rootfs-gen@.service:external file:external@0
unclassified root read internal/host/units/carlos-instance-rootfs-probe.service:external file:external@0
unclassified root read internal/host/units/carlos-instance-rootfs-worker@.service:external file:external@0
unclassified root read internal/host/units/carlos-instance-rootfs@.service:external file:external@0
unclassified root read internal/instancectl/client.go:Client.doBody file:ReadAll@719
unclassified root read internal/instancectl/gen.go:Server.genUnitOfExeKey file:Lstat@197
unclassified root read internal/instancectl/gen.go:Server.writeDropInIfChanged file:ReadFile@95
unclassified root read internal/instancectl/gen.go:hostMountLines data-literal@123,data-literal@203,data-literal@288
unclassified root read internal/instancectl/rootfs.go:Server.ensureRootfsExe file:Lstat@673,file:ReadFile@1102
unclassified root read internal/instancectl/rootfs.go:resolveRootfsLaunch file:Command@1150,file:Command@898,file:Command@917
unclassified root read internal/instancectl/rootfs.go:writeFileExcl file:CreateTemp@230
unclassified root read internal/instancectl/rootfs_identity_linux.go:fileIdentityOf file:Fstat@87
unclassified root read internal/instancectl/server.go:waitCgroupFileEmpty file:ReadFile@115
changed legacy declaration internal/activator/backend_exec.go:ExecBackend.LegacyRunGone
changed legacy declaration internal/activator/backend_exec.go:ExecBackend.killFailedGen
changed legacy declaration internal/activator/backend_exec.go:ExecBackend.spawnUnitFrom
changed legacy declaration internal/activator/backend_exec.go:ExecBackend.startUnitAt
changed legacy declaration internal/activator/backend_exec.go:ExecBackend.waitRoleCgroupEmpty
changed legacy declaration internal/activator/unitproc.go:ExecBackend.newUnitProc
changed legacy declaration internal/activator/unitproc.go:ExecBackend.newUnitProcFor
changed legacy declaration internal/instancectl/client.go:Client.do
changed legacy declaration internal/instancectl/cmd.go:Server.handleCmdStatus
changed legacy declaration internal/instancectl/gen.go:GenDropIn
changed legacy declaration internal/instancectl/server.go:Server.RunProbe
changed legacy declaration internal/instancectl/server.go:WaitCgroupEmpty
changed legacy declaration internal/instancectl/server.go:parseRequest
changed legacy declaration internal/instancectl/server.go:type Server
changed legacy declaration internal/registry/registry.go:Registry.migrate
changed legacy declaration internal/registry/registry.go:scanRoute
changed reclassified declaration internal/activator/backend_exec.go:ExecBackend.startSidecarUnitFrom
changed reclassified declaration internal/registry/registry.go:Registry.Upsert
make census-card shows what changed; make census-propose NAME=<branch> CLAIMS=<file> writes the overlay and its approval card (AGENTS.md, The census)
FAIL
FAIL github.com/carlosframework/platform/internal/provision 27.696s
ok github.com/carlosframework/platform/internal/publicrelease 0.025s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 6.677s
ok github.com/carlosframework/platform/internal/relay 7.273s
ok github.com/carlosframework/platform/internal/release 28.147s
ok github.com/carlosframework/platform/internal/rollout 4.916s
ok github.com/carlosframework/platform/internal/rolloutwords 0.014s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootfsimage 0.353s
ok github.com/carlosframework/platform/internal/rootparticipation 0.068s
ok github.com/carlosframework/platform/internal/rootstate 0.033s
ok github.com/carlosframework/platform/internal/runs 0.662s
ok github.com/carlosframework/platform/internal/s3compat 0.010s
ok github.com/carlosframework/platform/internal/safefile 0.007s
ok github.com/carlosframework/platform/internal/schedule 3.831s
ok github.com/carlosframework/platform/internal/seeds 0.011s
ok github.com/carlosframework/platform/internal/sim/engines 4.675s
ok github.com/carlosframework/platform/internal/sim/hostlife 12.551s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.907s
ok github.com/carlosframework/platform/internal/status 0.094s
ok github.com/carlosframework/platform/internal/steering 0.010s
ok github.com/carlosframework/platform/internal/stopevidence 0.382s
ok github.com/carlosframework/platform/internal/svcrole 0.005s
ok github.com/carlosframework/platform/internal/telemetry 0.020s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.004s
ok github.com/carlosframework/platform/internal/updatewords 0.006s
ok github.com/carlosframework/platform/internal/usage 0.010s
ok github.com/carlosframework/platform/internal/version 0.005s
ok github.com/carlosframework/platform/internal/vet 2.065s
FAIL
=== internal_console@0 (78 s)
ok github.com/carlosframework/platform/internal/console 76.386s
=== internal_console@1 (50 s)
ok github.com/carlosframework/platform/internal/console 48.526s
=== internal_console@2 (46 s)
ok github.com/carlosframework/platform/internal/console 44.244s
=== internal_console@3 (47 s)
ok github.com/carlosframework/platform/internal/console 45.495s
=== internal_console@4 (49 s)
ok github.com/carlosframework/platform/internal/console 47.116s
=== internal_console@5 (47 s)
ok github.com/carlosframework/platform/internal/console 45.997s
=== internal_console@6 (51 s)
ok github.com/carlosframework/platform/internal/console 49.955s
=== internal_console@7 (55 s)
ok github.com/carlosframework/platform/internal/console 53.439s
=== internal_host@0 (36 s)
ok github.com/carlosframework/platform/internal/host 34.045s
=== internal_host@1 (35 s)
ok github.com/carlosframework/platform/internal/host 32.732s
=== internal_host@2 (67 s)
ok github.com/carlosframework/platform/internal/host 64.108s
=== internal_host@3 (39 s)
ok github.com/carlosframework/platform/internal/host 36.785s
=== root@0 (73 s)
ok github.com/carlosframework/platform 70.786s
=== root@1 (52 s)
ok github.com/carlosframework/platform 50.255s
=== root@2 (63 s)
ok github.com/carlosframework/platform 61.294s
=== root@3 (61 s)
ok github.com/carlosframework/platform 59.068s
=== root@4 (62 s)
ok github.com/carlosframework/platform 59.899s
Failure summary:
--- rest
--- FAIL: TestPlan9B8RecordWriteCensus (1.20s)
plan9_b8_census_test.go:372: record-write census rejected 2 site(s); update the explicit inventory only after reviewing the source:
--- FAIL: TestPlan14RootReadCensus (16.18s)
plan14_root_reads_test.go:600: root read census: 66 discrepancies
FAIL github.com/carlosframework/platform/internal/provision 27.696s
FAILED: rest
make: *** [Makefile:19: test] Error 1
amadan: step gate failed: exit status 2
amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go| index b830c5313..191f6c998 100644 |
| --- a/internal/activator/backend_exec.go |
| +++ b/internal/activator/backend_exec.go |
| @@ -2506,7 +2506,7 @@ func (b *ExecBackend) spawnUnitFrom(ctx context.Context, rt registry.Route, sock |
| if err == nil { |
| // The unit ran and exited before root observed it: the build |
| // crashed at its start (errStartedThenExited, ruling 206). |
| - err = exitedUnobserved{unit: unitName(instancectl.RoleRun, rt.Host, gen)} |
| + err = exitedUnobserved{unit: runUnitName(rt.Host, gen)} |
| } |
| if kerr := b.killFailedGen(ctx, rt, gen); kerr != nil { |
| // Nothing is released: the generation may still be running. |
| @@ -2526,14 +2526,17 @@ func (b *ExecBackend) spawnUnitFrom(ctx context.Context, rt registry.Route, sock |
| // The server read the invocation after the start and found none: |
| // the unit already exited (plan 2 final review). Nothing is |
| // running, so there is nothing to stop; the wake failed. |
| - return fail(markStartUncertain(fmt.Errorf("%s: %w", rt.Host, exitedUnobserved{unit: unitName(instancectl.RoleRun, rt.Host, gen)}))) |
| + return fail(markStartUncertain(fmt.Errorf("%s: %w", rt.Host, exitedUnobserved{unit: runUnitName(rt.Host, gen)}))) |
| } |
| started := tier |
| if started == (cgroup.Limit{}) { |
| started = b.tier(rt) |
| } |
| - log.Printf("activator: %s: started %s memory.max=%d (%s)", rt.Host, unitName(instancectl.RoleRun, rt.Host, gen), started.MemoryMax, steps.String()) |
| - up := b.newUnitProcFor(rt, instancectl.RoleRun, gen, p, inv, ctl, logs) |
| + log.Printf("activator: %s: started %s memory.max=%d (%s)", rt.Host, runUnitName(rt.Host, gen), started.MemoryMax, steps.String()) |
| + up, err := b.newUnitProcFor(rt, instancectl.RoleRun, gen, p, inv, ctl, logs) |
| + if err != nil { |
| + return fail(err) |
| + } |
| // The file environment this generation launched with, kept on its |
| // handle: a run against it composes its environment from this, never |
| // from the files config sync has delivered since (merge ruling M3). |
| @@ -2600,7 +2603,7 @@ func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock s |
| } |
| // The spawn's own split, for the journal: the unit's launch (instance |
| // control's half included) and then the app's boot to its socket. |
| - log.Printf("activator: %s: %s launched in %s, healthy %s later", rt.Host, unitName(instancectl.RoleRun, rt.Host, n), |
| + log.Printf("activator: %s: %s launched in %s, healthy %s later", rt.Host, runUnitName(rt.Host, n), |
| launched.Sub(began).Round(time.Millisecond), time.Since(launched).Round(time.Millisecond)) |
| b.normalizeSocketMode(rt, sock, up) |
| return up, spawnVer, nil |
| @@ -2621,7 +2624,7 @@ func (b *ExecBackend) LegacyRunGone(ctx context.Context, rt registry.Route) erro |
| if b.Control == nil { |
| return fmt.Errorf("%s: instance control channel not configured", rt.Host) |
| } |
| - unit := unitName(instancectl.RoleRun, rt.Host, 0) |
| + unit := runUnitName(rt.Host, 0) |
| if err := b.Control.Stop(ctx, rt.Host, instancectl.RoleRun); err != nil { |
| return fmt.Errorf("stop %s: %w", unit, err) |
| } |
| @@ -2678,7 +2681,7 @@ func (b *ExecBackend) killFailedGen(ctx context.Context, rt registry.Route, gen |
| kctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), instancectl.DeadlineControl) |
| defer cancel() |
| if err := gc.StopGen(kctx, rt.Host, gen, true); err != nil { |
| - log.Printf("activator: %s: kill %s after a failed start: %v", rt.Host, unitName(instancectl.RoleRun, rt.Host, gen), err) |
| + log.Printf("activator: %s: kill %s after a failed start: %v", rt.Host, runUnitName(rt.Host, gen), err) |
| return err |
| } |
| return nil |
| @@ -3235,7 +3238,7 @@ func (b *ExecBackend) startSidecarUnitFrom(ctx context.Context, rt registry.Rout |
| logs.Close() |
| return nil, fmt.Errorf("%s: %s exited before it was observed", rt.Host, instancectl.UnitName(instancectl.RoleSidecar, rt.Host)) |
| } |
| - return b.newUnitProc(rt, instancectl.RoleSidecar, p, inv, nil, logs), nil |
| + return b.newUnitProc(rt, instancectl.RoleSidecar, p, inv, nil, logs) |
| } |
| // socketClient returns an HTTP client that dials an instance's unix socket |
diff --git a/internal/activator/rootfs_worker_unit_test.go b/internal/activator/rootfs_worker_unit_test.go| index 42242a944..b5967cc04 100644 |
| --- a/internal/activator/rootfs_worker_unit_test.go |
| +++ b/internal/activator/rootfs_worker_unit_test.go |
| @@ -21,3 +21,34 @@ func TestWaitRoleCgroupEmptyRefusesAWorker(t *testing.T) { |
| t.Fatal("a worker's wait was not refused") |
| } |
| } |
| + |
| +// unitName answers an error, never "", for a role with no binary unit, and |
| +// newUnitProcFor refuses before it watches anything. |
| +func TestUnitNameRefusesAWorker(t *testing.T) { |
| + t.Parallel() |
| + if u, err := unitName(instancectl.RoleWorker, "shop.example.com", 0); err == nil || u != "" { |
| + t.Fatalf("unitName(worker) = %q, %v; want an error", u, err) |
| + } |
| + if u, err := unitName(instancectl.RoleRun, "shop.example.com", 0); err != nil || u != instancectl.UnitName(instancectl.RoleRun, "shop.example.com") { |
| + t.Fatalf("unitName(run) = %q, %v", u, err) |
| + } |
| + if u := runUnitName("shop.example.com", 7); u != instancectl.GenUnitName("shop.example.com", 7) { |
| + t.Fatalf("runUnitName(7) = %q", u) |
| + } |
| + b := &ExecBackend{} |
| + if up, err := b.newUnitProcFor(registry.Route{Host: "shop.example.com"}, instancectl.RoleWorker, 0, instancePaths{}, "inv", nil, nil); err == nil || up != nil { |
| + t.Fatalf("newUnitProcFor(worker) = %v, %v; want a refusal", up, err) |
| + } |
| +} |
| + |
| +// mustUnitProc wraps newUnitProcFor for a test whose role always has a |
| +// unit: mustUnitProc(t)(b.newUnitProcFor(...)). |
| +func mustUnitProc(t *testing.T) func(*unitProc, error) *unitProc { |
| + return func(up *unitProc, err error) *unitProc { |
| + t.Helper() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return up |
| + } |
| +} |
diff --git a/internal/activator/unitproc.go b/internal/activator/unitproc.go| index b69a2eab4..d5304b1a8 100644 |
| --- a/internal/activator/unitproc.go |
| +++ b/internal/activator/unitproc.go |
| @@ -95,24 +95,41 @@ type launchKeyLayerProc interface { |
| // LaunchKeyLayer is the layer LaunchEnv's CARLOS_INSTANCE_KEY came from. |
| func (p *unitProc) LaunchKeyLayer() instancekey.Layer { return p.launchKeyLayer } |
| -func (b *ExecBackend) newUnitProc(rt registry.Route, role instancectl.Role, p instancePaths, inv string, ctl *edge.ControlListener, logs *logListener) *unitProc { |
| +func (b *ExecBackend) newUnitProc(rt registry.Route, role instancectl.Role, p instancePaths, inv string, ctl *edge.ControlListener, logs *logListener) (*unitProc, error) { |
| return b.newUnitProcFor(rt, role, 0, p, inv, ctl, logs) |
| } |
| -func (b *ExecBackend) newUnitProcFor(rt registry.Route, role instancectl.Role, gen uint64, p instancePaths, inv string, ctl *edge.ControlListener, logs *logListener) *unitProc { |
| - up := &unitProc{b: b, rt: rt, role: role, gen: gen, unit: unitName(role, rt.Host, gen), slice: instancectl.SliceName(rt.Host), |
| +// newUnitProcFor starts observing the unit a role of host runs as. A role |
| +// with no binary unit (a rootfs worker, rootfs apps plan 2) is refused |
| +// before anything watches "". |
| +func (b *ExecBackend) newUnitProcFor(rt registry.Route, role instancectl.Role, gen uint64, p instancePaths, inv string, ctl *edge.ControlListener, logs *logListener) (*unitProc, error) { |
| + unit, err := unitName(role, rt.Host, gen) |
| + if err != nil { |
| + return nil, err |
| + } |
| + up := &unitProc{b: b, rt: rt, role: role, gen: gen, unit: unit, slice: instancectl.SliceName(rt.Host), |
| invocation: inv, runDir: p.runDir, dataDir: p.dataDir, ctl: ctl, logs: logs, done: make(chan struct{})} |
| go b.observe(up) |
| - return up |
| + return up, nil |
| } |
| // unitName is the unit a role of host runs as: the original template's, or |
| -// generation gen's (zero-downtime spec §7.6). |
| -func unitName(role instancectl.Role, host string, gen uint64) string { |
| - if gen == 0 { |
| - return instancectl.UnitName(role, host) |
| +// generation gen's (zero-downtime spec §7.6). A role with no binary unit |
| +// (RoleWorker) is an error, never "". |
| +func unitName(role instancectl.Role, host string, gen uint64) (string, error) { |
| + if gen != 0 { |
| + return instancectl.GenUnitName(host, gen), nil |
| } |
| - return instancectl.GenUnitName(host, gen) |
| + if unit := instancectl.UnitName(role, host); unit != "" { |
| + return unit, nil |
| + } |
| + return "", fmt.Errorf("%s: role %s has no binary unit", host, role) |
| +} |
| + |
| +// runUnitName is unitName for the run role, which always has a unit. |
| +func runUnitName(host string, gen uint64) string { |
| + unit, _ := unitName(instancectl.RoleRun, host, gen) |
| + return unit |
| } |
| // stopRole stops one role's unit: the original template's through Control, |
diff --git a/internal/activator/unitproc_gen_test.go b/internal/activator/unitproc_gen_test.go| index 1d8394c42..a09d8d6aa 100644 |
| --- a/internal/activator/unitproc_gen_test.go |
| +++ b/internal/activator/unitproc_gen_test.go |
| @@ -77,7 +77,7 @@ func TestGenerationUnitProcStopsAndWatchesItsOwnUnit(t *testing.T) { |
| } |
| unit.set("active", "inv7", true, true) |
| ctl.stopHook = func(string, instancectl.Role) { unit.set("inactive", "inv7", false, false) } |
| - up := b.newUnitProcFor(rt, instancectl.RoleRun, 7, b.pathsFor(rt), "inv7", nil, nil) |
| + up := mustUnitProc(t)(b.newUnitProcFor(rt, instancectl.RoleRun, 7, b.pathsFor(rt), "inv7", nil, nil)) |
| ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) |
| defer cancel() |
| if err := up.Stop(ctx); err != nil { |
| @@ -127,7 +127,7 @@ func TestGenerationUnitProcConsumesOnlyItsOwnRecordAndLaunchFile(t *testing.T) { |
| } |
| } |
| unit.set("active", "inv7", true, true) |
| - up := b.newUnitProcFor(rt, instancectl.RoleRun, 7, p, "inv7", nil, nil) |
| + up := mustUnitProc(t)(b.newUnitProcFor(rt, instancectl.RoleRun, 7, p, "inv7", nil, nil)) |
| unit.set("inactive", "inv7", false, false) |
| waitFor(t, "the generation to exit", up.Exited) |
| for path, want := range map[string]bool{ |
| @@ -300,7 +300,7 @@ func TestGenerationNaturalExitFinalisesItsFiles(t *testing.T) { |
| t.Parallel() |
| ctl, unit, b, rt, launch := genProcFixture(t) |
| unit.set("active", "inv7", true, true) |
| - up := b.newUnitProcFor(rt, instancectl.RoleRun, 7, b.pathsFor(rt), "inv7", nil, nil) |
| + up := mustUnitProc(t)(b.newUnitProcFor(rt, instancectl.RoleRun, 7, b.pathsFor(rt), "inv7", nil, nil)) |
| unit.set("inactive", "inv7", false, false) |
| waitFor(t, "the generation to exit", up.Exited) |
| if want := "stopgen h.example.com 7 kill=false"; ctl.joined() != want { |
| @@ -325,7 +325,7 @@ func TestGenerationForeignInvocationExitFinalisesItsFiles(t *testing.T) { |
| ctl, unit, b, rt, launch := genProcFixture(t) |
| ctl.stopHook = func(string, instancectl.Role) { unit.set("inactive", "", false, false) } |
| unit.set("active", "inv7", true, true) |
| - up := b.newUnitProcFor(rt, instancectl.RoleRun, 7, b.pathsFor(rt), "inv7", nil, nil) |
| + up := mustUnitProc(t)(b.newUnitProcFor(rt, instancectl.RoleRun, 7, b.pathsFor(rt), "inv7", nil, nil)) |
| unit.set("active", "someone-else", true, true) |
| waitFor(t, "the generation to exit", up.Exited) |
| if want := "stopgen h.example.com 7 kill=false"; ctl.joined() != want { |
diff --git a/internal/instancectl/client.go b/internal/instancectl/client.go| index a3d653c1b..fc4a07054 100644 |
| --- a/internal/instancectl/client.go |
| +++ b/internal/instancectl/client.go |
| @@ -425,7 +425,9 @@ func (c *Client) startRootfsQuery(ctx context.Context, q url.Values, memoryMax i |
| switch { |
| case status == http.StatusForbidden: |
| return "", fmt.Errorf("%w: %v", ErrStagedImage, err) |
| - case status == statusGenUnproven && q.Has("gen"): |
| + case status == statusGenUnproven && (q.Has("gen") || q.Has("worker")): |
| + // A generation or worker whose refused start could not be proven |
| + // stopped: its files are kept and the unit may live. |
| return "", fmt.Errorf("%w: %w", ErrGenUnproven, err) |
| } |
| return "", err |
diff --git a/internal/instancectl/gen.go b/internal/instancectl/gen.go| index f2ea69c56..9759fe07f 100644 |
| --- a/internal/instancectl/gen.go |
| +++ b/internal/instancectl/gen.go |
| @@ -378,13 +378,15 @@ func (s *Server) markGenStarting(unit string, on bool) { |
| s.genMu.Lock() |
| defer s.genMu.Unlock() |
| if !on { |
| - delete(s.genStarting, unit) |
| + if s.genStarting[unit]--; s.genStarting[unit] <= 0 { |
| + delete(s.genStarting, unit) |
| + } |
| return |
| } |
| if s.genStarting == nil { |
| - s.genStarting = map[string]bool{} |
| + s.genStarting = map[string]int{} |
| } |
| - s.genStarting[unit] = true |
| + s.genStarting[unit]++ |
| } |
| // sweepGenLeftovers is the sweep's recovery for generation units that are |
| @@ -514,7 +516,7 @@ func (s *Server) sweepGenLeftover(ctx context.Context, unit string) { |
| // cannot write fresh files in between. |
| s.genMu.Lock() |
| defer s.genMu.Unlock() |
| - if s.genStarting[unit] { |
| + if s.genStarting[unit] > 0 { |
| return |
| } |
| if state := s.showValue(ctx, "ActiveState", unit); state != "inactive" && state != "failed" { |
diff --git a/internal/instancectl/rootfs.go b/internal/instancectl/rootfs.go| index 8c0e80387..7215a7689 100644 |
| --- a/internal/instancectl/rootfs.go |
| +++ b/internal/instancectl/rootfs.go |
| @@ -351,12 +351,13 @@ func (s *Server) ensureRootfsExe(key string, role Role, imagePath string, launch |
| // writeFileExcl writes data to a fresh temp file beside path (O_EXCL, so a |
| // link planted there is never followed) and renames it into place. |
| func writeFileExcl(path string, data []byte, mode os.FileMode) error { |
| - tmp := path + ".tmp" |
| - _ = os.Remove(tmp) |
| - f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_EXCL, mode) |
| + // A unique temp name (O_EXCL under the hood), so two starts of one unit |
| + // overlapping on the same layout never write through each other's file. |
| + f, err := os.CreateTemp(filepath.Dir(path), "."+filepath.Base(path)+".*.tmp") |
| if err != nil { |
| return err |
| } |
| + tmp := f.Name() |
| if _, err := f.Write(data); err != nil { |
| f.Close() |
| os.Remove(tmp) |
| @@ -584,9 +585,11 @@ func (s *Server) rootfsPreflight(w http.ResponseWriter, host string, role Role, |
| return RootfsLaunch{}, true |
| } |
| if rep, ok := s.ReadyRootfs(); !ok { |
| + // The first check that is not ok: a failing one, or the unverified |
| + // line a service with no probe image reports. |
| check := "not yet run" |
| for _, l := range rep.Lines { |
| - if l.Verdict == "fail" { |
| + if l.Verdict != "ok" { |
| check = l.Check |
| break |
| } |
| @@ -748,7 +751,7 @@ func (s *Server) startRootfs(w http.ResponseWriter, r *http.Request, req request |
| http.Error(w, msg, http.StatusBadGateway) |
| return |
| } |
| - inv, ok := s.rootfsPlaced(ctx, w, req.host, unit, launch.JIT, remove) |
| + inv, ok := s.rootfsPlaced(ctx, w, req.host, unit, launch.JIT, perStart, remove) |
| if !ok { |
| return |
| } |
| @@ -801,13 +804,25 @@ func rootfsMountFailure(out string) (string, bool) { |
| // the host slice (a unit that has already exited has none, so its |
| // configured Slice stands in); MemoryDenyWriteExecute must be no when the |
| // release declares jit and yes otherwise. On a mismatch the unit is |
| -// stopped and waited on, its jit drop-in removed, a reload marked owed |
| -// (the loaded configuration was stale even when no file was there), its |
| -// files taken, and the request answered 502 naming the property. ok is |
| -// false when it answered. |
| -func (s *Server) rootfsPlaced(ctx context.Context, w http.ResponseWriter, host, unit string, jit bool, remove func()) (inv string, ok bool) { |
| +// stopped and proven gone (endMisplaced), a reload marked owed (the loaded |
| +// configuration was stale even when no file was there), its files taken, |
| +// and the request answered 502 naming the property. A stop that cannot be |
| +// proven keeps the files and says so: a generation or worker (perStart) |
| +// answers statusGenUnproven, as abandonGen does, so the client knows the |
| +// unit may live. ok is false when it answered. |
| +func (s *Server) rootfsPlaced(ctx context.Context, w http.ResponseWriter, host, unit string, jit, perStart bool, remove func()) (inv string, ok bool) { |
| + var m map[string]string |
| end := func(msg string) (string, bool) { |
| - s.endMisplaced(ctx, host, unit, remove) |
| + if err := s.endMisplaced(ctx, unit, m["ControlGroup"]); err != nil { |
| + s.logf("instance-control: %s: %s; termination unproven, files kept: %v", host, msg, err) |
| + status := http.StatusBadGateway |
| + if perStart { |
| + status = statusGenUnproven |
| + } |
| + http.Error(w, msg+"; termination unproven: "+err.Error(), status) |
| + return "", false |
| + } |
| + remove() |
| s.logf("instance-control: %s: %s", host, msg) |
| http.Error(w, msg, http.StatusBadGateway) |
| return "", false |
| @@ -816,7 +831,7 @@ func (s *Server) rootfsPlaced(ctx context.Context, w http.ResponseWriter, host, |
| if err != nil { |
| return end("show: " + err.Error()) |
| } |
| - m := parseShow(out) |
| + m = parseShow(out) |
| slice := SliceName(host) |
| own := "/" + parentSlices + "/" + slice + "/" + unit |
| if cg := m["ControlGroup"]; cg != "" && cg != own || cg == "" && m["Slice"] != slice { |
| @@ -833,20 +848,31 @@ func (s *Server) rootfsPlaced(ctx context.Context, w http.ResponseWriter, host, |
| } |
| // endMisplaced stops a started unit the post-start check refused, on a |
| -// context the client's hang-up cannot cancel, waits for its cgroup, drops |
| -// its jit drop-in, marks the reload owed and takes its files. |
| -func (s *Server) endMisplaced(ctx context.Context, host, unit string, remove func()) { |
| - dctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), DeadlineControl+s.waitBudget()) |
| +// context the client's hang-up cannot cancel, and proves it gone: the stop |
| +// succeeds, systemd reports the unit inactive or failed, and the control |
| +// group the show reported (not the one under the host slice, which a |
| +// misplaced unit never had) is absent or empty. nil is the proof. The |
| +// reload is marked owed either way; the jit drop-in goes only with the |
| +// proof, so a unit that may still run keeps the file that describes it. |
| +func (s *Server) endMisplaced(ctx context.Context, unit, cg string) error { |
| + s.markJITReloadOwed() |
| + dctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), DeadlineControl+2*s.waitBudget()) |
| defer cancel() |
| - s.stopUnit(dctx, unit) |
| + if out, err := s.Systemctl(dctx, "stop", unit); err != nil { |
| + return errors.New("stop: " + strings.TrimSpace(out+" "+err.Error())) |
| + } |
| wctx, waitCancel := context.WithTimeout(dctx, s.waitBudget()) |
| defer waitCancel() |
| - if err := WaitCgroupEmpty(wctx, s.CgroupRoot, SliceName(host), unit, 50*time.Millisecond); err != nil { |
| - s.logf("instance-control: %s: %s: %v", host, unit, err) |
| + if err := s.waitInactiveOrFailed(wctx, unit, 50*time.Millisecond); err != nil { |
| + return err |
| + } |
| + if cg != "" && path.IsAbs(cg) && path.Clean(cg) == cg { |
| + if err := waitCgroupFileEmpty(wctx, filepath.Join(s.CgroupRoot, cg, "cgroup.events"), unit, 50*time.Millisecond); err != nil { |
| + return err |
| + } |
| } |
| s.removeJITDropIn(unit) |
| - s.markJITReloadOwed() |
| - remove() |
| + return nil |
| } |
| // startRootfsCmd is handleCmd's rootfs branch (run spec §5.4 for a rootfs |
| @@ -925,7 +951,7 @@ func (s *Server) startRootfsCmd(w http.ResponseWriter, r *http.Request, req requ |
| deadline := time.Now().Add(s.cmdStartWait()) |
| for { |
| if inv := s.showValue(ctx, "InvocationID", unit); inv != "" && inv != before { |
| - if _, ok := s.rootfsPlaced(ctx, w, req.host, unit, launch.JIT, remove); !ok { |
| + if _, ok := s.rootfsPlaced(ctx, w, req.host, unit, launch.JIT, false, remove); !ok { |
| return |
| } |
| s.logf("instance-control: %s: started %s (%s) from %s memory.max=%d", req.host, unit, launch.Process, launch.Version, req.tier.MemoryMax) |
diff --git a/internal/instancectl/rootfs_test.go b/internal/instancectl/rootfs_test.go| index 05cd8d3cc..9085d184b 100644 |
| --- a/internal/instancectl/rootfs_test.go |
| +++ b/internal/instancectl/rootfs_test.go |
| @@ -1749,3 +1749,210 @@ func TestRootfsDigestOfAnotherFileAtTheSamePathIsRehashed(t *testing.T) { |
| t.Fatalf("an image the manifest does not name was vouched for: %v", err) |
| } |
| } |
| + |
| +// Fix round 1, finding 1: the C5/C10 refusal takes a unit's files only once |
| +// the unit is proven gone: the stop succeeded, systemd reports it inactive, |
| +// and the control group the show reported (a misplaced unit's own, never |
| +// one under the host slice) is absent or empty. A stop that fails, or a |
| +// cgroup that stays populated, keeps the files and says so: a generation |
| +// or worker answers statusGenUnproven, which the client reads as |
| +// ErrGenUnproven; a run unit a 502 naming the unproven termination. |
| +func TestRootfsMisplacedUnitIsRemovedOnlyWhenProvenGone(t *testing.T) { |
| + misplaced := func(f *rootfsBox, unit string) string { |
| + return "InvocationID=" + strings.Repeat("ab", 16) + "\nSlice=system.slice\nControlGroup=/system.slice/" + unit + "\nMemoryDenyWriteExecute=no\n" |
| + } |
| + // Run unit, jit release (so a jit drop-in exists), stop fails: 502 naming |
| + // the unproven termination, the layout and the jit drop-in kept, a |
| + // reload owed all the same. |
| + f := rootfsFixture(t) |
| + f.restageManifest(t, func(m *release.Manifest) { m.JIT = true }) |
| + unit := RootfsUnitName(RoleRun, f.host) |
| + f.ctl.set(rootfsPostStartShow+" "+unit, misplaced(f, unit)) |
| + f.ctl.fail["stop "+unit] = errors.New("exit status 1") |
| + // systemd reads the unit inactive all the same (the stop job failed for |
| + // another reason): the failed stop alone must keep the files. |
| + f.ctl.set("show -p ActiveState --value "+unit, "inactive\n") |
| + status, body := f.post(t, "/start", f.startQuery(RoleRun, release.ProcessWeb, 0)) |
| + if status != 502 || !strings.Contains(body, "termination unproven") || !strings.Contains(body, "slice") { |
| + t.Fatalf("stop failed: %d %q", status, body) |
| + } |
| + for _, p := range []string{filepath.Join(f.srv.DropInDir, unit+".d", JITDropInName), filepath.Join(RootfsExeDir(f.srv.ExeRoot, provision.InstanceID(f.host), RoleRun), RootfsExeLaunch)} { |
| + if _, err := os.Stat(p); err != nil { |
| + t.Errorf("%s went although the stop was not proven: %v", p, err) |
| + } |
| + } |
| + if !f.srv.jitReloadOwed() { |
| + t.Error("no reload owed after a stale MemoryDenyWriteExecute") |
| + } |
| + // Run unit, stop succeeds but the reported control group stays populated: |
| + // unproven too. |
| + f = rootfsFixture(t) |
| + f.restageManifest(t, func(m *release.Manifest) { m.JIT = true }) |
| + f.srv.cgroupWaitBudget = 100 * time.Millisecond |
| + f.ctl.set(rootfsPostStartShow+" "+unit, misplaced(f, unit)) |
| + ev := filepath.Join(f.srv.CgroupRoot, "system.slice", unit, "cgroup.events") |
| + if err := os.MkdirAll(filepath.Dir(ev), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := os.WriteFile(ev, []byte("populated 1\n"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if status, body := f.post(t, "/start", f.startQuery(RoleRun, release.ProcessWeb, 0)); status != 502 || !strings.Contains(body, "termination unproven") { |
| + t.Fatalf("populated cgroup: %d %q", status, body) |
| + } |
| + f.ctl.mustHaveCall(t, "stop "+unit) |
| + if _, err := os.Stat(filepath.Join(f.srv.DropInDir, unit+".d", JITDropInName)); err != nil { |
| + t.Errorf("the jit drop-in went although processes remain: %v", err) |
| + } |
| + // The same, once the cgroup empties: proven, files taken, a plain 502. |
| + if err := os.WriteFile(ev, []byte("populated 0\n"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if status, body := f.post(t, "/start", f.startQuery(RoleRun, release.ProcessWeb, 0)); status != 502 || strings.Contains(body, "unproven") { |
| + t.Fatalf("empty cgroup: %d %q", status, body) |
| + } |
| + if _, err := os.Stat(filepath.Join(f.srv.DropInDir, unit+".d", JITDropInName)); err == nil { |
| + t.Error("the jit drop-in survived a proven stop") |
| + } |
| + // Generation, stop fails: statusGenUnproven, drop-in and exe directory |
| + // kept; through the client it is ErrGenUnproven. A worker the same. |
| + for _, c := range []struct { |
| + name string |
| + unit string |
| + dir string |
| + call func(f *rootfsBox) error |
| + }{ |
| + {"gen", RootfsGenUnitName(f.host, 2), GenKey(f.host, 2), func(f *rootfsBox) error { |
| + _, err := f.client(t).StartRootfs(context.Background(), f.host, 2, cgroup.TierChildren.MemoryMax, f.img(release.ProcessWeb)) |
| + return err |
| + }}, |
| + {"worker", WorkerUnitName(f.host, "jobs"), WorkerKey(f.host, "jobs"), func(f *rootfsBox) error { |
| + _, err := f.client(t).StartRootfsWorker(context.Background(), f.host, "jobs", cgroup.TierChildren.MemoryMax, f.img("jobs")) |
| + return err |
| + }}, |
| + } { |
| + f := rootfsFixture(t) |
| + f.ctl.set(rootfsPostStartShow+" "+c.unit, misplaced(f, c.unit)) |
| + f.ctl.fail["stop "+c.unit] = errors.New("exit status 1") |
| + f.ctl.set("show -p ActiveState --value "+c.unit, "inactive\n") |
| + err := c.call(f) |
| + if !errors.Is(err, ErrGenUnproven) || !strings.Contains(err.Error(), "termination unproven") { |
| + t.Fatalf("%s: %v, want ErrGenUnproven", c.name, err) |
| + } |
| + if _, err := os.Stat(filepath.Join(f.srv.DropInDir, c.unit+".d", DropInName)); err != nil { |
| + t.Errorf("%s: drop-in went although the stop failed: %v", c.name, err) |
| + } |
| + if _, err := os.Stat(filepath.Join(f.srv.ExeRoot, c.dir)); err != nil { |
| + t.Errorf("%s: exe directory went although the stop failed: %v", c.name, err) |
| + } |
| + // With the stop working, the files go and the answer is a 502 the |
| + // client does not read as unproven. |
| + delete(f.ctl.fail, "stop "+c.unit) |
| + if err := c.call(f); err == nil || errors.Is(err, ErrGenUnproven) { |
| + t.Fatalf("%s with a working stop: %v, want a plain refusal", c.name, err) |
| + } |
| + if _, err := os.Stat(filepath.Join(f.srv.ExeRoot, c.dir)); err == nil { |
| + t.Errorf("%s: exe directory survived a proven stop", c.name) |
| + } |
| + } |
| +} |
| + |
| +// Fix round 1, finding 2: the 503 names the first check that is not ok, |
| +// so a service with no probe image (the production stub, "unverified") |
| +// says so rather than "not yet run". |
| +func TestRootfsPreflightNamesTheFirstFailingCheck(t *testing.T) { |
| + f := rootfsFixture(t) |
| + f.srv.setRootfsReport(ProbeReport{Ready: false, Lines: []VerdictLine{{Check: "root-image", Verdict: "ok"}, {Check: "rootfs-probe", Verdict: "unverified", Detail: "no probe image on this service"}}}) |
| + if status, body := f.post(t, "/start", f.startQuery(RoleRun, release.ProcessWeb, 0)); status != 503 || !strings.Contains(body, "(rootfs-probe)") { |
| + t.Fatalf("unverified line: %d %q", status, body) |
| + } |
| + // The production stub itself. |
| + f.srv.ProbeImage = nil |
| + f.srv.runRootfsProbe(context.Background()) |
| + if status, body := f.post(t, "/start", f.startQuery(RoleRun, release.ProcessWeb, 0)); status != 503 || !strings.Contains(body, "(rootfs-probe)") { |
| + t.Fatalf("stub report: %d %q", status, body) |
| + } |
| + // A fresh service that has not probed at all. |
| + f.srv.mu.Lock() |
| + f.srv.rootfsProbed = false |
| + f.srv.mu.Unlock() |
| + if status, body := f.post(t, "/start", f.startQuery(RoleRun, release.ProcessWeb, 0)); status != 503 || !strings.Contains(body, "(rootfs-probe)") { |
| + t.Fatalf("not probed: %d %q", status, body) |
| + } |
| +} |
| + |
| +// Fix round 1, finding 4: a worker's drop-in is written only when its bytes |
| +// change. A second identical start leaves the file's mtime alone (systemd |
| +// would otherwise report NeedDaemonReload for the loaded unit); a file |
| +// whose bytes differ is rewritten. |
| +func TestRootfsWorkerDropInIsWrittenOnlyWhenItsBytesChange(t *testing.T) { |
| + f := rootfsFixture(t) |
| + unit := WorkerUnitName(f.host, "jobs") |
| + q := f.startQuery(RoleWorker, "jobs", 0) |
| + q.Set("worker", "jobs") |
| + if status, body := f.post(t, "/start", q); status != 200 { |
| + t.Fatalf("first start: %d %s", status, body) |
| + } |
| + path := filepath.Join(f.srv.DropInDir, unit+".d", DropInName) |
| + past := time.Now().Add(-time.Hour) |
| + if err := os.Chtimes(path, past, past); err != nil { |
| + t.Fatal(err) |
| + } |
| + // The unit is stopped by its stop verb between starts in production; here |
| + // the stop would remove the drop-in, so the second start is made with |
| + // the unit still loaded, which is exactly when the mtime matters. |
| + if status, body := f.post(t, "/start", q); status != 200 { |
| + t.Fatalf("second start: %d %s", status, body) |
| + } |
| + if fi, err := os.Stat(path); err != nil || !fi.ModTime().Equal(past) { |
| + t.Errorf("an identical drop-in was rewritten: mtime %v, %v", fi.ModTime(), err) |
| + } |
| + // Bytes that differ (a drop-in from an older build) are replaced. |
| + if err := os.WriteFile(path, []byte(WorkerDropIn(f.host, "jobs", cgroup.TierChildren)+"MemoryMax=1\n"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if status, body := f.post(t, "/start", q); status != 200 { |
| + t.Fatalf("third start: %d %s", status, body) |
| + } |
| + if got, _ := os.ReadFile(path); string(got) != WorkerDropIn(f.host, "jobs", cgroup.TierChildren) { |
| + t.Errorf("a stale drop-in was not rewritten:\n%s", got) |
| + } |
| +} |
| + |
| +// Fix round 1, finding 5: two overlapping starts of the same worker (its |
| +// name is reused) each hold the unit as starting; the first to finish must |
| +// not let the sweep take the files the other still needs. The sweep runs |
| +// while start A is inside its systemctl start, after start B has run whole |
| +// and systemd (the model) is told the unit is inactive. |
| +func TestOverlappingWorkerStartsKeepTheSecondStartsFiles(t *testing.T) { |
| + f := rootfsFixture(t) |
| + unit := WorkerUnitName(f.host, "jobs") |
| + q := f.startQuery(RoleWorker, "jobs", 0) |
| + q.Set("worker", "jobs") |
| + var swept sync.WaitGroup |
| + swept.Add(1) |
| + f.ctl.mu.Lock() |
| + f.ctl.before = map[string]func(){"start": func() { |
| + defer swept.Done() |
| + if status, body := f.post(t, "/start", q); status != 200 { |
| + t.Errorf("start B inside start A: %d %s", status, body) |
| + } |
| + f.ctl.set("show -p ActiveState --value "+unit, "inactive\n") |
| + f.srv.sweepGenLeftovers(context.Background(), nil) |
| + }} |
| + f.ctl.mu.Unlock() |
| + if status, body := f.post(t, "/start", q); status != 200 { |
| + t.Fatalf("start A: %d %s", status, body) |
| + } |
| + swept.Wait() |
| + for _, p := range []string{filepath.Join(f.srv.DropInDir, unit+".d", DropInName), filepath.Join(RootfsExeDir(f.srv.ExeRoot, WorkerKey(f.host, "jobs"), RoleWorker), RootfsExeLaunch)} { |
| + if _, err := os.Stat(p); err != nil { |
| + t.Errorf("%s was swept while a start of the unit was in flight: %v", p, err) |
| + } |
| + } |
| + // With both starts done, the same sweep takes the inactive unit's files. |
| + f.srv.sweepGenLeftovers(context.Background(), nil) |
| + if _, err := os.Stat(filepath.Join(f.srv.ExeRoot, WorkerKey(f.host, "jobs"))); err == nil { |
| + t.Error("the sweep left an inactive worker's files once no start was in flight") |
| + } |
| +} |
diff --git a/internal/instancectl/server.go b/internal/instancectl/server.go| index f46f0f580..e10c9209e 100644 |
| --- a/internal/instancectl/server.go |
| +++ b/internal/instancectl/server.go |
| @@ -137,8 +137,11 @@ type Server struct { |
| // genMu guards genStarting, the generation units a start is preparing |
| // now. The sweep never removes the files of one of them. |
| - genMu sync.Mutex |
| - genStarting map[string]bool |
| + genMu sync.Mutex |
| + // genStarting counts the starts of each unit in flight: worker names |
| + // are reused, so two overlapping starts of one unit must both finish |
| + // before the sweep may judge its files. |
| + genStarting map[string]int |
| // Legacy run start attempts per host (legacystart.go, #292): the |
| // starts in flight, the stops in progress, and each host's barrier, |
| @@ -1070,7 +1073,12 @@ func (s *Server) waitBudget() time.Duration { |
| // absent or reports "populated 0" in cgroup.events, polling every `every`, |
| // bounded by ctx and the caller's budget (design §6.5, codex R2-5). |
| func WaitCgroupEmpty(ctx context.Context, root, slice, unit string, every time.Duration) error { |
| - events := filepath.Join(root, parentSlices, slice, unit, "cgroup.events") |
| + return waitCgroupFileEmpty(ctx, filepath.Join(root, parentSlices, slice, unit, "cgroup.events"), unit, every) |
| +} |
| + |
| +// waitCgroupFileEmpty is WaitCgroupEmpty on one cgroup.events file, for a |
| +// cgroup that is not under the host slice (a misplaced unit's). |
| +func waitCgroupFileEmpty(ctx context.Context, events, unit string, every time.Duration) error { |
| for { |
| b, err := os.ReadFile(events) |
| switch { |