amadan
replica freshness: the alert rule, the replicator's level-at, the status shape (#284)
Summary
A replica alert rule (fires when the worst replica is more than 15 minutes overdue, clears below 10, names the database or host). lease.Replicator.LevelAt exposes litestream's last-successful-sync time, and activator.Service.ReplicaLevels reports it per awake host, plus sleeping hosts whose last stop is remembered as clean. The box status snapshot gains a private host-keyed route_replicas map.
Agent notes
Nothing feeds the rule or the map yet; the agent-side watcher and the box-wide checker come next. Design is amadan discussion #290.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6 files changed,
+218
−4
internal/activator/replicalevel.go+48 −0internal/activator/replicalevel_test.go+55 −0internal/lease/lease.go+8 −0internal/metrics/alert.go+35 −2internal/metrics/alert_test.go+38 −0internal/status/snapshot.go+34 −2
CI failed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.011s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 337.106s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.190s
ok github.com/carlosframework/platform/cmd/moveprobe 93.240s
ok github.com/carlosframework/platform/cmd/zddprobe 0.381s
ok github.com/carlosframework/platform/internal/activator 226.783s
ok github.com/carlosframework/platform/internal/agentsock 0.016s
ok github.com/carlosframework/platform/internal/alerting 0.003s
ok github.com/carlosframework/platform/internal/analytics 0.023s
ok github.com/carlosframework/platform/internal/appconfig 2.345s
ok github.com/carlosframework/platform/internal/apperr 0.071s
ok github.com/carlosframework/platform/internal/applog 0.131s
ok github.com/carlosframework/platform/internal/awsinv 0.005s
ok github.com/carlosframework/platform/internal/boxchoice 0.004s
ok github.com/carlosframework/platform/internal/boxid 1.017s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.011s
ok github.com/carlosframework/platform/internal/certwarm 0.002s
ok github.com/carlosframework/platform/internal/cgroup 0.690s
ok github.com/carlosframework/platform/internal/changelog 0.291s
ok github.com/carlosframework/platform/internal/clamd 0.055s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 362.258s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.554s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.008s
ok github.com/carlosframework/platform/internal/dbverify 0.005s
ok github.com/carlosframework/platform/internal/dnsrec 0.002s
ok github.com/carlosframework/platform/internal/docsite 0.005s
ok github.com/carlosframework/platform/internal/domains 2.163s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.005s
ok github.com/carlosframework/platform/internal/edge 25.956s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.007s
ok github.com/carlosframework/platform/internal/fleetapi 0.122s
ok github.com/carlosframework/platform/internal/gate 1.657s
ok github.com/carlosframework/platform/internal/genselect 0.015s
ok github.com/carlosframework/platform/internal/geo 0.018s
ok github.com/carlosframework/platform/internal/geoip 0.029s
ok github.com/carlosframework/platform/internal/host 94.197s
ok github.com/carlosframework/platform/internal/hostactors 0.485s
ok github.com/carlosframework/platform/internal/hostbox 2.120s
ok github.com/carlosframework/platform/internal/hostclaim 0.506s
ok github.com/carlosframework/platform/internal/hostevidence 0.005s
ok github.com/carlosframework/platform/internal/hostexec 4.809s
ok github.com/carlosframework/platform/internal/hostfence 0.032s
ok github.com/carlosframework/platform/internal/hostfiles 10.536s
ok github.com/carlosframework/platform/internal/hostfloor 0.012s
ok github.com/carlosframework/platform/internal/hostguard 1.002s
ok github.com/carlosframework/platform/internal/hostops 1.418s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.003s
ok github.com/carlosframework/platform/internal/hostquarantine 0.002s
ok github.com/carlosframework/platform/internal/hostrelocate 0.425s
ok github.com/carlosframework/platform/internal/hostresolve 0.136s
ok github.com/carlosframework/platform/internal/hostwrite 0.002s
ok github.com/carlosframework/platform/internal/infraview 0.002s
ok github.com/carlosframework/platform/internal/instancectl 1.815s
ok github.com/carlosframework/platform/internal/instanceexec 2.721s
ok github.com/carlosframework/platform/internal/instancefs 0.022s
ok github.com/carlosframework/platform/internal/instancekey 0.016s
ok github.com/carlosframework/platform/internal/instanceupdates 0.021s
ok github.com/carlosframework/platform/internal/layout 1.414s
ok github.com/carlosframework/platform/internal/lease 9.448s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.007s
ok github.com/carlosframework/platform/internal/ledger 0.015s
ok github.com/carlosframework/platform/internal/mail 0.168s
ok github.com/carlosframework/platform/internal/maintenance 0.083s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.513s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.090s
ok github.com/carlosframework/platform/internal/metrics 0.005s
ok github.com/carlosframework/platform/internal/moves 20.917s
ok github.com/carlosframework/platform/internal/moves/archive 0.750s
ok github.com/carlosframework/platform/internal/objstore 0.163s
ok github.com/carlosframework/platform/internal/ops 0.175s
ok github.com/carlosframework/platform/internal/park 0.201s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
2026/10/13 22:36:01 provision: listing apps/bab/blog/instances/: list refused — its records are unseen this pass
2026/10/13 22:36:01 provision: listing apps/bab/: listdirs refused — its records are unseen this pass
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: listing apps/bac/tito/instances/: list refused — its records are unseen this pass
2026/10/13 22:36:01 provision: listing apps/bac/tito/instances/: list refused — its records are unseen this pass
2026/10/13 22:36:01 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:01 provision: remove retired instance route shop.example: provision: claim admission unavailable
2026/10/13 22:36:01 provision: remove retired instance route shop.example: provision: claim admission unavailable
2026/10/13 22:36:01 provision: remove retired sidecar route worker.sidecar.internal: provision: claim admission unavailable
2026/10/13 22:36:01 provision: remove retired sidecar route worker.sidecar.internal: provision: claim admission unavailable
2026/10/13 22:36:01 provision: claim install acme.go.tito.io: provision: claim admission unavailable
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/13 22:36:01 provision: claim install bot.eleven.bac.sidecar.internal: provision: claim admission unavailable
2026/10/13 22:36:01 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 22:36:01 provision: claim install bot.eleven.bac.sidecar.internal: registry: claim row ownership changed or unknown
2026/10/13 22:36:01 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/13 22:36:01 provision: acme.go.tito.io: not minting this pass: the box's layout intent could not be read: no sealed config
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/13 22:36:01 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 22:36:01 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 22:36:01 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/13 22:36:01 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 22:36:01 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/13 22:36:01 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/13 22:36:01 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 22:36:01 provision: claim install bot.eleven.bac.sidecar.internal: registry: claim row ownership changed or unknown
2026/10/13 22:36:01 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 22:36:01 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:01 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:01 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 22:36:01 provision: skipping record apps/bac/tito/instances/new.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 22:36:01 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 22:36:01 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 22:36:01 provision: skipping record apps/bac/tito/instances/new.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 22:36:02 provision: minted instance route rb.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rc.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rf.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rg.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rh.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rj.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rk.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rl.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route ra.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rd.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route re.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route ri.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route ra.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rb.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rc.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route rd.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: skipping record apps/bac/tito/instances/rb.go.tito.io.json: read failed: read refused
2026/10/13 22:36:02 provision: skipping record apps/bac/tito/instances/rb.go.tito.io.json: read failed: read refused
2026/10/13 22:36:02 provision: minted instance route re.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.appd.example (app "appd" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.appg.example (app "appg" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.apph.example (app "apph" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.appb.example (app "appb" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.appc.example (app "appc" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.appe.example (app "appe" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.appf.example (app "appf" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.appi.example (app "appi" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.appj.example (app "appj" acct "bac")
2026/10/13 22:36:02 provision: minted instance route x.appa.example (app "appa" acct "bac")
2026/10/13 22:36:02 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 22:36:02 provision: minted instance route r0093.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0011.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0023.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0039.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0082.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0022.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0036.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0047.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0089.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0004.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0045.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0120.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0010.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0102.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0105.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0033.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0076.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0095.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0119.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0001.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0020.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0034.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0038.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0049.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0097.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0110.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0113.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0073.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0088.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0128.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0009.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0017.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0067.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0072.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0079.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0099.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0008.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0054.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0003.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0032.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0044.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0068.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0127.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0019.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0029.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0052.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0070.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0015.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0021.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0066.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0080.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0086.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0106.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0103.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0121.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0122.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0063.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0078.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0083.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0100.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0114.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0091.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0112.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0042.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0055.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0109.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0117.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0124.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0125.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0000.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0012.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0048.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0056.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0057.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0069.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0085.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0007.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0061.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0081.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0092.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0024.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0031.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0046.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0006.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0053.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0058.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0074.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0098.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0107.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0116.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0026.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0030.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0060.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0065.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0096.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0115.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0028.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0043.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0111.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0118.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0014.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0027.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0035.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0059.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0064.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0075.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0108.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0123.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0005.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0040.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0090.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0025.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0062.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0077.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0087.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0016.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0094.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0104.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0129.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0013.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0018.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0126.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0041.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0051.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0101.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0002.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0037.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0050.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0071.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:02 provision: minted instance route r0084.go.tito.io (app "tito" acct "bac")
--- FAIL: TestPlan14RootReadCensus (8.59s)
plan14_root_reads_test.go:664: root read census: 16 discrepancies
changed root read internal/host/units/carlos-litestream.service:external; review class, task, sites and digest
changed root read internal/ops/ops.go:runLitestream; review class, task, sites and digest
changed root read internal/status/snapshot.go:CollectBox; review class, task, sites and digest
changed root source envelope internal/activator
changed root source envelope internal/host
changed root source envelope internal/host/units
changed root source envelope internal/lease
changed root source envelope internal/metrics
changed root source envelope internal/ops
changed root source envelope internal/status
unclassified root read internal/activator/replicalevel.go:Service.ReplicaLevels registry:Replication@680,whole:Route@682
unclassified root read internal/host/restartwatch.go:platformUnitRestarts file:Command@60
unclassified root read internal/ops/ops.go:systemdDaemonReload file:Command@48
changed legacy declaration internal/host/units/carlos-litestream.service:external
changed legacy declaration internal/ops/ops.go:runLitestream
changed legacy declaration internal/status/snapshot.go:CollectBox
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route whole.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route console.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted service route ha.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted service route svc.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route shared.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: converged acme.go.tito.io channel stable -> edge
2026/10/13 22:36:11 provision: converged acme.go.tito.io channel edge -> canary/rehearsal
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: empty channel
2026/10/13 22:36:11 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: converged acme.go.tito.io environment "" -> "staging"
2026/10/13 22:36:11 provision: converged acme.go.tito.io environment "staging" -> ""
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: converged acme.go.tito.io steering "" -> "latency"
2026/10/13 22:36:11 provision: converged acme.go.tito.io steering "latency" -> ""
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: converged acme.go.tito.io hibernate false -> true
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: converged acme.go.tito.io hibernate true -> false (replication "daily" cleared: only a hibernating route's rung is read)
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: converged acme.go.tito.io status-named false -> true
2026/10/13 22:36:11 provision: converged acme.go.tito.io status-named true -> false
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted service route ha.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7001,127.0.0.1:7002" -> "127.0.0.1:7001,127.0.0.1:7003" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/13 22:36:11 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7001,127.0.0.1:7003" -> "127.0.0.1:7003,127.0.0.1:7001" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/13 22:36:11 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7003,127.0.0.1:7001" -> "127.0.0.1:7003,127.0.0.1:7001" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/healthz\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/13 22:36:11 provision: minted service route whole.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: converged whole.go.tito.io upstreams addr "127.0.0.1:9443" -> "", addrs "" -> "127.0.0.1:9443,127.0.0.1:9444" (probe "" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/13 22:36:11 provision: converged whole.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:9443,127.0.0.1:9444" -> "" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "")
2026/10/13 22:36:11 provision: whole.go.tito.io: record describes a different route SHAPE than the row (kind "instance"/"service" unit ""/"" socket "/run/carlos/whole.go.tito.io.sock"/"" db "/data/whole.go.tito.io.db"/"") — not converged in place; `carlos remove whole.go.tito.io` purges this route's capability grants (registry.Delete's contract, not a bug) before the next pass re-mints it, so treat that as a last resort, not a routine fix
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: minted service route whole.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: skipping accounts/bac/account.json: malformed account manifest: invalid character 'o' in literal null (expecting 'u')
2026/10/13 22:36:11 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: manifest bac/tito: read failed: provision: manifest bac/tito: invalid character 'o' in literal null (expecting 'u')
2026/10/13 22:36:11 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: disagrees with its own path (host="wrong-host.go.tito.io" account="bac" app="tito")
2026/10/13 22:36:11 provision: skipping record apps/bac/tito/instances/acme$evil.go.tito.io.json: host "acme$evil.go.tito.io" is not a hostname
2026/10/13 22:36:11 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:11 provision: skipping record apps/bac/tito/instances/acme$evil.go.tito.io.json: host "acme$evil.go.tito.io" is not a hostname
2026/10/13 22:36:11 provision: not removing acme$evil.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 22:36:12 provision: skipping record apps/bac/tito/instances/bad-service.go.tito.io.json: invalid shape: kind service with no addr: the platform only proxies to a service, so the record must name the upstream it proxies to
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: skipping record apps/bac/tito/instances/bad-service.go.tito.io.json: invalid shape: kind service with no addr: the platform only proxies to a service, so the record must name the upstream it proxies to
2026/10/13 22:36:12 provision: not removing bad-service.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 22:36:12 provision: skipping record apps/bac/tito/instances/no-channel.go.tito.io.json: empty channel
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: skipping record apps/bac/tito/instances/no-channel.go.tito.io.json: empty channel
2026/10/13 22:36:12 provision: not removing no-channel.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 22:36:12 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 22:36:12 provision: removed retired instance route ghost.go.tito.io (record gone)
2026/10/13 22:36:12 provision: duplicate host dup.go.tito.io: keeping apps/bac/tito/instances/dup.go.tito.io.json, skipping apps/bad/tito/instances/dup.go.tito.io.json
2026/10/13 22:36:12 provision: minted instance route dup.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 22:36:12 provision: removed retired instance route acme.go.tito.io (app in the trash)
2026/10/13 22:36:12 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 22:36:12 provision: removed retired static route site.go.tito.io (app in the trash)
2026/10/13 22:36:12 provision: trash marker for bac/tito unreadable: release: malformed trash marker for bac/tito: invalid character 'n' looking for beginning of object key string — not minting for this app
2026/10/13 22:36:12 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 22:36:12 provision: removed retired instance route acme.go.tito.io (app in the trash)
2026/10/13 22:36:12 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 22:36:12 provision: trash marker for bac/tito unreadable: release: malformed trash marker for bac/tito: invalid character 'n' looking for beginning of object key string — not minting for this app
2026/10/13 22:36:12 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 22:36:12 provision: converged acme.go.tito.io channel stable -> edge
2026/10/13 22:36:12 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 22:36:12 provision: converged acme.go.tito.io channel canary/rehearsal -> edge
2026/10/13 22:36:12 provision: converged acme.go.tito.io environment "staging" -> ""
2026/10/13 22:36:12 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 22:36:12 provision: acme.go.tito.io exists but is not the route this record describes (backing "unit" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/13 22:36:12 provision: acme.go.tito.io exists but is not the route this record describes (tunnel "pi-fleet" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/13 22:36:12 provision: acme.go.tito.io exists but is not the route this record describes (caps "port:25" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/13 22:36:12 provision: acme.go.tito.io exists but is not the route this record describes (platform (owned by the domains reconcile pass)) — leaving it alone
2026/10/13 22:36:12 provision: acme.go.tito.io exists but is not the route this record describes (canary (owned by the domains reconcile pass)) — leaving it alone
2026/10/13 22:36:12 provision: acme.go.tito.io exists but is not the route this record describes (db "/data/other.db"/"/data/acme.go.tito.io.db") — leaving it alone
2026/10/13 22:36:12 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 22:36:12 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/13 22:36:12 provision: dry-run: would converge owned.go.tito.io channel stable -> edge
2026/10/13 22:36:12 provision: dry-run: would converge owned.go.tito.io environment "" -> "staging"
2026/10/13 22:36:12 provision: dry-run: would converge owned.go.tito.io hibernate false -> true
2026/10/13 22:36:12 provision: dry-run: would converge owned.go.tito.io seed
2026/10/13 22:36:12 provision: dry-run: would mint instance route acme.go.tito.io (app "tito" acct "bac" channel "stable")
2026/10/13 22:36:12 provision: dry-run: would adopt existing route adopt.go.tito.io (app "tito" acct "bac") — this record would own its channel and environment
2026/10/13 22:36:12 provision: dry-run: would converge adopt.go.tito.io channel stable -> edge
2026/10/13 22:36:12 provision: dry-run: would converge adopt.go.tito.io environment "" -> "staging"
2026/10/13 22:36:12 provision: dry-run: would remove retired instance route gone.go.tito.io (record gone)
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: adopted existing route adopt.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 22:36:12 provision: converged adopt.go.tito.io channel stable -> edge
2026/10/13 22:36:12 provision: converged adopt.go.tito.io environment "" -> "staging"
2026/10/13 22:36:12 provision: converged owned.go.tito.io channel stable -> edge
2026/10/13 22:36:12 provision: converged owned.go.tito.io environment "" -> "staging"
2026/10/13 22:36:12 provision: converged owned.go.tito.io hibernate false -> true
2026/10/13 22:36:12 provision: removed retired instance route gone.go.tito.io (record gone)
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: adopted existing route story.carlosframework.com (app "story" acct "bad") — this record now owns its channel and environment
2026/10/13 22:36:12 provision: story.carlosframework.com exists but is not the route this record describes (kind "instance"/"static") — leaving it alone
2026/10/13 22:36:12 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted service route pub.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted service route pubs.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted service route priv.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route unit.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route exec.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted service route loop.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted service route named.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted static route pisite.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: removed retired static route site.go.tito.io (record gone)
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route acme-uk.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: converged acme.go.tito.io replicas "ie,uk" -> "uk"
2026/10/13 22:36:12 provision: converged acme.go.tito.io replicas "uk" -> ""
2026/10/13 22:36:12 provision: minted instance route acme-uk.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: converged acme-uk.go.tito.io replica mark true -> false
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: acme.go.tito.io: row is on the nested layout (db "/data/instances/acme.go.tito.io/acme.go.tito.io.db" socket "/run/carlos/instances/acme.go.tito.io/app.sock"); the record's shared paths are not drift
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:12 provision: acme.go.tito.io: not minting this pass: the box's layout intent could not be read: store down
2026/10/13 22:36:12 provision: router.go.tito.io: not minting this pass: the box's layout intent could not be read: store down
2026/10/13 22:36:12 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 22:36:12 provision: converged acme.go.tito.io channel stable -> edge
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
2026/10/13 22:36:12 provision: not removing acme.go.tito.io (record gone): participant of a running operation
2026/10/13 22:36:12 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/13 22:36:12 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
2026/10/13 22:36:12 provision: refusing to mint held route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/13 22:36:12 provision: held acme.go.tito.io landing by mv_acme (was ready)
2026/10/13 22:36:12 provision: acme.go.tito.io: held restoring by rb_other, not by mv_acme; leaving it
2026/10/13 22:36:12 provision: acme.go.tito.io: the row is not this pass's (app "other" acct ""); not holding it for mv_acme
2026/10/13 22:36:12 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 22:36:12 provision: dry-run: would mint instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac" channel "stable")
2026/10/13 22:36:12 provision: acme.go.tito.io: not held landing by mv_acme: the row's hold changed under this pass; the next pass judges it afresh
2026/10/13 22:36:12 provision: held acme.go.tito.io landing by mv_acme (was ready)
2026/10/13 22:36:13 provision: removed retired sidecar route gone.eleven.bac.sidecar.internal (record gone)
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: converged sidecar bot.eleven.bac.sidecar.internal args ["sidecar","run"] -> ["worker","run"]
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: converged sidecar bot.eleven.bac.sidecar.internal ceilings 300/3600 -> 60/120
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: converged sidecar bot.eleven.bac.sidecar.internal channel stable -> canary/eleven
2026/10/13 22:36:13 provision: converged sidecar bot.eleven.bac.sidecar.internal environment "other" -> "relay"
2026/10/13 22:36:13 provision: converged sidecar bot.eleven.bac.sidecar.internal hibernate true -> false
2026/10/13 22:36:13 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: read failed: failStore: reads disabled under apps/bac/eleven/sidecars/
2026/10/13 22:36:13 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: malformed: invalid character 'n' looking for beginning of object key string
2026/10/13 22:36:13 provision: not removing sidecar bot.eleven.bac.sidecar.internal: its record exists but could not be read cleanly this pass
2026/10/13 22:36:13 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: empty channel
2026/10/13 22:36:13 provision: skipping sidecar record apps/bac/eleven/sidecars/held.json: empty channel
2026/10/13 22:36:13 provision: not removing sidecar held.eleven.bac.sidecar.internal: its record exists but could not be read cleanly this pass
2026/10/13 22:36:13 provision: removed retired sidecar route old.eleven.bac.sidecar.internal (record gone)
2026/10/13 22:36:13 provision: bac/eleven has sidecar records but no sidecars manifest — not minting
2026/10/13 22:36:13 provision: bac/eleven is in the trash — not minting sidecar routes for it
2026/10/13 22:36:13 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 22:36:13 provision: sidecar hub.relay.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: minted sidecar route hub.relay.bac.sidecar.internal (app "relay" acct "bac" args ["relay"])
2026/10/13 22:36:13 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 22:36:13 provision: removed retired sidecar route bot.eleven.bac.sidecar.internal (record gone)
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 22:36:13 provision: skipping sidecar record apps/bac/relay/sidecars/hub.json: read failed: failStore: reads disabled under apps/bac/relay/sidecars/
2026/10/13 22:36:13 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/13 22:36:13 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 22:36:13 provision: dry-run: would mint sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" channel "stable" args ["sidecar","run"])
2026/10/13 22:36:13 provision: dry-run: would remove retired sidecar route gone.eleven.bac.sidecar.internal (record gone)
2026/10/13 22:36:13 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/13 22:36:13 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:13 provision: converged acme.go.tito.io update settings "{\"updates\":\"manual\",\"channel\":\"stable\"}" -> ""
2026/10/13 22:36:13 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:13 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:13 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/13 22:36:13 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/13 22:36:13 provision: acme.go.tito.io: not minted this pass: the app's update policy could not be read
2026/10/13 22:36:13 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 22:36:13 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/13 22:36:13 provision: acme.go.tito.io: not minted this pass: the app's update policy could not be read
2026/10/13 22:36:13 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
FAIL
FAIL github.com/carlosframework/platform/internal/provision 11.930s
ok github.com/carlosframework/platform/internal/publicrelease 0.006s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 3.999s
ok github.com/carlosframework/platform/internal/relay 7.228s
ok github.com/carlosframework/platform/internal/release 19.294s
ok github.com/carlosframework/platform/internal/rollout 3.672s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootparticipation 0.007s
ok github.com/carlosframework/platform/internal/rootstate 0.025s
ok github.com/carlosframework/platform/internal/s3compat 0.004s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.270s
ok github.com/carlosframework/platform/internal/sim/hostlife 9.191s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.182s
ok github.com/carlosframework/platform/internal/status 0.041s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.192s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.002s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.004s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.543s
FAIL
make: *** [Makefile:18: test] Error 1
amadan: step gate failed: exit status 2
amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/internal/activator/replicalevel.go b/internal/activator/replicalevel.go| new file mode 100644 |
| index 000000000..73bcbf202 |
| --- /dev/null |
| +++ b/internal/activator/replicalevel.go |
| @@ -0,0 +1,48 @@ |
| +package activator |
| + |
| +import "time" |
| + |
| +// ReplicaLevel is what the activator can say about one hibernating host's |
| +// replica without asking the bucket. |
| +type ReplicaLevel struct { |
| + // LevelAt is the last time the host's running replicator had nothing left |
| + // to ship (lease.Replicator.LevelAt). Zero when it has not managed that |
| + // since the wake, and for a parked host. |
| + LevelAt time.Time |
| + // Interval is the host's replication-class sync interval; 0 is stream. |
| + // A replica is not late until it trails by more than this. |
| + Interval time.Duration |
| + // Parked says the host is asleep and its last stop ended with a final |
| + // sync that succeeded: nothing can be waiting to ship. |
| + Parked bool |
| +} |
| + |
| +// ReplicaLevels reports every host this activator replicates itself: awake |
| +// hosts whose replicator can state its level, and sleeping hosts whose last |
| +// stop is remembered as clean. A unit-backed host is absent (the box-wide |
| +// replicator owns its database), and so is a sleeping host whose stop this |
| +// process did not see or could not prove: absent is "cannot say". |
| +func (a *Service) ReplicaLevels() map[string]ReplicaLevel { |
| + a.mu.Lock() |
| + defer a.mu.Unlock() |
| + out := map[string]ReplicaLevel{} |
| + for host, o := range a.stopHistory { |
| + if o != nil && o.replicaClosed && !o.uncertain { |
| + out[host] = ReplicaLevel{Parked: true} |
| + } |
| + } |
| + for host, e := range a.entries { |
| + // Any live entry replaces what the last stop said: the host is |
| + // waking, awake or stopping, and its replica can be behind again. |
| + delete(out, host) |
| + if e.state != actAwake || e.rep == nil { |
| + continue |
| + } |
| + rep, ok := e.rep.(interface{ LevelAt() time.Time }) |
| + if !ok { |
| + continue |
| + } |
| + out[host] = ReplicaLevel{LevelAt: rep.LevelAt(), Interval: replicationSyncInterval(e.route.Replication)} |
| + } |
| + return out |
| +} |
diff --git a/internal/activator/replicalevel_test.go b/internal/activator/replicalevel_test.go| new file mode 100644 |
| index 000000000..738f39c0d |
| --- /dev/null |
| +++ b/internal/activator/replicalevel_test.go |
| @@ -0,0 +1,55 @@ |
| +package activator |
| + |
| +import ( |
| + "context" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| +) |
| + |
| +type levelRep struct{ at time.Time } |
| + |
| +func (levelRep) Err() error { return nil } |
| +func (levelRep) Close(context.Context) error { return nil } |
| +func (r levelRep) LevelAt() time.Time { return r.at } |
| + |
| +type silentRep struct{} |
| + |
| +func (silentRep) Err() error { return nil } |
| +func (silentRep) Close(context.Context) error { return nil } |
| + |
| +func TestReplicaLevels(t *testing.T) { |
| + at := time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC) |
| + clean := &stopObservation{mainStop: true, mainExited: true, sidecarStop: true, sidecarExited: true, replicaClosed: true} |
| + a := &Service{ |
| + entries: map[string]*actEntry{ |
| + "awake.example.com": {host: "awake.example.com", state: actAwake, rep: levelRep{at}, route: registry.Route{Replication: replicationBatched}}, |
| + "fresh.example.com": {host: "fresh.example.com", state: actAwake, rep: levelRep{}}, |
| + "unit.example.com": {host: "unit.example.com", state: actAwake, rep: silentRep{}}, |
| + "waking.example.com": {host: "waking.example.com", state: actWaking}, |
| + "stopping.example.com": {host: "stopping.example.com", state: actStopping, rep: levelRep{at}}, |
| + "rewoken.example.com": {host: "rewoken.example.com", state: actWaking}, |
| + }, |
| + stopHistory: map[string]*stopObservation{ |
| + "parked.example.com": clean, |
| + "rewoken.example.com": clean, |
| + "unproven.example.com": {mainStop: true, mainExited: true, sidecarStop: true, sidecarExited: true, replicaClosed: true, uncertain: true}, |
| + "syncfailed.example.com": {mainStop: true, mainExited: true, sidecarStop: true, sidecarExited: true}, |
| + }, |
| + } |
| + got := a.ReplicaLevels() |
| + want := map[string]ReplicaLevel{ |
| + "awake.example.com": {LevelAt: at, Interval: 5 * time.Minute}, |
| + "fresh.example.com": {}, |
| + "parked.example.com": {Parked: true}, |
| + } |
| + if len(got) != len(want) { |
| + t.Fatalf("ReplicaLevels = %+v, want exactly %+v", got, want) |
| + } |
| + for host, w := range want { |
| + if g, ok := got[host]; !ok || g != w { |
| + t.Errorf("%s = %+v (present %v), want %+v", host, g, ok, w) |
| + } |
| + } |
| +} |
diff --git a/internal/lease/lease.go b/internal/lease/lease.go| index 61788e219..15a4eaf6a 100644 |
| --- a/internal/lease/lease.go |
| +++ b/internal/lease/lease.go |
| @@ -487,3 +487,11 @@ func Restore(ctx context.Context, dbPath, bucket, prefix, region string) (bool, |
| } |
| return true, nil |
| } |
| + |
| +// LevelAt is the last time a replica sync ended with nothing left to ship: |
| +// every change the database held when that sync began was in the replica when |
| +// it returned. litestream records it at the end of each sync that returns nil |
| +// (an idle database's ticks included, which make no S3 call), so the time |
| +// since it is an upper bound on how far the replica trails. Zero until the |
| +// first such sync after this replicator started. |
| +func (r *Replicator) LevelAt() time.Time { return r.db.LastSuccessfulSyncAt() } |
diff --git a/internal/metrics/alert.go b/internal/metrics/alert.go| index 489952dc4..fd37d28d5 100644 |
| --- a/internal/metrics/alert.go |
| +++ b/internal/metrics/alert.go |
| @@ -23,8 +23,16 @@ type Rules struct { |
| // observation's window before the restarts rule fires. It is a count, not |
| // a fraction, and the rule clears only on a reading of 0. |
| RestartCount float64 // fire >, default 3 |
| + // ReplicaSecs is how long a database's replica may be overdue (behind, |
| + // beyond its own sync interval) before the replica rule fires. Seconds, |
| + // like LagSecs; it clears replicaClearBy below. |
| + ReplicaSecs int64 // fire >, default 900, immediate |
| } |
| +// replicaClearBy is the replica rule's hysteresis: fired above 15 minutes |
| +// overdue, it clears below 10. |
| +const replicaClearBy = 300 |
| + |
| // DefaultRules are the thresholds shipped with the platform. |
| func DefaultRules() Rules { |
| return Rules{ |
| @@ -38,6 +46,7 @@ func DefaultRules() Rules { |
| // GeoIPSustain: geoip spec §10.4. |
| GeoIPSustain: 30 * time.Minute, |
| RestartCount: 3, |
| + ReplicaSecs: 900, |
| } |
| } |
| @@ -97,6 +106,12 @@ type Observation struct { |
| // tell yet. |
| Restarts *float64 |
| RestartsUnit string |
| + // ReplicaOverdueSecs is how far the box's worst replica is overdue: the |
| + // time since it was last level with its database, less that database's |
| + // sync interval. ReplicaOf names the database or host. 0 says every |
| + // replica the reporter can see is on time; nil says it can see none. |
| + ReplicaOverdueSecs *int64 |
| + ReplicaOf string |
| } |
| // RuleState is one rule's sustain/hysteresis bookkeeping. Since and FiredAt |
| @@ -115,7 +130,8 @@ type RuleState struct { |
| FiredAt time.Time `json:"fired_at,omitempty"` // set on fire, cleared on clear |
| Value float64 `json:"value,omitempty"` // reading at last transition |
| // Detail names what the reading was about, for the rules whose value |
| - // alone does not say: the unit for "restarts". Empty for the rest. |
| + // alone does not say: the unit for "restarts", the database or host for |
| + // "replica". Empty for the rest. |
| Detail string `json:"detail,omitempty"` |
| } |
| @@ -126,7 +142,7 @@ type AlertState struct { |
| V int `json:"v"` // 1 |
| Box string `json:"box"` |
| UpdatedAt time.Time `json:"updated_at"` |
| - Rules map[string]RuleState `json:"rules"` // "disk","mem","cpu","lag","geoip","restarts" |
| + Rules map[string]RuleState `json:"rules"` // "disk","mem","cpu","lag","geoip","restarts","replica" |
| } |
| // Event is emitted on a firing or clearing transition — never on every |
| @@ -307,5 +323,22 @@ func Evaluate(r Rules, st AlertState, obs Observation) (AlertState, []Event) { |
| } |
| } |
| + // replica: the reading is already a duration, so like lag it fires at |
| + // once. It names what is behind, and keeps that name until it clears even |
| + // if another replica becomes the worst meanwhile. |
| + if obs.ReplicaOverdueSecs != nil { |
| + val := float64(*obs.ReplicaOverdueSecs) |
| + cur := out.Rules["replica"] |
| + switch { |
| + case cur.Firing && val < float64(r.ReplicaSecs-replicaClearBy): |
| + of := cur.Detail |
| + out.Rules["replica"] = RuleState{Value: val} |
| + events = append(events, Event{Rule: "replica", Firing: false, Value: val, Box: st.Box, Detail: of}) |
| + case !cur.Firing && val > float64(r.ReplicaSecs): |
| + out.Rules["replica"] = RuleState{Firing: true, Since: obs.At, FiredAt: obs.At, Value: val, Detail: obs.ReplicaOf} |
| + events = append(events, Event{Rule: "replica", Firing: true, Value: val, Box: st.Box, Detail: obs.ReplicaOf}) |
| + } |
| + } |
| + |
| return out, events |
| } |
diff --git a/internal/metrics/alert_test.go b/internal/metrics/alert_test.go| index 92e0a6d4c..ec24213bf 100644 |
| --- a/internal/metrics/alert_test.go |
| +++ b/internal/metrics/alert_test.go |
| @@ -289,3 +289,41 @@ func TestRestartsRuleFiresAboveTheCountAndClearsOnlyAtZero(t *testing.T) { |
| t.Fatalf("a quiet rule emitted: %v", ev) |
| } |
| } |
| + |
| +// The replica rule fires once a replica is more than 15 minutes overdue, |
| +// names it, and clears below 10 minutes. No reading changes nothing. |
| +func TestReplicaRuleFiresWhenOverdueAndClearsWithHysteresis(t *testing.T) { |
| + r := DefaultRules() |
| + t0 := time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC) |
| + secs := func(v int64) *int64 { return &v } |
| + st := AlertState{V: 1, Box: "b"} |
| + |
| + st, ev := Evaluate(r, st, Observation{At: t0, ReplicaOverdueSecs: secs(900), ReplicaOf: "/data/shop.db"}) |
| + if len(ev) != 0 { |
| + t.Fatalf("fired at 15 minutes, not above: %v", ev) |
| + } |
| + st, ev = Evaluate(r, st, Observation{At: t0.Add(time.Minute), ReplicaOverdueSecs: secs(960), ReplicaOf: "/data/shop.db"}) |
| + if len(ev) != 1 || !ev[0].Firing || ev[0].Rule != "replica" || ev[0].Value != 960 || ev[0].Detail != "/data/shop.db" { |
| + t.Fatalf("above 15 minutes: %+v", ev) |
| + } |
| + // Another database is now the worst; the alert keeps the name it fired on. |
| + st, ev = Evaluate(r, st, Observation{At: t0.Add(2 * time.Minute), ReplicaOverdueSecs: secs(5000), ReplicaOf: "/data/other.db"}) |
| + if len(ev) != 0 || st.Rules["replica"].Detail != "/data/shop.db" { |
| + t.Fatalf("a change of worst replica re-fired or renamed: %v %+v", ev, st.Rules["replica"]) |
| + } |
| + st, ev = Evaluate(r, st, Observation{At: t0.Add(3 * time.Minute)}) |
| + if len(ev) != 0 || !st.Rules["replica"].Firing { |
| + t.Fatalf("no reading changed the rule: %v", ev) |
| + } |
| + st, ev = Evaluate(r, st, Observation{At: t0.Add(4 * time.Minute), ReplicaOverdueSecs: secs(700)}) |
| + if len(ev) != 0 || !st.Rules["replica"].Firing { |
| + t.Fatalf("cleared inside the hysteresis band: %v", ev) |
| + } |
| + st, ev = Evaluate(r, st, Observation{At: t0.Add(5 * time.Minute), ReplicaOverdueSecs: secs(0)}) |
| + if len(ev) != 1 || ev[0].Firing || ev[0].Detail != "/data/shop.db" { |
| + t.Fatalf("on time again must clear and name what had fired: %+v", ev) |
| + } |
| + if rs := st.Rules["replica"]; rs.Firing || rs.Detail != "" || !rs.FiredAt.IsZero() { |
| + t.Fatalf("state after clearing: %+v", rs) |
| + } |
| +} |
diff --git a/internal/status/snapshot.go b/internal/status/snapshot.go| index 4dd9ef44a..76329d6f6 100644 |
| --- a/internal/status/snapshot.go |
| +++ b/internal/status/snapshot.go |
| @@ -150,6 +150,12 @@ type Probes struct { |
| // out of Metrics.RouteDBBytes rather than published as 0. |
| DBBytes func(dbPath string) *int64 |
| + // Replicas is host -> what this box knows about that route's off-box |
| + // replica, already assembled by the caller from the activator's |
| + // replicators and the box-wide replica check. Carried straight through |
| + // to Metrics.RouteReplicas. A host the box cannot speak for is absent. |
| + Replicas map[string]RouteReplica |
| + |
| // Layout is the box's layout report (isolated-exec design §8.4): which |
| // mode the box last converged to and, per instance-kind route, the word |
| // the layout pass's own Classify gave it. The caller (internal/host's |
| @@ -591,6 +597,11 @@ type BoxMetrics struct { |
| // readers. A host whose file could not be sized is absent, never 0; the |
| // whole map is omitted when no route could be sized. |
| RouteDBBytes map[string]int64 `json:"route_db_bytes,omitempty"` |
| + // RouteReplicas is host -> the state of that route's off-box replica as |
| + // this box sees it (discussion #284). Hostname-keyed and private like |
| + // RouteDBBytes, and absent for a host the box cannot speak for: no entry |
| + // is "unknown", never "fine". |
| + RouteReplicas map[string]RouteReplica `json:"route_replicas,omitempty"` |
| // Layout is Probes.Layout carried straight through — the box's layout |
| // report, hostname-keyed like RouteVersions and RouteDBBytes above, and |
| // private for the identical reason: BuildFleet copies Box's fields into |
| @@ -699,6 +710,25 @@ type RouteCounts struct { |
| Asleep *int `json:"asleep"` // null when the socket table is unreadable |
| } |
| +// RouteReplica is one route's replica as its box sees it. At most one of |
| +// LevelAt, BehindSince and Parked says where the replica stands; none of them |
| +// set means the box replicates the route but cannot yet say (a host woken |
| +// moments ago, before its first sync). |
| +type RouteReplica struct { |
| + // LevelAt is the last time nothing on the box was waiting to ship for |
| + // this route: the replica held every change the database did. |
| + LevelAt *time.Time `json:"level_at,omitempty"` |
| + // BehindSince is set when the replica is known to trail the database, |
| + // and is the earliest time the box held a change the replica did not. |
| + BehindSince *time.Time `json:"behind_since,omitempty"` |
| + // Parked says the instance is asleep and its final sync succeeded, so |
| + // nothing can be waiting. |
| + Parked bool `json:"parked,omitempty"` |
| + // IntervalSecs is the route's sync interval (0 for a continuous one). A |
| + // replica is on time while it trails by no more than this. |
| + IntervalSecs int64 `json:"interval_secs,omitempty"` |
| +} |
| + |
| // Replication summarizes litestream's furthest-behind lag across this box's |
| // routes, banded to a healthy bool rather than the raw seconds — the raw |
| // number is still carried for the console's private view, but the public |
| @@ -1043,7 +1073,7 @@ func CollectBox(routes []registry.Route, p Probes, now time.Time) Box { |
| p.DiskFreeBytes != nil || p.Release != "" || p.Tunnel != "" || |
| p.InstanceID != "" || p.Platform != "" || p.SystemChannel != "" || p.System != nil || |
| len(routeVersions) > 0 || len(routeServing) > 0 || len(aliasReadings) > 0 || len(p.Upstreams) > 0 || |
| - len(routeDBBytes) > 0 || p.Layout != nil || len(p.ConfigRefusals) > 0 || len(p.ReleaseRefusals) > 0 || len(p.ProofWaits) > 0 || p.Rollout != nil || p.ConfigGuard != "" || |
| + len(routeDBBytes) > 0 || len(p.Replicas) > 0 || p.Layout != nil || len(p.ConfigRefusals) > 0 || len(p.ReleaseRefusals) > 0 || len(p.ProofWaits) > 0 || p.Rollout != nil || p.ConfigGuard != "" || |
| p.Alerts != nil || p.GeoIP != nil || len(p.BuildStates) > 0 || p.BuildStatesErr != "" || |
| p.UpdateSettings != 0 || len(p.Updates) > 0 { |
| box.Metrics = &BoxMetrics{ |
| @@ -1075,7 +1105,9 @@ func CollectBox(routes []registry.Route, p Probes, now time.Time) Box { |
| // alongside another known field above, and omitempty drops it. |
| Upstreams: p.Upstreams, |
| RouteDBBytes: routeDBBytes, |
| - Layout: p.Layout, |
| + // Built fresh by the caller every tick, like Upstreams. |
| + RouteReplicas: p.Replicas, |
| + Layout: p.Layout, |
| // A fresh copy: the guard's own slice is its to mutate. |
| ConfigRefusals: append([]ConfigRefusal(nil), p.ConfigRefusals...), |
| ConfigRefusalsTotal: p.ConfigRefusalsTotal, |