amadan
WIP: slice B as built on the withdrawn design, plus the console's alert detail (#284)
Summary
Everything built before Astra's design review of slice B came back (amadan #290): lease.Replicator.Level, activator.ReplicaLevels, the console's replication column in the JSON API, the CLI and the HTML table, and the alert detail for the restarts and replica rules. Also the box's alert sentences for restarts.
Agent notes
Branch replica-freshness-284 is cut here and keeps all of it for the revised design. The next commit on litestream-oom-284 takes the slice B parts back out so slice A can land alone. Known wrong per the review: Level() reads last-successful-sync (findings 1, 2), and the Parked state trusts a nil Close (finding 3). All user-visible strings are drafts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
18 files changed,
+615
−33
apimode.go+11 −0docs/site/cli.md+7 −0instances_cli.go+24 −3instances_cli_test.go+63 −0instances_e2e_test.go+3 −0internal/activator/replicalevel.go+32 −10internal/activator/replicalevel_test.go+9 −4internal/console/alertwatch.go+24 −4internal/console/apiinstances.go+83 −2internal/console/metricsview.go+4 −1internal/console/overview.go+34 −0internal/console/replicafresh_test.go+261 −0internal/console/server.go+9 −0internal/console/templates/app_instances.html+3 −2internal/console/templates/metrics.html+1 −1internal/host/alertsink.go+15 −0internal/host/alertsink_test.go+12 −0internal/lease/lease.go+20 −6
CI failed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.011s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
using http://127.0.0.1:42379 as a@keymail.dev
shipped titesite abc1234
using http://127.0.0.1:46413 as a@keymail.dev
using http://127.0.0.1:43733 as a@keymail.dev
shipped titogo abc1234
using http://127.0.0.1:46067 as vicky@keymail.dev
titogo (staging):
KEY VALUE
PORT 8080
--- FAIL: TestZeroDowntimeCopyIsApplied (0.00s)
approved_copy_zdd_test.go:72: zdd.list.header: instances_cli.go does not contain the approved text "HOST\\tREGION\\tCHANNEL\\tPOOL\\tINSTALLED\\tSERVING\\tHEALTH\\tUPSTREAMS\\tDB\\tCREATED\\tBY\\tREASON"
2026/10/13 23:54:03 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:10 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:16 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:22 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:27 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:29 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:31 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:33 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:35 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:37 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:39 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:41 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:41 console: api instances get bab/titogo/probe-titogo.bab.oncarlos.test: read timeout
2026/10/13 23:54:41 console: api instances get bab/titogo/probe-titogo.bab.oncarlos.test: read timeout
2026/10/13 23:54:43 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:45 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:49 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:53 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:57 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:54:57 edges: box "uk-box" has an edges file under control/edges/ but no status snapshot at all, so no region outside the home is automatically eligible until it reports; said once
2026/10/13 23:54:57 edges: 1 edges file(s) under control/edges/ but no fresh box snapshot matched any of them, so automatic region placement sees the home alone (are edges and status in different buckets? see CARLOS_STATUS_BUCKET/CARLOS_STATUS_PREFIX)
2026/10/13 23:54:57 console: an incomplete read of the edges files or box snapshots (edges complete true, snapshots complete false), so no region outside the home is automatically eligible this request; said once
using http://127.0.0.1:36261 as paul@keymail.dev
account: ops (bab)
set 1 env var(s) for titogo (default)
using http://127.0.0.1:36261 as paul@keymail.dev
account: ops (bab)
set 1 secret(s) for titogo (default)
account: ops (bab)
account: ops (bab)
using http://127.0.0.1:36261 as paul@keymail.dev
account: ops (bab)
config sync requested for titogo the files land within seconds; programs carlos owns restart themselves, and anything else keeps its old values until you restart it
2026/10/13 23:55:21 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/13 23:55:21 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/13 23:55:21 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/13 23:55:21 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/13 23:55:21 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/13 23:55:21 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/13 23:55:23 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/13 23:55:23 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=888
2026/10/13 23:55:23 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/13 23:55:23 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
using http://127.0.0.1:46789 as paul@keymail.dev
using http://127.0.0.1:46789 as paul@keymail.dev
using http://127.0.0.1:46789 as paul@keymail.dev
Querying a copy of acme.go.tito.io's database, up to transaction 3, written 2026-10-13 23:55:23 UTC. Nothing you run here reaches acme.go.tito.io.
2026/10/13 23:55:23 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:55:23 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:55:25 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:55:27 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:55:29 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:55:31 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:55:33 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:55:35 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:55:37 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
using http://127.0.0.1:44969 as paul@keymail.dev
account: ops (bab)
attached tickets.example.com -> default.go.tito.io
DNS: pointing
using http://127.0.0.1:44969 as paul@keymail.dev
account: ops (bab)
attached tickets.example.com -> default.go.tito.io
DNS: pointing
using http://127.0.0.1:44969 as paul@keymail.dev
account: ops (bab)
attached tickets.example.com -> default.go.tito.io
DNS: pointing
using http://127.0.0.1:44969 as paul@keymail.dev
account: ops (bab)
attached tickets.example.com -> default.go.tito.io
DNS: pointing
using http://127.0.0.1:38689 as paul@keymail.dev
account: ops (bab)
attached tickets.example.com -> default.go.tito.io
DNS: pointing
using http://127.0.0.1:38689 as paul@keymail.dev
account: ops (bab)
attached carloku.example.com -> default.go.tito.io
2026/10/13 23:56:04 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:56:35 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:56:43 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:57:24 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/13 23:57:29 domains: minted platform domain echoapp.bab.oncarlos.test (upstream app.local)
2026/10/13 23:57:29 domains: minted platform domain site.bab.oncarlos.test (upstream site.local)
2026/10/13 23:57:31 edge: static oldsite/stable: resolve channel: dirstore: apps/bab/oldsite/channels/stable.json: file does not exist — if this deployment predates account-scoped bucket keys its objects are still at apps/oldsite/, which the runtime no longer reads; move them once with: carlos accounts migrate --app oldsite --from legacy --to bab
2026/10/13 23:57:31 domains: minted platform domain blog.mdbbndjqx.oncarlos.test (upstream blog.kass.test)
2026/10/13 23:57:31 domains: minted platform domain blog.gdpkwhwrw.oncarlos.test (upstream blog.woodstar.test)
2026/10/13 23:57:31 domains: minted platform domain oldsite.bab.oncarlos.test (upstream old.test)
2026/10/13 23:57:31 edge: static oldsite/stable: resolve channel: dirstore: apps/bab/oldsite/channels/stable.json: file does not exist — if this deployment predates account-scoped bucket keys its objects are still at apps/oldsite/, which the runtime no longer reads; move them once with: carlos accounts migrate --app oldsite --from legacy --to bab
FAIL
FAIL github.com/carlosframework/platform 336.606s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.142s
ok github.com/carlosframework/platform/cmd/moveprobe 93.203s
ok github.com/carlosframework/platform/cmd/zddprobe 0.383s
ok github.com/carlosframework/platform/internal/activator 228.039s
ok github.com/carlosframework/platform/internal/agentsock 0.016s
ok github.com/carlosframework/platform/internal/alerting 0.005s
ok github.com/carlosframework/platform/internal/analytics 0.019s
ok github.com/carlosframework/platform/internal/appconfig 2.371s
ok github.com/carlosframework/platform/internal/apperr 0.074s
ok github.com/carlosframework/platform/internal/applog 0.148s
ok github.com/carlosframework/platform/internal/awsinv 0.005s
ok github.com/carlosframework/platform/internal/boxchoice 0.005s
ok github.com/carlosframework/platform/internal/boxid 0.763s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.012s
ok github.com/carlosframework/platform/internal/certwarm 0.002s
ok github.com/carlosframework/platform/internal/cgroup 0.709s
ok github.com/carlosframework/platform/internal/changelog 0.298s
ok github.com/carlosframework/platform/internal/clamd 0.054s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 363.937s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.063s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.009s
ok github.com/carlosframework/platform/internal/dbverify 0.004s
ok github.com/carlosframework/platform/internal/dnsrec 0.003s
ok github.com/carlosframework/platform/internal/docsite 0.004s
ok github.com/carlosframework/platform/internal/domains 2.143s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.003s
ok github.com/carlosframework/platform/internal/edge 25.895s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.007s
ok github.com/carlosframework/platform/internal/fleetapi 0.122s
ok github.com/carlosframework/platform/internal/gate 1.655s
ok github.com/carlosframework/platform/internal/genselect 0.015s
ok github.com/carlosframework/platform/internal/geo 0.014s
ok github.com/carlosframework/platform/internal/geoip 0.029s
ok github.com/carlosframework/platform/internal/host 96.604s
ok github.com/carlosframework/platform/internal/hostactors 0.309s
ok github.com/carlosframework/platform/internal/hostbox 2.107s
ok github.com/carlosframework/platform/internal/hostclaim 0.480s
ok github.com/carlosframework/platform/internal/hostevidence 0.005s
ok github.com/carlosframework/platform/internal/hostexec 4.861s
ok github.com/carlosframework/platform/internal/hostfence 0.030s
ok github.com/carlosframework/platform/internal/hostfiles 10.512s
ok github.com/carlosframework/platform/internal/hostfloor 0.012s
ok github.com/carlosframework/platform/internal/hostguard 1.006s
ok github.com/carlosframework/platform/internal/hostops 1.426s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.004s
ok github.com/carlosframework/platform/internal/hostquarantine 0.004s
ok github.com/carlosframework/platform/internal/hostrelocate 0.437s
ok github.com/carlosframework/platform/internal/hostresolve 0.138s
ok github.com/carlosframework/platform/internal/hostwrite 0.004s
ok github.com/carlosframework/platform/internal/infraview 0.020s
ok github.com/carlosframework/platform/internal/instancectl 1.812s
ok github.com/carlosframework/platform/internal/instanceexec 2.706s
ok github.com/carlosframework/platform/internal/instancefs 0.004s
ok github.com/carlosframework/platform/internal/instancekey 0.003s
ok github.com/carlosframework/platform/internal/instanceupdates 0.024s
ok github.com/carlosframework/platform/internal/layout 1.417s
ok github.com/carlosframework/platform/internal/lease 9.449s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.008s
ok github.com/carlosframework/platform/internal/ledger 0.010s
ok github.com/carlosframework/platform/internal/mail 0.160s
ok github.com/carlosframework/platform/internal/maintenance 0.077s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.542s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.080s
ok github.com/carlosframework/platform/internal/metrics 0.008s
ok github.com/carlosframework/platform/internal/moves 21.088s
ok github.com/carlosframework/platform/internal/moves/archive 0.772s
ok github.com/carlosframework/platform/internal/objstore 0.156s
ok github.com/carlosframework/platform/internal/ops 0.156s
ok github.com/carlosframework/platform/internal/park 0.206s
ok github.com/carlosframework/platform/internal/privatefile 0.005s
2026/10/13 23:53:36 provision: listing apps/bab/blog/instances/: list refused — its records are unseen this pass
2026/10/13 23:53:36 provision: listing apps/bab/: listdirs refused — its records are unseen this pass
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: listing apps/bac/tito/instances/: list refused — its records are unseen this pass
2026/10/13 23:53:36 provision: listing apps/bac/tito/instances/: list refused — its records are unseen this pass
2026/10/13 23:53:36 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:36 provision: remove retired instance route shop.example: provision: claim admission unavailable
2026/10/13 23:53:36 provision: remove retired instance route shop.example: provision: claim admission unavailable
2026/10/13 23:53:36 provision: remove retired sidecar route worker.sidecar.internal: provision: claim admission unavailable
2026/10/13 23:53:36 provision: remove retired sidecar route worker.sidecar.internal: provision: claim admission unavailable
2026/10/13 23:53:36 provision: claim install acme.go.tito.io: provision: claim admission unavailable
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/13 23:53:36 provision: claim install bot.eleven.bac.sidecar.internal: provision: claim admission unavailable
2026/10/13 23:53:36 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 23:53:36 provision: claim install bot.eleven.bac.sidecar.internal: registry: claim row ownership changed or unknown
2026/10/13 23:53:36 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/13 23:53:36 provision: acme.go.tito.io: not minting this pass: the box's layout intent could not be read: no sealed config
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/13 23:53:36 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 23:53:36 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 23:53:36 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/13 23:53:36 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 23:53:36 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: claim install acme.go.tito.io: registry: claim row ownership changed or unknown
2026/10/13 23:53:36 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/13 23:53:36 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["run"])
2026/10/13 23:53:36 provision: claim install bot.eleven.bac.sidecar.internal: registry: claim row ownership changed or unknown
2026/10/13 23:53:36 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 23:53:36 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:36 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 23:53:36 provision: skipping record apps/bac/tito/instances/new.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 23:53:36 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 23:53:36 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 23:53:36 provision: skipping record apps/bac/tito/instances/new.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 23:53:36 provision: minted instance route rb.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route re.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rh.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route ri.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rk.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rc.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rd.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rf.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rg.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rj.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rl.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route ra.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route ra.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rb.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rc.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route rd.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: skipping record apps/bac/tito/instances/rb.go.tito.io.json: read failed: read refused
2026/10/13 23:53:36 provision: skipping record apps/bac/tito/instances/rb.go.tito.io.json: read failed: read refused
2026/10/13 23:53:36 provision: minted instance route re.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.appa.example (app "appa" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.appb.example (app "appb" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.appc.example (app "appc" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.appe.example (app "appe" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.appd.example (app "appd" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.appf.example (app "appf" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.appg.example (app "appg" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.apph.example (app "apph" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.appi.example (app "appi" acct "bac")
2026/10/13 23:53:36 provision: minted instance route x.appj.example (app "appj" acct "bac")
2026/10/13 23:53:37 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 23:53:37 provision: minted instance route r0091.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0096.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0125.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0018.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0022.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0032.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0047.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0062.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0119.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0003.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0009.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0016.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0066.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0070.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0101.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0116.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0121.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0094.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0111.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0123.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0008.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0046.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0056.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0067.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0071.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0090.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0110.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0117.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0027.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0040.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0058.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0120.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0105.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0005.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0034.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0039.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0077.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0000.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0002.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0004.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0011.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0020.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0073.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0089.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0095.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0054.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0055.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0127.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0026.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0109.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0115.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0001.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0063.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0079.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0097.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0106.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0114.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0126.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0128.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0025.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0030.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0050.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0072.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0124.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0021.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0048.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0060.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0035.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0042.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0043.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0049.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0068.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0065.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0080.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0085.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0036.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0057.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0099.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0015.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0087.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0102.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0024.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0074.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0093.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0113.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0052.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0092.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0103.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0014.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0069.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0098.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0129.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0053.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0006.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0037.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0045.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0059.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0076.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0112.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0122.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0081.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0104.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0012.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0041.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0044.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0064.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0100.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0118.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0017.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0033.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0075.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0084.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0019.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0038.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0007.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0029.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0051.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0078.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0082.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0083.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0108.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0010.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0031.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0086.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0088.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0107.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0013.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0023.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0028.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:37 provision: minted instance route r0061.go.tito.io (app "tito" acct "bac")
--- FAIL: TestPlan14RootReadCensus (9.82s)
plan14_root_reads_test.go:664: root read census: 27 discrepancies
changed root read internal/console/apiinstances.go:Server.handleAPIInstanceList; review class, task, sites and digest
changed root read internal/console/server.go:Server.handleAppInstances; review class, task, sites and digest
changed root read internal/console/templates/app_instances.html:external; review class, task, sites and digest
changed root read internal/console/templates/metrics.html:external; review class, task, sites and digest
changed root read internal/host/units/carlos-litestream.service:external; review class, task, sites and digest
changed root read internal/ops/ops.go:runLitestream; review class, task, sites and digest
changed root read internal/status/snapshot.go:CollectBox; review class, task, sites and digest
changed root source envelope .
changed root source envelope internal/activator
changed root source envelope internal/console
changed root source envelope internal/console/templates
changed root source envelope internal/host
changed root source envelope internal/host/units
changed root source envelope internal/lease
changed root source envelope internal/metrics
changed root source envelope internal/ops
changed root source envelope internal/status
unclassified root read internal/activator/replicalevel.go:Service.ReplicaLevels registry:Replication@687,whole:Route@689
unclassified root read internal/host/restartwatch.go:platformUnitRestarts file:Command@60
unclassified root read internal/ops/ops.go:systemdDaemonReload file:Command@48
changed legacy declaration internal/console/apiinstances.go:Server.handleAPIInstanceList
changed legacy declaration internal/console/server.go:Server.handleAppInstances
changed legacy declaration internal/console/templates/app_instances.html:external
changed legacy declaration internal/console/templates/metrics.html:external
changed legacy declaration internal/host/units/carlos-litestream.service:external
changed legacy declaration internal/ops/ops.go:runLitestream
changed legacy declaration internal/status/snapshot.go:CollectBox
--- FAIL: TestPlan9B8RecordWriteCensus (0.49s)
plan9_b8_census_test.go:371: record-write census rejected 2 site(s); update the explicit inventory only after reviewing the source:
changed site internal/console/apiinstances.go:Server.handleAPIInstanceList: source differs from the reviewed digest
changed site internal/console/server.go:Server.handleAppInstances: source differs from the reviewed digest
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: refusing to mint route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route console.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted service route ha.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted service route svc.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route whole.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route shared.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: converged acme.go.tito.io channel stable -> edge
2026/10/13 23:53:47 provision: converged acme.go.tito.io channel edge -> canary/rehearsal
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: empty channel
2026/10/13 23:53:47 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: converged acme.go.tito.io environment "" -> "staging"
2026/10/13 23:53:47 provision: converged acme.go.tito.io environment "staging" -> ""
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: converged acme.go.tito.io steering "" -> "latency"
2026/10/13 23:53:47 provision: converged acme.go.tito.io steering "latency" -> ""
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: converged acme.go.tito.io hibernate false -> true
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: converged acme.go.tito.io hibernate true -> false (replication "daily" cleared: only a hibernating route's rung is read)
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: converged acme.go.tito.io status-named false -> true
2026/10/13 23:53:47 provision: converged acme.go.tito.io status-named true -> false
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted service route ha.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7001,127.0.0.1:7002" -> "127.0.0.1:7001,127.0.0.1:7003" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/13 23:53:47 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7001,127.0.0.1:7003" -> "127.0.0.1:7003,127.0.0.1:7001" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/13 23:53:47 provision: converged ha.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:7003,127.0.0.1:7001" -> "127.0.0.1:7003,127.0.0.1:7001" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "{\"path\":\"/healthz\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/13 23:53:47 provision: minted service route whole.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: converged whole.go.tito.io upstreams addr "127.0.0.1:9443" -> "", addrs "" -> "127.0.0.1:9443,127.0.0.1:9444" (probe "" -> "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}")
2026/10/13 23:53:47 provision: converged whole.go.tito.io upstreams addr "" -> "", addrs "127.0.0.1:9443,127.0.0.1:9444" -> "" (probe "{\"path\":\"/\",\"status\":200,\"interval_ms\":2000,\"fail_after\":3,\"rise_after\":2}" -> "")
2026/10/13 23:53:47 provision: whole.go.tito.io: record describes a different route SHAPE than the row (kind "instance"/"service" unit ""/"" socket "/run/carlos/whole.go.tito.io.sock"/"" db "/data/whole.go.tito.io.db"/"") — not converged in place; `carlos remove whole.go.tito.io` purges this route's capability grants (registry.Delete's contract, not a bug) before the next pass re-mints it, so treat that as a last resort, not a routine fix
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: minted service route whole.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: skipping accounts/bac/account.json: malformed account manifest: invalid character 'o' in literal null (expecting 'u')
2026/10/13 23:53:47 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: manifest bac/tito: read failed: provision: manifest bac/tito: invalid character 'o' in literal null (expecting 'u')
2026/10/13 23:53:47 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: disagrees with its own path (host="wrong-host.go.tito.io" account="bac" app="tito")
2026/10/13 23:53:47 provision: skipping record apps/bac/tito/instances/acme$evil.go.tito.io.json: host "acme$evil.go.tito.io" is not a hostname
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: skipping record apps/bac/tito/instances/acme$evil.go.tito.io.json: host "acme$evil.go.tito.io" is not a hostname
2026/10/13 23:53:47 provision: not removing acme$evil.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 23:53:47 provision: skipping record apps/bac/tito/instances/bad-service.go.tito.io.json: invalid shape: kind service with no addr: the platform only proxies to a service, so the record must name the upstream it proxies to
2026/10/13 23:53:47 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:47 provision: skipping record apps/bac/tito/instances/bad-service.go.tito.io.json: invalid shape: kind service with no addr: the platform only proxies to a service, so the record must name the upstream it proxies to
2026/10/13 23:53:47 provision: not removing bad-service.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 23:53:48 provision: skipping record apps/bac/tito/instances/no-channel.go.tito.io.json: empty channel
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: skipping record apps/bac/tito/instances/no-channel.go.tito.io.json: empty channel
2026/10/13 23:53:48 provision: not removing no-channel.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: skipping record apps/bac/tito/instances/acme.go.tito.io.json: malformed: provision: invalid maintenance projection or protected fields
2026/10/13 23:53:48 provision: not removing acme.go.tito.io: its record exists but could not be read cleanly this pass
2026/10/13 23:53:48 provision: removed retired instance route ghost.go.tito.io (record gone)
2026/10/13 23:53:48 provision: duplicate host dup.go.tito.io: keeping apps/bac/tito/instances/dup.go.tito.io.json, skipping apps/bad/tito/instances/dup.go.tito.io.json
2026/10/13 23:53:48 provision: minted instance route dup.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 23:53:48 provision: removed retired instance route acme.go.tito.io (app in the trash)
2026/10/13 23:53:48 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 23:53:48 provision: removed retired static route site.go.tito.io (app in the trash)
2026/10/13 23:53:48 provision: trash marker for bac/tito unreadable: release: malformed trash marker for bac/tito: invalid character 'n' looking for beginning of object key string — not minting for this app
2026/10/13 23:53:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 23:53:48 provision: removed retired instance route acme.go.tito.io (app in the trash)
2026/10/13 23:53:48 provision: bac/tito is in the trash — not minting instance routes for it
2026/10/13 23:53:48 provision: trash marker for bac/tito unreadable: release: malformed trash marker for bac/tito: invalid character 'n' looking for beginning of object key string — not minting for this app
2026/10/13 23:53:48 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 23:53:48 provision: converged acme.go.tito.io channel stable -> edge
2026/10/13 23:53:48 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 23:53:48 provision: converged acme.go.tito.io channel canary/rehearsal -> edge
2026/10/13 23:53:48 provision: converged acme.go.tito.io environment "staging" -> ""
2026/10/13 23:53:48 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 23:53:48 provision: acme.go.tito.io exists but is not the route this record describes (backing "unit" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/13 23:53:48 provision: acme.go.tito.io exists but is not the route this record describes (tunnel "pi-fleet" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/13 23:53:48 provision: acme.go.tito.io exists but is not the route this record describes (caps "port:25" (box-scoped, not expressible in a record)) — leaving it alone
2026/10/13 23:53:48 provision: acme.go.tito.io exists but is not the route this record describes (platform (owned by the domains reconcile pass)) — leaving it alone
2026/10/13 23:53:48 provision: acme.go.tito.io exists but is not the route this record describes (canary (owned by the domains reconcile pass)) — leaving it alone
2026/10/13 23:53:48 provision: acme.go.tito.io exists but is not the route this record describes (db "/data/other.db"/"/data/acme.go.tito.io.db") — leaving it alone
2026/10/13 23:53:48 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 23:53:48 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/13 23:53:48 provision: dry-run: would converge owned.go.tito.io channel stable -> edge
2026/10/13 23:53:48 provision: dry-run: would converge owned.go.tito.io environment "" -> "staging"
2026/10/13 23:53:48 provision: dry-run: would converge owned.go.tito.io hibernate false -> true
2026/10/13 23:53:48 provision: dry-run: would converge owned.go.tito.io seed
2026/10/13 23:53:48 provision: dry-run: would mint instance route acme.go.tito.io (app "tito" acct "bac" channel "stable")
2026/10/13 23:53:48 provision: dry-run: would adopt existing route adopt.go.tito.io (app "tito" acct "bac") — this record would own its channel and environment
2026/10/13 23:53:48 provision: dry-run: would converge adopt.go.tito.io channel stable -> edge
2026/10/13 23:53:48 provision: dry-run: would converge adopt.go.tito.io environment "" -> "staging"
2026/10/13 23:53:48 provision: dry-run: would remove retired instance route gone.go.tito.io (record gone)
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: adopted existing route adopt.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 23:53:48 provision: converged adopt.go.tito.io channel stable -> edge
2026/10/13 23:53:48 provision: converged adopt.go.tito.io environment "" -> "staging"
2026/10/13 23:53:48 provision: converged owned.go.tito.io channel stable -> edge
2026/10/13 23:53:48 provision: converged owned.go.tito.io environment "" -> "staging"
2026/10/13 23:53:48 provision: converged owned.go.tito.io hibernate false -> true
2026/10/13 23:53:48 provision: removed retired instance route gone.go.tito.io (record gone)
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: adopted existing route story.carlosframework.com (app "story" acct "bad") — this record now owns its channel and environment
2026/10/13 23:53:48 provision: story.carlosframework.com exists but is not the route this record describes (kind "instance"/"static") — leaving it alone
2026/10/13 23:53:48 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted service route pub.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted service route pubs.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route exec.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted service route loop.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted service route named.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted service route priv.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route unit.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted static route pisite.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted static route site.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: removed retired static route site.go.tito.io (record gone)
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route acme-uk.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: converged acme.go.tito.io replicas "ie,uk" -> "uk"
2026/10/13 23:53:48 provision: converged acme.go.tito.io replicas "uk" -> ""
2026/10/13 23:53:48 provision: minted instance route acme-uk.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: converged acme-uk.go.tito.io replica mark true -> false
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: acme.go.tito.io: row is on the nested layout (db "/data/instances/acme.go.tito.io/acme.go.tito.io.db" socket "/run/carlos/instances/acme.go.tito.io/app.sock"); the record's shared paths are not drift
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: minted instance route router.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:48 provision: router.go.tito.io: not minting this pass: the box's layout intent could not be read: store down
2026/10/13 23:53:48 provision: acme.go.tito.io: not minting this pass: the box's layout intent could not be read: store down
2026/10/13 23:53:48 provision: adopted existing route acme.go.tito.io (app "tito" acct "bac") — this record now owns its channel and environment
2026/10/13 23:53:48 provision: converged acme.go.tito.io channel stable -> edge
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
2026/10/13 23:53:48 provision: not removing acme.go.tito.io (record gone): participant of a running operation
2026/10/13 23:53:48 provision: removed retired instance route acme.go.tito.io (record gone)
2026/10/13 23:53:48 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
2026/10/13 23:53:48 provision: refusing to mint held route acme.go.tito.io (app "tito" acct "bac"): this box declares CARLOS_INBOUND=none (no-inbound) and cannot serve routed instance traffic
2026/10/13 23:53:48 provision: held acme.go.tito.io landing by mv_acme (was ready)
2026/10/13 23:53:48 provision: acme.go.tito.io: held restoring by rb_other, not by mv_acme; leaving it
2026/10/13 23:53:48 provision: acme.go.tito.io: the row is not this pass's (app "other" acct ""); not holding it for mv_acme
2026/10/13 23:53:48 provision: acme.go.tito.io exists but is not the route this record describes (app "other"/"tito") — leaving it alone
2026/10/13 23:53:48 provision: dry-run: would mint instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac" channel "stable")
2026/10/13 23:53:48 provision: acme.go.tito.io: not held landing by mv_acme: the row's hold changed under this pass; the next pass judges it afresh
2026/10/13 23:53:48 provision: held acme.go.tito.io landing by mv_acme (was ready)
2026/10/13 23:53:48 provision: removed retired sidecar route gone.eleven.bac.sidecar.internal (record gone)
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: converged sidecar bot.eleven.bac.sidecar.internal args ["sidecar","run"] -> ["worker","run"]
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: converged sidecar bot.eleven.bac.sidecar.internal ceilings 300/3600 -> 60/120
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: converged sidecar bot.eleven.bac.sidecar.internal channel stable -> canary/eleven
2026/10/13 23:53:49 provision: converged sidecar bot.eleven.bac.sidecar.internal environment "other" -> "relay"
2026/10/13 23:53:49 provision: converged sidecar bot.eleven.bac.sidecar.internal hibernate true -> false
2026/10/13 23:53:49 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: read failed: failStore: reads disabled under apps/bac/eleven/sidecars/
2026/10/13 23:53:49 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: malformed: invalid character 'n' looking for beginning of object key string
2026/10/13 23:53:49 provision: not removing sidecar bot.eleven.bac.sidecar.internal: its record exists but could not be read cleanly this pass
2026/10/13 23:53:49 provision: skipping sidecar record apps/bac/eleven/sidecars/bot.json: empty channel
2026/10/13 23:53:49 provision: skipping sidecar record apps/bac/eleven/sidecars/held.json: empty channel
2026/10/13 23:53:49 provision: not removing sidecar held.eleven.bac.sidecar.internal: its record exists but could not be read cleanly this pass
2026/10/13 23:53:49 provision: removed retired sidecar route old.eleven.bac.sidecar.internal (record gone)
2026/10/13 23:53:49 provision: bac/eleven has sidecar records but no sidecars manifest — not minting
2026/10/13 23:53:49 provision: bac/eleven is in the trash — not minting sidecar routes for it
2026/10/13 23:53:49 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 23:53:49 provision: sidecar hub.relay.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: minted sidecar route hub.relay.bac.sidecar.internal (app "relay" acct "bac" args ["relay"])
2026/10/13 23:53:49 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 23:53:49 provision: removed retired sidecar route bot.eleven.bac.sidecar.internal (record gone)
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: sidecar bot.eleven.bac.sidecar.internal is not placeable here: it asks for "no-inbound" placement and this box accepts inbound work (CARLOS_INBOUND="")
2026/10/13 23:53:49 provision: skipping sidecar record apps/bac/relay/sidecars/hub.json: read failed: failStore: reads disabled under apps/bac/relay/sidecars/
2026/10/13 23:53:49 provision: REFUSING to mint sidecar bot.eleven.bac.sidecar.internal: bot.eleven.bac.sidecar.internal-sidecar.env is staged on this box — the legacy file co-spawn and a native sidecar row are two mechanisms for one host; remove the file to migrate this host onto sidecar records
2026/10/13 23:53:49 provision: minted sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" args ["sidecar","run"])
2026/10/13 23:53:49 provision: dry-run: would mint sidecar route bot.eleven.bac.sidecar.internal (app "eleven" acct "bac" channel "stable" args ["sidecar","run"])
2026/10/13 23:53:49 provision: dry-run: would remove retired sidecar route gone.eleven.bac.sidecar.internal (record gone)
2026/10/13 23:53:49 provision: bot.eleven.bac.sidecar.internal exists but is not a sidecar row this pass owns (kind "sidecar" provisioned false app "eleven" acct "bac") — leaving it alone
2026/10/13 23:53:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:49 provision: converged acme.go.tito.io update settings "{\"updates\":\"manual\",\"channel\":\"stable\"}" -> ""
2026/10/13 23:53:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:49 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/13 23:53:49 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/13 23:53:49 provision: acme.go.tito.io: not minted this pass: the app's update policy could not be read
2026/10/13 23:53:49 provision: minted instance route acme.go.tito.io (app "tito" acct "bac")
2026/10/13 23:53:49 provision: manifest bac/tito: read failed: provision: manifest bac/tito: unexpected end of JSON input
2026/10/13 23:53:49 provision: acme.go.tito.io: not minted this pass: the app's update policy could not be read
2026/10/13 23:53:49 provision: minted instance route acme.go.tito.io held landing by mv_acme (app "tito" acct "bac")
FAIL
FAIL github.com/carlosframework/platform/internal/provision 13.150s
ok github.com/carlosframework/platform/internal/publicrelease 0.006s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 4.020s
ok github.com/carlosframework/platform/internal/relay 7.223s
ok github.com/carlosframework/platform/internal/release 19.586s
ok github.com/carlosframework/platform/internal/rollout 3.742s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.006s
ok github.com/carlosframework/platform/internal/rootparticipation 0.008s
ok github.com/carlosframework/platform/internal/rootstate 0.025s
ok github.com/carlosframework/platform/internal/s3compat 0.004s
ok github.com/carlosframework/platform/internal/safefile 0.003s
ok github.com/carlosframework/platform/internal/schedule 2.324s
ok github.com/carlosframework/platform/internal/sim/hostlife 9.362s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.363s
ok github.com/carlosframework/platform/internal/status 0.041s
ok github.com/carlosframework/platform/internal/steering 0.005s
ok github.com/carlosframework/platform/internal/stopevidence 0.228s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.002s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.006s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.554s
FAIL
make: *** [Makefile:18: test] Error 1
amadan: step gate failed: exit status 2
amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/apimode.go b/apimode.go| index 00177a653..dd5692eb1 100644 |
| --- a/apimode.go |
| +++ b/apimode.go |
| @@ -3224,6 +3224,13 @@ func (s *apiSession) instancesEnable(ctx context.Context, a apiAcct, app string, |
| // apiInstanceRecord mirrors provision.Record's wire shape — redeclared here |
| // rather than imported because this client renders records, never writes |
| // them, and the fields it renders are the contract worth pinning. |
| +// apiReplica is the list's replica object. State is "level", "behind", |
| +// "late", "parked" or "unknown"; BehindSecs is set for "behind" and "late". |
| +type apiReplica struct { |
| + State string `json:"state"` |
| + BehindSecs *int64 `json:"behind_secs,omitempty"` |
| +} |
| + |
| type apiInstanceRecord struct { |
| Host string `json:"host"` |
| Slug string `json:"slug"` |
| @@ -3289,6 +3296,10 @@ type apiInstanceRecord struct { |
| // measured it (instanceListRecord.DBBytes). nil when unknown, or from a |
| // console that predates the field — the DB column prints a dash. |
| DBBytes *int64 `json:"db_bytes,omitempty"` |
| + // Replica is where the instance's off-box replica stands as its box last |
| + // reported it (instanceListRecord.Replica). nil when unknown, or from a |
| + // console that predates the field: the REPLICA column prints a dash. |
| + Replica *apiReplica `json:"replica,omitempty"` |
| // PrivateStoreBucket is the bucket the CREATE door provisioned for this |
| // instance under the app's private-storage policy (issue #412). Set on |
| // exactly one response — the create that minted it — and empty |
diff --git a/docs/site/cli.md b/docs/site/cli.md| index e2bc11208..30f7ef9c0 100644 |
| --- a/docs/site/cli.md |
| +++ b/docs/site/cli.md |
| @@ -734,6 +734,13 @@ every few minutes, and asleep instances are measured too. A dash means no box |
| has measured it: the instance is not running anywhere yet, it has not created |
| its database, or it keeps its data somewhere CARLOS does not manage. |
| +The `REPLICA` column says whether each instance's off-box copy of that |
| +database is keeping up. `ok` means it is, or the instance is asleep and |
| +finished its last sync. A time such as `2h10m behind` means the copy has |
| +trailed the database for that long, past its sync interval and a grace of |
| +fifteen minutes. A dash means the instance's box has not said, which is not |
| +the same as `ok`. |
| + |
| `carlos instances list` shows two builds for each instance: the one installed |
| on its box, and the one serving its requests. They differ while a new build |
| rolls out, while an asleep instance waits for its next request, and after a |
diff --git a/instances_cli.go b/instances_cli.go| index a147fc534..7b5380651 100644 |
| --- a/instances_cli.go |
| +++ b/instances_cli.go |
| @@ -32,6 +32,7 @@ import ( |
| "net/url" |
| "os" |
| "strings" |
| + "time" |
| "github.com/carlosframework/platform/internal/appconfig" |
| "github.com/carlosframework/platform/internal/provision" |
| @@ -430,8 +431,28 @@ func upstreamsCell(ups []apiUpstreamState) string { |
| return fmt.Sprintf("%d/%d", up, len(ups)) |
| } |
| +// replicaCell is the REPLICA column. Only a replica that is late is worth |
| +// words: a level, parked or behind-but-on-time one is "ok", and anything the |
| +// box could not say (no object, an unknown state, an older console) is a dash, |
| +// never "ok". DRAFT copy, pending the owner's copy review. |
| +func replicaCell(r *apiReplica) string { |
| + if r == nil { |
| + return "-" |
| + } |
| + switch r.State { |
| + case "level", "parked", "behind": |
| + return "ok" |
| + case "late": |
| + if r.BehindSecs != nil { |
| + return fmtDur(time.Duration(*r.BehindSecs)*time.Second) + " behind" |
| + } |
| + return "late" |
| + } |
| + return "-" |
| +} |
| + |
| const ( |
| - instancesListHeader = "HOST\tREGION\tCHANNEL\tPOOL\tINSTALLED\tSERVING\tHEALTH\tUPSTREAMS\tDB\tCREATED\tBY\tREASON" |
| + instancesListHeader = "HOST\tREGION\tCHANNEL\tPOOL\tINSTALLED\tSERVING\tHEALTH\tUPSTREAMS\tDB\tREPLICA\tCREATED\tBY\tREASON" |
| instancesListRetryHint = "a build that failed is tried again on every box with carlos promote --app %s --retry <version> <channel>, or on one instance with carlos instances retry --app %s <host>" |
| ) |
| @@ -558,8 +579,8 @@ func runInstancesList(args []string) { |
| if rowHealth == "" { |
| rowHealth = "—" // an old console: the field predates it, not "unhealthy" |
| } |
| - fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", r.Host, r.Region, r.Channel, r.Pool, |
| - installedCell(r), servingCell(r), rowHealth, upstreamsCell(r.Upstreams), humanBytes(r.DBBytes), |
| + fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", r.Host, r.Region, r.Channel, r.Pool, |
| + installedCell(r), servingCell(r), rowHealth, upstreamsCell(r.Upstreams), humanBytes(r.DBBytes), replicaCell(r.Replica), |
| r.CreatedAt, r.CreatedBy, reasonCell(r)) |
| } |
| w.Flush() |
diff --git a/instances_cli_test.go b/instances_cli_test.go| index 395408395..772adcc2d 100644 |
| --- a/instances_cli_test.go |
| +++ b/instances_cli_test.go |
| @@ -9,8 +9,10 @@ package main |
| import ( |
| "context" |
| + "encoding/json" |
| "net/http" |
| "net/http/httptest" |
| + "strings" |
| "testing" |
| "github.com/carlosframework/platform/internal/provision" |
| @@ -97,6 +99,67 @@ func TestUpstreamsCell(t *testing.T) { |
| } |
| } |
| +// TestReplicaCell pins the REPLICA column: "ok" for a replica that is level, |
| +// parked or behind but on time, a duration only for a late one, and a dash |
| +// for everything the box did not say (never "ok"). |
| +func TestReplicaCell(t *testing.T) { |
| + secs := func(n int64) *int64 { return &n } |
| + cases := []struct { |
| + name string |
| + r *apiReplica |
| + want string |
| + }{ |
| + {"absent (an older console, or no box report)", nil, "-"}, |
| + {"unknown", &apiReplica{State: "unknown"}, "-"}, |
| + {"a state from a newer console", &apiReplica{State: "frozen"}, "-"}, |
| + {"level", &apiReplica{State: "level"}, "ok"}, |
| + {"parked", &apiReplica{State: "parked"}, "ok"}, |
| + {"behind", &apiReplica{State: "behind", BehindSecs: secs(120)}, "ok"}, |
| + {"late", &apiReplica{State: "late", BehindSecs: secs(2*3600 + 10*60)}, "2h10m behind"}, |
| + {"late by minutes", &apiReplica{State: "late", BehindSecs: secs(17*60 + 5)}, "17m05s behind"}, |
| + {"late without a figure", &apiReplica{State: "late"}, "late"}, |
| + } |
| + for _, c := range cases { |
| + if got := replicaCell(c.r); got != c.want { |
| + t.Errorf("replicaCell(%s) = %q, want %q", c.name, got, c.want) |
| + } |
| + } |
| +} |
| + |
| +// TestInstanceListDecodesReplica: the list's "replica" object decodes into |
| +// apiInstanceRecord.Replica, a record without the key decodes to nil (the |
| +// dash above), and the header puts REPLICA right after DB. |
| +func TestInstanceListDecodesReplica(t *testing.T) { |
| + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + json.NewEncoder(w).Encode(map[string]any{ |
| + "instances": []map[string]any{ |
| + {"host": "late.example.com", "replica": map[string]any{"state": "late", "behind_secs": 7800, "interval_secs": 60}}, |
| + {"host": "old.example.com"}, |
| + }, |
| + "truncated": false, "total": 2, |
| + }) |
| + })) |
| + defer ts.Close() |
| + |
| + s := newTestSession(ts.URL) |
| + recs, _, _, err := s.instanceList(context.Background(), apiAcct{Seg: "bah"}, "app", "", "") |
| + if err != nil { |
| + t.Fatalf("instanceList: %v", err) |
| + } |
| + if len(recs) != 2 { |
| + t.Fatalf("recs = %+v, want 2", recs) |
| + } |
| + if got := replicaCell(recs[0].Replica); got != "2h10m behind" { |
| + t.Errorf("late record cell = %q, want %q", got, "2h10m behind") |
| + } |
| + if recs[1].Replica != nil || replicaCell(recs[1].Replica) != "-" { |
| + t.Errorf("record without a replica key = %+v, want nil and a dash", recs[1].Replica) |
| + } |
| + if !strings.Contains(instancesListHeader, "\tDB\tREPLICA\t") { |
| + t.Errorf("header = %q, want REPLICA immediately after DB", instancesListHeader) |
| + } |
| +} |
| + |
| // TestParseAddrList pins the given-vs-absent split -addr/-addrs share |
| // (Task 8): a flag never named on the command line must return nil (the |
| // wire's "leave/default"), never mistaken for the empty list an explicitly |
diff --git a/instances_e2e_test.go b/instances_e2e_test.go| index 8a7720e5d..0f61b40ac 100644 |
| --- a/instances_e2e_test.go |
| +++ b/instances_e2e_test.go |
| @@ -203,6 +203,9 @@ func TestE2EInstancesThroughConsole(t *testing.T) { |
| if !strings.Contains(listOut, "UPSTREAMS") { |
| t.Fatalf("list stdout = %q, want an UPSTREAMS column header", listOut) |
| } |
| + if !strings.Contains(listOut, "REPLICA") { |
| + t.Fatalf("list stdout = %q, want a REPLICA column header", listOut) |
| + } |
| if !strings.Contains(listOut, haHost) { |
| t.Fatalf("list stdout = %q, want the multi-upstream host listed", listOut) |
| } |
diff --git a/internal/activator/replicalevel.go b/internal/activator/replicalevel.go| index 73bcbf202..074935aa5 100644 |
| --- a/internal/activator/replicalevel.go |
| +++ b/internal/activator/replicalevel.go |
| @@ -6,9 +6,12 @@ import "time" |
| // replica without asking the bucket. |
| type ReplicaLevel struct { |
| // LevelAt is the last time the host's running replicator had nothing left |
| - // to ship (lease.Replicator.LevelAt). Zero when it has not managed that |
| + // to ship (lease.Replicator.Level). Zero when it has not managed that |
| // since the wake, and for a parked host. |
| LevelAt time.Time |
| + // Empty says the database holds nothing a replicator could ship yet, so |
| + // a zero LevelAt is not a replica falling behind. |
| + Empty bool |
| // Interval is the host's replication-class sync interval; 0 is stream. |
| // A replica is not late until it trails by more than this. |
| Interval time.Duration |
| @@ -17,15 +20,27 @@ type ReplicaLevel struct { |
| Parked bool |
| } |
| +// leveller is the part of a running replicator ReplicaLevels reads. Only the |
| +// activator's own embedded replicator has it: a unit-backed host's Rep does |
| +// not, because the box-wide replicator owns that database. |
| +type leveller interface { |
| + Level() (at time.Time, empty bool) |
| +} |
| + |
| // ReplicaLevels reports every host this activator replicates itself: awake |
| // hosts whose replicator can state its level, and sleeping hosts whose last |
| -// stop is remembered as clean. A unit-backed host is absent (the box-wide |
| -// replicator owns its database), and so is a sleeping host whose stop this |
| -// process did not see or could not prove: absent is "cannot say". |
| +// stop is remembered as clean. A unit-backed host is absent, and so is a |
| +// sleeping host whose stop this process did not see or could not prove: |
| +// absent is "cannot say". |
| func (a *Service) ReplicaLevels() map[string]ReplicaLevel { |
| - a.mu.Lock() |
| - defer a.mu.Unlock() |
| + type awake struct { |
| + rep leveller |
| + interval time.Duration |
| + } |
| out := map[string]ReplicaLevel{} |
| + running := map[string]awake{} |
| + |
| + a.mu.Lock() |
| for host, o := range a.stopHistory { |
| if o != nil && o.replicaClosed && !o.uncertain { |
| out[host] = ReplicaLevel{Parked: true} |
| @@ -38,11 +53,18 @@ func (a *Service) ReplicaLevels() map[string]ReplicaLevel { |
| if e.state != actAwake || e.rep == nil { |
| continue |
| } |
| - rep, ok := e.rep.(interface{ LevelAt() time.Time }) |
| - if !ok { |
| - continue |
| + if rep, ok := e.rep.(leveller); ok { |
| + running[host] = awake{rep, replicationSyncInterval(e.route.Replication)} |
| } |
| - out[host] = ReplicaLevel{LevelAt: rep.LevelAt(), Interval: replicationSyncInterval(e.route.Replication)} |
| + } |
| + a.mu.Unlock() |
| + |
| + // Asked outside the lock: a replicator's position can cost a directory |
| + // read, and nothing here needs the entry to still be awake by the time |
| + // it answers. |
| + for host, r := range running { |
| + at, empty := r.rep.Level() |
| + out[host] = ReplicaLevel{LevelAt: at, Empty: empty, Interval: r.interval} |
| } |
| return out |
| } |
diff --git a/internal/activator/replicalevel_test.go b/internal/activator/replicalevel_test.go| index 738f39c0d..7841e0b1e 100644 |
| --- a/internal/activator/replicalevel_test.go |
| +++ b/internal/activator/replicalevel_test.go |
| @@ -8,11 +8,14 @@ import ( |
| "github.com/carlosframework/platform/internal/registry" |
| ) |
| -type levelRep struct{ at time.Time } |
| +type levelRep struct { |
| + at time.Time |
| + empty bool |
| +} |
| func (levelRep) Err() error { return nil } |
| func (levelRep) Close(context.Context) error { return nil } |
| -func (r levelRep) LevelAt() time.Time { return r.at } |
| +func (r levelRep) Level() (time.Time, bool) { return r.at, r.empty } |
| type silentRep struct{} |
| @@ -24,11 +27,12 @@ func TestReplicaLevels(t *testing.T) { |
| clean := &stopObservation{mainStop: true, mainExited: true, sidecarStop: true, sidecarExited: true, replicaClosed: true} |
| a := &Service{ |
| entries: map[string]*actEntry{ |
| - "awake.example.com": {host: "awake.example.com", state: actAwake, rep: levelRep{at}, route: registry.Route{Replication: replicationBatched}}, |
| + "awake.example.com": {host: "awake.example.com", state: actAwake, rep: levelRep{at: at}, route: registry.Route{Replication: replicationBatched}}, |
| "fresh.example.com": {host: "fresh.example.com", state: actAwake, rep: levelRep{}}, |
| + "empty.example.com": {host: "empty.example.com", state: actAwake, rep: levelRep{empty: true}}, |
| "unit.example.com": {host: "unit.example.com", state: actAwake, rep: silentRep{}}, |
| "waking.example.com": {host: "waking.example.com", state: actWaking}, |
| - "stopping.example.com": {host: "stopping.example.com", state: actStopping, rep: levelRep{at}}, |
| + "stopping.example.com": {host: "stopping.example.com", state: actStopping, rep: levelRep{at: at}}, |
| "rewoken.example.com": {host: "rewoken.example.com", state: actWaking}, |
| }, |
| stopHistory: map[string]*stopObservation{ |
| @@ -42,6 +46,7 @@ func TestReplicaLevels(t *testing.T) { |
| want := map[string]ReplicaLevel{ |
| "awake.example.com": {LevelAt: at, Interval: 5 * time.Minute}, |
| "fresh.example.com": {}, |
| + "empty.example.com": {Empty: true}, |
| "parked.example.com": {Parked: true}, |
| } |
| if len(got) != len(want) { |
diff --git a/internal/console/alertwatch.go b/internal/console/alertwatch.go| index afcd77f21..5d1c0118f 100644 |
| --- a/internal/console/alertwatch.go |
| +++ b/internal/console/alertwatch.go |
| @@ -308,6 +308,10 @@ func alertValueText(rule string, value float64) string { |
| case "geoip": |
| // A reason code, not a measurement (geoip spec §10.4). |
| return geoipReasonText(int(value)) |
| + case "restarts": |
| + return fmt.Sprintf("%d restarts in 10 minutes", int(value)) |
| + case "replica": |
| + return humanAge(time.Duration(value)*time.Second) + " overdue" |
| default: |
| return strconv.FormatFloat(value, 'g', -1, 64) |
| } |
| @@ -322,18 +326,34 @@ func alertMessage(box, rule string, rs metrics.RuleState, updated time.Time) (su |
| at = rs.FiredAt |
| } |
| stamp := at.UTC().Format(time.RFC3339) |
| + detail := alertDetailLine(rule, rs.Detail) |
| if rs.Firing { |
| subject = fmt.Sprintf("[carlos] %s: %s alert", box, rule) |
| - body = fmt.Sprintf("The %s alert is firing on %s.\n\nValue at the transition: %s\nTime (UTC): %s\n\n"+ |
| - "You are getting this because you hold the CARLOS fleet-operator role.\n", rule, box, alertValueText(rule, rs.Value), stamp) |
| + body = fmt.Sprintf("The %s alert is firing on %s.\n\nValue at the transition: %s\n%sTime (UTC): %s\n\n"+ |
| + "You are getting this because you hold the CARLOS fleet-operator role.\n", rule, box, alertValueText(rule, rs.Value), detail, stamp) |
| return subject, body |
| } |
| subject = fmt.Sprintf("[carlos] %s: %s cleared", box, rule) |
| - body = fmt.Sprintf("The %s alert has cleared on %s.\n\nValue at the transition: %s\nTime (UTC): %s\n\n"+ |
| - "You are getting this because you hold the CARLOS fleet-operator role.\n", rule, box, alertValueText(rule, rs.Value), stamp) |
| + body = fmt.Sprintf("The %s alert has cleared on %s.\n\nValue at the transition: %s\n%sTime (UTC): %s\n\n"+ |
| + "You are getting this because you hold the CARLOS fleet-operator role.\n", rule, box, alertValueText(rule, rs.Value), detail, stamp) |
| return subject, body |
| } |
| +// alertDetailLine is the mail's line naming what a rule's value was about, |
| +// with its newline, or "" when the rule carried no detail (a clear zeroes it, |
| +// and most rules never have one). DRAFT copy, pending the owner's copy review. |
| +func alertDetailLine(rule, detail string) string { |
| + switch { |
| + case detail == "": |
| + return "" |
| + case rule == "restarts": |
| + return "Unit: " + detail + "\n" |
| + case rule == "replica": |
| + return "Database: " + detail + "\n" |
| + } |
| + return "Detail: " + detail + "\n" |
| +} |
| + |
| // geoipReasonText names a geoip rule reason code (geoip spec §10.4). |
| func geoipReasonText(code int) string { |
| switch code { |
diff --git a/internal/console/apiinstances.go b/internal/console/apiinstances.go| index 3c4b3eb0c..c2c453c02 100644 |
| --- a/internal/console/apiinstances.go |
| +++ b/internal/console/apiinstances.go |
| @@ -195,6 +195,12 @@ type instanceListRecord struct { |
| // box has sized it: not minted yet, no database file yet, or a route |
| // whose database lives somewhere the platform does not know about. |
| DBBytes *int64 `json:"db_bytes,omitempty"` |
| + // Replica is where this instance's off-box replica stands as its box |
| + // last reported it (status.BoxMetrics.RouteReplicas, discussion #284), |
| + // under the same staleness rule as DBBytes. Absent when no fresh box |
| + // speaks for the host: that is "unknown", and a client must never read it |
| + // as "fine". |
| + Replica *replicaView `json:"replica,omitempty"` |
| // Rollout is this host's rollout as its box last reported it |
| // (zero-downtime spec §9.1), joined under StagedVersion's staleness rule. |
| // Absent when no fresh box reports one: a box from before plan 5, one |
| @@ -349,6 +355,7 @@ type boxJoin struct { |
| serving map[string]string |
| upstreams map[string][]status.UpstreamState |
| dbBytes map[string]int64 |
| + replicas map[string]replicaJoin |
| rollouts map[string]rolloutView |
| updates map[string]updatesJoinView |
| } |
| @@ -463,6 +470,75 @@ func (j boxJoin) dbSize(host string) *int64 { |
| return &n |
| } |
| +// replicaLateAfter is how far past its sync interval a replica may trail |
| +// before it is called late. It matches the box's replica alert rule |
| +// (metrics.DefaultRules().ReplicaSecs = 900) and internal/status's |
| +// replHealthyWithin, so the list, the alert and the status page agree on when |
| +// a replica has fallen behind. |
| +const replicaLateAfter = 15 * time.Minute |
| + |
| +// replicaJoin is one host's replica report with the time of the snapshot it |
| +// came from. |
| +type replicaJoin struct { |
| + r status.RouteReplica |
| + at time.Time |
| +} |
| + |
| +// replicaView is a replica as a member sees it. State is one of "level", |
| +// "behind", "late", "parked" or "unknown"; BehindSecs is set only for "behind" |
| +// and "late". |
| +type replicaView struct { |
| + State string `json:"state"` |
| + LevelAt *time.Time `json:"level_at,omitempty"` |
| + BehindSince *time.Time `json:"behind_since,omitempty"` |
| + BehindSecs *int64 `json:"behind_secs,omitempty"` |
| + IntervalSecs int64 `json:"interval_secs,omitempty"` |
| +} |
| + |
| +// replicaState is the one place a box's replica report becomes a word. now is |
| +// the caller's clock, not read here, so a table of cases can pin it. A |
| +// parked replica is level by construction (its final sync succeeded). A |
| +// replica with BehindSince trails by the time since then. One with only |
| +// LevelAt trails by at most the time since then, so that is the bound the |
| +// word is judged on. A report with neither says nothing yet. A replica is on |
| +// time while it trails by no more than its own interval; replicaLateAfter is |
| +// the slack past that. |
| +func replicaState(r status.RouteReplica, now time.Time) (state string, behind time.Duration) { |
| + interval := time.Duration(r.IntervalSecs) * time.Second |
| + switch { |
| + case r.Parked: |
| + return "parked", 0 |
| + case r.BehindSince != nil: |
| + behind = now.Sub(*r.BehindSince) |
| + if behind-interval > replicaLateAfter { |
| + return "late", behind |
| + } |
| + return "behind", behind |
| + case r.LevelAt != nil: |
| + behind = now.Sub(*r.LevelAt) |
| + if behind-interval > replicaLateAfter { |
| + return "late", behind |
| + } |
| + return "level", behind |
| + } |
| + return "unknown", 0 |
| +} |
| + |
| +// replica is the join's answer for host, nil when no fresh box reported one. |
| +func (j boxJoin) replica(host string, now time.Time) *replicaView { |
| + rj, ok := j.replicas[host] |
| + if !ok { |
| + return nil |
| + } |
| + state, behind := replicaState(rj.r, now) |
| + v := &replicaView{State: state, LevelAt: rj.r.LevelAt, BehindSince: rj.r.BehindSince, IntervalSecs: rj.r.IntervalSecs} |
| + if state == "behind" || state == "late" { |
| + secs := int64(behind / time.Second) |
| + v.BehindSecs = &secs |
| + } |
| + return v |
| +} |
| + |
| // loadBoxJoin builds boxJoin from every NON-STALE box's own |
| // private Metrics block (status/boxes/*.json), through status.LoadBoxes — |
| // the same loader the operator fleet page's HostRows reads through |
| @@ -596,7 +672,7 @@ func recordFleets(recs []provision.Record) map[string]bool { |
| // has its boxes reads them once. |
| func joinBoxes(boxes []status.Box, now time.Time) boxJoin { |
| j := boxJoin{versions: map[string]string{}, serving: map[string]string{}, upstreams: map[string][]status.UpstreamState{}, |
| - dbBytes: map[string]int64{}, rollouts: map[string]rolloutView{}} |
| + dbBytes: map[string]int64{}, replicas: map[string]replicaJoin{}, rollouts: map[string]rolloutView{}} |
| for _, b := range boxes { |
| if b.Metrics == nil || status.BoxStale(b, now, status.SnapshotPeriod) { |
| continue |
| @@ -613,6 +689,9 @@ func joinBoxes(boxes []status.Box, now time.Time) boxJoin { |
| for host, n := range b.Metrics.RouteDBBytes { |
| j.dbBytes[host] = n |
| } |
| + for host, r := range b.Metrics.RouteReplicas { |
| + j.replicas[host] = replicaJoin{r: r, at: b.GeneratedAt} |
| + } |
| j.foldRollout(b, true) |
| j.foldUpdates(b, true) |
| } |
| @@ -766,7 +845,8 @@ func (s *Server) handleAPIInstanceList(w http.ResponseWriter, r *http.Request) { |
| acct.Sqid, app, len(skipped), skipped) |
| relevant = nil |
| } |
| - join, joinComplete := s.loadRolloutJoin(ctx, acct.Sqid, time.Now(), relevant) |
| + now := time.Now() |
| + join, joinComplete := s.loadRolloutJoin(ctx, acct.Sqid, now, relevant) |
| served := loadEdgesJoin(ctx, s.cfg.Releases) |
| // The app's update policy, read once: an absent manifest is no policy, |
| // and a read error is logged and treated the same. The list never fails |
| @@ -807,6 +887,7 @@ func (s *Server) handleAPIInstanceList(w http.ResponseWriter, r *http.Request) { |
| matched = append(matched, instanceListRecord{Record: rec, |
| StagedVersion: join.versions[rec.Host], Health: h, Upstreams: join.upstreams[rec.Host], |
| ServedBy: served[rec.Host], Gated: rec.Gate != "", DBBytes: join.dbSize(rec.Host), |
| + Replica: join.replica(rec.Host, now), |
| Rollout: join.rollout(rec.Host, acct.Sqid, app), |
| UpdateReport: join.update(rec.Host, acct.Sqid, app), |
| SettingsDefault: provision.EffectiveUpdates(rec, policy).Default()}) |
diff --git a/internal/console/metricsview.go b/internal/console/metricsview.go| index 7f1c70307..2a1028146 100644 |
| --- a/internal/console/metricsview.go |
| +++ b/internal/console/metricsview.go |
| @@ -60,6 +60,9 @@ type AlertRow struct { |
| Box, Rule string |
| Value float64 |
| FiredAt time.Time |
| + // Detail names what the reading was about (RuleState.Detail): the unit |
| + // for restarts, the database or host for replica. Empty for other rules. |
| + Detail string |
| } |
| // metricsCacheTTL is how soon after a successful sweep LoadMetrics considers |
| @@ -840,7 +843,7 @@ func (s *Server) sweepAlerts(ctx context.Context) ([]AlertRow, error) { |
| if !rs.Firing { |
| continue |
| } |
| - rows = append(rows, AlertRow{Box: box, Rule: name, Value: rs.Value, FiredAt: rs.FiredAt}) |
| + rows = append(rows, AlertRow{Box: box, Rule: name, Value: rs.Value, FiredAt: rs.FiredAt, Detail: rs.Detail}) |
| } |
| } |
| return rows, nil |
diff --git a/internal/console/overview.go b/internal/console/overview.go| index 03db7eba9..9229a8ed4 100644 |
| --- a/internal/console/overview.go |
| +++ b/internal/console/overview.go |
| @@ -5,7 +5,9 @@ |
| package console |
| import ( |
| + "fmt" |
| "strings" |
| + "time" |
| "github.com/carlosframework/platform/internal/registry" |
| ) |
| @@ -232,6 +234,38 @@ type InstanceRow struct { |
| // plain word — the column's whole job. |
| HealthWord string |
| Restartable bool |
| + |
| + // ReplicaText is the Replica column's cell, "" when the route's box has |
| + // not said where its off-box replica stands. ReplicaLate marks the one |
| + // state worth drawing the eye to. Both come from attachReplicas. |
| + ReplicaText string |
| + ReplicaLate bool |
| +} |
| + |
| +// attachReplicas fills each row's Replica cell from the box join. A host no |
| +// fresh box speaks for keeps an empty cell: unknown is blank, never "ok". |
| +func attachReplicas(rows []InstanceRow, join boxJoin, now time.Time) { |
| + for i := range rows { |
| + rows[i].ReplicaText, rows[i].ReplicaLate = replicaCellText(join.replica(rows[i].Host, now)) |
| + } |
| +} |
| + |
| +// replicaCellText is the Replica column's wording, the only place it lives. |
| +// DRAFT copy, pending the owner's copy review. |
| +func replicaCellText(v *replicaView) (text string, late bool) { |
| + if v == nil { |
| + return "", false |
| + } |
| + switch v.State { |
| + case "level", "parked", "behind": |
| + return "ok", false |
| + case "late": |
| + if v.BehindSecs != nil { |
| + return fmt.Sprintf("%s behind", humanAge(time.Duration(*v.BehindSecs)*time.Second)), true |
| + } |
| + return "late", true |
| + } |
| + return "", false |
| } |
| // assembleInstanceRows joins the app's routes with the sweep's readings, in |
diff --git a/internal/console/replicafresh_test.go b/internal/console/replicafresh_test.go| new file mode 100644 |
| index 000000000..134c21833 |
| --- /dev/null |
| +++ b/internal/console/replicafresh_test.go |
| @@ -0,0 +1,261 @@ |
| +package console |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "net/http" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/metrics" |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/status" |
| +) |
| + |
| +// TestReplicaState pins every branch of the one function that turns a box's |
| +// replica report into a word, including the edges: exactly 15 minutes past |
| +// the interval is not yet late, and the interval itself is respected. |
| +func TestReplicaState(t *testing.T) { |
| + now := time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC) |
| + ago := func(d time.Duration) *time.Time { v := now.Add(-d); return &v } |
| + cases := []struct { |
| + name string |
| + r status.RouteReplica |
| + want string |
| + wantBehind time.Duration |
| + }{ |
| + {"parked", status.RouteReplica{Parked: true}, "parked", 0}, |
| + {"parked wins over a stale level", status.RouteReplica{Parked: true, LevelAt: ago(48 * time.Hour)}, "parked", 0}, |
| + {"parked wins over behind", status.RouteReplica{Parked: true, BehindSince: ago(48 * time.Hour)}, "parked", 0}, |
| + {"behind, recent", status.RouteReplica{BehindSince: ago(2 * time.Minute)}, "behind", 2 * time.Minute}, |
| + {"behind, exactly 15m past a zero interval", status.RouteReplica{BehindSince: ago(15 * time.Minute)}, "behind", 15 * time.Minute}, |
| + {"behind, one second over 15m", status.RouteReplica{BehindSince: ago(15*time.Minute + time.Second)}, "late", 15*time.Minute + time.Second}, |
| + {"behind, long interval 23h", status.RouteReplica{BehindSince: ago(23 * time.Hour), IntervalSecs: 86400}, "behind", 23 * time.Hour}, |
| + {"behind, long interval exactly 24h15m", status.RouteReplica{BehindSince: ago(24*time.Hour + 15*time.Minute), IntervalSecs: 86400}, "behind", 24*time.Hour + 15*time.Minute}, |
| + {"behind, long interval 24h16m", status.RouteReplica{BehindSince: ago(24*time.Hour + 16*time.Minute), IntervalSecs: 86400}, "late", 24*time.Hour + 16*time.Minute}, |
| + {"behind wins over level", status.RouteReplica{BehindSince: ago(time.Minute), LevelAt: ago(time.Hour)}, "behind", time.Minute}, |
| + {"level, recent", status.RouteReplica{LevelAt: ago(time.Minute)}, "level", time.Minute}, |
| + {"level, exactly 15m", status.RouteReplica{LevelAt: ago(15 * time.Minute)}, "level", 15 * time.Minute}, |
| + {"level, one second over 15m", status.RouteReplica{LevelAt: ago(15*time.Minute + time.Second)}, "late", 15*time.Minute + time.Second}, |
| + {"level, long interval 23h", status.RouteReplica{LevelAt: ago(23 * time.Hour), IntervalSecs: 86400}, "level", 23 * time.Hour}, |
| + {"level, long interval 24h16m", status.RouteReplica{LevelAt: ago(24*time.Hour + 16*time.Minute), IntervalSecs: 86400}, "late", 24*time.Hour + 16*time.Minute}, |
| + {"nothing known", status.RouteReplica{}, "unknown", 0}, |
| + {"interval alone says nothing", status.RouteReplica{IntervalSecs: 60}, "unknown", 0}, |
| + } |
| + for _, c := range cases { |
| + got, behind := replicaState(c.r, now) |
| + if got != c.want || behind != c.wantBehind { |
| + t.Errorf("%s: replicaState = %q, %v; want %q, %v", c.name, got, behind, c.want, c.wantBehind) |
| + } |
| + } |
| +} |
| + |
| +// TestReplicaCellText pins the table's wording: unknown is blank, never "ok". |
| +func TestReplicaCellText(t *testing.T) { |
| + secs := func(n int64) *int64 { return &n } |
| + cases := []struct { |
| + name string |
| + v *replicaView |
| + want string |
| + wantLate bool |
| + }{ |
| + {"no report", nil, "", false}, |
| + {"unknown", &replicaView{State: "unknown"}, "", false}, |
| + {"level", &replicaView{State: "level"}, "ok", false}, |
| + {"parked", &replicaView{State: "parked"}, "ok", false}, |
| + {"behind", &replicaView{State: "behind", BehindSecs: secs(300)}, "ok", false}, |
| + {"late", &replicaView{State: "late", BehindSecs: secs(2*3600 + 10*60)}, "2h10m behind", true}, |
| + {"late without a figure", &replicaView{State: "late"}, "late", true}, |
| + {"a state from the future", &replicaView{State: "frozen"}, "", false}, |
| + } |
| + for _, c := range cases { |
| + got, late := replicaCellText(c.v) |
| + if got != c.want || late != c.wantLate { |
| + t.Errorf("%s: replicaCellText = %q, %v; want %q, %v", c.name, got, late, c.want, c.wantLate) |
| + } |
| + } |
| +} |
| + |
| +// TestAPIInstanceListReplica: replica joins from a FRESH box's |
| +// Metrics.RouteReplicas, is absent for a host the box did not speak for, and |
| +// is dropped for a host whose only reporting box has gone stale, the same |
| +// rules as db_bytes. |
| +func TestAPIInstanceListReplica(t *testing.T) { |
| + fx, token := instanceFixture(t, "code-inst-replica-state") |
| + enableInstances(t, fx, "titogo", titoManifest()) |
| + u := instancesURL(fx, "bab", "titogo") |
| + for _, h := range []string{"shop.go.tito.io", "cafe.go.tito.io", "late.go.tito.io", "stale.go.tito.io"} { |
| + if code, _, out := rawReq(t, http.MethodPost, u, token, |
| + `{"host":"`+h+`","region":"ie"}`); code != http.StatusCreated { |
| + t.Fatalf("create %s = %d %s", h, code, out) |
| + } |
| + } |
| + now := time.Now() |
| + fresh := status.CollectBox(nil, status.Probes{MachineID: "box1"}, now) |
| + levelAt := now.Add(-time.Minute).UTC().Truncate(time.Second) |
| + behindSince := now.Add(-3 * time.Hour).UTC().Truncate(time.Second) |
| + fresh.Metrics = &status.BoxMetrics{RouteReplicas: map[string]status.RouteReplica{ |
| + "shop.go.tito.io": {LevelAt: &levelAt, IntervalSecs: 60}, |
| + "late.go.tito.io": {BehindSince: &behindSince}, |
| + }} |
| + if err := status.WriteBox(context.Background(), fx.rel, fresh); err != nil { |
| + t.Fatal(err) |
| + } |
| + stale := status.CollectBox(nil, status.Probes{MachineID: "box2"}, now.Add(-time.Hour)) |
| + stale.Metrics = &status.BoxMetrics{RouteReplicas: map[string]status.RouteReplica{ |
| + "stale.go.tito.io": {LevelAt: &levelAt}, |
| + }} |
| + if err := status.WriteBox(context.Background(), fx.rel, stale); err != nil { |
| + t.Fatal(err) |
| + } |
| + |
| + code, _, out := rawReq(t, http.MethodGet, u, token, "") |
| + if code != http.StatusOK { |
| + t.Fatalf("list = %d %s", code, out) |
| + } |
| + var list struct { |
| + Instances []map[string]any `json:"instances"` |
| + } |
| + if err := json.Unmarshal([]byte(out), &list); err != nil { |
| + t.Fatalf("list body %s: %v", out, err) |
| + } |
| + got := map[string]map[string]any{} |
| + for _, r := range list.Instances { |
| + got[r["host"].(string)] = r |
| + } |
| + shop, _ := got["shop.go.tito.io"]["replica"].(map[string]any) |
| + if shop == nil || shop["state"] != "level" || shop["interval_secs"] != float64(60) || shop["level_at"] != levelAt.Format(time.RFC3339) { |
| + t.Errorf("shop replica = %v, want a level object with its level_at and interval", got["shop.go.tito.io"]["replica"]) |
| + } |
| + if _, has := shop["behind_secs"]; has { |
| + t.Errorf("a level replica carries behind_secs: %v", shop) |
| + } |
| + late, _ := got["late.go.tito.io"]["replica"].(map[string]any) |
| + if late == nil || late["state"] != "late" || late["behind_since"] != behindSince.Format(time.RFC3339) { |
| + t.Fatalf("late replica = %v, want a late object", got["late.go.tito.io"]["replica"]) |
| + } |
| + if secs, _ := late["behind_secs"].(float64); secs < 3*3600 || secs > 3*3600+60 { |
| + t.Errorf("late behind_secs = %v, want about 3 hours", late["behind_secs"]) |
| + } |
| + for _, h := range []string{"cafe.go.tito.io", "stale.go.tito.io"} { |
| + if v, ok := got[h]["replica"]; ok { |
| + t.Errorf("%s replica = %v, want the key absent", h, v) |
| + } |
| + } |
| +} |
| + |
| +// TestAppInstancesTableReplicaColumn: the Instances tab shows the Replica |
| +// column from the same join, "ok" for a replica that is level, a flagged |
| +// duration for a late one, and nothing for a host no box speaks for. |
| +func TestAppInstancesTableReplicaColumn(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + for _, h := range []string{"level.kass.test", "late.kass.test", "mute.kass.test"} { |
| + fx.addRoute(t, registry.Route{Host: h, App: "kass", Account: "bab", Kind: "instance", Status: "ready"}) |
| + } |
| + now := time.Now() |
| + levelAt := now.Add(-time.Minute) |
| + behindSince := now.Add(-2*time.Hour - 10*time.Minute) |
| + box := status.CollectBox(nil, status.Probes{MachineID: "box1"}, now) |
| + box.Metrics = &status.BoxMetrics{RouteReplicas: map[string]status.RouteReplica{ |
| + "level.kass.test": {LevelAt: &levelAt}, |
| + "late.kass.test": {BehindSince: &behindSince}, |
| + }} |
| + if err := status.WriteBox(context.Background(), fx.rel, box); err != nil { |
| + t.Fatal(err) |
| + } |
| + c := signInPaul(t, fx) |
| + st, _, body := get(t, c, fx.u("/apps/kass/instances")) |
| + if st != http.StatusOK { |
| + t.Fatalf("GET instances = %d:\n%s", st, body) |
| + } |
| + if !strings.Contains(body, `<th scope="col">Replica</th>`) { |
| + t.Fatalf("no Replica column header:\n%s", body) |
| + } |
| + row := func(host string) string { |
| + i := strings.Index(body, ">"+host+"</a>") |
| + if i < 0 { |
| + t.Fatalf("no row for %s:\n%s", host, body) |
| + } |
| + j := strings.Index(body[i:], "</tr>") |
| + return body[i : i+j] |
| + } |
| + if r := row("level.kass.test"); !strings.HasSuffix(strings.TrimSpace(r), "<td>ok</td>") { |
| + t.Errorf("level row has no ok cell:\n%s", r) |
| + } |
| + if r := row("late.kass.test"); !strings.Contains(r, `chip-health-attention">2h10m behind</span>`) { |
| + t.Errorf("late row has no flagged duration:\n%s", r) |
| + } |
| + if r := row("mute.kass.test"); !strings.HasSuffix(strings.TrimSpace(r), "<td></td>") { |
| + t.Errorf("a host no box speaks for must have an empty replica cell:\n%s", r) |
| + } |
| +} |
| + |
| +// TestAlertTextForRestartsAndReplica: the two newer rules say what their |
| +// value means, and an unknown rule still prints a bare number. |
| +func TestAlertTextForRestartsAndReplica(t *testing.T) { |
| + for _, c := range []struct { |
| + rule string |
| + value float64 |
| + want string |
| + }{ |
| + {"restarts", 5, "5 restarts in 10 minutes"}, |
| + {"replica", 4320, "1h12m overdue"}, |
| + {"replica", 905, "15m overdue"}, |
| + {"somefuturerule", 3, "3"}, |
| + } { |
| + if got := alertValueText(c.rule, c.value); got != c.want { |
| + t.Errorf("alertValueText(%q, %v) = %q, want %q", c.rule, c.value, got, c.want) |
| + } |
| + } |
| +} |
| + |
| +// TestAlertMessageDetailLine: a firing alert that carries a detail names it |
| +// right after the value, worded for its rule; a clear (the box zeroes the |
| +// state) and a rule with no detail add no line. |
| +func TestAlertMessageDetailLine(t *testing.T) { |
| + at := time.Date(2026, 10, 13, 9, 0, 0, 0, time.UTC) |
| + firing := func(detail string) metrics.RuleState { |
| + return metrics.RuleState{Firing: true, Value: 3, FiredAt: at, Detail: detail} |
| + } |
| + for _, c := range []struct { |
| + rule, detail, want string |
| + }{ |
| + {"restarts", "carlos-litestream.service", "Value at the transition: 3 restarts in 10 minutes\nUnit: carlos-litestream.service\nTime (UTC)"}, |
| + {"replica", "/data/shop/app.db", "Database: /data/shop/app.db\nTime (UTC)"}, |
| + {"somefuturerule", "thing", "Value at the transition: 3\nDetail: thing\nTime (UTC)"}, |
| + {"restarts", "", "Value at the transition: 3 restarts in 10 minutes\nTime (UTC)"}, |
| + } { |
| + _, body := alertMessage("box-a", c.rule, firing(c.detail), at) |
| + if !strings.Contains(body, c.want) { |
| + t.Errorf("%s/%q body missing %q:\n%s", c.rule, c.detail, c.want, body) |
| + } |
| + } |
| + _, body := alertMessage("box-a", "restarts", metrics.RuleState{}, at) |
| + if strings.Contains(body, "Unit:") { |
| + t.Errorf("a clear invented a detail line:\n%s", body) |
| + } |
| +} |
| + |
| +// TestSweepAlertsCarriesDetail: the dashboard's row keeps the rule's detail. |
| +func TestSweepAlertsCarriesDetail(t *testing.T) { |
| + af := newAlertFixture(t) |
| + af.writeAlert(t, metrics.AlertState{ |
| + V: 1, Box: "box-a", UpdatedAt: time.Now(), |
| + Rules: map[string]metrics.RuleState{ |
| + "restarts": {Firing: true, Value: 4, FiredAt: time.Now(), Detail: "carlos-litestream.service"}, |
| + "disk": {Firing: true, Value: 0.9, FiredAt: time.Now()}, |
| + }, |
| + }) |
| + rows, err := af.srv.sweepAlerts(context.Background()) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + got := map[string]string{} |
| + for _, r := range rows { |
| + got[r.Rule] = r.Detail |
| + } |
| + if len(rows) != 2 || got["restarts"] != "carlos-litestream.service" || got["disk"] != "" { |
| + t.Fatalf("rows = %+v", rows) |
| + } |
| +} |
diff --git a/internal/console/server.go b/internal/console/server.go| index d0b388eae..15e96185c 100644 |
| --- a/internal/console/server.go |
| +++ b/internal/console/server.go |
| @@ -3328,6 +3328,15 @@ func (s *Server) handleAppInstances(w http.ResponseWriter, r *http.Request) { |
| p.AppMetrics = s.appResourceMetrics(r.Context(), p.Detail, acct) |
| q := parseInstanceQuery(r.URL.Query()) |
| rows := assembleInstanceRows(p.Detail, acct, p.AppMetrics) |
| + // The Replica column is best-effort like the Database tile: a slow or |
| + // failed status read leaves the cells blank (loadBoxJoin logs it), never |
| + // the page failing. |
| + if len(rows) > 0 { |
| + joinCtx, cancelJoin := context.WithTimeout(r.Context(), instanceStoreTimeout) |
| + now := time.Now() |
| + attachReplicas(rows, loadBoxJoin(joinCtx, s.cfg.Releases, now), now) |
| + cancelJoin() |
| + } |
| for _, row := range rows { |
| if row.Restartable { |
| p.CanRestartApp = true |
diff --git a/internal/console/templates/app_instances.html b/internal/console/templates/app_instances.html| index b3904fcbb..b96a2f3dd 100644 |
| --- a/internal/console/templates/app_instances.html |
| +++ b/internal/console/templates/app_instances.html |
| @@ -31,11 +31,11 @@ |
| {{if .InstanceRows}} |
| <div class="tablewrap"> |
| <table class="instances"> |
| - <thead><tr><th scope="col">Address</th><th scope="col">Health</th><th scope="col">Version</th><th scope="col">Channel</th><th scope="col">CPU</th><th scope="col">Memory</th></tr></thead> |
| + <thead><tr><th scope="col">Address</th><th scope="col">Health</th><th scope="col">Version</th><th scope="col">Channel</th><th scope="col">CPU</th><th scope="col">Memory</th><th scope="col">Replica</th></tr></thead> |
| {{if .InstanceQuery.Group}} |
| {{range .InstanceGroups}} |
| <tbody> |
| - <tr class="instance-group"><th colspan="6" scope="rowgroup"><code>{{.Channel}}</code> <span class="small">{{$.InstanceCopy.Target}}: {{if .TargetErr}}—{{else if .TargetVersion}}<code>{{.TargetVersion}}</code>{{else}}—{{end}} · Instances: {{len .Rows}} / {{.Total}}</span></th></tr> |
| + <tr class="instance-group"><th colspan="7" scope="rowgroup"><code>{{.Channel}}</code> <span class="small">{{$.InstanceCopy.Target}}: {{if .TargetErr}}—{{else if .TargetVersion}}<code>{{.TargetVersion}}</code>{{else}}—{{end}} · Instances: {{len .Rows}} / {{.Total}}</span></th></tr> |
| {{range .Rows}}{{template "instance_row" (instanceRowPage $ .)}}{{end}} |
| </tbody> |
| {{end}} |
| @@ -109,5 +109,6 @@ hibernated instance state stays in the old placement's bucket.</p> |
| <td><span class="cell-value">{{pct .Row.Metrics.CPUFrac}}</span>{{if .Row.Metrics.SparkCPU}}<svg class="cellspark" aria-hidden="true" viewBox="0 0 240 40" preserveAspectRatio="none"><polyline fill="none" stroke="currentColor" points="{{.Row.Metrics.SparkCPU}}"/></svg>{{end}}</td> |
| <td><span class="cell-value">{{bytesOrDash .Row.Metrics.MemBytes}}</span>{{if .Row.Metrics.SparkMem}}<svg class="cellspark" aria-hidden="true" viewBox="0 0 240 40" preserveAspectRatio="none"><polyline fill="none" stroke="currentColor" points="{{.Row.Metrics.SparkMem}}"/></svg>{{end}}</td> |
| {{else}}<td colspan="2"><span class="small">no data</span></td>{{end}} |
| + <td>{{if .Row.ReplicaLate}}<span class="chip chip-health-attention">{{.Row.ReplicaText}}</span>{{else}}{{.Row.ReplicaText}}{{end}}</td> |
| </tr> |
| {{end}} |
diff --git a/internal/console/templates/metrics.html b/internal/console/templates/metrics.html| index b11fcbec6..5e7c1a5ac 100644 |
| --- a/internal/console/templates/metrics.html |
| +++ b/internal/console/templates/metrics.html |
| @@ -9,7 +9,7 @@ |
| {{range .MetricAlerts}} |
| <tr> |
| <td>{{.Box}}</td> |
| - <td>{{.Rule}}</td> |
| + <td>{{.Rule}}{{if .Detail}} ({{.Detail}}){{end}}</td> |
| <td class="error">{{alertVal .Rule .Value}}</td> |
| <td>{{fmtTime .FiredAt}}</td> |
| </tr> |
diff --git a/internal/host/alertsink.go b/internal/host/alertsink.go| index 2a1509634..6b91e2b0d 100644 |
| --- a/internal/host/alertsink.go |
| +++ b/internal/host/alertsink.go |
| @@ -365,6 +365,8 @@ func alertSentence(ev metrics.Event) string { |
| return fmt.Sprintf("mail relay lag is %d seconds on %s", int64(ev.Value), ev.Box) |
| case "geoip": |
| return geoipAlertSentence(ev) |
| + case "restarts": |
| + return fmt.Sprintf("%s restarted %d times in %d minutes on %s", restartsAlertUnit(ev), int64(ev.Value), int(restartWindow.Minutes()), ev.Box) |
| default: |
| return fmt.Sprintf("%s is firing on %s", ev.Rule, ev.Box) |
| } |
| @@ -382,6 +384,8 @@ func alertClearSentence(ev metrics.Event) string { |
| return fmt.Sprintf("mail relay lag is back to %d seconds on %s", int64(ev.Value), ev.Box) |
| case "geoip": |
| return fmt.Sprintf("geoip on %s is back to normal", ev.Box) |
| + case "restarts": |
| + return fmt.Sprintf("%s has stopped restarting on %s", restartsAlertUnit(ev), ev.Box) |
| default: |
| return fmt.Sprintf("%s has cleared on %s", ev.Rule, ev.Box) |
| } |
| @@ -397,10 +401,21 @@ func alertLevel(r metrics.Rules, rule string) (string, bool) { |
| return pct(r.CPUFrac), true |
| case "lag": |
| return fmt.Sprintf("%d seconds", r.LagSecs), true |
| + case "restarts": |
| + return fmt.Sprintf("more than %d restarts in %d minutes", int64(r.RestartCount), int(restartWindow.Minutes())), true |
| } |
| return "", false |
| } |
| +// restartsAlertUnit is the unit a restarts event is about. An event with no |
| +// detail (one persisted by an older build) still has to read as a sentence. |
| +func restartsAlertUnit(ev metrics.Event) string { |
| + if ev.Detail == "" { |
| + return "a platform unit" |
| + } |
| + return ev.Detail |
| +} |
| + |
| func alertSustain(r metrics.Rules, rule string) time.Duration { |
| switch rule { |
| case "mem": |
diff --git a/internal/host/alertsink_test.go b/internal/host/alertsink_test.go| index d44bea351..5853e0aae 100644 |
| --- a/internal/host/alertsink_test.go |
| +++ b/internal/host/alertsink_test.go |
| @@ -56,6 +56,18 @@ func TestFormatAlertFireAndClear(t *testing.T) { |
| subjectHas: []string{"carlos alert cleared", "disk", "30%", "ada-stargazing"}, |
| bodyHas: []string{"back", "30%", "ada-stargazing"}, |
| }, |
| + { |
| + name: "restarts fires and names the unit", |
| + ev: metrics.Event{Rule: "restarts", Firing: true, Value: 6, Box: "box-a", Detail: "carlos-litestream.service"}, |
| + subjectHas: []string{"carlos alert", "carlos-litestream.service", "restarted 6 times", "10 minutes", "box-a"}, |
| + bodyHas: []string{"carlos-litestream.service", "more than 3 restarts in 10 minutes", "clears"}, |
| + }, |
| + { |
| + name: "restarts clears", |
| + ev: metrics.Event{Rule: "restarts", Firing: false, Value: 0, Box: "box-a", Detail: "carlos-litestream.service"}, |
| + subjectHas: []string{"carlos alert cleared", "carlos-litestream.service", "stopped restarting", "box-a"}, |
| + bodyHas: []string{"carlos-litestream.service", "stopped restarting"}, |
| + }, |
| { |
| name: "mem fires with its sustain window", |
| ev: metrics.Event{Rule: "mem", Firing: true, Value: 0.94, Box: "box-a"}, |
diff --git a/internal/lease/lease.go b/internal/lease/lease.go| index 15a4eaf6a..dea5e3f7c 100644 |
| --- a/internal/lease/lease.go |
| +++ b/internal/lease/lease.go |
| @@ -488,10 +488,24 @@ func Restore(ctx context.Context, dbPath, bucket, prefix, region string) (bool, |
| return true, nil |
| } |
| -// LevelAt is the last time a replica sync ended with nothing left to ship: |
| -// every change the database held when that sync began was in the replica when |
| -// it returned. litestream records it at the end of each sync that returns nil |
| +// Level reports how current the replica is without asking the bucket. |
| +// |
| +// at is the last time a replica sync ended with nothing left to ship: every |
| +// change the database held when that sync began was in the replica when it |
| +// returned. litestream records it at the end of each sync that returns nil |
| // (an idle database's ticks included, which make no S3 call), so the time |
| -// since it is an upper bound on how far the replica trails. Zero until the |
| -// first such sync after this replicator started. |
| -func (r *Replicator) LevelAt() time.Time { return r.db.LastSuccessfulSyncAt() } |
| +// since it is an upper bound on how far the replica trails. It is zero until |
| +// the first such sync after this replicator started. |
| +// |
| +// empty says the database has no position yet: nothing has been written that |
| +// litestream could ship, and it records no sync for such a database however |
| +// long it runs. A zero at with empty false is a replicator that has data and |
| +// has not yet managed to ship it. |
| +func (r *Replicator) Level() (at time.Time, empty bool) { |
| + at = r.db.LastSuccessfulSyncAt() |
| + if !at.IsZero() { |
| + return at, false |
| + } |
| + pos, err := r.db.Pos() |
| + return at, err == nil && pos.IsZero() |
| +} |