eleven / ocho Public

status: local box's own row also prefers its ownership-filtered snapshot (#852)

The previous fix (#851) partitioned a PEER's reported hib count by HRW
ownership, but the local box's own row was still computed by a completely
separate code path (build_fleet's own `ss`+registry sweep), so it kept
reporting every mesh-replicated route it knew about — including ones a peer
now owns. Live after deploying #851: box A's row still showed all 20 routes
while box B's newly-filtered row showed 6, so the two rows summed to 26, not
the true 20 — the same overlap the fix was meant to remove, just smaller.

box A's own Go process already publishes an ownership-filtered snapshot of
itself to S3 for peers to read (computeBoxStats via PutStats). build_fleet
now reads that back for its OWN row too, falling back to the local ss sweep
only when mesh is off, unpublished yet, or stale.

Co-authored-by: Claude <noreply@anthropic.com>

James Adam 3 months ago dc6c93e78ed51d57b81aee2c5b23e736c2b1cccf parent cfa72d8

3 files changed, +71 −18

deploy/status-publish.sh +58 −17
test/status-contract.test.mjs +12 −0
test/status-fixtures/fleet-multi-host.json +1 −1

diff --git a/deploy/status-publish.sh b/deploy/status-publish.sh

index d5b04d3..5ff9f39 100755

--- a/deploy/status-publish.sh

+++ b/deploy/status-publish.sh

@@ -366,8 +366,21 @@ box_label() {

# isn't mesh-configured, has no live peers, or any AWS call fails; a missing

# peer here just means the Fleet page undercounts by one box, never a

# publish failure.

+# As a side effect (one listing, not two), also writes THIS box's own

+# self-published stats body to $TMP/mesh-self-stats-result.json (removed if

+# unpublished/stale/mesh off) — computeBoxStats (meshstats.go) already

+# ownership-partitions this box's own hib count the same way it does for a

+# peer, so build_fleet can prefer this over its own local `ss` sweep instead

+# of re-deriving an unfiltered view that double-counts against a peer's row.

+# A file, not a global var: the caller captures this function's stdout with

+# `$(...)`, which runs it in a subshell — a global assignment made in there

+# never survives back to the parent shell, but a file write does.

mesh_peer_boxes() {

+ local self_out="$TMP/mesh-self-stats-result.json"

+ rm -f "$self_out"

if [ -n "$DRY_RUN" ]; then

+ if [ -f "$FIXDIR/mesh-self-stats.json" ]; then cp "$FIXDIR/mesh-self-stats.json" "$self_out"; fi

if [ -f "$FIXDIR/mesh-peers.json" ]; then cat "$FIXDIR/mesh-peers.json"; else echo '[]'; fi

return

@@ -423,11 +436,14 @@ mesh_peer_boxes() {

key=$(printf '%s' "$line" | awk '{print $4}')

[ -n "$key" ] || continue

[ "$date_part $time_part" '>' "$cutoff" ] || continue # stale — treat like a dead mesh member

- if [ -n "$self" ] && [ "$(b64url_decode "$key")" = "$self" ]; then continue; fi

tmp="$TMP/mesh-stat-$(printf '%s' "$key" | tr '/' '_').json"

aws s3 cp "s3://$bucket/${mesh_prefix}-stats/$key" "$tmp" --region "$region" >/dev/null 2>&1 || continue

body=$(cat "$tmp" 2>/dev/null) || continue

[ -n "$body" ] || continue

+ if [ -n "$self" ] && [ "$(b64url_decode "$key")" = "$self" ]; then

+ printf '%s' "$body" > "$self_out"

+ continue

+ fi

peers=$(sqj "SELECT json_insert(

json('$(printf '%s' "$peers" | sed "s/'/''/g")'), '\$[#]',

json('$(printf '%s' "$body" | sed "s/'/''/g")'))")

@@ -538,21 +554,37 @@ build_fleet() {

# Labels are whimsical name-action pairs derived from a HASH of the machine

# id ("ada-stargazing") — stable per box, and never peer URLs, IPs,

# hostnames, or the machine id itself. Each box's awake/asleep entry here is

- # its OWN locally-verified split — this box's is the `ss` check above, a

- # peer's is its OWN self-published snapshot (mesh_peer_boxes above), already

- # banded/counted-only by the time it left that box — never re-derived or

- # second-guessed here. (Below, the top-level `instances.awake`/`asleep`

- # fields are summed across every box in this array — a per-box figure here

- # is deliberately this box's own view, not the fleet total.) The optional

- # region is the operator-set coarse continent + country word pair (see

- # PRIVACY) — absent env, absent key, never guessed; escaped like the version

- # strings so a quote can't break out of the SQL literal. A peer's region

- # (if its own LCHAT_BOX_REGION is set — meshstats.go's boxRegion) rides

- # through automatically: mesh_peer_boxes inserts each peer's whole

- # self-published object verbatim, so an extra key needs no bash change

- # here. `always_on` stays a genuinely local-box-only concept, though — no

- # peer publishes one.

+ # its OWN locally-verified split. mesh_peer_boxes must run FIRST: besides

+ # every peer's own snapshot, it also writes THIS box's own self-published

+ # one to $TMP/mesh-self-stats-result.json — computeBoxStats (meshstats.go)

+ # already filters that to routes this box currently HRW-owns, so preferring

+ # it here

+ # (over the raw local `ss` sweep above) keeps this box's row from

+ # double-reporting a mesh-replicated route a peer now owns. Falls back to

+ # the local sweep when mesh is off, unpublished yet, or stale — same

+ # "unknowns degrade gracefully" rule as the rest of this file. (Below, the

+ # top-level `instances.awake`/`asleep` fields are summed across every box in

+ # this array — a per-box figure here is this box's own ownership-scoped

+ # view, not the fleet total.) The optional region is the operator-set coarse

+ # continent + country word pair (see PRIVACY) — absent env, absent key,

+ # never guessed; escaped like the version strings so a quote can't break out

+ # of the SQL literal. A peer's region (if its own LCHAT_BOX_REGION is set —

+ # meshstats.go's boxRegion) rides through automatically: mesh_peer_boxes

+ # inserts each peer's whole self-published object verbatim, so an extra key

+ # needs no bash change here. `always_on` stays a genuinely local-box-only

+ # concept, though — no peer publishes one.

local boxes peer_boxes region_kv=""

+ peer_boxes=$(mesh_peer_boxes)

+ local self_stats_file="$TMP/mesh-self-stats-result.json"

+ if [ -s "$self_stats_file" ]; then

+ local self_esc self_awake

+ self_esc=$(cat "$self_stats_file" | sed "s/'/''/g")

+ self_awake=$(sqj "SELECT json_extract('$self_esc', '\$.awake')")

+ if [ "$self_awake" != null ] && [ -n "$self_awake" ]; then

+ awake_json=$self_awake

+ asleep_json=$(sqj "SELECT json_extract('$self_esc', '\$.asleep')")

+ fi

# Resolved here, not at file top: DRY_RUN/argv must be parsed first so the

# dry-run's no-live-probes guard actually guards (and the dry-run fixture's

# direct BOX_REGION= assignment still wins over a re-resolve).

@@ -567,7 +599,6 @@ build_fleet() {

'asleep', json('$asleep_json'),

'load', json('$load_json')$region_kv))")

- peer_boxes=$(mesh_peer_boxes)

boxes=$(sqj "SELECT json_group_array(json(value)) FROM (

SELECT value FROM json_each('$(printf '%s' "$boxes" | sed "s/'/''/g")')

UNION ALL

@@ -1323,7 +1354,10 @@ EOF

# self-published stats snapshot, exactly the shape internal/activator/

# heartbeat.go's PutStats writes and mesh_peer_boxes reads back. Default

# (unset) keeps hosts.count at 1, byte-identical to the single-box shape —

- # the mesh's own "absence changes nothing" convention, mirrored here.

+ # the mesh's own "absence changes nothing" convention, mirrored here. This

+ # also seeds THIS box's own self-published stats fixture

+ # (mesh-self-stats.json), gated the same way: when mesh is off, this box

+ # never self-publishes either, so the single-box shape must stay untouched.

if [ "${STATUS_DRY_MESH_PEER:-}" = 1 ]; then

# awake=1 here, not some arbitrary number: this peer only ever runs

# instances mesh-replicated FROM this box's own registry, so its awake

@@ -1336,6 +1370,13 @@ EOF

# merge's raw passthrough carries an extra key with no bash-side change.

cat > "$FIXDIR/mesh-peers.json" <<'EOF'

[{"label":"turing-orienteering","awake":1,"asleep":1,"load":"busy","region":"UK · London"}]

+EOF

+ # awake=1/asleep=2 (3 total) here, one fewer than the local `ss` sweep's

+ # own awake=2/asleep=2 (4 total) above — proves build_fleet actually

+ # prefers this ownership-filtered snapshot over its own unfiltered local

+ # sweep, not just reads it and ignores it.

+ cat > "$FIXDIR/mesh-self-stats.json" <<'EOF'

+{"label":"hypatia-sketching","awake":1,"asleep":2,"load":"comfortable","region":"EU · Germany"}

EOF

diff --git a/test/status-contract.test.mjs b/test/status-contract.test.mjs

index 3fd22cc..dd44511 100644

--- a/test/status-contract.test.mjs

+++ b/test/status-contract.test.mjs

@@ -443,6 +443,18 @@ await test("multi-host fixture merges the local box with a live mesh peer", () =

assert.notEqual(local.label, peer.label, "the two boxes must have distinct labels");

});

+// The local box's own row must prefer its self-published, ownership-filtered

+// snapshot (mesh-self-stats.json in the dry-run fixture: awake=1) over its

+// raw local `ss` sweep (which would show awake=2, see the single-box `fleet`

+// fixture) — otherwise this box's own row still overlaps a peer's, even once

+// the peer's row is correctly filtered.

+await test("the local box's own row prefers its self-published stats over its raw ss sweep", () => {

+ const [local] = fleetMH.hosts.boxes;

+ assert.equal(local.awake, 1, "local.awake must come from the self-published snapshot, not the unfiltered ss sweep (2)");

+ assert.notEqual(local.awake, fleet.hosts.boxes[0].awake,

+ "a regression would make the mesh-off and mesh-on local rows identical again");

+});

// A host genuinely running on a mesh peer has no local socket on this box, so

// the top-level instances.awake/asleep must sum every box's own verified

// split, not just this box's — otherwise a peer-hosted instance reads as

diff --git a/test/status-fixtures/fleet-multi-host.json b/test/status-fixtures/fleet-multi-host.json

index 22c2521..776ada9 100644

--- a/test/status-fixtures/fleet-multi-host.json

+++ b/test/status-fixtures/fleet-multi-host.json

@@ -1 +1 @@

-{"generated_at":"2026-07-13T22:58:58Z","instances":{"total":6,"systemd":2,"hibernating":4,"awake":3,"asleep":1},"hosts":{"count":2,"boxes":[{"label":"hypatia-sketching","always_on":2,"awake":2,"asleep":2,"load":"comfortable","region":"EU · Germany"},{"label":"turing-orienteering","awake":1,"asleep":1,"load":"busy","region":"UK · London"}]},"capacity":{"level":"plenty"},"versions":[{"version":"v100","count":3},{"version":"v101","count":2},{"version":"unknown","count":1}],"services":{"home":true,"push":true,"cloud":true,"connect":false},"replication":{"healthy":true,"lag_seconds":42}}

+{"generated_at":"2026-07-13T23:21:02Z","instances":{"total":6,"systemd":2,"hibernating":4,"awake":2,"asleep":2},"hosts":{"count":2,"boxes":[{"label":"hypatia-sketching","always_on":2,"awake":1,"asleep":2,"load":"comfortable","region":"EU · Germany"},{"label":"turing-orienteering","awake":1,"asleep":1,"load":"busy","region":"UK · London"}]},"capacity":{"level":"plenty"},"versions":[{"version":"v100","count":3},{"version":"v101","count":2},{"version":"unknown","count":1}],"services":{"home":true,"push":true,"cloud":true,"connect":false},"replication":{"healthy":true,"lag_seconds":42}}