amadan
spec: dashboard truthful status + scale to 100s of instances
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+145
−0
docs/superpowers/specs/2026-08-10-dashboard-truth-and-scale-design.md+145 −0
diff --git a/docs/superpowers/specs/2026-08-10-dashboard-truth-and-scale-design.md b/docs/superpowers/specs/2026-08-10-dashboard-truth-and-scale-design.md| new file mode 100644 |
| index 000000000..d34b186e0 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-10-dashboard-truth-and-scale-design.md |
| @@ -0,0 +1,145 @@ |
| +# App dashboard: truthful status + scale to 100s of instances |
| + |
| +2026-08-10. Prompted by the titogo Overview on the tito console showing 18 |
| +false "Not responding" flags, a false app-wide "This app is not deploying" |
| +banner, and a "2 of 20 responding" tile beside "20 all working" — while every |
| +flagged address served 2xx/3xx in under a second with the correct |
| +`X-Carlos-Version`. The page must also anticipate Tito's model: one app |
| +minting per-customer/per-event instances into the hundreds. |
| + |
| +Scope decision (Paul): both halves in one design, truth first — search and |
| +filter over lying statuses just scales the lie. Growth shape: 100s of |
| +*dynamic instances* on one app; channels and named routes stay dozens-at-most. |
| + |
| +## Diagnosis (verified live + in code) |
| + |
| +- The health word comes from a 48h metrics sweep matched to routes only by |
| + the managed unit conventions — `carlos-app@<host>.service` or the |
| + delegated-exec cgroup name (`internal/console/metricsview.go`, |
| + `OwnerAppMetrics`). Routes running under custom units (tito's `titogo@*`, |
| + `titogo-canary@*`) never match, so `Reporting()` is false and `healthWord` |
| + (`internal/console/overview.go`) falls through to "Not responding" for |
| + instance-kind routes. The route rows can carry their real unit name since |
| + #110 (`unit` column, used only for restart binding); the sweep ignores it. |
| +- `AppView.AccountMismatch` collapses route-account stamps to one app-wide |
| + boolean (last-writer-wins per app name, `internal/console/view.go`) and |
| + renders "this app is not deploying". On tito, live hosts serve `bae`'s |
| + edge version — promotes made on the page demonstrably reach the fleet, so |
| + the app-wide claim is false; some subset of route rows is stamped `bab`. |
| +- The Web addresses tile measures DNS-pointing + TLS issuance per domain |
| + (`domainNeedsAttention`), not liveness; its "all working" copy invites the |
| + liveness reading, colliding with the Running tile. |
| +- Real defect found alongside: `titogo.bae.platform.tito.io` serves 404 with |
| + no version header — operational, tracked separately (see Rollout). |
| +- The console has zero search/filter/pagination machinery anywhere; every |
| + list is unbounded or hard-capped. This design introduces the pattern. |
| + |
| +## Design |
| + |
| +### 1. Real readings (box) + honest vocabulary (console) |
| + |
| +Box metrics sweep: in addition to `carlos-app@*` and exec cgroups, sample |
| +the cgroup of any registry route whose `unit` column is stamped. Emit into |
| +the existing readings shape keyed by host, exactly like managed units — |
| +additive; nothing that parses readings today changes. A stamped unit that |
| +does not exist or has no cgroup emits no reading and never fails the sweep |
| +(`ValidUnitName` already rejects malformed names at stamp time). |
| + |
| +Console `healthWord`, for a silent, non-hibernating instance-kind route: |
| + |
| +- Route is expected to report — the box itself runs its process (managed |
| + `carlos-app@` unit or delegated exec child) or the route has a stamped |
| + unit — → **"Not responding"**, bad tone, as today. Now true. |
| +- Otherwise → **"No readings"**, neutral tone (like "Asleep"), with |
| + instance-page guidance: stamp the unit (`carlos route … -unit`) to get |
| + health here. |
| + |
| +Tiles: |
| + |
| +- Running: no-readings routes stop counting against the app — |
| + "2 of 2 reporting · 18 no readings" (the "· N no readings" clause only |
| + when non-zero), not "2 of 20". |
| +- Web addresses: "20 — DNS & certs OK" instead of "all working". |
| +- Resources: unchanged (already sums reporting copies only; gains real |
| + numbers as routes are stamped). |
| + |
| +### 2. Account-mismatch banner at route granularity |
| + |
| +Replace the app-wide boolean with the actual list of mismatched routes |
| +(host + stamped account). |
| + |
| +- **All** non-platform routes mismatched → keep the strong "not deploying" |
| + banner, now naming the account and the hosts. |
| +- **Some** mismatched → scoped warning, no "not deploying" claim: |
| + "N addresses are stamped to account `X` and won't follow promotes made |
| + here: <hosts> — fix on Settings." |
| + |
| +On tito this converts a false alarm into a precise pointer at the stale |
| +`bab`-stamped rows, which is also the diagnostic for the cleanup. |
| + |
| +### 3. Scale: rollup + exceptions on Overview, workbench on Running |
| + |
| +Overview flow diagram — per column (EDGE, BETA, LIVE, each canary): |
| + |
| +- ≤8 addresses: list all, exactly as today. Small apps are unchanged. |
| +- >8: rollup line "142 addresses · 3 need attention", then only the |
| + needs-attention addresses (Not responding / Failed / stuck Setting up), |
| + capped at 5, then "+K more → Running". Healthy and Asleep are counted, |
| + not listed. |
| + |
| +Running tab — server-rendered, GET-param driven (bookmarkable, no JS |
| +dependency): |
| + |
| +- `q=` substring match on host; `health=` filter by health word; |
| +- cursor pagination ordered by host, page size 50 (`cursor=` opaque, |
| + offset-free); |
| +- header shows per-health-word totals, which double as filter links. |
| + |
| +API parity (the page is a view of the API): the instances API gains the |
| +same `q`, `health`, `limit`, `cursor` params; the response additively gains |
| +`total`, per-health counts, and `next_cursor` — additive per the wire-shape |
| +rule: everything that parses today's shape keeps parsing (new fields only, |
| +no field renamed, moved, or re-typed). |
| + |
| +CLI parity: `carlos instances list -match <substr> -health <word>` with the |
| +same server-side semantics, printing totals — no local-disk registry walk, |
| +works with zero box access. |
| + |
| +Channels grid: structurally unchanged (bounded by team practice); its |
| +per-column address lists get the same ≥8 rollup for consistency. |
| + |
| +### 4. Rollout |
| + |
| +1. Branch off origin/main `db76025`; land box sweep + console + API + CLI. |
| +2. Deploy boxes first (sampling stamped units is harmless while nothing is |
| + stamped), then consoles. |
| +3. Stamp tito's ~18 custom-unit routes (`carlos route … -unit |
| + titogo@….service` / `titogo-canary@….service`, run as carlos on the |
| + box, one-time). Route records (PR #137) makes stamps durable once |
| + merged; not a gate. |
| +4. Verify against the thing changed: titogo Overview shows canaries |
| + Running with real CPU/memory; Running tile reads "N of N reporting"; |
| + verify with the deployed console, not a cached page. |
| +5. Follow-ups, precisely named by the new banner/tiles: re-stamp or migrate |
| + the `bab`-stamped route rows; investigate the |
| + `titogo.bae.platform.tito.io` 404. |
| + |
| +Filed as a gap, not built: edge-observed liveness (router-reported |
| +per-route last-success as a truth source) — the truest signal, deferred for |
| +its status wire-shape cost. |
| + |
| +### 5. Testing |
| + |
| +- `healthWord` table tests gain the "No readings" state and the |
| + expected-to-report distinction. |
| +- Overview-stats tests pin the new Running-tile arithmetic and the |
| + Web-addresses copy semantics. |
| +- Banner tests: none / some / all routes mismatched, hosts named in output. |
| +- API handler tests: `q`/`health`/`cursor` combinations, additive-response |
| + compatibility (no new params → prior shape). |
| +- Box sweep test: stamped unit with fake cgroup dir; stamped-but-absent |
| + unit emits nothing and does not fail the sweep. |
| +- CLI list tests build env through `carlosTestEnv` with `cmd.Dir` in a |
| + scratch dir. |
| +- Gate: `go build ./... && go vet ./... && go test ./... -count=1`, gofmt |
| + on touched files, darwin cross-compile. |