carlos / platform Public

No CLI or API answers "who is using what right now" — the operator's per-account view is browser-only Issue open

"Which account is using what, right now?" has no CLI or API answer — the only view is a browser page, so answering it means reading S3 by hand.

What happened

Asked to check whether one member (james@keymail.dev, account bac, app jam) was hogging resources on the flagship. There is no verb for it. Every door that holds the answer is shut to a terminal:

  • GET /fleet/metrics and GET /fleet/economics/accounts are the operator's live per-box/per-account views, and both gate on s.session(r) then IsOperator (internal/console/server.go:1910, :1976). A CLI bearer token is not a session, so both answer a bare http.NotFound — correctly, since the route must not be probeable, but it leaves no token-shaped way in.
  • carlos economics backhaul is the one operator verb that prints a per-account table, and it lists only accounts that backhauled something. Four of the deployment's accounts appear; the rest, including the one being asked about, do not.
  • carlos status --live reports boxes, not tenants: release, tunnel, mem, load, disk-free per box, nothing per account or per instance.
  • carlos instances with no sub-verb reads /data/registry.db — box-local, and it lists routes rather than what they are consuming.

So the answer came out of the bucket with aws s3 cp: metrics/boxes/<box>/raw/<hour>.json for the live per-unit cgroup readings, usage/accounts/<sqid>/<month>.json for the meters, usage/edge/<box>/days/<date>.json for requests and egress. That is precisely the move AGENTS.md's "operate via API and CLI" rule says is a product gap rather than a path.

Why it matters beyond one question

The data is all there and all published — the metrics loop writes per-unit cpu/mem_bytes every 60s, and the usage rollups are per account per month. The only thing missing is a door. Anyone answering a capacity, billing or noisy-neighbour question today either opens a browser or learns the bucket layout, and the second one is how an agent ends up with AWS credentials for a question that should have cost one command.

Proposed shape

carlos status --instances (signed-in, operator-gated like economics backhaul, same bare-404 on a non-operator token), printing one row per awake instance across the fleet — box, host, account, app, CPU cores, memory, and how long it has been resident:

BOX             HOST                          ACCOUNT  APP     CPU    MEM     AWAKE
annie-welding   amadan.net                    bah      amadan-hub  0.017  351MB   9d
annie-welding   ips.bdn.oncarlos.com          bdn      ips         0.003  284MB   4h
annie-welding   jam.bac.oncarlos.com          bac      jam         0.000   26MB   24h

--account <sqid> to narrow it, and a --usage sibling (or carlos economics accounts) for the month-to-date meter table the browser page already renders, which would close #182's neighbour at the same time.

The console side needs one new /api/cli/... door reading what LoadMetrics and LoadEconomics already return; nothing new has to be measured or stored.

Evidence

Measured 2026-08-30 on the flagship: jam was resident 24h continuously at ~26MB and 0.000 cores, month-to-date 23 instance-hours against bab's 3,161 — a question that took one aws s3 sweep of four boxes' metrics and nine accounts' usage objects to answer, and that a member could not have asked about their own app at all.

---

Mirrored from GitHub #370, which I filed before remembering that platform issues belong here. Closing that one with a pointer to this.

Built it, on branch status-instances. CI green.

carlos status --instances carlos status --instances --account bac

BOX HOST ACCOUNT APP CPU MEM annie-welding amadan.net bah amadan-hub 0.017 351MB annie-welding jam.bac.oncarlos.com bac jam 0.000 26MB (awake instances only — a hibernating instance has no process to measure)

Console-backed, so it needs no AWS credentials — which was the whole point. Nothing new is measured or stored, exactly as you said: the metrics loop already writes per-unit cpu and mem every 60s, LoadMetrics already sweeps them for the browser page, and unitOwner already knows how to attribute a sampled cgroup to a route's account. The only missing piece was a door.

GET /api/cli/status/instances, gated cliUser → IsOperator → byte-identical 404, the shape economics backhaul and email pause use. Registered with an empty role set, so no service credential of any role reaches it: the answer names every account's instances on every box, and there is no app-scoped reading of that question. Added to closedDoors with that reason.

One implementation note: rather than write a second matcher, I split unitOwner's rule out as routeForUnit and shared it. This listing needs the route's HOST and not only its owner, and a private second notion of which route a cgroup belongs to is how a listing ends up disagreeing with the economics pass about whose the cost is.

One column from your sketch is missing: AWAKE. I could not build it honestly. Nothing in the tree publishes a process start time — not metrics.UnitSample, not status.BoxMetrics, not the route — so the only available answer is "first seen in the 48-hour sweep", which under-reports anything older than that and resets on any agent restart or history gap. Given that residency is exactly what you were investigating (jam resident 24h at 26MB and 0.000 cores), a column that quietly under-reports it seemed worse than one that is missing. A real one needs a start-time field on the box's own sample; say if you want that and I will add it.

Two things the output says out loud rather than leaving to inference: a unit matching no route is labelled (platform) — the console's own unit, the agent, system units — rather than dropped, because an operator asking what is on the box wants the box's own overhead in the picture; and the hibernating-instance caveat prints under every answer, so a short list is never read as the whole tenancy.

Your --usage sibling is not built. It is a different door onto LoadEconomics, and worth doing, but it is the month-to-date meter table rather than "right now" and I would rather it landed as its own change than widened this one.

Triage, 2026-09-05. status-instances reviewed. CI green. Verdict: merge with fixes. No data leak found. One attribution bug and one test that asserts nothing.

Confirmed sound, since this is a door onto every account's tenancy and that is the part worth being sure about:

  • Gate order matches the shape economics backhaul uses, byte for byte: cliRoute (bearer, service refusal) → cliUser → IsOperator → http.NotFound. No CSRF or requireFreshAuth applies to a bearer door, consistent with every other operator CLI verb.
  • The empty role set genuinely excludes every service credential — svcrole.Set{}.Permits returns false for all roles including "" — and closedDoors walks it.
  • --account filtering is server-side, so the client never receives other accounts' rows.
  • The routeForUnit split is behaviour-preserving for unitOwner: same first-match iteration, same return on hit and miss. No misbilling risk, which was the thing I most wanted to rule out about sharing that matcher.

The attribution bug is the one to fix before this lands. The route join is box-local — openPassRegistry opens the console's own box registry — while the metrics sweep it reads is fleet-wide. On a multi-box deployment every unit sampled on another box matches no route, falls through to (platform), and is charged to bab. So the answer to "who is using what right now" would report a whole box's tenants as platform overhead.

Inert today, because both deployments are single-box — and sweepUsage already documents this exact trap with the structural fix tracked as #137. But the new file's own SCOPE comment claims the opposite, and this door's whole purpose is to be the thing an operator trusts instead of reading the bucket. The comment should say what is true, and the caveat belongs in the CLI output beside the hibernating line you already print.

The --account test is vacuous. No test in internal/console writes anything under metrics/boxes/, so LoadMetrics returns zero boxes and the assertion loop never runs — deleting the filter would still pass. One fixture box with two units would pin the filter, the join, and the (platform) label at once.

Smaller: the "cannot be told from a door that does not exist" claim is overstated — a POST to the path returns 405 with an Allow header from the mux, before any auth, to anyone. Pre-existing for every method-scoped cliRoute including email pause, so I would correct the claim rather than re-architect, but the test asserts a property the door does not have. Also --account is sqid-only while the rest of the CLI takes sqid-or-name, so a name silently answers "no awake instances".

Merge-order warning: this branch and deploy-build-probe (#13) both rewrite the same closed-door assertion from 17 to 18. Whichever lands second conflicts, and the mechanical resolution is red — it has to become 19 with both reasons listed.

Your AWAKE column: the reasoning for leaving it out is right, and the fix is small enough to name. Nothing publishes a process start time, but the activator already holds spawnVersion per child and has no accessor — the same gap #13 runs into from the other side. One start-time field on the box's own sample would serve both. Worth doing as its own change.