carlos / platform Public

Error monitoring: where it stands (handoff, 2026-09-25) Issue open

Handoff for the error-monitoring work Bill started on 2026-09-24: seeing errors across instances, grouping them, and giving customers a reference support can look up. Written so another session, or a person, can pick it up cold on any machine.

Landed on main (amadan, at 8c54ec7; CI green)

  • Availability tab (was GitHub #417). Edge 502/503/504 move off Errors onto their own tab, grouped by cause, not by path. - Cause vocabulary: refused, reset, timeout, noupstream, fleetdark, tunnel, static, proxy, upstream, abandoned. - Platform lifecycle events (failed wakes, exits) move there too. - carlos availability. Spec: docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md §11. - Reviewed by Opus plus three Fable rounds; every finding fixed and mutation-tested.
  • Parallel chunk read (was GitHub #418). ReadChunks fetches 16 chunks at a time and assembles them in serial order. The Errors tab was spending about 5 s of about 7 s reading a day of log chunks one at a time.
  • Request-reference spec, approved by Bill: docs/superpowers/specs/2026-09-24-request-reference-design.md. Format R-XXXX-XXXX-XXXX (30 bits of seconds, good to 2060, and 30 random). Bill's four rulings are in §8, Fable's review in §10. Not built.
  • v0.22.0 operator note in README.md. The public release notes draft is in the description of the closed branch release-notes-v0.22.0.
  • AGENTS.md: amadan is the source of truth. Land with amadan branch merge on green; no GitHub PRs.
  • amadan workflow on for this repo, and the agent skill installed at .claude/skills/amadan/SKILL.md.

Blocking the deploy

  1. v0.22.0 is not cut. latest.txt still says v0.21.0. Tag amadan main (8c54ec7); the recipe is README "Releasing the platform". It needs the SSM signing keys.
  2. tito's console runs a custom build. X-Carlos-Version: cf02a4d5-tito is a commit on fix/tito-platform-domains-compatible (Paul, 2026-09-15), not on main. It will not pick up v0.22.0 by itself. Someone has to rebuild it from main, or merge main into that branch, and redeploy. Question for Paul: does that branch still need to exist?
  3. Rolling the boxes (carlos system update) needs an ops-account owner. Bill's CLI login gets "not found" on carlos system status.

How to tell it is live

  1. curl -s https://apt.carlosframework.com/latest.txt prints v0.22.0.
  2. carlos system status --channel canary|stable shows every box current on v0.22.0.
  3. The console's X-Carlos-Version changes from cf02a4d5-tito, and titogo's app page has an Availability sub-tab.
  4. After carlos update, carlos availability --app titogo --since 24h answers. New 502s carry a cause. carlos errors --app titogo --since 24h is faster than the 5.4 s it took before.

Next, in order

  1. Request references, step 1: mint, carry across hops and the tunnel, return on every response, show on edge error pages (spec §7). Waiting on the deploy by Bill's choice.
  2. Skip chunks with no error lines. Boxes record each chunk's highest level, so the errors read never opens chunks that have nothing in them.
  3. Alerts on a new cause or fingerprint, plus resolve/mute. Already asked for in discussion #137.
  4. The server door. Apps post exceptions with stacks, for grouping by root cause.

Known gaps (documented, not fixed)

  • Endpoints that hold their response headers for more than 5 s by design (long polls) file a standing "the visitor gave up waiting for the instance" row.
  • A client aborting mid-upload can be filed as reset, which blames the instance.
  • A far agent whose status store failed to open files nothing, and the front box no longer covers it. A client leaving after 5 s can be filed as abandoned on both boxes.

Housekeeping

  • AGENTS.md is now partly wrong. It says amadan branch merge has no squash mode, so squash on the branch first. The current CLI has amadan branch merge -squash. Worth a one-line fix.
  • GitHub main is behind amadan (38639c1e against 8c54ec7). Bill chose to leave it, expecting it to catch up; nothing observed mirrors it so far.
  • Setting up a new machine: - go install amadan.net/amadan/amadan/cmd/amadan@latest, and the same for git-remote-amadan. - Use the agent credential in ~/.amadan-claude: copy it across, or amadan agent token create claude-code. - Run amadan ledger init -level prompts -local in the checkout.