Error monitoring: where it stands (handoff, 2026-09-25) Issue open
Handoff for the error-monitoring work Bill started on 2026-09-24: seeing errors across instances, grouping them, and giving customers a reference support can look up. Written so another session, or a person, can pick it up cold on any machine.
Landed on main (amadan, at 8c54ec7; CI green)
- Availability tab (was GitHub #417). Edge 502/503/504 move off Errors onto their own tab, grouped by cause, not by path. - Cause vocabulary: refused, reset, timeout, noupstream, fleetdark, tunnel, static, proxy, upstream, abandoned. - Platform lifecycle events (failed wakes, exits) move there too. -
carlos availability. Spec:docs/superpowers/specs/2026-09-07-errors-and-analytics-design.md§11. - Reviewed by Opus plus three Fable rounds; every finding fixed and mutation-tested. - Parallel chunk read (was GitHub #418).
ReadChunksfetches 16 chunks at a time and assembles them in serial order. The Errors tab was spending about 5 s of about 7 s reading a day of log chunks one at a time. - Request-reference spec, approved by Bill:
docs/superpowers/specs/2026-09-24-request-reference-design.md. FormatR-XXXX-XXXX-XXXX(30 bits of seconds, good to 2060, and 30 random). Bill's four rulings are in §8, Fable's review in §10. Not built. - v0.22.0 operator note in README.md. The public release notes draft is in the description of the closed branch
release-notes-v0.22.0. - AGENTS.md: amadan is the source of truth. Land with
amadan branch mergeon green; no GitHub PRs. amadan workflow onfor this repo, and the agent skill installed at.claude/skills/amadan/SKILL.md.
Blocking the deploy
- v0.22.0 is not cut.
latest.txtstill says v0.21.0. Tag amadanmain(8c54ec7); the recipe is README "Releasing the platform". It needs the SSM signing keys. - tito's console runs a custom build.
X-Carlos-Version: cf02a4d5-titois a commit onfix/tito-platform-domains-compatible(Paul, 2026-09-15), not onmain. It will not pick up v0.22.0 by itself. Someone has to rebuild it frommain, or mergemaininto that branch, and redeploy. Question for Paul: does that branch still need to exist? - Rolling the boxes (
carlos system update) needs an ops-account owner. Bill's CLI login gets "not found" oncarlos system status.
How to tell it is live
curl -s https://apt.carlosframework.com/latest.txtprints v0.22.0.carlos system status --channel canary|stableshows every boxcurrenton v0.22.0.- The console's
X-Carlos-Versionchanges fromcf02a4d5-tito, and titogo's app page has an Availability sub-tab. - After
carlos update,carlos availability --app titogo --since 24hanswers. New 502s carry a cause.carlos errors --app titogo --since 24his faster than the 5.4 s it took before.
Next, in order
- Request references, step 1: mint, carry across hops and the tunnel, return on every response, show on edge error pages (spec §7). Waiting on the deploy by Bill's choice.
- Skip chunks with no error lines. Boxes record each chunk's highest level, so the errors read never opens chunks that have nothing in them.
- Alerts on a new cause or fingerprint, plus resolve/mute. Already asked for in discussion #137.
- The server door. Apps post exceptions with stacks, for grouping by root cause.
Known gaps (documented, not fixed)
- Endpoints that hold their response headers for more than 5 s by design (long polls) file a standing "the visitor gave up waiting for the instance" row.
- A client aborting mid-upload can be filed as
reset, which blames the instance. - A far agent whose status store failed to open files nothing, and the front box no longer covers it. A client leaving after 5 s can be filed as
abandonedon both boxes.
Housekeeping
- AGENTS.md is now partly wrong. It says
amadan branch mergehas no squash mode, so squash on the branch first. The current CLI hasamadan branch merge -squash. Worth a one-line fix. - GitHub
mainis behind amadan (38639c1e against 8c54ec7). Bill chose to leave it, expecting it to catch up; nothing observed mirrors it so far. - Setting up a new machine: -
go install amadan.net/amadan/amadan/cmd/amadan@latest, and the same forgit-remote-amadan. - Use the agent credential in~/.amadan-claude: copy it across, oramadan agent token create claude-code. - Runamadan ledger init -level prompts -localin the checkout.