| 1 | # 🤖 Public status dashboards — fleet, books, economics, investing, security, carbon, contributors, prompts, SSH access & music |
| 2 | |
| 3 | World-readable pages, published every 5 minutes from the router box: the |
| 4 | running half of #550 Part 11/12's transparency direction (the append-only |
| 5 | public *ledger* repo is still roadmap — see `docs/planet-scale.md`). |
| 6 | |
| 7 | - **Fleet status** (`index.html` + `fleet.json`) — how many instances exist, |
| 8 | how many are always-on vs hibernating, how many of the hibernating ones are |
| 9 | awake right now, the serving hosts and allocation capacity (as abstract |
| 10 | band words — see below), the version distribution, whether the shared |
| 11 | services (home / push / cloud / connect) are up, and fleet-backup |
| 12 | replication health. |
| 13 | - **The books** (`books.html` + `books.json`) — two clearly-headed halves. |
| 14 | **Income**, in aggregate: active paid subscriptions (total and by tier — |
| 15 | tier names are the public price book), MRR in EUR, and gross collected this |
| 16 | month. **Expenditure**, itemized per provider (AWS, Hetzner, Anthropic API, |
| 17 | claude.ai, DNSimple, Apple Developer). Each section renders independently: |
| 18 | a degraded income feed shows its notice inside the Income section and never |
| 19 | hides Expenditure. |
| 20 | - **Tier economics** (`economics.html` + `economics.json`) — the cost-model |
| 21 | half of the books: the public price book, storage allowances, input rates, |
| 22 | and per-tier revenue/cost/break-even estimates, published so #562's |
| 23 | commitment — *the price is the cost plus resilience* — is a checkable |
| 24 | claim. The JSON is **committed data** (`web/status/economics.json`), a |
| 25 | hand-derived snapshot of `docs/tier-economics.md` (the repo is private; |
| 26 | this page is what makes the doc actually public), republished as-is. See |
| 27 | *Keeping the tier-economics page current* below. |
| 28 | - **Investing** (`funding.html` + `funding.json` + `waterfall.js`) — the |
| 29 | funding model (`docs/funding.md` — the repo is private; this page is what |
| 30 | makes the model public) and its open ledger. The JSON is **committed data** |
| 31 | like economics — empty (`adopted: false`) until the model passes legal + |
| 32 | tax review and a series deliberately opens; thereafter every series, note |
| 33 | (pseudonymous ids only), and quarterly waterfall computation publishes |
| 34 | here. `waterfall.js` is the algorithm's **normative reference |
| 35 | implementation**, served from the same directory so the page's interactive |
| 36 | simulator imports and runs the very file it publishes — an audit is a |
| 37 | re-execution (Node-side consumers use the `tools/waterfall.mjs` re-export; |
| 38 | vectors `test/waterfall-vectors.json`). |
| 39 | - **The carbon tally** (`carbon.html` + `carbon.json`) — the running estimate |
| 40 | of the CO₂e cost of the LLM tokens spent *building* Eleven, as a cumulative |
| 41 | chart + ledger with everyday equivalents. Unlike the other two feeds, the |
| 42 | JSON is **committed data** (`web/status/carbon.json`, accrued by |
| 43 | `tools/carbon-tally.mjs` — see *Keeping the carbon tally accruing* below); |
| 44 | the box timer just republishes it with the pages. Methodology + error bars: |
| 45 | `docs/comparison-carbon.md`. |
| 46 | - **Contributors** (`contributors.html` + `contributors.json`) — who builds |
| 47 | Eleven, humans and AI models on one metric: humans as GitHub commit authors, |
| 48 | models as `Co-Authored-By` trailers in git history. Committed data like the |
| 49 | carbon tally (`web/status/contributors.json`, refreshed by |
| 50 | `tools/contributors.mjs`); republished as-is. **Contributors own their row**: |
| 51 | bios seed from the GitHub profile, then edits to the committed JSON win — |
| 52 | customise your `bio` (or set `"hidden": true` to leave the index) by PR, and |
| 53 | the tool preserves those fields on every refresh. |
| 54 | - **The security ledger** (`security.html` + `security.json`) — posture, never |
| 55 | a live exploit map. Four sections: the E2EE commitment (the #550 Part 12 |
| 56 | invariants in member language), dependency hygiene (govulncheck against the |
| 57 | exact shipped binary — **status + count only**, see the disclosure rule |
| 58 | below), box/service hardening as a checklist of facts, and the deliberate |
| 59 | architectural edges (link previews, GIF search, plaintext branding, |
| 60 | bots-as-members, the content-blind-store trade), honestly named. The JSON is |
| 61 | **committed data** written at *build* time by `deploy/security-scan.sh` |
| 62 | (govulncheck needs the Go toolchain, which the box deliberately lacks); the |
| 63 | box republishes it as-is, and an absent `security.json` renders as "scan |
| 64 | pending" — never a fake clean. Vulnerability reports: |
| 65 | team@elevenmessenger.com (swap to a dedicated security@ mailbox when one exists). |
| 66 | - **The prompt registry** (`prompts.html` + the `prompts/` corpus in the |
| 67 | bucket) — the provenance record of AI-assisted work: whole Claude Code |
| 68 | session transcripts, credentials redacted and every redaction annotated in |
| 69 | place. Accrued data like the carbon tally, but far too large to commit: a |
| 70 | contributor runs `tools/prompt-registry.mjs` and uploads the corpus straight |
| 71 | to the bucket (see *The prompt registry* below). The page is a committed |
| 72 | viewer, republished with the others. |
| 73 | - **The SSH access ledger** (`ssh.html` + `ssh-sessions.json` + |
| 74 | `ssh-entries.json`) — every operator SSH session to the router box, on the |
| 75 | public record with a plain-language "why". Two halves: a standalone box-side |
| 76 | collector (`deploy/ssh-ledger-collect.sh`, its own 5-minute timer) publishes |
| 77 | session *metadata* from sshd's journal, and small PR-reviewed entry files |
| 78 | (`web/status/ssh/*.json`) carry the human reason. See *The SSH access |
| 79 | ledger* below. |
| 80 | - **The music ledger** (`music.html` + `music.json` + `music-YYYY-MM.json`) — |
| 81 | what was playing on contributor machines while Eleven was built: the |
| 82 | soundtrack companion to the prompt ledger. Committed data accrued per |
| 83 | session by `tools/music-ledger.mjs`, sharded by month (the page loads one |
| 84 | month at a time) with album art deduped per album and inlined as data URIs. |
| 85 | See *The music ledger* below. |
| 86 | |
| 87 | Pieces: generator `deploy/status-publish.sh` (box timer, mirrors |
| 88 | `deploy/fleet-backup.sh`), viewer pages `web/status/`, bucket + IAM |
| 89 | `infra/prod/status.tf`, contract test `test/status-contract.test.mjs`. All the |
| 90 | pages share one stylesheet, `web/status/status.css` (the design tokens and |
| 91 | common components — tiles, bar lists, notes, footer), and one masthead+nav |
| 92 | builder, `web/status/status-nav.js`: each page ships two empty mounts the |
| 93 | builder fills, grouping the dashboards (Operations / Money / Ledgers / Plan — |
| 94 | collapsed behind a Menu disclosure on narrow screens) and marking the current |
| 95 | page from the URL, so the nav can't drift between pages. (`milestones.html` — |
| 96 | the curated timeline of key moments, each entry deep-linking its ledger |
| 97 | session via `prompts.html#s=<id>` — joined the nav under *Plan* when the nav |
| 98 | was grouped; before that it was a nav-orphaned sub-page of the prompt ledger. |
| 99 | Still hand-appended as moments land, never generated.) Each page keeps only a |
| 100 | small inline `<style>` for its own bits. Both shared files are same-origin in |
| 101 | the status bucket (no third-party request — the privacy line holds), and the |
| 102 | publish script uploads them *before* the pages so a page is never served |
| 103 | ahead of its styles or its nav. |
| 104 | |
| 105 | ## The privacy rules (non-negotiable) |
| 106 | |
| 107 | Everything published is public, forever (the bucket is versioned). The |
| 108 | aggregation boundary lives in the generator and holds by construction: |
| 109 | |
| 110 | - **Fleet: aggregates only.** Counts, version strings (already public via the |
| 111 | signed release feed), booleans. **Never a hostname** — member spaces |
| 112 | obviously, but also `test-*` canaries; the simple rule is *no hostnames at |
| 113 | all*. Never per-space member/message counts, never anything keyed by space. |
| 114 | - **Capacity + hosts: abstract band words only** (Paul's call: bucketed words, |
| 115 | not needles). `capacity.level` is `plenty` / `getting-full` / `at-capacity`; |
| 116 | a box's `load` is `comfortable` / `busy` / `critical`. **Never** remaining |
| 117 | slot counts, free GB, percentages, or cap values — the caps and ratios are |
| 118 | read internally to pick the word and die in the generator. Host labels are |
| 119 | whimsical name-action pairs hashed from the machine id (`ada-stargazing`) — stable per box; never peer URLs, IPs, hostnames, or the machine id itself. |
| 120 | A box's optional `region` is coarse by design: an operator-set continent + |
| 121 | country pair (e.g. `EU · Germany`) — never a city, datacenter, IP, or |
| 122 | hostname; unset, the field is omitted (never guessed). The local box (the |
| 123 | one running the publisher) reads `STATUS_BOX_REGION`; a mesh peer publishes |
| 124 | its own via `LCHAT_BOX_REGION` (set in its own `/data/router.env`, read by |
| 125 | the router binary itself) — two different env vars on two different code |
| 126 | paths, since a peer's region rides the mesh stats side-channel, never the |
| 127 | publisher script. |
| 128 | - **Books: aggregated income only** (decision #19 in #550: income publishes |
| 129 | aggregated, never per-payer; invariant 12: no PII in our stores). Never |
| 130 | customer ids, emails, subscription ids, per-payer lines, or timestamps of |
| 131 | individual payments. |
| 132 | - **Economics: the cost model, never live utilization.** Prices, storage |
| 133 | allowances, unit rates, and per-tier estimates are fleet-level constants, |
| 134 | published on purpose. Two source-doc numbers deliberately do NOT publish: |
| 135 | the per-box instance cap and the box's disk size — either one, combined |
| 136 | with the fleet feed's public instance count, would reconstruct the |
| 137 | remaining-slot needle the capacity band words exist to hide. Keep the slot |
| 138 | rate's denominator qualitative ("full instance capacity"), never the cap. |
| 139 | - **Expenses: itemized by design** — decision #19's other half: our spend is |
| 140 | ours to disclose, so it publishes one line per provider with the real |
| 141 | amount, in that provider's *own billing currency* (never FX-converted — no |
| 142 | exchange rate to misstate). Still never account ids, invoice ids, workspace |
| 143 | ids, or resource-level lines — provider word + amount + currency is the |
| 144 | entire schema, and the contract test allowlists it. A provider that can't |
| 145 | be honestly resolved is omitted, never zeroed or guessed. Disbursements |
| 146 | belong to the future ledger repo, not these dashboards. |
| 147 | - **Contributors: public GitHub identity only, owned by its subject.** Login, |
| 148 | display name, avatar, profile link, commit count — all already public on |
| 149 | GitHub — plus a bio the contributor writes. Never emails (the contract test |
| 150 | greps), never private activity. Anyone listed can hide themselves with |
| 151 | `"hidden": true` in the committed JSON. |
| 152 | - **Carbon: aggregate token counts and kg estimates only.** Never per-session |
| 153 | data, never prompt or transcript content in *this feed* — the tally reads |
| 154 | token *usage numbers* from local transcripts and publishes only their sums. |
| 155 | (Transcript *content* publishes only through the prompt registry, below, |
| 156 | under its own explicitly different bar.) |
| 157 | - **The prompt registry publishes transcripts on purpose — its bar is |
| 158 | credentials-only** (Paul's decision, 2026-07-12; it supersedes the |
| 159 | no-hostnames rule for this feed alone). Transcripts are a contributor's own |
| 160 | working record, published *whole* as provenance: hostnames and ops narrative |
| 161 | stay (instance hostnames are already public record via Certificate |
| 162 | Transparency logs), while every secret, token, key, and the gitignored |
| 163 | CLAUDE.local.md runbook are redacted with an in-place annotation saying why. |
| 164 | The build is fail-closed: `tools/prompt-registry.mjs` re-scans its own |
| 165 | output with the same detectors and refuses to emit the corpus on any hit. |
| 166 | Publishing a machine's registry is **that contributor's choice** — the tool |
| 167 | only reads the machine it runs on, and nothing here auto-collects anyone's |
| 168 | transcripts. Each contributor also keeps a **small personal redaction |
| 169 | budget** (`--personal <file>`: literal strings or `/regex/`, one per line — |
| 170 | a `/regex/` line also matches itself literally, so a path-shaped literal is |
| 171 | never silently reinterpreted as a pattern): |
| 172 | things removed for personal reasons, no justification owed. They publish as |
| 173 | `⟦redacted: personal⟧` and are **counted per session like every other |
| 174 | category** — the public count is the budget's honesty mechanism; a row with |
| 175 | three personal redactions reads very differently from one with three |
| 176 | hundred. |
| 177 | - **Sessions are opt-in by default, with two session-level escape hatches** |
| 178 | (`--sessions <file>`): `withhold <id> <reason>` publishes a **tombstone** |
| 179 | (date, prompt count, reason — no content), `omit <id>` leaves only the |
| 180 | aggregate omitted count. Reasons vocabulary: `personal-safety` (sharing |
| 181 | would cause or attract harm to a person), `security` (would reveal secrets |
| 182 | whose exposure could cause harm, incl. national security), `third-party` |
| 183 | (the session centres on an identifiable person who didn't opt into the |
| 184 | ledger — the blanket email redaction hides addresses, but a session *about* |
| 185 | someone withholds whole), `beyond-intent` (consequences beyond revealing |
| 186 | the human's intent — the thing transparency is for). **One case is mandatory: centralised-service security.** A session |
| 187 | whose content could compromise the push relay, home, cloud/billing, |
| 188 | connect, the tunnel gateway, or the router box is withheld *whole* — |
| 189 | credential redaction is not enough when the narrative itself maps the |
| 190 | defences (where key material lives, how backups restore, what the sandbox |
| 191 | permits). Sweep before every accrual and review the hits: |
| 192 | |
| 193 | ```sh |
| 194 | find ~/.claude/projects -maxdepth 2 -name '*.jsonl' -path '*<repo>*' -print0 | |
| 195 | while IFS= read -r -d '' f; do |
| 196 | n=$(grep -c -E 'AGE-SECRET-KEY|secrets-vault|sops -d |vault-bundle|/data/push\.env|LCHAT_APNS_KEY|control\.token|litestream\.env|sk_live_|whsec_|LCHAT_STRIPE_SECRET|/data/acme|LCHAT_DNSIMPLE_TOKEN|LoadCredential' "$f") |
| 197 | [ "${n:-0}" -ge 3 ] && echo "withhold $(basename "$f" .jsonl) security" |
| 198 | done >> <your --sessions file> |
| 199 | ``` |
| 200 | |
| 201 | The ≥3 threshold is deliberately conservative — over-withholding is |
| 202 | recoverable (drop the line, rebuild), publication to a versioned public |
| 203 | bucket is not. |
| 204 | - **Security: status + count, never a name — the disclosure line.** The |
| 205 | ledger publishes posture: dependency findings ship as `clean` or "N |
| 206 | advisories under remediation" — **never** the CVE/GHSA/OSV ids, affected |
| 207 | packages, versions, or code locations of anything *unpatched* (naming an |
| 208 | open hole is a responsible-disclosure violation and an attacker roadmap; |
| 209 | once patched it's history and may be named, e.g. in a changelog). No secret |
| 210 | inventory, no internal component versions beyond the build sha the signed |
| 211 | release feed already publishes, and nothing that locates a box — hardening |
| 212 | publishes as facts ("firewall default-deny", "SSH key-only"), never |
| 213 | addresses. The full govulncheck report stays on the operator's terminal; |
| 214 | the summarizer refuses to write an output containing an id-shaped string, |
| 215 | and the contract test greps for every id grammar. |
| 216 | - **The SSH access ledger: metadata only, published on close.** Who (operator |
| 217 | handle via public-key fingerprint — public key material, safe to publish), |
| 218 | when, connection count, and a wireguard-or-public source word. **Never |
| 219 | commands, command output, or session content** — the collector doesn't even |
| 220 | read them — and client IPs are read only to pick the source word, dying in |
| 221 | the collector like the capacity numbers do. Rows publish only after a |
| 222 | session has been quiet ≥20 minutes, so the page is never a live |
| 223 | "operator connected right now" beacon. The human why-entries hold the |
| 224 | prompt registry's redaction bar: what, not how; never credentials or |
| 225 | emails; security-sensitive sessions publish as `(withheld: security)` |
| 226 | rather than vanishing (see *The SSH access ledger* below). |
| 227 | - Adding a field? It must be an aggregate over the whole fleet / all payers, |
| 228 | or it doesn't ship (the prompt registry, a whole-transcript feed by design, |
| 229 | is the one standing exception). The `--dry-run` self-check and the contract |
| 230 | test both fail if a fixture hostname or per-payer id reaches the output. |
| 231 | |
| 232 | ## What's in the JSONs |
| 233 | |
| 234 | `fleet.json`: |
| 235 | |
| 236 | ```json |
| 237 | { |
| 238 | "generated_at": "2026-07-11T13:00:00Z", |
| 239 | "instances": { "total": 6, "systemd": 2, "hibernating": 4, "awake": 2, "asleep": 2 }, |
| 240 | "hosts": { "count": 1, "boxes": [ { "label": "ada-stargazing", "region": "EU · Germany", "always_on": 2, "awake": 2, "asleep": 2, "load": "comfortable" } ] }, |
| 241 | "capacity": { "level": "plenty" }, |
| 242 | "versions": [ { "version": "v100", "count": 3 } ], |
| 243 | "services": { "home": true, "push": true, "cloud": true, "connect": true }, |
| 244 | "replication": { "healthy": true, "lag_seconds": 42 } |
| 245 | } |
| 246 | ``` |
| 247 | |
| 248 | `awake`/`asleep` split the hibernating set by whether each host's unix socket |
| 249 | is currently *bound and listening* (`ss -xlH`) — socket-file existence is not |
| 250 | sound (instances never unlink on exit, so slept hosts leave stale files) and |
| 251 | dialing the socket would touch the instance; the kernel socket table is |
| 252 | side-effect-free. `lag_seconds` approximates replication lag as seconds since |
| 253 | the litestream replicator last logged (null when unknown). Both `awake` and |
| 254 | `asleep` are null if `ss` is unavailable — unknowns publish as null, never as |
| 255 | guesses. |
| 256 | |
| 257 | `capacity.level` is the worst of four internally-measured signals, each banded |
| 258 | at the same lines (**plenty** below 70%, **getting-full** 70–90%, |
| 259 | **at-capacity** above 90%): the materialized-instance ratio against |
| 260 | `LCHAT_MAX_INSTANCES`, the registered-space ratio against |
| 261 | `LCHAT_MAX_REGISTERED` (both mirror `controld.go`'s `atCapacity()`, including |
| 262 | its hibernate-creates branch and code defaults — 40 / 5 GB floor / 10×), disk |
| 263 | proximity to the `LCHAT_MIN_FREE_GB` floor (at or below the floor is |
| 264 | at-capacity outright), and memory utilization (`/proc/meminfo`). The caps are |
| 265 | read from `/data/control.env`. A box's `load` word bands memory utilization |
| 266 | the same way (`comfortable` / `busy` / `critical`, null when unknown). Only |
| 267 | the words publish — never the ratios, GB, or caps behind them. `hosts` |
| 268 | aggregates across every live mesh box, not just the one running the |
| 269 | publisher: each member self-publishes its own capacity snapshot as a sibling |
| 270 | object next to its membership heartbeat (a slower, one-minute cadence — this |
| 271 | is visibility, not coordination), and the publisher folds every live peer's |
| 272 | snapshot into the array alongside its own. Each box's own awake/asleep count |
| 273 | is filtered to the hosts it currently owns by the mesh's placement hash, so a |
| 274 | host re-homed to a peer is never reported as "asleep" by more than one box — |
| 275 | `instances.awake`/`asleep` at the top level sum every box's own count, which |
| 276 | is exactly why that filtering matters. Single-box behavior (no live mesh |
| 277 | peer) is unchanged: `hosts.count` stays `1`, byte-identical to before the |
| 278 | mesh existed. Per-box `region` rides the same self-published snapshot |
| 279 | (`LCHAT_BOX_REGION`, see the privacy rules above) — a peer with no region set |
| 280 | simply omits the field, same as the local box's own `STATUS_BOX_REGION` seam. |
| 281 | |
| 282 | `books.json` (or `{"available": false, "reason": "…"}` when billing isn't |
| 283 | configured / Stripe is unreachable): |
| 284 | |
| 285 | ```json |
| 286 | { |
| 287 | "generated_at": "2026-07-11T13:00:00Z", |
| 288 | "available": true, |
| 289 | "currency": "eur", |
| 290 | "month": "2026-07", |
| 291 | "subscriptions": { "active": 3, "by_tier": [ { "tier": "friends", "count": 2 } ] }, |
| 292 | "mrr_cents": 1600, |
| 293 | "collected_month_cents": 1200, |
| 294 | "connect": { "gmv_month_cents": 1450, "commission_month_cents": 145 }, |
| 295 | "expenses": { "month": "2026-07", "providers": [ |
| 296 | { "provider": "aws", "month_to_date_cents": 1234, "currency": "usd" }, |
| 297 | { "provider": "hetzner", "monthly_cents": 2094, "currency": "eur" }, |
| 298 | { "provider": "anthropic", "month_to_date_cents": 250, "currency": "usd" }, |
| 299 | { "provider": "claude.ai", "to_date_cents": 229000, "currency": "eur" }, |
| 300 | { "provider": "dnsimple", "month_to_date_cents": 2450, "currency": "usd" }, |
| 301 | { "provider": "apple", "yearly_cents": 9900, "currency": "eur" } |
| 302 | ] } |
| 303 | } |
| 304 | ``` |
| 305 | |
| 306 | MRR sums each active subscription item's `unit_amount × quantity` normalized |
| 307 | to a month (yearly ÷ 12, etc.), EUR items only. Collected-this-month is the |
| 308 | gross of positive `charge`/`payment` balance transactions since the month |
| 309 | started (refunds and payouts excluded — it's gross income, not net cash). |
| 310 | |
| 311 | `connect` is spaces selling to their own members through Stripe Connect |
| 312 | (in-space subscriptions charge directly on the space admin's connected |
| 313 | account, with an application fee to the platform — `subscriptions.go`). |
| 314 | Those charges never appear in the two reads above, so the fee list is the |
| 315 | window: `commission_month_cents` sums this month's application-fee amounts |
| 316 | (the platform's actual income from these sales), and `gmv_month_cents` sums |
| 317 | each fee's expanded charge amount — the gross volume the spaces sold. The |
| 318 | GMV is published deliberately (Paul's call: if we know it, it's public) but |
| 319 | it is the spaces' money, not Eleven's — which is why neither figure folds |
| 320 | into `collected_month_cents`. Same gross semantics as the rest: positive |
| 321 | EUR amounts, refunds don't net. |
| 322 | |
| 323 | `expenses` is infrastructure spend, itemized per provider — the deliberate |
| 324 | mirror image of income's aggregation (see the privacy rules above). Four |
| 325 | provider shapes exist, exactly one amount key each: metered providers carry |
| 326 | `month_to_date_cents`, flat monthly subscriptions `monthly_cents`, flat |
| 327 | yearly ones `yearly_cents`, and a cumulative operator-attested figure |
| 328 | `to_date_cents` — each as billed, in its own billing currency (a yearly fee |
| 329 | is published yearly, never divided into a fake monthly; a lifetime figure is |
| 330 | published to date, never smeared into months). |
| 331 | |
| 332 | - **aws** — month-to-date UnblendedCost from Cost Explorer (a us-east-1 |
| 333 | global endpoint, regardless of where the bucket lives). CE bills ~$0.01 |
| 334 | per request and the publisher runs every 5 minutes, so the result is |
| 335 | cached in `$DATA_DIR/status-costs-cache.json` (per-provider entries with |
| 336 | `fetched_at`) and refetched only when older than |
| 337 | `STATUS_COSTS_MAX_AGE_HOURS` (default 6) or from a previous month. A |
| 338 | failed refetch serves the same-month stale cache; failure with no cache |
| 339 | omits the entry. |
| 340 | - **hetzner** — the flat-rate Hetzner Cloud servers. Hetzner has no billing |
| 341 | API, but the servers API prices each server: with `STATUS_HETZNER_TOKEN` |
| 342 | (a **read-only** Cloud API token) the generator computes the account's |
| 343 | monthly server cost from `GET /v1/servers` (pagination followed via |
| 344 | `meta.pagination.next_page`) — each server's embedded |
| 345 | `server_type.prices[]` entry for its own location, `price_monthly.gross` |
| 346 | summed, rounded once at the total. The prices carry no currency field, so |
| 347 | the published `eur` is the documented constant (Hetzner bills in EUR). |
| 348 | The figure is **server compute only** — primary IPs, backups, volumes, |
| 349 | and traffic overages price via the separate `/pricing` endpoint and are |
| 350 | deliberately not included. Same cache treatment as AWS but with 24h |
| 351 | default freshness (a flat price list changes ~never; |
| 352 | `STATUS_COSTS_MAX_AGE_HOURS` still overrides). Exactly one price entry |
| 353 | must match per server and every amount must parse cleanly, or the fetch |
| 354 | fails — and a failed figure is omitted, never misstated. The operator-set |
| 355 | constant `STATUS_HETZNER_MONTHLY_CENTS` (EUR) **overrides the API** when |
| 356 | set — operator truth wins; neither set omits the provider. |
| 357 | - **anthropic** — the bot sidecars' API spend: month-to-date from the Admin |
| 358 | API cost report (`GET /v1/organizations/cost_report`, daily buckets summed; |
| 359 | amounts arrive as decimal cents, USD). Authenticated with |
| 360 | `STATUS_ANTHROPIC_ADMIN_KEY` — an *Admin* key with read-only usage scope, |
| 361 | not the sidecars' completion key. Same cache treatment as AWS. |
| 362 | - **claude.ai** — the claude.ai subscriptions: no API for them, and the |
| 363 | honest figure the operator can attest is **lifetime spend**, not a |
| 364 | monthly — so the operator-set constant `STATUS_CLAUDE_TO_DATE_CENTS` |
| 365 | (EUR) publishes as `to_date_cents`, the cumulative shape. It's an |
| 366 | estimate by nature, which is why the page labels it |
| 367 | "claude.ai · to date (approx.)". |
| 368 | - **dnsimple** — the product domains (elevenmessenger.com, 11ch.at, |
| 369 | 11factor.org): month-to-date from the billing-charges API |
| 370 | (`GET https://api.dnsimple.com/v2/{account}/billing/charges` with |
| 371 | `start_date`/`end_date`; `collected` charges summed — refunds are not |
| 372 | spend). Amounts arrive as decimal dollar strings with **no currency field**, |
| 373 | so the published `usd` is the documented constant (DNSimple's price book is |
| 374 | USD), not parsed. Authenticated with `STATUS_DNSIMPLE_TOKEN` + |
| 375 | `STATUS_DNSIMPLE_ACCOUNT` — an account API token, read-only use here, never |
| 376 | echoed. Same cache treatment as AWS; more than one page of charges in a |
| 377 | month counts as failure (a partial sum never publishes). |
| 378 | - **apple** — the Apple Developer Program membership: fixed yearly with no |
| 379 | billing API, so the operator-set constant `STATUS_APPLE_YEARLY_CENTS` |
| 380 | (EUR), published as billed (`yearly_cents`) — never divided into a fake |
| 381 | monthly. |
| 382 | |
| 383 | No providers → the whole `expenses` object is omitted. Expenses ride the |
| 384 | degraded income-unavailable shape too — a Stripe outage doesn't hide spend |
| 385 | the generator can still honestly state. |
| 386 | |
| 387 | `carbon.json` (committed at `web/status/carbon.json`, republished as-is): an |
| 388 | `entries[]` ledger of measurement periods — `period_start`/`period_end`, |
| 389 | `source` (which contributor machine, as an anonymous name-pair label — same |
| 390 | no-hostnames rule as the fleet's box labels), `basis` |
| 391 | (`measured` | `extrapolated`), `sessions`, `output_tokens`, |
| 392 | `fresh_input_tokens`, `cache_read_tokens`, `kg_low`/`kg_central`/`kg_high` — |
| 393 | plus the conversion `factors_mg_co2e_per_output_token`, a `totals` block, and |
| 394 | a `not_counted` honesty string. The contract test pins the arithmetic: totals |
| 395 | must equal the ledger sum, each entry's kg must derive from its tokens × the |
| 396 | factors, and each source's periods must be chronological and non-overlapping. |
| 397 | |
| 398 | ## Refreshing the contributors index |
| 399 | |
| 400 | `node tools/contributors.mjs --write` (needs `gh` authenticated) refreshes |
| 401 | commit counts, avatars (embedded as data URIs — the pages allow no external |
| 402 | hosts), and profile links, preserving `bio`/`name`/`hidden`. Run it when the |
| 403 | index looks stale — after a burst of merges, or when a new contributor lands — |
| 404 | commit, and copy `web/status/contributors.{html,json}` to the box's |
| 405 | status-pages dir. No hook nudges for this one: unlike the carbon tally there |
| 406 | is no pruning window, so a stale count costs nothing and catches up on the |
| 407 | next refresh. |
| 408 | |
| 409 | ## Keeping the tier-economics page current |
| 410 | |
| 411 | `web/status/economics.json` is a hand-derived snapshot of |
| 412 | `docs/tier-economics.md` — the formulas are the contract, the numbers a |
| 413 | snapshot. Whenever prices, storage allowances, the free-space quota, or the |
| 414 | input rates change (a Stripe price-book edit, a provider rate move, a quota |
| 415 | default actually shipping), update BOTH in the same change: re-derive the |
| 416 | doc's tables, mirror them into the JSON (and the page's static prose if a |
| 417 | reading shifted), and run `test/status-contract.test.mjs` — it pins net |
| 418 | against the Stripe fee line, profit/margin consistency, and that every |
| 419 | tier's allowance stays below its break-even. Then copy |
| 420 | `web/status/economics.{html,json}` to the box's `/opt/lchat/status-pages/`; |
| 421 | the next 5-minute publish uploads them. No timer regenerates this feed — a |
| 422 | stale snapshot is the failure mode, so treat a pricing change as incomplete |
| 423 | until the snapshot moves with it. |
| 424 | |
| 425 | ## Keeping the carbon tally accruing |
| 426 | |
| 427 | The tally's raw material is the local Claude Code transcripts on **each |
| 428 | contributor's machine**, and those are **pruned after roughly a week** — a |
| 429 | missed week is lost to the ledger and has to be extrapolated by hand (like |
| 430 | the tally's first entry). Every contributor machine accrues its own entries |
| 431 | under a stable anonymous `source` label (hashed from user@host; the tool |
| 432 | prints yours; override with `LCHAT_CARBON_SOURCE`). |
| 433 | |
| 434 | **The nudge is automatic.** A `SessionStart` hook in `.claude/settings.json` |
| 435 | runs `node tools/carbon-tally.mjs check` (fast — no transcript sweep): when |
| 436 | this machine's newest entry is older than 3 days (`LCHAT_CARBON_STALE_DAYS`), |
| 437 | the session is told to accrue. The accrual itself: |
| 438 | |
| 439 | 1. `node tools/carbon-tally.mjs --write` (from the repo root) — sweeps this |
| 440 | machine's transcripts and appends a `measured` entry. Cutoffs are |
| 441 | **per-source** and consult both the working tree and `origin/main`, so |
| 442 | repeated runs and parallel machines never double-count. |
| 443 | 2. Commit the updated `web/status/carbon.json` — prefer a tiny standalone PR, |
| 444 | merged promptly: two long-lived branches accruing the same window is the |
| 445 | one race the cutoffs can't see, and the contract test fails main on the |
| 446 | overlap (fix: drop the younger duplicate entry, re-run `--write`). |
| 447 | 3. `./deploy/ops pages push` — copies the committed `carbon.json` to the box's |
| 448 | `/opt/lchat/status-pages/` (add `carbon.html` as an argument if the page |
| 449 | changed); the next 5-minute publish uploads it. |
| 450 | |
| 451 | The conversion factors live in the JSON so a recalibration (better public |
| 452 | data) is one committed change; if the factors change, recompute the whole |
| 453 | ledger's kg columns rather than mixing eras (`docs/comparison-carbon.md` |
| 454 | carries the sources and the "as of" date). |
| 455 | |
| 456 | ## The prompt registry |
| 457 | |
| 458 | The provenance record of AI-assisted work: whole Claude Code session |
| 459 | transcripts, published at `prompts.html` with credentials redacted and every |
| 460 | redaction annotated in place (the redaction bar and its rationale: *The |
| 461 | privacy rules* above). Like the carbon tally, the raw material is the local |
| 462 | transcripts that **prune after roughly a week** — accrue before they age out |
| 463 | or the sessions are lost to the registry. |
| 464 | |
| 465 | Where it stands today: **Paul's sessions, published by his own choice.** The |
| 466 | direction is that every contribution built with AI includes its prompt |
| 467 | transcript, but that becomes a *requirement* on other contributors only once |
| 468 | agreed with them — don't encode it as a rule anywhere until that conversation |
| 469 | has happened. |
| 470 | |
| 471 | **Bucket layout is per-contributor** — corpora can never clobber each other: |
| 472 | |
| 473 | ``` |
| 474 | prompts/contributors.json ← the manifest: [{ slug, name }] the page merges |
| 475 | prompts/<slug>/index.json ← one prefix per contributor (slug = kebab-cased |
| 476 | prompts/<slug>/prompts.json --contributor name); everything the tool |
| 477 | prompts/<slug>/issues.json builds lands under the contributor's own slug |
| 478 | prompts/<slug>/<session>.jsonl |
| 479 | prompts/<slug>/<session>.sum.json |
| 480 | ``` |
| 481 | |
| 482 | The tool merges this machine's entry into the *live* manifest (a public read — |
| 483 | no creds needed) before writing contributors.json, so uploading it never drops |
| 484 | other contributors. If it prints the OFFLINE warning, don't upload the |
| 485 | manifest from that build — re-run online. |
| 486 | |
| 487 | **How another contributor joins** (their choice, per the policy above): run |
| 488 | the tool on their machine with `--contributor "Their Name"` (plus their own |
| 489 | `--personal` / `--sessions` files and the security sweep — same rules for |
| 490 | everyone), then publish their prefix one of two ways: |
| 491 | |
| 492 | - **With bucket creds** (mint a scoped IAM key allowing PutObject on |
| 493 | `prompts/<their-slug>/*` + `prompts/contributors.json` only): run the two |
| 494 | `aws s3` commands the tool prints — `sync` of their own prefix, `cp` of the |
| 495 | manifest. |
| 496 | - **Without creds**: `tar czf corpus.tgz -C <out>/prompts <slug> |
| 497 | contributors.json` and hand the tarball to an operator, who extracts and |
| 498 | runs the same two commands from the router box (`set -a; . /data/status.env; |
| 499 | set +a` for the creds). The operator never edits the corpus — the |
| 500 | contributor's build is what publishes, fail-closed scan and all. |
| 501 | |
| 502 | **The nudge is automatic here too.** The same `SessionStart` hook runs |
| 503 | `node tools/prompt-registry.mjs check` (fast — reads the last build's |
| 504 | `generated_at`, no sweep or redaction): when this machine's corpus is older |
| 505 | than 3 days (`LCHAT_REGISTRY_STALE_DAYS`) *and* transcripts have been touched |
| 506 | since, the session is told to accrue. A machine that has never built a corpus |
| 507 | at the default `--out` (`~/.cache/eleven-registry`) hears nothing — publishing |
| 508 | is opt-in (see above), so the nudge follows that choice, never precedes it. |
| 509 | The check can't run the accrual itself: the security sweep and the withhold |
| 510 | decisions are judgment calls, so a session acts on the nudge the way it acts |
| 511 | on the carbon one. |
| 512 | |
| 513 | Accruing a machine's sessions: |
| 514 | |
| 515 | 1. `node tools/prompt-registry.mjs --out <dir>` (from the repo root) — sweeps |
| 516 | this machine's transcripts for this repo (worktree sessions included), |
| 517 | redacts, and writes `<dir>/prompts/` (the redacted JSONL corpus + |
| 518 | `index.json`). The build is **fail-closed**: outputs are re-scanned with |
| 519 | the same detectors and the corpus is not emitted on any hit. Publishing is |
| 520 | whole-corpus-per-machine for now; per-session accrual can come later if |
| 521 | two machines ever need to merge into one index. |
| 522 | 2. **Summarise new sessions before publishing — dispatch haiku subagents.** |
| 523 | A keyless build carries prior summaries/gists forward but leaves *new* |
| 524 | sessions bare (the index falls back to first prompts), so don't upload |
| 525 | straight from step 1. Fan haiku subagents out over the **redacted** build |
| 526 | output only — never the raw transcripts: one-line session summaries from |
| 527 | each new session's prompts (≤90 chars, ending with the session's most |
| 528 | relevant GitHub URL — THE SUMMARY RULE in the tool's header), and ≤80-char |
| 529 | reply gists for replies missing from `<id>.sum.json` (keys are |
| 530 | `parsedLine:block`, reconstructable from the redacted JSONL: assistant text |
| 531 | blocks >200 chars). Write the results into the out dir (`index.json` |
| 532 | `summary` fields + merged `<id>.sum.json`), then **re-run step 1**: the |
| 533 | rebuild carries them through today's redaction rules and the fail-closed |
| 534 | verifier, and hashes exactly what publishes. |
| 535 | 3. Upload the corpus straight to the status bucket (it's far too large to |
| 536 | commit — a quarter GB and growing). The tool prints the exact commands: |
| 537 | `aws s3 sync <dir>/prompts/<slug> s3://eleven-status-<account>/prompts/<slug> |
| 538 | --delete --cache-control max-age=300`, then `aws s3 cp` the manifest (run |
| 539 | on the box with the status-publisher creds — `set -a; . /data/status.env; |
| 540 | set +a` — or anywhere with equivalent PutObject rights). Sync **only your |
| 541 | own prefix**; the manifest upload is a `cp`, never a sync with `--delete` |
| 542 | at the `prompts/` root, which would erase other contributors. |
| 543 | 4. Commit the hash ledger entry (the tool prints the exact command): the |
| 544 | corpus's `manifest.sha256.json` goes to the **public** |
| 545 | `elevenmessenger/releases` repo at `prompt-registry/<slug>.json` via |
| 546 | `gh api -X PUT`. Its git history is the registry's tamper-evidence — a |
| 547 | few KB per accrual, and any after-the-fact change to a published |
| 548 | transcript shows as a dated hash change. (The manifest also uploads with |
| 549 | the corpus, so the bucket is self-describing.) |
| 550 | 5. `prompts.html` itself is a committed viewer page: copy it to the box's |
| 551 | `/opt/lchat/status-pages/` like the other pages; the 5-minute publish |
| 552 | re-uploads it. |
| 553 | |
| 554 | ## The SSH access ledger |
| 555 | |
| 556 | Every operator SSH session to the router box, on the public record with a |
| 557 | plain-language reason — the prompt registry's idea applied to hands-on-the-box |
| 558 | access, prompted by the WireGuard hardening (public port 22 closed; operators |
| 559 | connect over the wg admin tunnel, `172.22.22.0/24`). Two halves: |
| 560 | |
| 561 | - **Automatic truth — the box-side collector.** `deploy/ssh-ledger-collect.sh` |
| 562 | (installed at exactly `/opt/lchat/ssh-ledger-collect.sh`, run every 5 minutes |
| 563 | by `lchat-ssh-ledger.timer`) reads *new* sshd journal events via a persisted |
| 564 | cursor (`/data/ssh-ledger/` — survives journald rotation), parses accepted |
| 565 | publickey logins, and groups them into sessions per (user, key fingerprint): |
| 566 | a reconnect within 15 minutes joins the session, and a session publishes |
| 567 | only after 20 quiet minutes — **publish-on-close**, so the page never |
| 568 | advertises a live connection. Rows append to the cumulative |
| 569 | `ssh-sessions.json` (`{id, operator, fingerprint, start, end, connections, |
| 570 | source}`; `source` is `wireguard` or `public`), deduped on id across |
| 571 | re-runs. Operator handles come from the committed fingerprint map |
| 572 | `web/status/ssh-operators.json`; an unmapped fingerprint publishes as |
| 573 | `unknown key` — deliberately render-worthy, an unexplained key is the |
| 574 | signal the page exists to surface. The collector is **deliberately |
| 575 | standalone** from `deploy/status-publish.sh` (it uploads its own outputs with |
| 576 | the same status-publisher creds, `/data/status.env`, on its own timer — so a |
| 577 | publish-script change never gates the ledger). |
| 578 | - **The human "why" — PR-reviewed entry files.** One JSON per entry in |
| 579 | `web/status/ssh/` (changelog-style — parallel PRs never conflict): |
| 580 | `{date, operator, window, why, refs}`. **The convention: any session, |
| 581 | Claude or human, that SSHes to the box adds an entry in the same or an |
| 582 | adjacent PR.** PR review is the redaction gate, holding the prompt |
| 583 | registry's bar: summaries are *what, not how*; never credentials or emails; |
| 584 | a security-sensitive session publishes as `"why": "(withheld: security)"` |
| 585 | rather than vanishing. Format + full convention: `web/status/ssh/README.md`. |
| 586 | The entries land on the box via the normal viewer-page scp step, and the |
| 587 | collector merges them into one `ssh-entries.json` so the page fetches |
| 588 | exactly two JSONs. |
| 589 | |
| 590 | `ssh.html` joins the halves client-side — same operator + same UTC date, |
| 591 | preferring an overlapping (else nearest) time window; one entry may cover |
| 592 | several sessions. **A session with no matching entry renders loudly as "no |
| 593 | summary recorded"** — not an error, the enforcement: unexplained access should |
| 594 | look unexplained. The page also carries the honest caveat in its policy card: |
| 595 | the box publishes its own ledger, so this is transparency, not tamper-proof |
| 596 | forensics (the versioned bucket keeps history of after-the-fact edits, but a |
| 597 | root attacker could stop or falsify the collector). |
| 598 | |
| 599 | Deploying it (the box side; needs `jq` — `apt install jq`): |
| 600 | |
| 601 | 1. **The collector, its units, and the viewer page ride the binary.** `lchat |
| 602 | bootstrap` (fresh box) or `sudo -n lchat bootstrap --collectors-only` |
| 603 | (refresh over WireGuard — no root shell) writes |
| 604 | `/opt/lchat/ssh-ledger-collect.sh`, its `lchat-ssh-ledger.{service,timer}` |
| 605 | units, and the viewer `ssh.html` into `/opt/lchat/status-pages/`, |
| 606 | `daemon-reload`s, and enables the timer. There is no root scp — the hardened |
| 607 | box has no root shell to copy files in, so the install path is the binary |
| 608 | itself. (`--collectors-only` skips the router entirely, so a refresh causes |
| 609 | no member-facing blip.) |
| 610 | 2. **Creds**: the collector uploads with the status-publisher creds in |
| 611 | `/data/status.env` (same key the status publisher uses) — see *Install |
| 612 | (router box)*. |
| 613 | 3. **The operator data JSON is NOT embedded** (it changes independently of the |
| 614 | binary): the fingerprint map `ssh-operators.json` and the PR-reviewed |
| 615 | why-entries (`web/status/ssh/*.json`) are collector/operator-produced and |
| 616 | land in `/opt/lchat/status-pages/` (and `…/ssh/`) via the operator's normal |
| 617 | status-page copy, not `bootstrap`. A session published before its entry |
| 618 | arrives shows "no summary recorded", and an unmapped fingerprint publishes |
| 619 | as `unknown key`, until the map/entry lands and the next collect pass runs. |
| 620 | 4. **Verify**: `systemctl list-timers lchat-ssh-ledger.timer`, |
| 621 | `journalctl -u lchat-ssh-ledger -n 20`, then load `ssh.html` (rows appear |
| 622 | only once a session has closed and gone 20 minutes quiet). |
| 623 | |
| 624 | A new operator key = add its fingerprint to `web/status/ssh-operators.json` by |
| 625 | PR *and* re-scp it, or their sessions publish as `unknown key`. Rotating the |
| 626 | box's SSH setup doesn't touch the ledger — it only reads the journal. |
| 627 | |
| 628 | ## The music ledger |
| 629 | |
| 630 | 🤖 What was playing on contributor machines while Eleven was built — the |
| 631 | soundtrack companion to the prompt ledger (`music.html` + |
| 632 | `music.json`, committed data like the carbon tally). A session logs |
| 633 | Apple Music locally (one TSV row per track change, sampled every 20s), and |
| 634 | the contributor accrues it with: |
| 635 | |
| 636 | ```sh |
| 637 | node tools/music-ledger.mjs <session.tsv> --listener "Full Name" |
| 638 | ``` |
| 639 | |
| 640 | The tool filters skips (a track must hold ≥60s to publish), resolves |
| 641 | song.link/album.link ids once via the iTunes Search API, and inlines album |
| 642 | art as data URIs — the page makes **no external requests** (same rule as |
| 643 | contributor avatars). Rows are attributed by the `ssh-operators.json` |
| 644 | naming, and each carries an honest choice word: a queue or radio names |
| 645 | itself; a bare catalog stream carries no signal because Music's scripting |
| 646 | interface can't distinguish a direct pick from autoplay. Publishing is |
| 647 | deliberate: run the tool, review the JSON diff (it's your music |
| 648 | history), commit by PR. |
| 649 | |
| 650 | 🤖 **Streaming (issue #1224).** `--stream` skips the release train entirely: |
| 651 | each track appends as a hash-chained entry to the `ledgers/music/` stream in |
| 652 | the status bucket (`tools/ledgerchain.mjs` is the chain spec, |
| 653 | `tools/ledger-verify.mjs music` walks and re-hashes it — no credentials |
| 654 | needed, the whole chain is public-read), album art goes up once per album as |
| 655 | a content-addressed blob, and the derived views the page reads |
| 656 | (`ledgers/music/music.json` + month shards) rebuild in place — live in |
| 657 | seconds. The page prefers the stream and falls back to the committed files. |
| 658 | Writes use the scoped `status-ledger-writer` key (`infra/prod/status.tf`: |
| 659 | PutObject under `ledgers/` only, no deletes — tamper-EVIDENT via the chain, |
| 660 | matching the SSH ledger's honesty about what it can prove). The generic |
| 661 | appender is `tools/ledger-append.mjs`; any future ledger streams the same |
| 662 | way. |
| 663 | |
| 664 | ## The fleet hardening dashboard |
| 665 | |
| 666 | 🤖 Per box in the mesh, whether SSH is locked down — on the public record |
| 667 | (`hardening.html` + `hardening.json`), the access ledger's transparency idea |
| 668 | one level down at the machines themselves. Prompted by the fleet growing a |
| 669 | second mesh box that could still have public SSH / root login open: an |
| 670 | unhardened box must be *visible*, not buried in an operator's head. Two halves, |
| 671 | mirroring the ledger's automatic-truth split: |
| 672 | |
| 673 | - **The local self-read (authoritative).** `deploy/hardening-posture.sh` |
| 674 | (installed at exactly `/opt/lchat/hardening-posture.sh`, run every 5 minutes |
| 675 | by `lchat-hardening.timer`, as root) reads the **local** box's own posture |
| 676 | from its OS: `ufw status` → `firewall_ssh_scope` (`wireguard-only` when port |
| 677 | 22 is allowed only over `wg0` and there is no bare public rule / |
| 678 | `internet-facing` the moment a bare `22/tcp … ALLOW … Anywhere` rule exists / |
| 679 | `unknown` when ufw is inactive or unreadable), `sshd -T` → `root_login` + |
| 680 | `password_auth`, `systemctl is-active unattended-upgrades` → `auto_updates`, |
| 681 | `ip link show wg0` → `wireguard`, and the ufw first line → `firewall_active`. |
| 682 | A box is the truth about *itself*. |
| 683 | - **The peer `:22` probe (a vantage-dependent hint).** The same run enumerates |
| 684 | the mesh — membership objects in the hibernation bucket |
| 685 | (`LCHAT_HIBERNATE_BUCKET`/`_PREFIX`/`_REGION` from `/data/router.env`, listed |
| 686 | under `<prefix>/mesh/members/`, joined to `<prefix>/mesh/members-stats/` for |
| 687 | each box's whimsical `label` + coarse `region`, exactly as |
| 688 | `status-publish.sh`'s `mesh_peer_boxes` reads them) — and probes each **peer** |
| 689 | box's port 22 from the outside: a connect-and-banner-grab, never a login. |
| 690 | Because an external probe only sees the fleet from the probing box's vantage |
| 691 | (a firewall may admit one box and deny another), it is read conservatively: |
| 692 | only a returned `SSH-…` banner is a confident **exposed**; a clean refusal or |
| 693 | a connect timeout is **not reachable**; a completed connection with no SSH |
| 694 | banner — or anything that isn't a clean refusal — is **inconclusive**, a warn |
| 695 | state, never a pass. The self box uses its *local* read (it never self-probes |
| 696 | `:22` — that would hairpin through its own NAT). |
| 697 | |
| 698 | `hardening.json` is `{generated, boxes:[…]}`, boxes **sorted problems-first** |
| 699 | (internet-facing, then unknown, then hardened), each `{label, region, is_self, |
| 700 | ssh_internet_facing, detail, …sub-states…, source}`. **No hostnames ever** — a |
| 701 | peer's real host is read only to point the probe and dies in the process; a box |
| 702 | is named by the same whimsical `label` the Fleet page shows. |
| 703 | |
| 704 | **The contract (enforced page-wide): unknown is loud, never a guessed pass.** A |
| 705 | signal the collector can't read is `"unknown"` (or a JSON `null` |
| 706 | `ssh_internet_facing`) and `hardening.html` renders it as a bordered warn chip — |
| 707 | an unreadable firewall or an un-self-reported peer must look *unresolved*, not |
| 708 | green. **The honest caveat** (in the page's policy card, the same framing as |
| 709 | `ssh.html`): the box publishes its own report, so this is transparency, not |
| 710 | tamper-proof forensics; the outside `:22` probe is a best-effort hint, and the |
| 711 | definitive answer for any box is that box self-reading its own ufw once it runs |
| 712 | the collector. The bucket is versioned, so after-the-fact edits leave history. |
| 713 | |
| 714 | Deploying it (the box side; needs `jq` — `apt install jq`): |
| 715 | |
| 716 | 1. **The collector, its units, and the viewer page ride the binary.** `lchat |
| 717 | bootstrap` (fresh box) or `sudo -n lchat bootstrap --collectors-only` |
| 718 | (refresh over WireGuard — no root shell) writes |
| 719 | `/opt/lchat/hardening-posture.sh`, its `lchat-hardening.{service,timer}` |
| 720 | units, and `hardening.html` into `/opt/lchat/status-pages/`, |
| 721 | `daemon-reload`s, and enables the timer. There is no root scp — the hardened |
| 722 | box has no root shell to copy files in. |
| 723 | 2. **Creds**: the collector reads mesh membership and uploads with the |
| 724 | status-publisher creds in `/data/status.env` (those need read access to the |
| 725 | hibernation bucket — the same read the Fleet page relies on). |
| 726 | 3. `hardening.json` is **generated on the box** by the collector each pass — |
| 727 | never embedded; only the viewer `hardening.html` rides the binary (the |
| 728 | shared `status.css` too, published alongside). |
| 729 | 4. **Verify**: `systemctl list-timers lchat-hardening.timer`, |
| 730 | `journalctl -u lchat-hardening -n 20`, then load `hardening.html`. |
| 731 | |
| 732 | Test mode is documented in the script header: point every input at a fixture |
| 733 | (`HARDENING_UFW_CMD`, `HARDENING_SSHD_CMD`, `HARDENING_SYSTEMCTL_CMD`, |
| 734 | `HARDENING_IP_CMD`, `HARDENING_ROUTER_ENV`, `HARDENING_MEMBERS_FILE`), pin |
| 735 | `HARDENING_NOW`, set `HARDENING_NO_UPLOAD=1`, and it builds `hardening.json` |
| 736 | without touching the box, S3, or the network — no `:22` probe, no upload. |
| 737 | |
| 738 | ## Install (router box) |
| 739 | |
| 740 | 1. **Apply the infra** (deliberate, named — not yet applied): |
| 741 | `AWS_PROFILE=… tofu -chdir=infra/prod apply`, then mint the publisher key: |
| 742 | `aws iam create-access-key --user-name status-publisher` (keys never land |
| 743 | in tofu state). |
| 744 | 2. **Creds on the box**: write `/data/status.env` with |
| 745 | `AWS_ACCESS_KEY_ID=…` / `AWS_SECRET_ACCESS_KEY=…` (the status-publisher |
| 746 | key also covers the Cost Explorer read), plus the optional expense vars: |
| 747 | `STATUS_HETZNER_TOKEN=…` (a Hetzner Cloud API token minted **read-only** |
| 748 | in the project's Security → API tokens — it only lists servers here, and |
| 749 | is never echoed; store it in 1Password like the other per-service |
| 750 | tokens) and/or `STATUS_HETZNER_MONTHLY_CENTS=…` (operator constant — |
| 751 | overrides the token's API-computed figure when both are set), |
| 752 | `STATUS_CLAUDE_TO_DATE_CENTS=…` (cumulative lifetime claude.ai spend — |
| 753 | an operator-attested estimate, labelled "(approx.)" on the page), |
| 754 | `STATUS_APPLE_YEARLY_CENTS=…`, `STATUS_ANTHROPIC_ADMIN_KEY=…` (an |
| 755 | `sk-ant-admin…` key with read-only usage scope — never the sidecars' |
| 756 | completion key), and `STATUS_DNSIMPLE_TOKEN=…` + `STATUS_DNSIMPLE_ACCOUNT=…` |
| 757 | (a DNSimple account API token — read-only use here, never echoed — plus |
| 758 | the numeric account id). `chown lchat:lchat`, `chmod 0600`. |
| 759 | 3. **Pages + the publish script on the box**: the viewer `.html` pages, the |
| 760 | shared `status.css`, the static page scripts (`waterfall.js`), AND the main |
| 761 | `status-publish.sh` itself all ride the binary — `lchat bootstrap` (or |
| 762 | `bootstrap --collectors-only`) writes them into `/opt/lchat/status-pages/` |
| 763 | (pages) and `/opt/lchat/status-publish.sh` (the script), so there's no scp |
| 764 | for them, and the script's per-page upload set can't drift stale on the box |
| 765 | (that drift is how funding + roadmap uploads went missing — fixed by |
| 766 | embedding the script alongside the other collectors, 2026-07-20). So a new |
| 767 | page = add the `.html` to `web/status/` + its `upload` line in |
| 768 | `deploy/status-publish.sh`, and a release carrying both + a |
| 769 | `bootstrap --collectors-only` is the whole deploy. Only the data JSON + icon |
| 770 | still copy in (`economics.json`, `funding.json`, `carbon.json`, |
| 771 | `contributors.json`, `security.json`, `icon-192.png`) — those are |
| 772 | generated/asset files, deliberately not embedded. Post-hardening note for |
| 773 | those copies: the dir is group-writable for `lchat-operators`, but the |
| 774 | *files* in it aren't — replace by scp-ing to `<name>.new` and `mv`-ing over |
| 775 | (directory write allows the rename). `status.css` is the shared stylesheet |
| 776 | every page links (uploaded before the pages by the publish script). |
| 777 | 4. **Install the timer** (first stand-up only — the script itself now rides the |
| 778 | binary, so this is just its systemd unit): `/opt/lchat/status-publish.sh |
| 779 | install` (run once via a root path at bootstrap). The unit runs as the |
| 780 | `lchat` user (least privilege: registry + cloud env are lchat-owned; |
| 781 | `systemctl is-active` / `ss` are unprivileged; the journal read rides |
| 782 | `SupplementaryGroups=systemd-journal`). |
| 783 | 5. **Verify**: `systemctl list-timers lchat-status.timer`, |
| 784 | `journalctl -u lchat-status -n 20`, then load the page. |
| 785 | |
| 786 | The dashboards live at |
| 787 | `https://eleven-status-<account-id>.s3.<region>.amazonaws.com/index.html` |
| 788 | (the `status_url` tofu output; account `427880590669`, region `eu-west-1` → |
| 789 | `https://eleven-status-427880590669.s3.eu-west-1.amazonaws.com/index.html`, |
| 790 | books at `…/books.html`). A friendlier hostname (e.g. |
| 791 | `status.elevenmessenger.com` via CloudFront or a CNAME-compatible front) is |
| 792 | deliberately later work — the pages use relative URLs so they work at any |
| 793 | origin unchanged. |
| 794 | |
| 795 | ## Testing |
| 796 | |
| 797 | - `./deploy/status-publish.sh --dry-run <dir>` builds both JSONs from |
| 798 | self-constructed fake registry/Stripe/provider-billing fixtures (exercising |
| 799 | pagination and the awake/asleep split) and fails on invalid JSON or a |
| 800 | privacy leak — including any digit inside the capacity subtree, or a |
| 801 | fixture workspace/request id surviving into the expenses. No box state, no |
| 802 | network, no upload. `STATUS_DRY_CAPACITY=getting-full|at-capacity` swaps in |
| 803 | the capacity fixture set for that band (default: plenty). |
| 804 | `STATUS_DRY_EXPENSES=omit` removes every expense source (asserts the whole |
| 805 | object is omitted); `STATUS_DRY_EXPENSES=stale-cache` fails all four API |
| 806 | fetches against a seeded, past-max-age same-month cache (asserts the stale |
| 807 | figures serve) — so the cache seams are testable end-to-end. The default |
| 808 | run computes hetzner from a two-page servers fixture (asserts pagination, |
| 809 | the per-location price match, and the cents rounding); |
| 810 | `STATUS_DRY_EXPENSES=hetzner-constant` sets the operator constant with the |
| 811 | fixture still resolvable (asserts the constant wins and the API is never |
| 812 | called). |
| 813 | - `./deploy/security-scan.sh` (run at build time, alongside |
| 814 | `deploy/build-linux.sh`) builds and scans the shipped linux binary with |
| 815 | govulncheck (`-mode=binary`; installed on the fly if absent) and writes the |
| 816 | **redacted** `web/status/security.json` — commit it with the release. Full |
| 817 | findings print to the terminal only. `--dry-run <dir>` folds a |
| 818 | self-constructed govulncheck fixture (fake GO-/CVE-/GHSA- ids, a |
| 819 | module-level informational that must not count) into `<dir>/security.json` |
| 820 | and fails on any id or package path surviving — regenerate the committed |
| 821 | test fixture with `./deploy/security-scan.sh --dry-run test/status-fixtures`. |
| 822 | - `node test/status-contract.test.mjs` (in `npm test`) pins the pages' FIELDS |
| 823 | manifests against the dry-run fixtures in `test/status-fixtures/` — if the |
| 824 | generator's shape and the pages drift, it fails. It also allowlists the |
| 825 | hosts/capacity schema **and** the expenses schema (any extra key fails, so |
| 826 | a precise number or an account id can't ship), pins one fleet fixture per |
| 827 | capacity band (`fleet-getting-full.json`, `fleet-at-capacity.json`), pins |
| 828 | expense omission (`books-no-expenses.json`), and pins hetzner's |
| 829 | API-computed cents + constant override (`books.json` / |
| 830 | `books-hetzner-constant.json`). Regenerate fixtures from a fresh dry-run |
| 831 | after changing the generator (the per-band ones via `STATUS_DRY_CAPACITY`, |
| 832 | the no-expenses one via `STATUS_DRY_EXPENSES=omit`, the hetzner-constant |
| 833 | one via `STATUS_DRY_EXPENSES=hetzner-constant`). For the security ledger it |
| 834 | allowlists the hygiene schema (status word + integer counts + ISO timestamp |
| 835 | + build sha, nothing else), requires status and count to agree, and greps |
| 836 | both the dry-run fixture and the committed `web/status/security.json` for |
| 837 | advisory ids in every grammar (`GO-`, `CVE-`, `GHSA-`) — the redaction |
| 838 | guard. |
| 839 | |
amadan