| 1 | # 🤖 Managed AI on provisioned spaces — design |
| 2 | |
| 3 | Status: **design agreed 2026-07-14** (decisions below are Paul's). Built so |
| 4 | far: phase 0 (#806 fix + Go invite mint, PRs #864/#865), phase 1 (the usage |
| 5 | pipe, PR #867), phase 2 (attach-bot + the onboarding opt-in, #858). Phases |
| 6 | 3–5 are unbuilt. This is the plan for making the |
| 7 | onboarding "enable AI" answer *do* something on self-serve spaces: provision a |
| 8 | sidecar, meter what it spends, bill it honestly, and eventually move the whole |
| 9 | bot trust plane off the serving box. |
| 10 | |
| 11 | Related: `docs/bots.md` (trust model), `docs/member-bots.md`, |
| 12 | `docs/cloud-provisioner.md` (the attach-bot gap this closes), |
| 13 | `docs/planet-scale.md` (co-spawn + the sidecar invariant), |
| 14 | `docs/tier-economics.md` ("AI is never bundled"), issues #562 (pricing), |
| 15 | #668 (usage metering pipe), #685 (evict the trust plane), #806 (sidecar |
| 16 | self-provisioning bug). |
| 17 | |
| 18 | ## The gap today |
| 19 | |
| 20 | The onboarding toggles are real gates — `bots_enabled` 403s new-agent creation |
| 21 | (`main.go`) and member-bot charters (`memberbots.go`); `ai_enabled` is a |
| 22 | genuine kill-switch the sidecar honors on every reconnect |
| 23 | (`internal/bot/brain.go`, via `GET /api/bot-config`). But a freshly |
| 24 | provisioned `<sqid>.11ch.at` space has **no sidecar and no model key**, so |
| 25 | answering "yes, enable AI" produces nothing: Eleven never appears. The |
| 26 | provisioner chain (`cloud.go` → `controld.go` → `addInstance`) contains zero |
| 27 | bot hooks. |
| 28 | |
| 29 | Meanwhile most of the machinery already exists: |
| 30 | |
| 31 | - `lchat add-bot` (`bot.go`) automates invite minting, the `-bot.env` file, |
| 32 | ownership, and the unit — or defers to co-spawn on hibernating hosts. |
| 33 | - The activator **co-spawns** sidecars with their instance and makes sidecar |
| 34 | traffic activity-invisible (`internal/bot/sidecarmark.go`), so a managed |
| 35 | sidecar can't hold a space awake or wake it. |
| 36 | - The storage meter (`meter.go`) is a proven per-space measure → record → |
| 37 | enforce loop keyed on the cloud ledger. |
| 38 | - Per-bot model choice already ships (`bot_configs.model`, admin UI menu, |
| 39 | re-read on reconnect); charters carry a per-task override. |
| 40 | |
| 41 | ## Decisions (Paul, 2026-07-14) |
| 42 | |
| 43 | 1. **Eleven only** is provisioned for cloud spaces. Admins can add further |
| 44 | agents through the existing admin "Add an agent" path; on managed spaces |
| 45 | that path must feed the managed sidecar automatically (see |
| 46 | *Roster follows the admin* below). |
| 47 | 2. **Bedrock is phased**, not first: phases 0–3 ship on the existing direct |
| 48 | Anthropic key; Bedrock lands with the off-box VPC tier where IAM roles and |
| 49 | VPC endpoints actually pay off. |
| 50 | 3. **Small included allowance funded from a capped pool** — free AI is a |
| 51 | bounded trial, never an unbounded subsidy. |
| 52 | 4. **On-box first, then iterate** — sidecars stay co-located with instances |
| 53 | (today's trust posture) until the bot-host tier exists. |
| 54 | 5. **Bring-your-own-key is an advanced admin option** — paste-a-key first, |
| 55 | "click to connect" via an OAuth-style flow when a provider supports one. |
| 56 | |
| 57 | ## Trust model and disclosure |
| 58 | |
| 59 | Managed AI means *we* run a machine that holds the bot's private key and can |
| 60 | decrypt exactly the threads the bot is in. That is already the disclosed |
| 61 | exception-by-membership in `docs/bots.md` and already true for the |
| 62 | hand-provisioned fleet — but self-serve opt-in turns it into a consumer-facing |
| 63 | promise, so: |
| 64 | |
| 65 | - **AI is opt-out by default.** The setup flow writes `ai_enabled=false` |
| 66 | unless the admin takes a specific opt-in step. (Deterministic bots — the |
| 67 | command floor, no model, nothing sent anywhere — can stay default-on.) |
| 68 | - **The opt-in copy discloses plainly**, in the lock-and-key voice: the |
| 69 | helper's key lives on Eleven's machines, and it can read the chats it's |
| 70 | invited to — that's what makes it able to help. Membership stays visible in |
| 71 | the roster. |
| 72 | - The invariants hold throughout: bot code never runs in the server process, |
| 73 | the sidecar stays a pure outbound client, and **the instance server never |
| 74 | stores a usable model key** (load-bearing for BYO key, below). |
| 75 | |
| 76 | ## Phase 0 — zero-touch provisioning that works |
| 77 | |
| 78 | Prerequisites for everything else; both PR-sized. |
| 79 | |
| 80 | - **Fix #806**: `bot run` must redeem the invite when a session path is set, |
| 81 | the file is absent, and the matching invite env var is present (option 1 in |
| 82 | the issue — it also covers the co-spawn path, and it is what makes keys |
| 83 | *born on the host that runs the bot*, which phase 4 relies on). |
| 84 | - **Mint agent invites in Go.** `add-bot` shells out to |
| 85 | `agent/mint-invite.mjs`, dragging Node onto any box that provisions. Port |
| 86 | the mint to the control binary. |
| 87 | |
| 88 | ## Phase 1 — meter before money (#668) |
| 89 | |
| 90 | Measure first, price from data. No billing in this phase. |
| 91 | |
| 92 | - Parse the `usage` block in `internal/bot/anthropic.go` (today |
| 93 | `anthResponse` doesn't even deserialize it). Record all four counters — |
| 94 | input, output, cache-write, cache-read — they're priced differently. |
| 95 | - Aggregate in the sidecar per (day, model, dimension); flush to a new cloud |
| 96 | `POST /api/usage` authed by a **per-instance meter token** minted at create |
| 97 | time (mint/list/revoke pattern copied from the push relay). Ledger: |
| 98 | `usage(space_id, day, dimension, quantity)` beside `storage_bytes`. |
| 99 | - Add bot-activity dimensions (replies, charter runs) on the same pipe. |
| 100 | - Run silently on the existing sidecar fleet for a couple of weeks; the |
| 101 | allowance and add-on price in phase 3 are set from this data. |
| 102 | |
| 103 | The meter flush doubles as the **budget heartbeat**: the response to |
| 104 | `POST /api/usage` carries the space's current AI budget state, so no new |
| 105 | cloud→instance channel is needed for enforcement (phase 3). |
| 106 | |
| 107 | ## Phase 2 — the opt-in that provisions (on-box) |
| 108 | |
| 109 | - **Onboarding**: the extras step's AI question becomes an explicit opt-in |
| 110 | chip with the disclosure copy; declining (or skipping) writes |
| 111 | `ai_enabled=false`. Opting in triggers provisioning *during* setup — |
| 112 | "Eleven is joining your space…" — and Eleven's welcome DM lands seconds |
| 113 | later. The same opt-in surfaces in the Admin app for existing spaces (note: |
| 114 | the Admin mini-app's `setFeatures` bridge is currently dead code — wire it |
| 115 | up here). |
| 116 | - **Signal path**: instance → cloud over the meter-token channel ("this space |
| 117 | wants AI") → cloud calls a new controld `POST /instances/{host}/bot`, which |
| 118 | runs the `add-bot` logic: mint the Eleven invite, stage |
| 119 | `/data/<host>-bot.env` with the fleet `ANTHROPIC_API_KEY`, chown. On a |
| 120 | hibernating host that's all it does — co-spawn picks the sidecar up (the |
| 121 | activator needs one small hook: notice a newly staged `-bot.env` while the |
| 122 | child is already awake, so opt-in doesn't wait for the next sleep/wake |
| 123 | cycle). |
| 124 | - **Roster follows the admin**: when an admin adds another agent on a managed |
| 125 | space, the same signal path appends that identity's invite to the bot env |
| 126 | and restarts the sidecar; post-#806 it self-provisions. The sidecar's fixed |
| 127 | two-identity env shape (`LCHAT_INVITE` / `LCHAT_ELEVEN_INVITE`) generalizes |
| 128 | to a numbered or manifest form as part of this. |
| 129 | - Latency budget: chip-tap to welcome DM in seconds, not sweeps — this is why |
| 130 | the signal is a push over the meter channel, not a stats-poll. |
| 131 | |
| 132 | ## Phase 3 — usage accrues to the subscription |
| 133 | |
| 134 | - **Stripe Billing Meters** on the existing per-space subscription |
| 135 | (`billing.go` — the provisioner's Stripe integration, not the in-space |
| 136 | Connect one). Metered price = provider cost × a published markup, recorded |
| 137 | in `docs/tier-economics.md` like every other rate. AI becomes the first |
| 138 | real marginal-cost line in the public books. |
| 139 | - **Included allowance from a capped pool**: quantities under the per-space |
| 140 | monthly allowance never forward to Stripe; a **fleet-wide monthly AI |
| 141 | budget** (mirror of the `creates_log` creation-budget circuit breaker) |
| 142 | bounds aggregate subsidized spend no matter how many free spaces opt in. |
| 143 | Allowance and pool sizes come from phase 1 data. Cost reality check at |
| 144 | today's list prices: a typical cached Sonnet reply runs ~1–1.5¢, so ~100 |
| 145 | replies/month ≈ €1.2/month against Family's €4.68/*year* net — the |
| 146 | allowance is trial-sized by necessity, and "AI is never bundled" stands. |
| 147 | - **Exhaustion behavior**: the meter-flush response flips the space to |
| 148 | budget-exhausted; the sidecar's existing `AIEnabled` gate grows that state — |
| 149 | the bot drops to the deterministic command floor and says so once, honestly. |
| 150 | Per the conduct rule (knowledge.md §20), Eleven describes the upgrade path |
| 151 | and never posts a payment link. |
| 152 | - **Abuse guards land with billing, not after**: a per-space daily reply |
| 153 | ceiling independent of budget (`minEvery` floors charter schedules but |
| 154 | nothing bounds conversational replies), and loop protection stays audited. |
| 155 | |
| 156 | ## Phase 4 — the bot-host tier: egress-only VPC (#685) |
| 157 | |
| 158 | The "hardened off-box sidecar tier" from #550 Part 4 v2; sidecars follow the |
| 159 | commons, not the provider. |
| 160 | |
| 161 | - **Sandbox: systemd, not Docker.** The bot unit already carries near-maximal |
| 162 | hardening (empty capability set, `ProtectSystem=strict`, |
| 163 | `MemoryDenyWriteExecute` — possible because the sidecar is pure Go — syscall |
| 164 | filter, `MemoryMax`), and the sidecar runs no untrusted code: member-app |
| 165 | code executes in the *client's* sandboxed iframe, charters are bounded tool |
| 166 | loops. Docker would add a daemon and an image pipeline — against the |
| 167 | one-boring-binary rule — for less isolation than the unit provides. The |
| 168 | real gap is that **co-spawned** sidecars inherit the router's sandbox; |
| 169 | moving them to a dedicated host puts each space back under its own hardened |
| 170 | unit (optionally per-space `DynamicUser`). |
| 171 | - **Host shape**: EC2 (Graviton, instance-role pattern from |
| 172 | `infra/dev/compute.tf`) in a **private subnet — no public IP, no inbound |
| 173 | anything**; egress via NAT restricted to 443. The sidecar is already a pure |
| 174 | outbound client; the `X-Lchat-Sidecar` marker means off-box sidecars can't |
| 175 | wake or hold instances awake, and charter due-time wakes already work from |
| 176 | off-box (`POST /api/charter-next-due`). The IaC boundary test holds: tofu |
| 177 | builds the box and role; the `lchat` binary (a bot-host mode) learns which |
| 178 | spaces it hosts by outbound-polling the control plane. |
| 179 | - **Keys are born on the bot host.** Provisioning hands the host one-time |
| 180 | invites; the sidecar self-provisions (#806 fix), so bot private keys never |
| 181 | transit anywhere. Combined with #685 this evicts bot keys *and* the model |
| 182 | key from the serving box. |
| 183 | - Lifecycle v1: off-box sidecars run always-on (tens of MB each; one small |
| 184 | host carries dozens; scale horizontally before getting clever). |
| 185 | - Ops debt that must close before this scales: **#811** — awake bot-sidecar |
| 186 | instances never pick up auto-updates. |
| 187 | |
| 188 | ## Phase 5 — the provider seam: Bedrock, local models, catalog |
| 189 | |
| 190 | One small interface in `internal/bot` (today's client becomes implementation |
| 191 | #1): |
| 192 | |
| 193 | 1. **Anthropic direct** — the existing hand-rolled Messages client; stays the |
| 194 | default for BYO key and self-hosters (`ANTHROPIC_BASE_URL` seam exists). |
| 195 | 2. **Bedrock** — for the managed VPC tier. The draw is not model choice: |
| 196 | it's **IAM-role auth** (no `ANTHROPIC_API_KEY` exists anywhere for managed |
| 197 | spaces — the shared-key blast radius disappears) and **VPC interface |
| 198 | endpoints** (model traffic never touches the public internet). Same |
| 199 | Messages body under SigV4 — `home/sigv4.go` proves the no-dependency |
| 200 | signer pattern; model ids gain the `anthropic.` prefix. Check eu-west-1 / |
| 201 | EU cross-region profile availability — EU-resident inference is a real |
| 202 | selling point given bots see plaintext. |
| 203 | 3. **OpenAI-compatible local** — the SLM companion direction: a small model |
| 204 | in the same private network, called by the sidecar. First workload to |
| 205 | move: the addressing classifier (hardcoded Haiku today, `brain.go`) — |
| 206 | highest volume, lowest difficulty; moving it in-network cuts both spend |
| 207 | and plaintext egress. |
| 208 | |
| 209 | **Model management**: per-bot choice already exists; the improvement is a |
| 210 | fleet-served **model catalog** (menu entries + ids) so `admin.js` stops |
| 211 | hardcoding model ids and fleet-wide swaps/deprecations need no client |
| 212 | redeploy. Keep the menu small: default, better, cheaper. |
| 213 | |
| 214 | ## Bring your own key (advanced admin option) |
| 215 | |
| 216 | - **The invariant shapes the design**: compromising the server must not yield |
| 217 | the model key (CLAUDE.md), so a BYO key is **never persisted in the |
| 218 | instance DB**. Set flow: admin UI → instance (held in memory only, never |
| 219 | stored, never logged) → cloud over the meter channel → controld writes it |
| 220 | into `/data/<host>-bot.env` → sidecar restart. Same at-rest posture as the |
| 221 | fleet key today; treated like the admin token in transit. |
| 222 | - **BYO spaces don't meter and aren't billed** for AI (#668's rule: their |
| 223 | key, their cost). The allowance/budget machinery disarms; `ai_enabled` |
| 224 | still works as the kill switch. |
| 225 | - **Click to connect**: Anthropic offers no third-party OAuth key |
| 226 | provisioning today, so v1 is paste-a-key (Anthropic key, Anthropic wire |
| 227 | format). The **connect broker** (`lchat-connect`, which already holds OAuth |
| 228 | client pairs for GitHub/DNSimple) is the natural seam when a provider |
| 229 | supports key issuance by consent — OpenRouter's PKCE flow exists today if |
| 230 | we ever want it, with its own disclosure (a third party would see the |
| 231 | bot's plaintext); an Anthropic OAuth flow is a watch item. |
| 232 | - Advanced means advanced: buried in the admin console, not in onboarding. |
| 233 | - v2 candidates: per-space Bedrock via cross-account role (Business tier), |
| 234 | arbitrary OpenAI-compatible base URL + key for self-managed local models. |
| 235 | |
| 236 | ## Kill switches and guards (summary) |
| 237 | |
| 238 | - Per-space: `ai_enabled` (exists, enforced in the sidecar), budget |
| 239 | exhaustion (phase 3), daily reply ceiling (phase 3). |
| 240 | - Fleet: the subsidy-pool circuit breaker (phase 3); env kill on bot hosts. |
| 241 | - Egress: charter/research URL fetches from our infrastructure need an |
| 242 | egress policy on the bot host (a prompt-injected charter fetching internal |
| 243 | endpoints is SSRF from our VPC) — the serving box's unfurl SSRF guard has |
| 244 | no sidecar equivalent today. |
| 245 | - Terms: one pass over Anthropic/Bedrock commercial terms (resale with |
| 246 | markup, data-retention posture we advertise) before the price goes public. |
| 247 | |
| 248 | ## Sequencing |
| 249 | |
| 250 | Phase 0 → 1 → 2 ship independently and in order; 3 needs 1's data and 2's |
| 251 | opt-in; 4 and 5 ride the #685 arc and don't block 2–3. Native parity is |
| 252 | mostly free (onboarding/admin render through the webview) but the opt-in flow |
| 253 | gets an explicit iOS verify. `docs/tier-economics.md` gains the AI rate line |
| 254 | when phase 3 prices are set, and its stale "a bot-sidecar space never sleeps" |
| 255 | note gets reconciled with co-spawn (#670) and the observed #811 behavior. |
| 256 | |
amadan