amadan
docs: managed-AI provisioning design — opt-in, metering, billing, VPC bot tier, BYO key (#855)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+253
−0
docs/ai-provisioning.md+253 −0
diff --git a/docs/ai-provisioning.md b/docs/ai-provisioning.md| new file mode 100644 |
| index 0000000..287a23e |
| --- /dev/null |
| +++ b/docs/ai-provisioning.md |
| @@ -0,0 +1,253 @@ |
| +# 🤖 Managed AI on provisioned spaces — design |
| + |
| +Status: **design agreed 2026-07-14** (decisions below are Paul's); nothing in |
| +this document is built unless a phase says so. This is the plan for making the |
| +onboarding "enable AI" answer *do* something on self-serve spaces: provision a |
| +sidecar, meter what it spends, bill it honestly, and eventually move the whole |
| +bot trust plane off the serving box. |
| + |
| +Related: `docs/bots.md` (trust model), `docs/member-bots.md`, |
| +`docs/cloud-provisioner.md` (the attach-bot gap this closes), |
| +`docs/planet-scale.md` (co-spawn + the sidecar invariant), |
| +`docs/tier-economics.md` ("AI is never bundled"), issues #562 (pricing), |
| +#668 (usage metering pipe), #685 (evict the trust plane), #806 (sidecar |
| +self-provisioning bug). |
| + |
| +## The gap today |
| + |
| +The onboarding toggles are real gates — `bots_enabled` 403s new-agent creation |
| +(`main.go`) and member-bot charters (`memberbots.go`); `ai_enabled` is a |
| +genuine kill-switch the sidecar honors on every reconnect |
| +(`internal/bot/brain.go`, via `GET /api/bot-config`). But a freshly |
| +provisioned `<sqid>.11ch.at` space has **no sidecar and no model key**, so |
| +answering "yes, enable AI" produces nothing: Eleven never appears. The |
| +provisioner chain (`cloud.go` → `controld.go` → `addInstance`) contains zero |
| +bot hooks. |
| + |
| +Meanwhile most of the machinery already exists: |
| + |
| +- `lchat add-bot` (`bot.go`) automates invite minting, the `-bot.env` file, |
| + ownership, and the unit — or defers to co-spawn on hibernating hosts. |
| +- The activator **co-spawns** sidecars with their instance and makes sidecar |
| + traffic activity-invisible (`internal/bot/sidecarmark.go`), so a managed |
| + sidecar can't hold a space awake or wake it. |
| +- The storage meter (`meter.go`) is a proven per-space measure → record → |
| + enforce loop keyed on the cloud ledger. |
| +- Per-bot model choice already ships (`bot_configs.model`, admin UI menu, |
| + re-read on reconnect); charters carry a per-task override. |
| + |
| +## Decisions (Paul, 2026-07-14) |
| + |
| +1. **Eleven only** is provisioned for cloud spaces. Admins can add further |
| + agents through the existing admin "Add an agent" path; on managed spaces |
| + that path must feed the managed sidecar automatically (see |
| + *Roster follows the admin* below). |
| +2. **Bedrock is phased**, not first: phases 0–3 ship on the existing direct |
| + Anthropic key; Bedrock lands with the off-box VPC tier where IAM roles and |
| + VPC endpoints actually pay off. |
| +3. **Small included allowance funded from a capped pool** — free AI is a |
| + bounded trial, never an unbounded subsidy. |
| +4. **On-box first, then iterate** — sidecars stay co-located with instances |
| + (today's trust posture) until the bot-host tier exists. |
| +5. **Bring-your-own-key is an advanced admin option** — paste-a-key first, |
| + "click to connect" via an OAuth-style flow when a provider supports one. |
| + |
| +## Trust model and disclosure |
| + |
| +Managed AI means *we* run a machine that holds the bot's private key and can |
| +decrypt exactly the threads the bot is in. That is already the disclosed |
| +exception-by-membership in `docs/bots.md` and already true for the |
| +hand-provisioned fleet — but self-serve opt-in turns it into a consumer-facing |
| +promise, so: |
| + |
| +- **AI is opt-out by default.** The setup flow writes `ai_enabled=false` |
| + unless the admin takes a specific opt-in step. (Deterministic bots — the |
| + command floor, no model, nothing sent anywhere — can stay default-on.) |
| +- **The opt-in copy discloses plainly**, in the lock-and-key voice: the |
| + helper's key lives on Eleven's machines, and it can read the chats it's |
| + invited to — that's what makes it able to help. Membership stays visible in |
| + the roster. |
| +- The invariants hold throughout: bot code never runs in the server process, |
| + the sidecar stays a pure outbound client, and **the instance server never |
| + stores a usable model key** (load-bearing for BYO key, below). |
| + |
| +## Phase 0 — zero-touch provisioning that works |
| + |
| +Prerequisites for everything else; both PR-sized. |
| + |
| +- **Fix #806**: `bot run` must redeem the invite when a session path is set, |
| + the file is absent, and the matching invite env var is present (option 1 in |
| + the issue — it also covers the co-spawn path, and it is what makes keys |
| + *born on the host that runs the bot*, which phase 4 relies on). |
| +- **Mint agent invites in Go.** `add-bot` shells out to |
| + `agent/mint-invite.mjs`, dragging Node onto any box that provisions. Port |
| + the mint to the control binary. |
| + |
| +## Phase 1 — meter before money (#668) |
| + |
| +Measure first, price from data. No billing in this phase. |
| + |
| +- Parse the `usage` block in `internal/bot/anthropic.go` (today |
| + `anthResponse` doesn't even deserialize it). Record all four counters — |
| + input, output, cache-write, cache-read — they're priced differently. |
| +- Aggregate in the sidecar per (day, model, dimension); flush to a new cloud |
| + `POST /api/usage` authed by a **per-instance meter token** minted at create |
| + time (mint/list/revoke pattern copied from the push relay). Ledger: |
| + `usage(space_id, day, dimension, quantity)` beside `storage_bytes`. |
| +- Add bot-activity dimensions (replies, charter runs) on the same pipe. |
| +- Run silently on the existing sidecar fleet for a couple of weeks; the |
| + allowance and add-on price in phase 3 are set from this data. |
| + |
| +The meter flush doubles as the **budget heartbeat**: the response to |
| +`POST /api/usage` carries the space's current AI budget state, so no new |
| +cloud→instance channel is needed for enforcement (phase 3). |
| + |
| +## Phase 2 — the opt-in that provisions (on-box) |
| + |
| +- **Onboarding**: the extras step's AI question becomes an explicit opt-in |
| + chip with the disclosure copy; declining (or skipping) writes |
| + `ai_enabled=false`. Opting in triggers provisioning *during* setup — |
| + "Eleven is joining your space…" — and Eleven's welcome DM lands seconds |
| + later. The same opt-in surfaces in the Admin app for existing spaces (note: |
| + the Admin mini-app's `setFeatures` bridge is currently dead code — wire it |
| + up here). |
| +- **Signal path**: instance → cloud over the meter-token channel ("this space |
| + wants AI") → cloud calls a new controld `POST /instances/{host}/bot`, which |
| + runs the `add-bot` logic: mint the Eleven invite, stage |
| + `/data/<host>-bot.env` with the fleet `ANTHROPIC_API_KEY`, chown. On a |
| + hibernating host that's all it does — co-spawn picks the sidecar up (the |
| + activator needs one small hook: notice a newly staged `-bot.env` while the |
| + child is already awake, so opt-in doesn't wait for the next sleep/wake |
| + cycle). |
| +- **Roster follows the admin**: when an admin adds another agent on a managed |
| + space, the same signal path appends that identity's invite to the bot env |
| + and restarts the sidecar; post-#806 it self-provisions. The sidecar's fixed |
| + two-identity env shape (`LCHAT_INVITE` / `LCHAT_ELEVEN_INVITE`) generalizes |
| + to a numbered or manifest form as part of this. |
| +- Latency budget: chip-tap to welcome DM in seconds, not sweeps — this is why |
| + the signal is a push over the meter channel, not a stats-poll. |
| + |
| +## Phase 3 — usage accrues to the subscription |
| + |
| +- **Stripe Billing Meters** on the existing per-space subscription |
| + (`billing.go` — the provisioner's Stripe integration, not the in-space |
| + Connect one). Metered price = provider cost × a published markup, recorded |
| + in `docs/tier-economics.md` like every other rate. AI becomes the first |
| + real marginal-cost line in the public books. |
| +- **Included allowance from a capped pool**: quantities under the per-space |
| + monthly allowance never forward to Stripe; a **fleet-wide monthly AI |
| + budget** (mirror of the `creates_log` creation-budget circuit breaker) |
| + bounds aggregate subsidized spend no matter how many free spaces opt in. |
| + Allowance and pool sizes come from phase 1 data. Cost reality check at |
| + today's list prices: a typical cached Sonnet reply runs ~1–1.5¢, so ~100 |
| + replies/month ≈ €1.2/month against Family's €4.68/*year* net — the |
| + allowance is trial-sized by necessity, and "AI is never bundled" stands. |
| +- **Exhaustion behavior**: the meter-flush response flips the space to |
| + budget-exhausted; the sidecar's existing `AIEnabled` gate grows that state — |
| + the bot drops to the deterministic command floor and says so once, honestly. |
| + Per the conduct rule (knowledge.md §20), Eleven describes the upgrade path |
| + and never posts a payment link. |
| +- **Abuse guards land with billing, not after**: a per-space daily reply |
| + ceiling independent of budget (`minEvery` floors charter schedules but |
| + nothing bounds conversational replies), and loop protection stays audited. |
| + |
| +## Phase 4 — the bot-host tier: egress-only VPC (#685) |
| + |
| +The "hardened off-box sidecar tier" from #550 Part 4 v2; sidecars follow the |
| +commons, not the provider. |
| + |
| +- **Sandbox: systemd, not Docker.** The bot unit already carries near-maximal |
| + hardening (empty capability set, `ProtectSystem=strict`, |
| + `MemoryDenyWriteExecute` — possible because the sidecar is pure Go — syscall |
| + filter, `MemoryMax`), and the sidecar runs no untrusted code: member-app |
| + code executes in the *client's* sandboxed iframe, charters are bounded tool |
| + loops. Docker would add a daemon and an image pipeline — against the |
| + one-boring-binary rule — for less isolation than the unit provides. The |
| + real gap is that **co-spawned** sidecars inherit the router's sandbox; |
| + moving them to a dedicated host puts each space back under its own hardened |
| + unit (optionally per-space `DynamicUser`). |
| +- **Host shape**: EC2 (Graviton, instance-role pattern from |
| + `infra/dev/compute.tf`) in a **private subnet — no public IP, no inbound |
| + anything**; egress via NAT restricted to 443. The sidecar is already a pure |
| + outbound client; the `X-Lchat-Sidecar` marker means off-box sidecars can't |
| + wake or hold instances awake, and charter due-time wakes already work from |
| + off-box (`POST /api/charter-next-due`). The IaC boundary test holds: tofu |
| + builds the box and role; the `lchat` binary (a bot-host mode) learns which |
| + spaces it hosts by outbound-polling the control plane. |
| +- **Keys are born on the bot host.** Provisioning hands the host one-time |
| + invites; the sidecar self-provisions (#806 fix), so bot private keys never |
| + transit anywhere. Combined with #685 this evicts bot keys *and* the model |
| + key from the serving box. |
| +- Lifecycle v1: off-box sidecars run always-on (tens of MB each; one small |
| + host carries dozens; scale horizontally before getting clever). |
| +- Ops debt that must close before this scales: **#811** — awake bot-sidecar |
| + instances never pick up auto-updates. |
| + |
| +## Phase 5 — the provider seam: Bedrock, local models, catalog |
| + |
| +One small interface in `internal/bot` (today's client becomes implementation |
| +#1): |
| + |
| +1. **Anthropic direct** — the existing hand-rolled Messages client; stays the |
| + default for BYO key and self-hosters (`ANTHROPIC_BASE_URL` seam exists). |
| +2. **Bedrock** — for the managed VPC tier. The draw is not model choice: |
| + it's **IAM-role auth** (no `ANTHROPIC_API_KEY` exists anywhere for managed |
| + spaces — the shared-key blast radius disappears) and **VPC interface |
| + endpoints** (model traffic never touches the public internet). Same |
| + Messages body under SigV4 — `home/sigv4.go` proves the no-dependency |
| + signer pattern; model ids gain the `anthropic.` prefix. Check eu-west-1 / |
| + EU cross-region profile availability — EU-resident inference is a real |
| + selling point given bots see plaintext. |
| +3. **OpenAI-compatible local** — the SLM companion direction: a small model |
| + in the same private network, called by the sidecar. First workload to |
| + move: the addressing classifier (hardcoded Haiku today, `brain.go`) — |
| + highest volume, lowest difficulty; moving it in-network cuts both spend |
| + and plaintext egress. |
| + |
| +**Model management**: per-bot choice already exists; the improvement is a |
| +fleet-served **model catalog** (menu entries + ids) so `admin.js` stops |
| +hardcoding model ids and fleet-wide swaps/deprecations need no client |
| +redeploy. Keep the menu small: default, better, cheaper. |
| + |
| +## Bring your own key (advanced admin option) |
| + |
| +- **The invariant shapes the design**: compromising the server must not yield |
| + the model key (CLAUDE.md), so a BYO key is **never persisted in the |
| + instance DB**. Set flow: admin UI → instance (held in memory only, never |
| + stored, never logged) → cloud over the meter channel → controld writes it |
| + into `/data/<host>-bot.env` → sidecar restart. Same at-rest posture as the |
| + fleet key today; treated like the admin token in transit. |
| +- **BYO spaces don't meter and aren't billed** for AI (#668's rule: their |
| + key, their cost). The allowance/budget machinery disarms; `ai_enabled` |
| + still works as the kill switch. |
| +- **Click to connect**: Anthropic offers no third-party OAuth key |
| + provisioning today, so v1 is paste-a-key (Anthropic key, Anthropic wire |
| + format). The **connect broker** (`lchat-connect`, which already holds OAuth |
| + client pairs for GitHub/DNSimple) is the natural seam when a provider |
| + supports key issuance by consent — OpenRouter's PKCE flow exists today if |
| + we ever want it, with its own disclosure (a third party would see the |
| + bot's plaintext); an Anthropic OAuth flow is a watch item. |
| +- Advanced means advanced: buried in the admin console, not in onboarding. |
| +- v2 candidates: per-space Bedrock via cross-account role (Business tier), |
| + arbitrary OpenAI-compatible base URL + key for self-managed local models. |
| + |
| +## Kill switches and guards (summary) |
| + |
| +- Per-space: `ai_enabled` (exists, enforced in the sidecar), budget |
| + exhaustion (phase 3), daily reply ceiling (phase 3). |
| +- Fleet: the subsidy-pool circuit breaker (phase 3); env kill on bot hosts. |
| +- Egress: charter/research URL fetches from our infrastructure need an |
| + egress policy on the bot host (a prompt-injected charter fetching internal |
| + endpoints is SSRF from our VPC) — the serving box's unfurl SSRF guard has |
| + no sidecar equivalent today. |
| +- Terms: one pass over Anthropic/Bedrock commercial terms (resale with |
| + markup, data-retention posture we advertise) before the price goes public. |
| + |
| +## Sequencing |
| + |
| +Phase 0 → 1 → 2 ship independently and in order; 3 needs 1's data and 2's |
| +opt-in; 4 and 5 ride the #685 arc and don't block 2–3. Native parity is |
| +mostly free (onboarding/admin render through the webview) but the opt-in flow |
| +gets an explicit iOS verify. `docs/tier-economics.md` gains the AI rate line |
| +when phase 3 prices are set, and its stale "a bot-sidecar space never sleeps" |
| +note gets reconciled with co-spawn (#670) and the observed #811 behavior. |