| new file mode 100644 |
| index 0000000..431c63c |
| --- /dev/null |
| +++ b/docs/operator-vault-migration.md |
| @@ -0,0 +1,369 @@ |
| +# 🤖 Retiring sops → the shared operator vault on Eleven |
| + |
| +The plan of record for moving the fleet's operator secrets off the sops+age |
| +S3-bundle vault (`docs/secrets-vault.md`, RUNBOOK.md "Secrets vault") and onto a |
| +**shared, locked Eleven vault** — the Vault mini-app with the biometric lock |
| +layer (`docs/locked-vaults.md`), running in a dedicated bot-less space. This |
| +doc is the security analysis and the migration runbook; the design lives in |
| +`docs/locked-vaults.md`, operator-specific bindings (which space, which |
| +hostnames) go in RUNBOOK.md when the migration executes. |
| + |
| +**This plan was written, then adversarially red-teamed against the shipped code |
| +(three independent reviewers: crypto/protocol, operational/DR, macOS-keychain). |
| +The verdict shaped it: the confidentiality story is sound, but the first draft |
| +overclaimed disaster-recovery parity and audit accountability, and the vault is |
| +a *DR regression vs sops* until the recovery escrow (§"Build prerequisites") is |
| +built. So the bar is explicit: no real secret moves until the prerequisites |
| +ship AND the restore drill passes on a cold machine.** sops stays the vault of |
| +record, in parallel, for a full cycle after that. |
| + |
| +## Why move at all |
| + |
| +sops+age works and is secure, but its *sharing* is painful: each operator seals |
| +their whole repo into an age-sealed S3 bundle slot (`sync.sh push`) and pulls |
| +the others' (`sync.sh pull`), an asynchronous, manual, merge-prone dance. The |
| +Eleven vault replaces that with the product's own always-on E2EE transport: one |
| +shared thread, every operator folds the same state live, with an append-only |
| +change log. It is also load-bearing dogfooding — "our own ops secrets ride |
| +Eleven" is the strongest statement of trust in the encryption. What the move |
| +must **not** do is weaken secrecy or recoverability. The rest of this doc proves |
| +where it does and doesn't, and what has to be built to close the gaps. |
| + |
| +## Does the unlock work shared between two operators? (yes — this is the core) |
| + |
| +A locked vault has **one vault keypair**. Its private half is wrapped once per |
| +enrolled unlocker into the keyring — N operators = N wraps of the *same* private |
| +key, each openable only by that operator's own biometric-gated KEK: |
| + |
| +- **Paul** unlocks with his passkey (browser, WebAuthn PRF → KEK) or his Mac's |
| + Touch ID keychain key (CLI). Two devices → two wraps, both his. |
| +- **James** unlocks with *his* passkey / *his* Mac's keychain key — different |
| + wraps, opening the same vault private key. |
| + |
| +Unlock is therefore **per-operator, per-device, and entirely local**: the |
| +in-memory key and the "lock now" / auto-expiry re-mask are properties of *one |
| +device's session*, never shared. Each operator seals new secrets to the public |
| +key with no unlock at all (adding is free); each reads only after their own |
| +biometric. **Onboarding the second operator** is the request/grant ceremony, |
| +built and tested end-to-end (vectors + full-ceremony crypto test + live |
| +round-trip test): Paul locks first (his key is the first wrap); James |
| +`eleven vault enroll` (mints his own keychain KEK, posts a `vault.request`); |
| +Paul **approves** with one biometric (out-of-band verified, §"Grant-approval"); |
| +James `eleven vault finish` (lands his own durable wrap). The vault private key |
| +never transits the server in the clear at any step — only wraps and |
| +ephemeral-sealed grants, all opaque blobs. **Verified sound** by the crypto |
| +review: `ephWrapped` gates completion (only the requester finishes), the grant |
| +is ECIES-bound to the request's ephemeral public key (a replayed/mismatched |
| +grant fails to open, never misdirects the key), and a grant can only leak the |
| +key to a party an already-unlocked granter chose to wrap to. |
| + |
| +## Security analysis |
| + |
| +Threat model is CLAUDE.md's: *if the server is compromised, the attacker gets |
| +nothing.* Each property is stated against that bar and against "is this at least |
| +as good as sops." Claims marked **[verified]** survived the code red-team; |
| +claims marked **[gap]** are honest weaknesses the plan must carry. |
| + |
| +### 1. Server / infrastructure compromise — **[verified] holds the rule, equal to sops** |
| + |
| +The server stores only the thread-key wraps (ECIES to each member's user key), |
| +the sealed values (ECIES to the vault public key), and the keyring wraps (the |
| +vault private key under each operator's KEK). None is a usable key; the vault |
| +private key is never present on the server in any form. A full server compromise |
| +yields ciphertext — the same nothing sops's S3 gives. A hostile server can drop |
| +or replay whole ciphertext events (rollback — acknowledged as advisory; #573 key |
| +transparency is the systemic answer) but cannot read, forge a value, or maul a |
| +wrap (no key, GCM tag). The space is bot-less (§6), so no relay/push/announcer |
| +path touches it. |
| + |
| +### 2. Stolen or **locked/powered-off** operator device (no code-exec) — **[verified] better than sops** |
| + |
| +sops's root of trust is a **plaintext file on disk** (`~/.config/sops/age/ |
| +keys.txt`, mode 600) — a stolen powered-off laptop, a disk image, or a backup |
| +that captured the file reads the age key and then every secret. The Eleven |
| +vault has **no plaintext *vault* key at rest**: the vault private key exists |
| +only as wraps, and on macOS the CLI KEK sits in the login keychain, *encrypted |
| +at rest*. A **locked or powered-off** Mac, a raw disk copy, or a `~/.eleven` |
| +copy taken cold yields the operator's user key (hence names + the change log) |
| +but **no values** — the KEK to open a wrap is unrecoverable without the login |
| +password. **This is the class where the vault genuinely beats sops.** |
| + |
| +Two honest bounds on this win: |
| +- The protection for a *locked* device is the login keychain's **at-rest |
| + encryption**, not the biometric. On an **unlocked, unattended** Mac the KEK is |
| + readable directly (the read path carries no LocalAuthentication context — |
| + `eleven/vaultkek_darwin.go`), so an unlocked walk-up device reduces to §3. |
| +- "No plaintext key at rest" is true of the **vault** key only. The operator's |
| + **user** key is a plaintext 0600 file in `~/.eleven` (`store.go`: "every file |
| + is a private key in JSON clothing") — same posture as sops's age key. It |
| + exposes names/change-log/any-unsealed-item, never sealed values. |
| + |
| +### 3. Malware running AS the operator — **[gap, = sops] no better, stated plainly** |
| + |
| +On macOS the CLI's KEK is a plain login-keychain item gated by an **in-process |
| +LocalAuthentication check**, *not* a Secure-Enclave `kSecAccessControl` ACL |
| +(enclave binding needs a signed binary with an application-identifier |
| +entitlement, which breaks the anyone-can-`go build` rule — see |
| +`docs/locked-vaults.md`; the read path in `vaultkek_darwin.go` sets no access |
| +control). So code running as the operator can read the KEK from the keychain |
| +*without* the LA prompt, combine it with `~/.eleven`, and decrypt everything. |
| +**sops is exactly as exposed** — the same malware reads `keys.txt`. Not a |
| +regression; both mitigate the same way (protect operator machines: WireGuard + |
| +named accounts + sudoers allowlist on the box; release keys offline). *Upgrade |
| +path (not a blocker):* a **signed** `eleven` build can move the KEK to a |
| +genuinely enclave-bound item without changing the wrap format (#950). |
| + |
| +### 4. Ex-operator / revocation — **[gap, = sops] ratchet unchanged** |
| + |
| +Removal does **not** retroactively protect anything an ex-operator saw: no |
| +rekey-on-leave (`db.go`), and they *read* the values while enrolled. |
| +`vault.revoke` drops their keyring wrap (hygiene), but removal means **rotating |
| +the real secrets** — identical to sops's `.sops.yaml` recipient ratchet. Do not |
| +mistake revoke for containment. (See §9 for the *offensive* use of revoke.) |
| + |
| +### 5. Bootstrap circularity + DR — **[gap → closed only by the recovery escrow]** |
| + |
| +This is the finding that reshaped the plan. `eleven vault get -db <replica>` |
| +reads a vault from a raw DB replica with **no server** (CI-proven, |
| +`vault_offline_test.go`; the offline reader imports only `database/sql` + |
| +`internal/e2ee` — genuinely network-free). The *crypto* loop is broken. But the |
| +red-team showed the *operational* loop is **not** closed by the first draft: |
| +offline restore requires three inputs that, as designed, have **no fresh-machine |
| +source**, making the vault a **strict DR regression vs sops** in the |
| +all-machines-lost corner: |
| + |
| +1. **The keychain KEK** — Mac-local, in no backup (never synced), on the |
| + never-in-vault list. On a fresh Mac, `get -db` finds no KEK and `enroll` |
| + needs a live server → dead end. |
| +2. **`~/.eleven`** for the ops space — a fresh identity isn't a DB member, and |
| + joining needs a live server. Required to unwrap the thread key at all. |
| +3. **The S3 read credentials** — today only in sops + on the box (the |
| + `litestream` IAM key in `/data/litestream.env`); there is **no independent |
| + 1Password copy**, despite the first draft asserting one. |
| + |
| +sops, by contrast, recovers from `1Password age key + any bundle`, on any fresh |
| +**Linux** box, by one operator. The Eleven vault, in the same disaster, recovers |
| +**only if a warm enrolled operator device survived** — worse than sops. That is |
| +unacceptable for the fleet's root secrets, so the migration is **gated on a |
| +recovery escrow** that restores sops-parity (§"Build prerequisites" R1–R3): |
| + |
| +- A **recovery wrap** (`kind: "recovery"`) of the vault private key under a KEK |
| + derived (PBKDF2, reusing `crypto.js` `deriveBrandingKey`) from a |
| + **high-entropy recovery passphrase** escrowed in 1Password — openable on |
| + **any** machine (incl. Linux / `CGO_ENABLED=0`), no keychain, no biometric. |
| +- A **dedicated recovery member identity** whose `~/.eleven` is escrowed in |
| + 1Password — so a fresh machine can unwrap the thread key and read the log. |
| +- The **S3 read credentials** escrowed as an independent 1Password resident (not |
| + only sops + box). |
| + |
| +With those three in 1Password + the S3 replica, any operator recovers on any |
| +machine — genuine parity with sops. **Never-in-the-vault (hard list):** the |
| +recovery passphrase, the recovery identity's `~/.eleven`, the S3 read creds, |
| +WireGuard/SSH/human passkeys, release + Sparkle signing keys, each operator's |
| +personal `~/.eleven` and keychain KEKs, the box control token, canary envs. |
| +Litmus: *"could I need this with every server, including this one, dead?"* → yes |
| +means offline/1Password, never the vault. |
| + |
| +### 6. Bot-key exposure — **[verified] eliminated by construction, but deployment-enforced only** |
| + |
| +A bot is an E2EE member whose key lives on the box; a bot in the ops vault would |
| +put a decrypting key on the box and collapse §1. So the space runs **no bot and |
| +no sidecar** (`add-instance` without `add-bot`; no `lchat-bot@` unit), member |
| +invites off, admins = operators only. **Caveat the red-team raised:** this is |
| +deployment discipline, not code — a later `add-bot` on this host would silently |
| +re-introduce the exposure. The runbook adds a standing check (§runbook) that the |
| +ops host has no `lchat-bot@` unit and no agent member in its roster. |
| + |
| +### 7. Sessions-are-public (Claude transcripts) — **[gap on the read side] intake safe, read is a new footgun** |
| + |
| +Intake is airtight and matches sops: values enter only via the masked form or |
| +`eleven vault set` (hidden prompt / `--from-file`) in the operator's **own** |
| +terminal; **Claude is never a member of the ops thread** (bot-less, operators- |
| +only). But the red-team flagged a *new* exposure sops doesn't have: `eleven |
| +vault get` prints the value to **stdout** for command substitution, and box work |
| +here is routinely Claude-driven. A `get X | ssh box …`, a `set --from-file |
| +<file-Claude-just-wrote>`, or **the DR drill itself** (`get -db … name`) run |
| +inside a Claude-driven shell puts the secret in a public transcript — weaker |
| +than sops, whose `sops -d` writes a root-only file, not stdout. **Mitigations |
| +(R4 + rule):** add `eleven vault get -o <file>` writing a root-only file (never |
| +stdout), make it the *only* form used in any Claude-adjacent or deploy context, |
| +and keep the bare stdout `get` for interactive human use only. The gate/DR drill |
| +uses `-o`. |
| + |
| +### 8. Grant-approval trust — **[verified] out-of-band verify defeats the only forge path** |
| + |
| +Approving a `vault.request` hands the vault key to whoever's ephemeral public |
| +key is in the request. The request rides *inside* the thread-key-encrypted, |
| +GCM-authenticated blob, and the bot-less server holds no thread key — so it can |
| +neither forge a request nor substitute the ephemeral key. The only forge path is |
| +a **member / compromised operator session** posting a request with an |
| +attacker-chosen ephemeral key *and* display name. Mitigation: the approver |
| +confirms **out-of-band** (over the messenger, a call) that the person is really |
| +requesting now, before approving — the same discipline sops requires before |
| +adding a `.sops.yaml` recipient. The crypto review confirmed this works |
| +*precisely because it routes around the forgeable display name* (§8-adjacent: |
| +audit attribution is not authenticated — see §9). |
| + |
| +### 9. Insider / compromised-session abuse — **[gap] worse than "nuisance"; audit is not accountable** |
| + |
| +Three real limits the first draft rounded off, all confirmed against the code: |
| + |
| +- **Audit attribution is self-asserted and unverifiable.** The actor |
| + (`uid`/`name`) is written by the sender *inside* the encrypted event; the |
| + server's authoritative `messages.user_id` is never exposed to any reader |
| + (`hub.go`, `GET /api/messages`). Any member can post a `vault.*` event |
| + attributed to another operator, and no shipped surface (web Access panel, |
| + `eleven vault list`, offline `-db`) can detect or repair it. So the change log |
| + records **what changed, with an *unauthenticated* actor hint** — it is **not** |
| + the tamper-proof accountability the first draft's "attributed / better than |
| + sops" implied. (It does not weaken §8: out-of-band verify goes around the |
| + name.) |
| +- **Revoke can permanently lock everyone out — including DR.** Nothing binds a |
| + wrap to an operator or stops revoking *others'* wraps; the only guard is |
| + "never the last wrap." A compromised session can enroll its own wrap and |
| + revoke every other down to one → legitimate operators lose live *and* offline |
| + read (`get -db` opens only surviving wraps). This is **permanent lockout**, |
| + not a nuisance; recovery = re-provision, or fold the raw append-only log to |
| + before the malicious revoke, or the **recovery escrow** (R1 — the recovery |
| + wrap must be **un-revokable**, so griefing can never remove the escrow path). |
| +- **Destructive overwrite / downgrade.** Any member can `vault.change` any item |
| + without knowing its value, and on a locked vault a plaintext `value` downgrades |
| + a sealed item (rendered flagged; CLI "!" marker). Recoverable from the raw log |
| + / parallel sops, but it is *destroy/downgrade*, not just spam. |
| + |
| +Confidentiality is already lost the moment a session is compromised, so these |
| +are integrity/availability harms on top of that — but they must be named, and |
| +they are why R1's recovery wrap is un-revokable and why sops stays a live |
| +parallel backstop through the cutover. |
| + |
| +### 10. Offline read authenticates *value content*, not *metadata* — **[gap, bounded]** |
| + |
| +`get -db` has no signature over the DB file. AES-GCM authenticates each blob, but |
| +ECIES-to-a-public-key is free: an attacker who tampers a raw replica can seal a |
| +chosen thread key to the member's (public) `users.public_key` and re-author the |
| +log. **Sealed values are safe** (a forged value / substituted vault key won't |
| +open without the reader's KEK — fails loudly). **Names, the change log, and any |
| +unsealed item are not** — a tampered replica returns attacker-controlled |
| +metadata silently. Bounded by keeping the *fetch path* trustworthy (S3 creds |
| +stay offline, §5), not by crypto. State it: offline read authenticates value |
| +*content* when sealed, not metadata. |
| + |
| +### Summary (honest) |
| + |
| +| Threat | sops+age | Eleven locked vault | |
| +|---|---|---| |
| +| Server / S3 compromise | ciphertext only | ciphertext only (**equal**) | |
| +| Stolen **locked/off** device, disk/backup | plaintext age key → all secrets | KEK encrypted at rest → **no values** (**better**) | |
| +| Unlocked walk-up / malware as operator | reads age key → all | reads KEK → all (**equal**) | |
| +| Ex-operator | rotate real secrets | rotate real secrets (**equal**) | |
| +| DR: all machines + fleet dead | 1Password age key, any Linux box, solo | **needs the recovery escrow (R1–R3)** to match; without it, **worse** | |
| +| Insider audit / accountability | none (files) | change log with **unauthenticated** actor hint (**≈ equal**, not "attributed") | |
| +| Insider lockout (revoke) | n/a (files) | permanent lockout unless recovery wrap is un-revokable (**gap → R1**) | |
| +| Secret at deploy/read time | `sops -d` → root-only file | `get` → **stdout footgun**; needs `get -o` (**R4**) | |
| +| Sharing ergonomics | manual sealed bundles | live shared thread (**better**) | |
| + |
| +Net: **better or equal on confidentiality and sharing; a DR regression and an |
| +audit-accountability overclaim until the build prerequisites ship.** With |
| +R1–R4, it reaches parity-or-better on every axis. |
| + |
| +## Build prerequisites (BLOCKING — no real secret moves until these ship) |
| + |
| +Filed as issues; each is small and reuses existing primitives. |
| + |
| +- **R1 — Recovery wrap + un-revokable (#995).** A `kind: "recovery"` keyring wrap: the |
| + vault private key AES-GCM-wrapped under a PBKDF2-derived KEK from a |
| + high-entropy passphrase (reuse `crypto.js` `deriveBrandingKey`/PBKDF2, 600k |
| + iters). Reducer: treat a `recovery` wrap like any wrap for opening, but make it |
| + **un-revokable** (extend the last-wrap guard) so §9's revoke-lockout can't |
| + remove it. Web: a "Create recovery key" flow (passphrase → wrap) in the Access |
| + panel. CLI: `eleven vault recovery-wrap` to mint it. Cross-platform: the open |
| + path must work under `CGO_ENABLED=0` (no keychain), so recovery never needs a |
| + Mac. |
| +- **R2 — `eleven vault get -db --recovery` (#996).** Offline read that prompts for the |
| + recovery passphrase (PBKDF2 KEK) instead of the keychain, so a **cold machine |
| + / Linux box** can recover. Pairs with R1. |
| +- **R3 — Recovery-identity + S3-cred escrow (ops, not code).** A dedicated |
| + recovery member enrolled in the ops space; its `~/.eleven` and the S3 read |
| + creds sealed into 1Password as independent residents. Documented in RUNBOOK. |
| +- **R4 — `eleven vault get -o <file>` (#997).** Write the value to a root-only file |
| + (mode 600), never stdout; make it the only form used in deploy/DR/Claude- |
| + adjacent contexts. (Optional companion: hold the web unlock key as a |
| + non-extractable `CryptoKey` — closes the §2 XSS caveat; low priority given the |
| + bot-less space.) |
| + |
| +## The gate (nothing real moves before this passes) |
| + |
| +Revised per the DR review — it must exercise the *real* disaster, not a warm |
| +shortcut: |
| + |
| +1. Stand up the ops space bot-less; lock it; enrol both operators; **create the |
| + R1 recovery wrap** and escrow R3. |
| +2. Seed a few **throwaway** secrets; confirm both operators read them. |
| +3. **From S3 only** (no `/data` copy shortcut), from the **correct bucket** — a |
| + streamed space replicates to the activator/hibernation bucket under a lease, |
| + which the fleet-backup recipe *excludes*; confirm the restore config points |
| + at the streamed replica, not the fleet-backup bucket. |
| +4. **Assert freshness:** a secret written just before the fetch is present in the |
| + restored replica (guards the "streamed DB never picked up" trap). |
| +5. `eleven vault get -db --recovery -o <file>` **on a cold machine with no |
| + enrolled keychain KEK** (ideally a Linux box), using only the 1Password escrow |
| + (recovery passphrase + recovery `~/.eleven` + S3 creds). The value comes back |
| + with the ops server stopped. |
| +6. Confirm a **wrong** identity, **wrong** KEK, and **wrong** passphrase each |
| + yield nothing. |
| + |
| +Only when 1–6 pass do real secrets move; sops stays canonical in parallel for a |
| +full cycle afterward. |
| + |
| +## Migration runbook |
| + |
| +1. **Create the space, bot-less.** `lchat add-instance <ops-host>` (no |
| + `add-bot`, invites off). Admin token → 1Password. Add a standing check that |
| + the host has no `lchat-bot@` unit and no agent in its roster (§6). Space |
| + specifics → RUNBOOK.md. |
| +2. **Streamed replication;** verify the S3 replica appears in the *correct* |
| + bucket (gate step 3). |
| +3. **Lock + enrol + escrow.** Paul locks first (§"shared"); James via |
| + request → out-of-band-verified approve (§8) → finish; both enrol a CLI |
| + keychain key on each recovery machine; **mint the R1 recovery wrap** and place |
| + the R3 escrow in 1Password. |
| +4. **Pass the gate** (above) with throwaway secrets, cold-machine recovery |
| + included. |
| +5. **Seed for real.** For each `/data/*.env` / service token in sops **not** on |
| + the never-in-vault list: `eleven vault set --from-file <file> <name>` (hidden, |
| + own terminal). **`router.env` is split first** — its S3/AWS creds are |
| + never-in-vault (they fetch the replica), while its DNS-01 token *is* vault- |
| + worthy; migrate the DNS-01 token as its own item and keep the S3 creds out |
| + (the red-team's file-vs-secret granularity trap). Binary bot envs stay base64 |
| + for now. |
| +6. **Parallel run with reconciliation.** Both vaults hold the same secrets for a |
| + full cycle; edits go to **both**. Because values are sealed (`list` shows |
| + names only; each `get` costs a biometric), divergence is invisible to |
| + inspection — so **before the flip, reconcile every item** `vault get -o` vs |
| + `sops -d`, per secret, and record it. |
| +7. **Flip canonical** only after step 6's reconciliation passes. Update |
| + `docs/secrets.md` / RUNBOOK.md to name the Eleven vault as vault of record; |
| + keep sops intact as fallback. Announce to operators. |
| +8. **Decommission later, deliberately.** Only after a real operating cycle, never |
| + by deleting the age key or bundles — the archived sops bundle remains the |
| + independent home of R3's escrow *unless* R3 truly duplicated it into |
| + 1Password (verify it did before pruning anything). |
| + |
| +## Rollback |
| + |
| +sops runs untouched in parallel through the whole migration, so rollback is |
| +"keep using sops" — no restore, no data motion. The Eleven ops space tears down |
| +(`lchat remove <ops-host> --stop` + `/data` delete) with zero sops impact. The |
| +only point of no return is step 8's deliberate decommission, gated on a real |
| +operating cycle — never a deadline. |
| + |
| +## What stays true afterward |
| + |
| +- The never-in-the-vault list (§5) is permanent, in this doc and the RUNBOOK, |
| + and now explicitly includes the recovery passphrase and recovery `~/.eleven`. |
| +- Adding/removing an operator is a ratchet (§4): removal = rotate secrets. |
| +- Every secret enters via a hidden prompt in an operator's own terminal (§7); |
| + reads in any deploy/Claude-adjacent context use `get -o` (R4), never stdout; |
| + Claude never joins the thread. |
| +- The recovery wrap is un-revokable (R1) and its escrow (R3) is the independent, |
| + any-machine root of trust that keeps this from being a DR regression. |
amadan