eleven / ocho Public

Download this file

1	# 🤖 The secrets vault
2
3	Where the operator's server-side secrets are stored and recovered from.
4	`docs/secrets.md` is the inventory — *what* each secret authorizes, its blast
5	radius, how to rotate it; this doc is the storage scheme. Per the inventory's
6	rule, operator specifics (the actual remote, box paths, key locations) live in
7	`CLAUDE.local.md`, not here.
8
9	## The scheme: sealed env files in a small git repo
10
11	The vault is a tiny private git repository holding one file per service —
12	mirroring the env files the services actually run from — with every value
13	sealed by [SOPS](https://github.com/getsops/sops) using an
14	[age](https://age-encryption.org) recipient. The vault is the **copy of
15	record**: the env files deployed on a box are decryptions of it. The editing
16	flow is `sops <file>` (decrypts into your editor, re-seals on save), commit,
17	copy the decrypted result to the box, restart the service.
18
19	Why this shape and not a secrets-manager service (Vault/OpenBao/cloud KMS):
20
21	- **No new always-on service.** A secrets daemon is itself a security-critical
22	process with its own unseal problem, and it puts every service's boot behind
23	its availability. Env files on disk have no failure modes; the vault only
24	changes where they're *kept*, not how they're *consumed*.
25	- **Two single-binary tools + git.** `sops` and `age` are static binaries; the
26	whole flow is readable in one sitting and operable by hand — the second
27	rule (built to outlive AI) applied to operations.
28	- **Versioned and diffable.** SOPS seals values but leaves names readable, so
29	a diff shows *which* secret changed without showing what it changed to.
30	Rotation history, review, and "what did this env look like last month" come
31	free from git. This replaces hand-maintained password-manager items as the
32	recovery path — the drift where a shipped secret never made it into the
33	backup story is what this design exists to end.
34
35	Key material: one age keypair **per operator**, and every file encrypts to
36	all operator recipients — co-operators are peers with full access, which is
37	the bus-factor point: either operator can recover and run the fleet from
38	their own key alone. The **public** (recipient) keys are not secrets and are
39	committed in the vault's `.sops.yaml` (comma-separated on the rule); each
40	**private** key is that operator's root of trust — it never enters the vault
41	and lives in their password manager plus their machines. Private keys are
42	never shared, not even between operators: a new operator mints their own
43	pair and sends only the public half, over an authenticated channel.
44
45	Membership is a ratchet. Adding an operator is one commit (`.sops.yaml` +
46	`sops updatekeys` over every file). Removing one is **not** the reverse:
47	re-wrapping only stops future reads, so a departure or key compromise means
48	rotating the underlying secrets themselves, per their `docs/secrets.md`
49	entries. Grant accordingly.
50
51	A co-operator's vault access is necessary but not sufficient for full
52	continuity: the things deliberately kept *out* of the vault — the release and
53	Sparkle signing keys, provider-account sign-ins — need their own succession
54	path (password-manager sharing or emergency access), arranged operator-side.
55
56	## What belongs in it — and what never does
57
58	**In:** every secret from `docs/secrets.md` whose rotation story is "swap the
59	env and restart" — instance admin tokens, relay tokens, Stripe keys, OAuth
60	client secrets, provisioner tokens, the DNS-01 token, sidecar Anthropic keys.
61
62	**Never:**
63
64	- **The release signing key and the Sparkle key.** The inventory keeps these
65	offline, used only at publish time, and the vault must not quietly undo
66	that: aggregating them here would mean a single age-key compromise could
67	become a ship-code-to-the-fleet compromise. They stay out, permanently.
68	- **The age private key itself** — the vault can't hold its own root of trust.
69	- **Human credentials** — passkeys, SSH keys, sign-in sessions. Those belong
70	in a password manager and its SSH agent, which keep private keys off disk;
71	a file vault would be a downgrade for them.
72
73	## Plaintext residents: the operator runbook
74
75	Not everything in the vault is sealed. `RUNBOOK.md` — the shared operator
76	runbook (hosting, instances, DNS, deploys, releases) — lives in the repo as
77	plaintext, because it changes constantly, every operator edits it, and sealed
78	markdown would make diffs opaque and merges unresolvable. What makes this
79	safe is one absolute rule: **a plaintext resident holds locations and
80	procedures only, never a secret value** — the same contract the runbook kept
81	in its previous life as an operator's gitignored CLAUDE.local.md. Each
82	operator's CLAUDE.local.md imports the runbook at session start
83	(`@~/elevenmessenger/secrets-vault/RUNBOOK.md`), so agents on every operator
84	machine load the one shared copy instead of drifting private ones.
85
86	## Where it lives
87
88	**Only on operator machines.** Not on GitHub, not on any third-party forge,
89	and not on the production box either — the machine most exposed to attack
90	holds no copy. Git is distributed, so the vault needs no host at all: the
91	remote of record is a bare repository on the primary operator's machine, and
92	**every clone is a complete backup**. Co-operators pull and push through an
93	authenticated tunnel served off that machine (the product's own relay
94	fronting git smart-HTTP behind a token — the tunnel hostname is public,
95	since transcripts are public; the token, exchanged only over the messenger,
96	is the entire access control). Redundancy that leaves operator machines is
97	**ciphertext only**: a daily job bundles the repo, seals the bundle to the
98	operator recipients with age, and parks it in versioned object storage — so
99	even the plaintext runbook resident never leaves an operator's disk
100	readable. The forge features a host would add (issues, PRs, CI) are exactly
101	the features a secrets repo must never use. Which machine, which paths,
102	where the bundles land: the runbook itself.
103
104	Disaster recovery reduces to two things from different places: the age private
105	key (password manager) and any clone or bundle of the vault. Together they
106	reconstruct every service env; separately each is useless.
107
108	## Why the vault is private, even though every value is sealed
109
110	The crypto would survive publication — age is sound, and files are sealed *as
111	if* the repo were public (that discipline is free, and covers an accidental
112	visibility flip). It stays private anyway, deliberately:
113
114	- **Publishing is irreversible, so security would collapse to one key,
115	forever.** A public repo is forked, mirrored, and archived within minutes
116	and can never be unpublished. If the age key is *ever* compromised — years
117	later, via a stolen laptop or a bad backup — every version of every secret
118	ever committed becomes readable, and rotation can't recall ciphertext the
119	internet already has. Private, an attacker needs the repo *and* the key:
120	two independent failures instead of one.
121	- **Mistake tolerance would drop to zero.** The realistic failure isn't broken
122	crypto; it's a plaintext slip — a file committed before sealing, a
123	`.sops.yaml` rule that misses a key. Private, that's a calm rotation.
124	Public, bots watching the commit firehose for `sk_live_`/`sk-ant-` patterns
125	harvest it in seconds — faster than a revert.
126	- **Sealed values still leak the map.** SOPS leaves structure readable by
127	design (that's what makes diffs reviewable): env var names, service
128	filenames, and commit history would advertise which providers we depend on
129	and when keys rotate — and when they don't. Useful diff metadata for us is
130	reconnaissance for an attacker.
131	- **Public buys nothing.** The repo's only audience is the operator and their
132	deploy tooling. The product's openness story lives in the product's repos;
133	an ops vault gains no users, contributors, or auditors from being public —
134	there is no benefit on the other side of the scale.
135
136	## Membership policy: operators, not contributors
137
138	Vault membership tracks exactly one thing: **who operates production**. A
139	recipient reads every value, so the grant is equivalent to root on the fleet
140	— extend it only to someone you would hand root. Contributors are a
141	different circle: they get the code repo, the docs, and disposable
142	`test-*.11ch.at` canaries — never the vault. The two circles may coincide at
143	any given moment; the policy exists so vault access never rides along with
144	contributor onboarding by default. Every addition is deliberate, and every
145	exit costs a full rotation (membership is a ratchet, above).
146
147	## The OpenBao trigger
148
149	Sealed-files-in-git has one honest gap: git records write history but no
150	*read* audit, and grants are standing rather than short-lived leases.
151	Acceptable at two operators; wrong at ten. **Stated intent: when the
152	operator circle grows past a handful, this vault migrates to OpenBao** (the
153	MPL-2.0, Linux Foundation fork of Vault — still a single Go binary), which
154	buys read audit logs, leases, and per-path policy at the price of an
155	always-on service with an unseal ceremony. The current design was chosen
156	knowing this and doesn't resist the move: recipients map to policies, files
157	map to secret paths, and the rotation recipes in `docs/secrets.md` are
158	engine-agnostic.
159
160	## Bootstrapping
161
162	```sh
163	age-keygen -o vault.agekey # store the key per the runbook, then shred the file
164	git init --bare ~/path/vault.git # on the primary operator's machine — not a server
165	# in a fresh clone:
166	cat > .sops.yaml <<EOF
167	creation_rules:
168	- age: <public key from age-keygen>
169	EOF
170	sops <service>.env # create/edit — seals on save
171	git add -A && git commit -m 'seed' && git push
172	```
173
174	Deploying a change: `sops -d <service>.env` to a root-only temp file, copy it
175	to the box's env path, restart the unit, delete the temp file. Adding a
176	secret: seal it here **and** add it to `docs/secrets.md` in the same change —
177	the inventory's same-PR rule is what keeps the two in step.
178