| 1 | # 🤖 CARLOS technical blueprint |
| 2 | |
| 3 | The stack and infrastructure shared by the family. Concrete values (instance |
| 4 | types, timeouts, cap numbers) are the family's defaults — snapshots, not |
| 5 | contracts; the shapes are the contract. |
| 6 | |
| 7 | > **Scope (re-drawn 2026-08-17):** this file is the *under-the-hood and |
| 8 | > self-hosting* reference — what the platform does on an app's behalf, |
| 9 | > and the complete recipe for running without it. An app **on** the |
| 10 | > platform (hosted or self-hosted deployment) hand-rolls none of "The |
| 11 | > carlos core", "Replication", the deploy recipe, or config delivery: |
| 12 | > those are driven through the `carlos` CLI — see |
| 13 | > [platform.md](platform.md). Bullets marked **Automatic on rastrillo** |
| 14 | > are enforced by the framework. The Go server, SQLite, frontend, and |
| 15 | > crypto sections bind every app regardless. |
| 16 | |
| 17 | ## Go server |
| 18 | |
| 19 | - One module, one static binary: `CGO_ENABLED=0`, cross-compiled |
| 20 | (`GOOS=linux GOARCH=arm64`), `-ldflags "-s -w -X main.buildVersion=$(git |
| 21 | rev-parse --short HEAD)"`. A deploy is one file you scp. |
| 22 | - `main.go` is subcommand dispatch and environment reading only — everything |
| 23 | real lives in `internal/`, grouped by concern. "A new concern gets a new |
| 24 | small package, not a new file in the biggest one." |
| 25 | - Subcommands: a platform-deployed rastrillo app needs none — the binary |
| 26 | speaks `--socket`/`--db` via `rastrillo.Run` and everything operational |
| 27 | is a `carlos` verb. Self-hosted-without-platform apps grow their own |
| 28 | (`serve`, `router`, `add`/`remove`/`instances`, `ops …`, `version`); |
| 29 | ops verbs are code, not runbooks — and delete any ops verb the moment |
| 30 | it can report success while changing nothing (one did, for weeks). |
| 31 | - HTTP is stdlib: `http.NewServeMux` with method+pattern routes |
| 32 | (`mux.HandleFunc("GET /api/thing", …)`), `writeJSON`/`httpError` helpers, |
| 33 | `http.MaxBytesReader` on every body. Wire all routes in one |
| 34 | `Server.handler()` so httptest exercises the real router. |
| 35 | - Config: flags with `<APP>_*` env fallback. **Secrets are env-only, never |
| 36 | flags** (argv is visible in `ps`). On the platform they are delivered by |
| 37 | `carlos env`/`carlos secrets` (sealed, converged in seconds); ssh-stdin |
| 38 | or 0600 `EnvironmentFile`s is the off-platform form only. |
| 39 | - Dependency floor: `modernc.org/sqlite` + `golang.org/x/crypto` (autocert). |
| 40 | Everything else earns its place. Hand-roll small API clients (S3 SigV4, |
| 41 | Stripe form-posts) instead of importing SDKs; if usage grows past a |
| 42 | handful of calls, swap in the SDK rather than growing the hand-rolled one. |
| 43 | |
| 44 | ## SQLite rules (hard-won) |
| 45 | |
| 46 | - `modernc.org/sqlite` (pure Go — cgo would break the static binary). |
| 47 | - One database file per instance; a separate registry DB for the router. |
| 48 | - **Automatic on rastrillo.** Open with: `busy_timeout` set **before** |
| 49 | `journal_mode=WAL` (the other order crashes with SQLITE_BUSY under |
| 50 | concurrent open), then `SetMaxOpenConns(1)`, then migrate. |
| 51 | - **Automatic on rastrillo.** Migrations are idempotent `CREATE TABLE IF |
| 52 | NOT EXISTS` plus an additive `ALTER` list whose errors are ignored only |
| 53 | when the message contains "duplicate column". Deploying new code over |
| 54 | an old DB must always be safe. Never delete data to update. |
| 55 | - Cross-process reads: a long-lived modernc connection can serve a stale WAL |
| 56 | snapshot of another process's commits — read fleet-visible state (the |
| 57 | route table) over a fresh connection. |
| 58 | - Registry change detection: poll a SQL fingerprint over a fresh connection |
| 59 | (`PRAGMA data_version` is blind to cross-process commits), and reload on |
| 60 | the *first* fingerprint too — baselining it away loses routes added during |
| 61 | startup. |
| 62 | - Scale by adding instances, not replicas of one. SQLite is not the |
| 63 | bottleneck you think: 200k attendees measured at 29 MB, FTS5 search <1 ms. |
| 64 | Don't shard for "performance". |
| 65 | |
| 66 | ## The carlos core (`internal/carlos`) |
| 67 | |
| 68 | **This whole section is the platform's job now** (`carlosframework/platform`, |
| 69 | live), not something an app hand-rolls anymore — regardless of whether it |
| 70 | uses rastrillo. Kept below as the reference for self-hosting outside the |
| 71 | platform, or for understanding what it's doing on an app's behalf. |
| 72 | |
| 73 | - **Registry**: SQLite table of `host → unix socket` routes (+ optional |
| 74 | slug, kind `instance|service`, lifecycle status, version). It doubles as |
| 75 | the ACME allowlist: a host nobody provisioned never gets a certificate, so |
| 76 | pointing DNS at the box entitles you to nothing. |
| 77 | - **Router**: the only process bound to :80/:443. Autocert HTTP-01 with |
| 78 | `HostPolicy` = registry lookup; per-host certs, no wildcards until the |
| 79 | ~50-certs/week Let's Encrypt budget demands DNS-01. Lazy |
| 80 | `httputil.ReverseProxy` per socket with a unix `DialContext`; a retrying |
| 81 | dialer (only `ECONNREFUSED`/`ENOENT`, window sized past the instance drain |
| 82 | time) masks restarts during deploys; swap the whole route table atomically |
| 83 | on reload. For streaming responses set `FlushInterval = -1`. The router |
| 84 | answers `/healthz` and `/api/version` for itself, and **every instance |
| 85 | must also serve `GET /api/version` reporting its build sha** — deploy |
| 86 | verification polls all of them. |
| 87 | - **Instances**: systemd template units (`app@<host>.service`), one per |
| 88 | account, `Restart=always`, socket in a tmpfiles.d-managed dir (or under |
| 89 | `/data` so a runtime-dir wipe can't yank sockets). Instances survive a |
| 90 | router restart because they are not its children. |
| 91 | - If the router proxies identity, strip those headers from every inbound |
| 92 | request before minting them (test that forgery is stripped). Unix socket |
| 93 | paths must stay under ~104 bytes (`sun_path`) — tests use |
| 94 | `os.MkdirTemp("", "xx")`, not `t.TempDir()`. |
| 95 | - Provisioning order: make the directory, create the owner, start the |
| 96 | instance, wait for it to actually answer, and only then publish the route |
| 97 | — publishing earlier makes the owner's first visit a 502. |
| 98 | - Hibernation (live and default-on for the platform's provisioned |
| 99 | instances; build it yourself only off-platform): park sleeping |
| 100 | instances' state in S3 under a single-writer lease (one object that is |
| 101 | both lock and manifest, expiring, with fenced commit), wake on request, |
| 102 | sleep via one sweep goroutine — never a timer per instance. |
| 103 | |
| 104 | ## Replication |
| 105 | |
| 106 | **Also the platform's job now**, for the same reason as the carlos core |
| 107 | above — Litestream is configured and run by the platform's host agent, |
| 108 | not by each app. |
| 109 | |
| 110 | - **Litestream streams every data-bearing SQLite DB to S3. No exceptions.** |
| 111 | "A box is disposable; the bucket is not." Adopted family-wide after a real |
| 112 | data-loss incident. |
| 113 | - The litestream config is *generated* by an ops verb on a timer, never |
| 114 | hand-edited: litestream accepts a glob in `path` and then silently |
| 115 | replicates nothing — that exact failure happened on first provision and |
| 116 | was caught only by checking the bucket. |
| 117 | - Restore drills are automated: a systemd timer on the box (daily) restores |
| 118 | every replica to a private tmp dir and requires |
| 119 | `PRAGMA integrity_check = ok` and a non-empty schema. "A backup nobody has |
| 120 | ever restored is not a backup — it is a folder in a bucket and a feeling." |
| 121 | - The bucket is versioned, private, per-product, with IAM scoped to exactly |
| 122 | it. Deploys never touch `/data/*.db*`. |
| 123 | - Canaries can restore the prod replica (`--seed-prod`) so review happens on |
| 124 | real data — for E2EE apps the canary holds ciphertext it cannot read. |
| 125 | |
| 126 | ## The home vault (single sign-in across instances) |
| 127 | |
| 128 | Instance-per-account creates a sign-in problem by construction: a person |
| 129 | with three instances must not need three credentials. This has tripped up |
| 130 | every app in the family; solve it on day one, not after. The answer is |
| 131 | **home** — a small companion service (Eleven's home server, ported whole |
| 132 | into siblings rather than reinvented) giving one passkey sign-in that spans |
| 133 | every instance: |
| 134 | |
| 135 | - Home is anonymous and content-blind. It stores credential public keys |
| 136 | and **one padded, sealed blob per person** — the vault of instance |
| 137 | addresses and wrapped keys, sealed in the browser — and nothing else. |
| 138 | Read the schema as the privacy policy. |
| 139 | - Instance addresses ride the URL `#fragment` on the way through home, so |
| 140 | home's server never learns which instances a person uses. |
| 141 | - When instance hostnames are hidden from users entirely, the app runs on |
| 142 | home's origin and calls the instance cross-origin with bearer tokens — |
| 143 | the address bar shows home, the content flows direct. **Home never |
| 144 | proxies instance traffic**: a proxy would make home load-bearing and |
| 145 | hand it a durable session→instance join. Where instances are directly |
| 146 | addressable instead, home is only the vault, not the front door. |
| 147 | - Home is glue (factor XI): thin, blind, replaceable. Losing it costs a |
| 148 | convenience, never content — the instances still hold everything. |
| 149 | - Passkeys are scoped per relying party, so pick home's hostname |
| 150 | deliberately and early: renaming an RP later is a multi-phase drill |
| 151 | (legacy RP accepted for sign-in, never registration, so the crossover |
| 152 | drains), not an edit. Never share home's origin or cookie domain with |
| 153 | anything that carries third-party content. |
| 154 | - Server-trust apps (Tito's model) reach the same one-sign-in outcome |
| 155 | differently: identity lives beside the router on the credential origin |
| 156 | (the root domain), and reaches instances only as trusted headers the |
| 157 | router strips from every client request before minting its own. |
| 158 | |
| 159 | ## Frontend |
| 160 | |
| 161 | - **Server-generated markup is the default, not the fallback.** The core |
| 162 | flow ships with zero JS and must work without it (confirm pages for |
| 163 | destructive actions, form posts); JS is progressive enhancement on top. |
| 164 | - **Hide the machinery.** Every instance has a URL, a key, a cipher — the |
| 165 | person sees none of them by default. Instance hostnames are deliberately |
| 166 | meaningless (sqids, hidden behind home); no nerdspeak in the default |
| 167 | flow — no "instance", "key", "encrypt", "URL" in member-facing copy |
| 168 | (enforceable as a word blacklist in the browser drive); never show an |
| 169 | identifier where a name belongs — prompt for the name instead. The |
| 170 | scope is the *default flow* only: settings, docs, the self-hosting |
| 171 | path and the published trade-offs speak plainly — this hides the |
| 172 | machinery from the person who didn't ask, it never conceals how the |
| 173 | system works from the person who did. |
| 174 | - When JS is needed: a small self-contained ES module — one file, one |
| 175 | concern, own state, no globals, no bundler. If a module needs another |
| 176 | module's internals, that's a server round-trip or a redesign, not an |
| 177 | import. |
| 178 | - **Automatic on rastrillo** (the same test, carried into `carlos vet`). |
| 179 | 300-line cap per module, enforced by a test; at most one named |
| 180 | coordinator module (the store/app shell that wires modules together and |
| 181 | owns no policy of its own) may carry a higher test-enforced cap. Caps |
| 182 | only ratchet down — raising a number to make a build pass is the exact |
| 183 | move the test exists to prevent. |
| 184 | - Storage and network are injected, never reached for — unit tests hand in |
| 185 | fakes; `localStorage` access from a leaf module is a test failure. |
| 186 | - Never `innerHTML` untrusted content — build DOM nodes with a helper whose |
| 187 | falsy-dropping filter guards the leaked-value bug class. |
| 188 | - When a view genuinely needs reactivity, the family's one sanctioned |
| 189 | reactive dependency is **VanJS, vendored as readable source**: views |
| 190 | return `{ el, update }` and build DOM through the same falsy-dropping |
| 191 | helper — in van bindings too, never bare tags. Anything heavier than |
| 192 | VanJS is a redesign, not a dependency. |
| 193 | - Assets via `go:embed`, cache-busted by build version (`?v=<sha>` or |
| 194 | fingerprinted names) so a deploy is never a hard refresh. |
| 195 | - If a JS dependency is truly needed, vendor it as readable source. No font |
| 196 | CDNs — self-host woff2. No toasts; feedback is inline. |
| 197 | |
| 198 | ## Crypto and identity defaults |
| 199 | |
| 200 | - **Automatic on rastrillo** (`rastrillo/crypto`, opt-in — an app that |
| 201 | isn't E2EE imports nothing from it). Platform primitives only: WebCrypto |
| 202 | in the browser, stdlib/x/crypto in Go. One 32-byte seed → HKDF-SHA256 |
| 203 | with domain-separated info strings (`app/purpose/v1`) → purpose keys. |
| 204 | ECDH P-256 → HKDF → AES-GCM for envelopes; ECIES to wrap per-thread/ |
| 205 | content keys. |
| 206 | - Secrets that must not reach the server ride the URL `#fragment`. |
| 207 | - Identity: passkeys with the WebAuthn PRF extension wrapping the seed |
| 208 | (possession of the PRF output is the security boundary, not the server) — |
| 209 | or, for server-trust apps, magic link + mandatory TOTP. Either way: 256-bit |
| 210 | random tokens, stored only as SHA-256, never logged; rate limits keyed by |
| 211 | email, IP, and user. **Partial on rastrillo:** `rastrillo/webauthn` |
| 212 | provides registration/assertion verification primitives (ES256 only, no |
| 213 | attestation checking); the PRF-extension seed-wrapping and home-vault |
| 214 | flow described here remain hand-built. |
| 215 | - **Automatic on rastrillo** for the crypto core specifically — the same |
| 216 | golden-vector CI discipline, promoted to the framework so it protects |
| 217 | every app that uses it from one shared fixture instead of a private |
| 218 | copy each. When the protocol exists in two or more languages (Go + JS + |
| 219 | Swift), golden vectors generated by the Go side are the spec: every |
| 220 | implementation must reproduce them, CI/`go test` compares the copies, |
| 221 | and a rule change regenerates the vectors in the same commit. Vectors |
| 222 | are not snapshots — never update them to make a test pass. |
| 223 | - Enumeration resistance where the domain is sensitive: auth endpoints |
| 224 | answer identically whatever happened. |
| 225 | |
| 226 | ## Boxes and deploys (self-hosting outside the platform only) |
| 227 | |
| 228 | **On the platform, this whole section is `carlos deploy`** — ship, |
| 229 | promote, watch `X-Carlos-Version` — and box choice, TLS, and systemd |
| 230 | hardening are the platform operator's concern. What follows is the |
| 231 | recipe the platform automated, for running without it. |
| 232 | |
| 233 | - One or two tiny ARM boxes: AWS t4g.nano (prod) / t4g.micro (dev) in |
| 234 | eu-west-1, or a single Hetzner box. Dedicated cloud account per product |
| 235 | with a named CLI profile — and never another product's profile ("Leave |
| 236 | woodstar alone!"). A Hetzner box still replicates to an S3 bucket in the |
| 237 | product's own AWS account — the box choice doesn't change the bucket |
| 238 | story. |
| 239 | - IaC in OpenTofu: two boxes, DNS, one bucket, IAM. State in a versioned S3 |
| 240 | bucket with `use_lockfile` once more than one machine applies. On |
| 241 | hand-built boxes: `lifecycle { ignore_changes = [ami, user_data]; |
| 242 | prevent_destroy = true }` — an AMI-drift replacement once deleted root |
| 243 | volumes. Never `tofu apply` from a checkout older than main: state removal |
| 244 | destroys resources. |
| 245 | - TLS on-box: autocert inside the router with the registry as allowlist. |
| 246 | (The static-binary-Caddy-in-front fallback is historical — pre-router |
| 247 | apps used it; nothing current does.) No load balancer, no CDN, no |
| 248 | containers. |
| 249 | - systemd hardening on every unit: dedicated user, empty |
| 250 | `CapabilityBoundingSet` (router alone keeps `CAP_NET_BIND_SERVICE`), |
| 251 | `NoNewPrivileges`, `ProtectSystem=strict` with narrow `ReadWritePaths`, |
| 252 | `PrivateTmp`, `MemoryDenyWriteExecute`, `MemoryMax`, `Restart=always`. |
| 253 | - Access: prefer SSM Session Manager with no SSH port at all; on SSH boxes, |
| 254 | operator accounts over WireGuard, root bootstrap-only, sudoers as a |
| 255 | committed reviewed allowlist. |
| 256 | - **Deploy = build, install, restart, VERIFY.** Fetch `origin/main`, build |
| 257 | exactly that commit in a throwaway git worktree (local edits structurally |
| 258 | cannot ship; there is no way to deploy a branch — that is a feature), scp |
| 259 | to a uniquely-named staged file, `install -m755`, restart router + all |
| 260 | instances, then poll `/api/version` until every instance socket reports |
| 261 | the deployed sha — a healthy service alone is not evidence the build won. |
| 262 | Dev first; prod after dev verifies. |
| 263 | - After every deploy, run the committed headless-browser console check |
| 264 | against the live host (see process.md). |
| 265 | - Mail, when needed, is SES with per-sender tenants so one user's bounces |
| 266 | can't poison everyone's deliverability. Secrets come from a password |
| 267 | manager (1Password/Bitwarden) at deploy time. |
| 268 | |
amadan