| new file mode 100644 |
| index 000000000..99e3a379f |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-01-tito-production-infra-design.md |
| @@ -0,0 +1,366 @@ |
| +# 🤖 Tito Production Infra — the First Self-Hosted CARLOS Deployment |
| + |
| +> 🤖 This document was written by an LLM (Claude), on the ideas, instruction, |
| +> and editing of humans, per [factor X](https://11factor.org/#x). |
| + |
| +**Status:** Concept spec, approved in brainstorming 2026-08-01. Revised the |
| +same day: the original draft placed the *flagship* deployment in Tito's AWS |
| +account with Tito as its first dedicated tenant. The deployment topology |
| +decision (platform spec §12) supersedes that framing — **Tito runs its own |
| +single-tenant CARLOS deployment in the Tito AWS account**, as one of the two |
| +use-cases that extract and prove the self-hosted story. The infrastructure |
| +below is unchanged by the reframe; only ownership and tenancy language moved. |
| + |
| +**Relationship to the platform spec:** the |
| +[CARLOS platform design](2026-08-01-carlos-platform-design.md) defines the |
| +substrate in the abstract. This document pins one deployment of it to the |
| +ground — provider, regions, buckets, DNS, money — for Tito. Where the two |
| +could diverge, the platform spec wins; this document only chooses among |
| +options it already allows. |
| + |
| +## 1. Purpose |
| + |
| +Stand up Tito's own deployment of the CARLOS platform as real, multi-region |
| +production infrastructure, with Tito Go as its (only) tenant. Goals, |
| +verbatim from the brainstorm: |
| + |
| +- All orchestration just works — no runbook between "customer signs up" |
| + and "instance serving." |
| +- Creating an account places an instance on a host, in seconds. |
| +- Hosts auto-update themselves. |
| +- Multi-region from day one: US, EU, IE, Asia, Australia. |
| +- Customers choose their region, and the choice is a data-at-rest promise. |
| +- Shared PDF rendering (one Gotenberg, not five). |
| +- A mesh design, and a written-down position on multi-edge. |
| + |
| +## 2. Decisions (settled in brainstorming, 2026-08-01) |
| + |
| +1. **All-AWS fleet.** One provider, one image story, IAM roles instead of |
| + keys on disk, per-region buckets. The required regions (IE, AU) rule |
| + out Hetzner regardless; uniformity is the "orchestration just works" |
| + property. The existing Hetzner box survives only until cutover (§12). |
| +2. **Control plane homes in IE (eu-west-1).** One edge serves |
| + `go.tito.io` — sign-in, all admin, Classic-style `/{slug}` URLs — |
| + from Ireland. Cross-region admin pays one proxied hop; buyer traffic |
| + never does (§6). Replicated edges are a platform-native later, not a |
| + day-one build. |
| +3. **Self-serve signup.** Sign-up on `go.tito.io` (existing identity |
| + flow), pick slug + region, instance ready in single-digit seconds. The |
| + operator CLI is the same provisioner core with a different front. |
| +4. **Activator runtime model, wholesale.** Instances are host-agent |
| + children under the single-writer object-store lease, replicated |
| + continuously, hibernated by default. Placing an instance is a data |
| + write, never host surgery. (This is the platform's host agent; Tito |
| + retires systemd-per-instance and the generated litestream config.) |
| +5. **Residency = data at rest in-region.** The instance database, WAL |
| + stream, parked copy, and blobs live in the chosen region's bucket. |
| + Requests may transit the IE edge; email sends via SES eu-west-1; PDFs |
| + render in transit in IE. Matches the platform's residency definition |
| + (spec §6): rest and execution in-region, transit unconstrained. |
| +6. **One box + Elastic IP per region.** The `carlos` host agent/router is |
| + the front door; no load balancers. A dead box is a ~10-minute |
| + restore-from-bucket runbook, rare by construction. Growth within a |
| + region is another host in the pool, not new machinery. |
| + |
| +### Reconciliations with the platform spec |
| + |
| +The brainstorm reached three mechanisms via Eleven that the platform spec |
| +supersedes; outcomes stand, mechanisms defer to the platform: |
| + |
| +- **Auto-update** is release **channels + adopt-on-wake** (spec §4), not |
| + Eleven's hourly Ed25519-manifest timer. Releases are content-addressed |
| + with build attestations; advancing `stable` is one lease-guarded pointer |
| + write; hosts roll instances at natural boundaries. Bake windows become |
| + staged promotion (`canary → beta → stable`), which is the same safety |
| + property expressed as pipeline shape rather than clock. |
| +- **One deployment, not five meshes.** Hosts carry a region attribute and |
| + belong to pool `tito`; the routing table is derived from the bucket. |
| + There is no per-region membership machinery and no global directory |
| + service to build — the deployment bucket already is one. |
| +- **Placement is deliberately dumb** (spec §6): an instance wakes on |
| + whichever eligible host answers first, weighted by load. Eleven's HRW |
| + hashing is not carried over; eligibility (pool ∩ residency, enforced at |
| + lease-claim) does the real work, and hibernation makes placement |
| + mistakes cheap. |
| + |
| +## 3. Regions |
| + |
| +| Code | AWS region | Role | |
| +|---|---|---| |
| +| `us` | us-east-1 | instance region | |
| +| `eu` | eu-central-1 (Frankfurt) | instance region | |
| +| `ie` | eu-west-1 (Dublin) | instance region **+ edge/control plane** | |
| +| `ap` | ap-southeast-1 (Singapore) | instance region | |
| +| `au` | ap-southeast-2 (Sydney) | instance region | |
| + |
| +Region codes are customer-visible (signup picker, relocation, invoices' |
| +"where your data lives" line). IE being both the deployment home and an |
| +ordinary instance region is deliberate — Tito is Irish, and the existing |
| +S3/SES footprint is already eu-west-1. |
| + |
| +## 4. Buckets and IAM (residency, enforced by construction) |
| + |
| +All in the Tito AWS account (**665106592142**): |
| + |
| +- **`carlos-tito-platform-665106592142`** (eu-west-1) — the deployment |
| + bucket (spec §3): apps, releases, channels, environment profiles, |
| + placement pools, instance records, host heartbeats, the shared cert |
| + cache (KMS-encrypted). Routing metadata only — "the edge stays global; |
| + routing metadata is not resident data" (spec §6). Nothing |
| + customer-shaped rests here. |
| +- **`carlos-tito-<region>-665106592142`** ×5 — the regional data buckets: |
| + parked databases, WAL streams, single-writer leases (`lock.json` beside |
| + the data it guards), blobs. **This bucket is the residency promise.** |
| + In this single-tenant deployment, pool `tito` is the only pool, and the |
| + regional bucket split is what carries residency; the same shape serves |
| + the flagship's dedicated tiers (spec §6). |
| +- **`carlos-tfstate-665106592142`** — OpenTofu state (versioned, SSE, |
| + native locking), per the house pattern (woodstar/slopbox/keymail). |
| + |
| +IAM makes residency structural rather than behavioral: each region's host |
| +role is scoped to the deployment bucket **plus its own region's data |
| +bucket and nothing else**. A Sydney host cannot claim a Dublin lease |
| +because it cannot touch the Dublin bucket — the wrong-region wake the |
| +spec requires to "fail loudly" fails at the IAM layer, below any code we |
| +could get wrong. Litestream's replication lease doubles as the takeover |
| +guard, as in Eleven: a host that wrongly believes it owns an instance |
| +loses the conditional-write race, loudly. |
| + |
| +## 5. Hosts |
| + |
| +One Graviton box (t4g.medium to start) + Elastic IP per region, from a |
| +stock arm64 Debian AMI + cloud-init; the `carlos` binary in host-agent |
| +role is the only meaningful software on it. Security group: 80/443 world, |
| +SSH from admin IPs; inter-host traffic is authenticated at the app layer |
| +(fleet token over TLS), not by network topology. Host OS patching is |
| +unattended-upgrades plus occasional AMI refresh via tofu. Releases ship |
| +per-target binaries (arm64 for the fleet, amd64 for dev and the Hetzner |
| +interim), which the platform's content-addressed release format already |
| +carries. |
| + |
| +The box is disposable by construction — all durable state is in buckets — |
| +so there is no backup story for hosts at all, only for buckets |
| +(versioning + the daily restore drill Tito already runs, generalized). |
| + |
| +## 6. Routing, DNS, and certs |
| + |
| +`tito.io` is already a Route53 zone; no delegation needed. |
| + |
| +**Tofu owns the static skeleton:** `go.tito.io` → IE edge EIP; regional |
| +aliases `<region>.go.tito.io` → each region's EIP; wildcard |
| +`*.go.tito.io` → IE as the catch-all. **The platform owns dynamic |
| +records:** account creation writes `<slug>.go.tito.io` CNAME → its |
| +region's alias (TTL 60); relocation flips it; the wildcard catches |
| +not-yet-propagated or unknown hosts at the edge, which proxies or 404s. |
| +Hard boundary: nothing account-shaped ever enters tofu state, so |
| +`tofu apply` is always safe and never touches a customer. |
| + |
| +Traffic shapes: |
| + |
| +- **Buyer traffic** (account subdomains, later custom domains) resolves |
| + directly to the owning region's host and terminates there. In-region |
| + latency; no dependency on the edge being up. |
| +- **Organizer/admin + root-URL traffic** (`go.tito.io/...`) terminates at |
| + the IE edge, which serves control-plane surfaces itself and proxies |
| + instance-bound requests to the owning region's host — one hop, |
| + fleet-token-authenticated. Tito's `X-Tito-Auth-*` rule tightens |
| + accordingly: instances trust those headers only from their own host's |
| + socket; hosts trust them only on the authenticated inter-host channel; |
| + every host strips them from public ingress. |
| + |
| +**Certs:** Let's Encrypt rate-limits per registered domain (`tito.io`), |
| +so per-host HTTP-01 across five regions would share one 50-certs/week |
| +budget. Instead the edge issues a `*.go.tito.io` wildcard via DNS-01 |
| +against Route53 and every host consumes it through the KMS-encrypted |
| +cert cache in the deployment bucket (the `cert cache = S3` seam Eleven |
| +reserved for exactly the day two boxes share issuance). Custom domains, |
| +when they land, use per-host HTTP-01 on the owning region's host — their |
| +own registered domains, no shared budget. |
| + |
| +## 7. Provisioning and placement |
| + |
| +Signup lives on the edge (existing magic-link + TOTP identity flow), |
| +collects slug + region, and calls the provisioner in-process — the |
| +activator model means no privileged host surgery, so no separate root |
| +daemon. The provisioner: validates the slug against the reserved list, |
| +writes the instance record (app `tito`, pool `tito`, residency |
| +`<region>`, channel `stable`) to the deployment bucket, creates the |
| +Route53 record, and seeds the instance database in-region on first wake |
| +(schema + starter content — the `titogo seed` path). Eligible host |
| +agents race the lease; first claim wins. Target: account serving in |
| +single-digit seconds. Abuse controls day one stay boring: TOTP-enrolled |
| +identity required, per-IP and per-identity rate limits, reserved slugs. |
| + |
| +The operator CLI verb is the same core; support can do everything signup |
| +can, including create accounts in regions before self-serve exposes them. |
| + |
| +## 8. Runtime: park, wake, update |
| + |
| +The platform host agent, as specced (§3, §4): instances are children |
| +under the object-store lease; continuous WAL streaming to the regional |
| +bucket; restore only when local disk lacks the database (local disk is a |
| +warm cache); hibernate after a generous idle window (30m to start — |
| +event accounts sell for weeks and sleep for months, and hibernation is |
| +what makes five small boxes hold thousands of accounts); due-time wakes |
| +cover scheduled sends and on-sale opens, so a sleeping instance never |
| +misses its own schedule. |
| + |
| +Updates are channel adoption (§2 reconciliation): `stable` advances, and |
| +host agents roll instances at wake or in a drain-and-restart sweep for |
| +long-running ones. A fleet with three releases live is a normal state; |
| +rollback is the pointer moved backwards. Tito canaries subscribe to |
| +`canary/*` channels and are ordinary instances — the per-session canary |
| +discipline in titogo's CLAUDE.md becomes `carlos canary`. |
| + |
| +## 9. svc/pdf: Gotenberg |
| + |
| +The platform spec's PDF/render service (§7) — **seeing, therefore |
| +stateless by construction** — instantiated as one Gotenberg behind Caddy |
| +on its own t4g.small in IE: pinned image, basic auth, security group |
| +admitting exactly the fleet's EIPs, no bucket grant, no database binding, |
| +no volume. Tito instances reach it through their environment profile; |
| +PDF work is async attachment generation, so cross-region latency is |
| +invisible and one region suffices. Renders transit; nothing rests — |
| +residency untouched. |
| + |
| +Honest v1 note: Gotenberg is a Docker container, not a CARLOS app, so |
| +day one it is leased infrastructure bound as a service endpoint (the |
| +attached-database pattern: provisioned and bound by the platform, not |
| +reinvented by it). Wrapping it in a `svc/pdf` CARLOS front can come |
| +later if the service class mechanics want it; a second box is |
| +`count = 2`, not a design change. |
| + |
| +## 10. Relocation |
| + |
| +Copy the instance's prefix from source to destination regional bucket |
| +(excluding the lease object, or the destination would 503 until TTL), |
| +flip the record's residency, update DNS. The propagation window needs no |
| +replication — lease + proxy-to-owner cover it at the cost of one hop. |
| +Support-run day one; self-serve later if demand exists. This is also the |
| +residency escape hatch: a customer who chose wrong fixes it with one |
| +command, and at-rest data genuinely moves. |
| + |
| +## 11. Multi-edge: considered, deliberately deferred |
| + |
| +Three different things hide under "multi-edge": |
| + |
| +1. **Static assets at CDN edge** — cheap, compatible, worth doing when |
| + event pages get heavy. CloudFront over regional origins for static |
| + paths only. No architectural cost; land any time. |
| +2. **Replicated edges** (control-plane surfaces served near the user) — |
| + platform-native eventually ("edges are replicated; sessions ride with |
| + identity," spec §8), blocked on replicated identity. The IE-home |
| + decision costs far-region organizers ~200–300ms on admin surfaces, |
| + acceptable for admin. Revisit when organizer geography demands. |
| +3. **Instance-level multi-edge** (one account served from several |
| + hosts) — for Eleven this is plausible because its storage is |
| + append-only, `mergeable` in manifest terms. **Tito's core write is |
| + the opposite:** inventory decrement under contention is exactly what |
| + doesn't merge, and the platform never merges an `exclusive` store |
| + (spec §5, §10). On-sale spikes are served by measured SQLite headroom |
| + (200k-attendee events with room to spare) and, if ever needed, a |
| + waiting-room page — not by distributing the writer. **Verdict: |
| + likely never; recorded here so nobody relitigates it casually.** |
| + |
| +## 12. Cutover from the Hetzner box |
| + |
| +The current box becomes the IE region's content, not its hardware: stand |
| +up IE, litestream-restore the registry+identity and each instance |
| +database into it (the cutover *is* a restore drill — the path is proven |
| +daily), flip `go.tito.io` + wildcard from 62.238.45.61 to the IE EIP, |
| +watch, retire the box, delete its bucket prefixes. Existing accounts get |
| +residency `ie`. This lands only after the activator model is proven on |
| +canaries; until then the Hetzner box runs today's systemd model |
| +untouched, consuming releases as an interim update client if convenient. |
| + |
| +## 13. The tofu stack |
| + |
| +`infra/tito/` in this repo (deployment infra is platform-repo property; |
| +the same modules should instantiate the Eleven and flagship deployments, |
| +spec §12), house pattern: state in `carlos-tfstate-665106592142`, |
| +`use_lockfile`, AWS provider ~> 6.0, profile `titogo`. Modules: |
| + |
| +- `region` ×5 — box, EIP, SG, regional data bucket, IAM role, regional |
| + DNS alias; IE instantiated with `control_plane = true`. |
| +- `deployment` — the deployment bucket, KMS key for the cert cache, |
| + release/signing parameters. |
| +- `svc-pdf` — the Gotenberg box, Caddy, SG of the fleet's EIPs. |
| +- Static DNS records in the existing `tito.io` zone. |
| + |
| +Boundary, restated as a rule: **tofu owns boxes, buckets, IAM, and |
| +static DNS; the platform owns apps, releases, instances, and dynamic |
| +DNS.** |
| + |
| +## 14. Cost and blast radius |
| + |
| +Steady state ≈ **$180/month**: 5 × t4g.medium ≈ $135, svc/pdf t4g.small |
| +≈ $13, EIPs/S3/Route53/KMS ≈ $20–30. Blast radius by construction: a |
| +region's box dying strands only that region's instances until the |
| +restore runbook completes; the edge dying strands admin, signup, and |
| +root-URL pages but **not buyer traffic** — subdomains resolve directly |
| +to regional hosts, which keep serving. That asymmetry (buyers keep |
| +buying while admin is down) is the availability story, worth saying out |
| +loud in the status-page vocabulary. |
| + |
| +## 15. Rollout slices |
| + |
| +Deployment-focused slices, roughly ordered; the platform's own build |
| +(edge, host agent, channels — extracted from Eleven's proven activator, |
| +updater, and mesh code) is sliced in this repo's plans, and titogo's |
| +app-side conformance (manifest, park/wake, dropping systemd assumptions) |
| +in titogo's: |
| + |
| +1. `infra/tito/` skeleton: IE region, deployment + tfstate buckets, |
| + svc/pdf, static DNS. Pure tofu; Tito's invoice work can consume |
| + Gotenberg immediately, before any platform code exists. |
| +2. Platform core proven with titogo as the first app on a miniature |
| + canary deployment (spec §9's gate). |
| +3. titogo-as-app: manifest, `carlos vet` conformance, activator adoption. |
| +4. Remaining four regions (tofu instantiation). |
| +5. Provisioner + self-serve signup + dynamic DNS. |
| +6. Hetzner → IE cutover; retire the box. |
| +7. Relocation command. |
| + |
| +Each slice stands alone; the fleet is never half-migrated within one. |
| + |
| +## 16. Open and resolved questions |
| + |
| +Resolved by the 2026-08-01 topology decision (platform spec §12): |
| + |
| +- ~~**Flagship co-tenancy.**~~ Eleven does not join this deployment, and |
| + neither does anyone else: Tito's deployment is single-tenant and |
| + self-hosted. The shared flagship is a separate deployment in a separate |
| + account, serving Keymail, Slopbox, Woodstar, and Kass. |
| +- ~~**AWS account.**~~ This deployment lives in Tito's account |
| + (665106592142), which keeps its existing S3/SES footprint and its |
| + 265k/day production SES quota. The flagship gets its own account |
| + (platform spec §12). |
| + |
| +Still open: |
| + |
| +1. **Library or binary?** Does titogo swap `internal/carlos` for the |
| + platform as a Go library, or run beside the `carlos` binary and speak |
| + its contracts? (The titogo CLAUDE.md extraction plan assumed library; |
| + the platform spec's one-binary-many-roles shape suggests either |
| + works.) |
| +2. **Identity unification.** Tito's root-domain identity (registry DB, |
| + magic link + TOTP) versus the platform edge "owning identity and |
| + sessions" (spec §3) — merge timing, and whether Tito's identity is |
| + the seed of the platform's or a tenant of it. |
| +3. **Email regionality.** SES sends from eu-west-1 for all regions |
| + today; fine under transit-unconstrained residency, but AU/US sending |
| + identities may eventually be wanted for deliverability, not |
| + residency. |
| + |
| +## 17. Provenance |
| + |
| +Concretized from the CARLOS platform design (same day) and titogo's |
| +brainstorming session of 2026-08-01: provider, control-plane home, |
| +signup model, runtime model, residency semantics, and region-entry |
| +topology were each chosen from explicit alternatives with trade-offs. |
| +Reframed the same day from "flagship with Tito as tenant" to "Tito's own |
| +self-hosted deployment" by the topology decision. Eleven's |
| +built-and-proven mesh/activator/updater/relocation code |
| +(`docs/planet-scale.md`, `docs/instance-router.md`, |
| +`docs/multi-edge.md` in elevenmessenger/messenger) is the implementation |
| +quarry throughout. |
amadan