amadan
docs: design record for the Rails-behind-the-edge PoC (Classic/Pro staging)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+191
−0
docs/superpowers/specs/2026-08-11-rails-behind-carlos-edge-poc-design.md+191 −0
diff --git a/docs/superpowers/specs/2026-08-11-rails-behind-carlos-edge-poc-design.md b/docs/superpowers/specs/2026-08-11-rails-behind-carlos-edge-poc-design.md| new file mode 100644 |
| index 000000000..7e74d81ef |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-11-rails-behind-carlos-edge-poc-design.md |
| @@ -0,0 +1,191 @@ |
| +# Rails behind the CARLOS edge: the Classic/Pro proof of concept |
| + |
| +**Date:** 2026-08-11 |
| +**Status:** Approved design, staging slice first |
| +**Prompted by:** James running miren (miren.dev, a container deployment |
| +tool) behind the carlos router. Question under test: can the CARLOS tito |
| +deployment front the two big Rails apps — Tito Classic and Tito Pro — well |
| +enough that migrating them off Kubernetes onto a much simpler all-CARLOS |
| +deployment is a real option rather than a slogan. |
| + |
| +## Context |
| + |
| +Tito Classic and Tito Pro run today on Kubernetes, with Postgres, Redis and |
| +Memcached dependencies, in the **same AWS account and region as the CARLOS |
| +tito boxes** (665106592142, eu-west-1). CI already builds Docker images for |
| +the k8s deployment; those images are amd64 today, but we *want* arm64 and |
| +can build arm64 variants as needed — so the PoC standardizes on arm64 |
| +(Graviton boxes, matching the rest of the fleet) rather than inheriting |
| +x86. |
| + |
| +CARLOS today has no container runtime and its release format is a single |
| +static binary — a Rails app cannot be a CARLOS *release*. But the edge |
| +already fronts processes it didn't build: a route can proxy to an arbitrary |
| +TCP `host:port` (`-kind service -addr`, `internal/registry/registry.go`), |
| +and the svc-pdf box (`platform-infrastructure/tito/svc-pdf.tf`) is the live |
| +precedent for "a Docker-shaped dependency on its own tofu-provisioned box, |
| +trusted via security group". The PoC composes exactly those two precedents |
| +and deliberately nothing new in the platform. |
| + |
| +Two facts remove whole workstreams: `*.platform.tito.io` already resolves |
| +to the IE box's EIP (`tito/dns.tf`, wildcard A record), and the edge mints |
| +per-host HTTP-01 certs automatically the moment a route exists. DNS and TLS |
| +are zero work. |
| + |
| +## Decisions taken during design |
| + |
| +- **Runtime on the boxes: bare box — docker + systemd** (svc-pdf pattern |
| + scaled up). Miren was considered (closest to James's PoC) and rejected as |
| + a second control plane pointing away from "simpler all-CARLOS"; Kamal was |
| + rejected because it requires SSH (the fleet is deliberately SSM-only) and |
| + inserts its own proxy. |
| +- **Reuse the k8s images** — the PoC runs what CI builds, rebuilt for |
| + arm64; no separate build pipeline, no source checkouts on boxes. |
| +- **Staging first.** Boxes wired to STAGING Postgres/Redis/Memcached now; |
| + production boxes are written in tofu in the same PR but applied only |
| + after the staging PoC convinces us. |
| +- **Full process tree** — web + workers + cron, so the boxes are true |
| + parallel replicas of the k8s deployment sharing its queues and cache. |
| + Workers pulling real staging jobs from the shared Redis is the point, |
| + not a hazard. |
| +- **Reuse existing staging dependencies** rather than on-box replacements; |
| + an on-box memcached container is a documented fallback only. |
| + |
| +## Design |
| + |
| +### 1. Boxes |
| + |
| +Two new EC2 instances in `platform-infrastructure/tito/`, one `.tf` + |
| +cloud-init file each, copying the svc-pdf shape: |
| + |
| +- `rails-classic-staging` and `rails-pro-staging` |
| +- Graviton (arm64) — e.g. `t4g.large` to start, sized honestly after the |
| + first week; `lifecycle { ignore_changes = [ami] }` (the floating-AMI |
| + replace trap is recorded in svc-pdf.tf) |
| +- Own security group each; ingress **only** the app port from the IE box's |
| + SG (`module.host_ie.security_group_id`) — same trust boundary as svc-pdf |
| +- IAM: `AmazonSSMManagedInstanceCore` + ECR pull on the Classic/Pro |
| + repositories + `ssm:GetParameter` on `/tito/rails-poc/<app>-staging/*`. |
| + Nothing else — no bucket grants. |
| +- No SSH. All operation via SSM, like the rest of the fleet. |
| + |
| +`rails-classic-prod.tf` / `rails-pro-prod.tf` are written in the same PR, |
| +`count = 0` (or commented apply instructions) until the staging PoC passes. |
| + |
| +### 2. On each box |
| + |
| +Cloud-init installs docker + the ECR credential helper, materializes the |
| +app env from SSM Parameter Store (`/tito/rails-poc/<app>-staging/env`, |
| +written to `/etc/rails-poc/<app>.env`, 0600), and writes systemd units — |
| +one per process type, each a `docker run` on the pinned image tag: |
| + |
| +- `rails-web.service` — the Rails/puma web process, port published on the |
| + box's private IP |
| +- `rails-worker.service` — background workers (one unit per worker class |
| + if the k8s deployment splits them) |
| +- `rails-cron.service` — clock/scheduler process, if the app has one |
| +- all `Restart=always`, logs to journald |
| + |
| +The env file carries DATABASE_URL / REDIS_URL / MEMCACHED / SECRET_KEY_BASE |
| +etc., populated once by hand from the k8s staging secrets (values never in |
| +tofu, never in the repo). |
| + |
| +### 3. Images: arm64 |
| + |
| +The k8s images are amd64 today. The PoC builds arm64 variants of the same |
| +Dockerfiles (`docker buildx --platform linux/arm64`, pushed to the same ECR |
| +repos under an `-arm64` tag suffix or a multi-arch manifest). One-off |
| +manual builds are fine to start; the follow-up is CI building multi-arch so |
| +the PoC boxes track the same tags k8s deploys. If an image turns out to |
| +have amd64-only native gems or base layers, fixing that is in scope for the |
| +PoC — it is exactly the kind of migration work the PoC exists to surface. |
| + |
| +### 4. Dependencies |
| + |
| +Staging Postgres/Redis/Memcached are reached by adding ingress rules on |
| +their security groups admitting the new boxes' SGs. **Preflight before |
| +apply:** confirm the staging RDS/Elasticache share a VPC with the default |
| +VPC these boxes land in (same account/region is confirmed; the svc-pdf |
| +pattern uses the default VPC). If EKS keeps its data stores in its own |
| +VPC, place the PoC boxes in that VPC instead — a variable on the box |
| +resources, not a redesign — or peer. |
| + |
| +### 5. Edge routing |
| + |
| +On the IE box (via SSM, as user `carlos`): |
| + |
| +``` |
| +carlos add -kind service -addr <classic-private-ip>:<port> classic.platform.tito.io |
| +carlos add -kind service -addr <pro-private-ip>:<port> pro.platform.tito.io |
| +``` |
| + |
| +Gotchas carried from exploration: |
| + |
| +- Confirm which registry the edge reads before adding routes — the tito |
| + box carries two (`/data/registry.db` platform, `/data/titogo/registry.db` |
| + titogo's own). |
| +- Never set `-hibernate` on these routes: hibernation drives the exec/unit |
| + backends and their `--socket/--db/healthz` contract, which these boxes |
| + don't speak. |
| +- Repointing an upstream = re-run `carlos add` (it upserts); there is no |
| + `carlos route -addr` setter. |
| +- Addr-upstream routes are "externally managed by design" |
| + (`internal/host/socket_advice.go`) — no config delivery, no version |
| + stamping, no log shipping, no managed restarts. That is expected and is |
| + part of what the PoC measures (see §7). |
| + |
| +### 6. Deploys |
| + |
| +A small script per app, run over SSM RunCommand from a developer laptop |
| +(or CI later): pull the new image tag → restart the units in order → |
| +poll the health endpoint → report. No SSH, no new tooling. The tag |
| +deployed is whatever CI pushed for k8s staging, so the two deployments |
| +stay comparable during the parallel-run phase. |
| + |
| +### 7. What the PoC deliberately surfaces (the roadmap fallout) |
| + |
| +CARLOS gives these routes routing + TLS and nothing else. Every gap met is |
| +a named candidate roadmap item, and this list is the actual deliverable of |
| +the migration evaluation: |
| + |
| +1. **No image/bundle release kind** — a CARLOS release is one static |
| + binary; ExecStart is pinned to `/opt/carlos/live/%i` |
| + (`internal/caps/render.go`). Running Rails as a first-class app needs a |
| + bundle/image release kind or an equivalent. |
| +2. **No attached-database primitive** — the platform design spec already |
| + sketches `attached` engines (Postgres provisioned and bound by the |
| + platform, `docs/superpowers/specs/2026-08-01-carlos-platform-design.md` |
| + §storage); the objstore primitive |
| + (`2026-08-10-object-storage-primitive-design.md`) is the implementation |
| + template. |
| +3. **No config/env delivery to addr-upstream routes** — the PoC hand-rolls |
| + SSM Parameter Store; a platform answer would ride the existing |
| + `appconfig` rails. |
| +4. **No log shipping for external processes** — journald-only on the PoC |
| + boxes; CARLOS logs stage 2 (unit tenants) is the relevant seam. |
| +5. **No health/restart/metrics for off-box upstreams** — `-unit` stamping |
| + only works for processes on the edge box itself. |
| + |
| +## Out of scope |
| + |
| +- Any platform code change (release kinds, primitives) — the PoC is |
| + precedent-composition only. |
| +- Cutting any real traffic over from k8s; DNS for the apps' real customer |
| + domains is untouched. |
| +- Production apply (written, not applied). |
| +- Multi-arch CI (follow-up; one-off buildx builds carry the PoC). |
| + |
| +## Verification |
| + |
| +1. `tofu plan` **targeted** to the new resources — the tito tree carries |
| + known drift from open PRs and a full apply would strip live SMTP |
| + ingress (recorded in `tito/object-store.tf`'s commit message). |
| +2. Boxes reachable via SSM; units running; env files 0600. |
| +3. `https://classic.platform.tito.io` and `https://pro.platform.tito.io` |
| + serve the real staging apps with valid certs. |
| +4. A staging background job enqueued through the k8s deployment is |
| + observed processed by a PoC-box worker (proves shared-queue replica |
| + behavior). |
| +5. One deploy exercised end-to-end over SSM (new tag → restart → healthy). |
| +6. A week running alongside k8s untouched; then the production-apply |
| + decision. |