Rails behind the CARLOS edge: the Classic/Pro proof of concept

Date: 2026-08-11 Status: Approved design, staging slice first Prompted by: James running miren (miren.dev, a container deployment tool) behind the carlos router. Question under test: can the CARLOS tito deployment front the two big Rails apps — Tito Classic and Tito Pro — well enough that migrating them off Kubernetes onto a much simpler all-CARLOS deployment is a real option rather than a slogan.

Context

Tito Classic and Tito Pro run today on Kubernetes, with Postgres, Redis and Memcached dependencies, in the same AWS account and region as the CARLOS tito boxes (665106592142, eu-west-1). Buildkite already builds Docker images for the k8s deployment; those images are amd64, and the PoC runs them as-is on x86 boxes (Paul, 2026-08-11: these boxes are disposable — they'll be turned off after the evaluation, so rebuilding for arm64 buys nothing here). arm64 only matters if Rails-on-CARLOS graduates to a first-class deployment on the Graviton fleet, and that is recorded as roadmap fallout (§7), not PoC work.

CARLOS today has no container runtime and its release format is a single static binary — a Rails app cannot be a CARLOS release. But the edge already fronts processes it didn't build: a route can proxy to an arbitrary TCP host:port (-kind service -addr, internal/registry/registry.go), and the svc-pdf box (platform-infrastructure/tito/svc-pdf.tf) is the live precedent for "a Docker-shaped dependency on its own tofu-provisioned box, trusted via security group". The PoC composes exactly those two precedents and deliberately nothing new in the platform.

Two facts remove whole workstreams: *.platform.tito.io already resolves to the IE box's EIP (tito/dns.tf, wildcard A record), and the edge mints per-host HTTP-01 certs automatically the moment a route exists. DNS and TLS are zero work.

Decisions taken during design

• Runtime on the boxes: bare box — docker + systemd (svc-pdf pattern scaled up). Miren was considered (closest to James's PoC) and rejected as a second control plane pointing away from "simpler all-CARLOS"; Kamal was rejected because it requires SSH (the fleet is deliberately SSM-only) and inserts its own proxy.
• Reuse the k8s images unmodified — the PoC runs exactly what Buildkite builds (amd64); no separate build pipeline, no arm64 rebuilds, no source checkouts on boxes.
• Staging first. Boxes wired to STAGING Postgres/Redis/Memcached now; production boxes are written in tofu in the same PR but applied only after the staging PoC convinces us.
• Full process tree — web + workers + cron, so the boxes are true parallel replicas of the k8s deployment sharing its queues and cache. Workers pulling real staging jobs from the shared Redis is the point, not a hazard.
• Reuse existing staging dependencies rather than on-box replacements; an on-box memcached container is a documented fallback only.

Design

1. Boxes

Two new EC2 instances in platform-infrastructure/tito/, one .tf + cloud-init file each, copying the svc-pdf shape:

• rails-classic-staging and rails-pro-staging
• x86 (amd64), because that is what the Buildkite images are — t3a.large to start (t4g is Graviton-only, so the burstable x86 class is t3/t3a), sized honestly after the first week; an amd64 Debian AMI data source (the fleet's shared AMI is arm64), with lifecycle { ignore_changes = [ami] } (the floating-AMI replace trap is recorded in svc-pdf.tf)
• Own security group each; ingress only the app port from the IE box's SG (module.host_ie.security_group_id) — same trust boundary as svc-pdf
• IAM: AmazonSSMManagedInstanceCore + ECR pull on the Classic/Pro repositories + ssm:GetParameter on /tito/rails-poc/<app>-staging/*. Nothing else — no bucket grants.
• No SSH. All operation via SSM, like the rest of the fleet.

rails-classic-prod.tf / rails-pro-prod.tf are written in the same PR, count = 0 (or commented apply instructions) until the staging PoC passes.

2. On each box

Cloud-init installs docker + the ECR credential helper, materializes the app env from SSM Parameter Store (/tito/rails-poc/<app>-staging/env, written to /etc/rails-poc/<app>.env, 0600), and writes systemd units — one per process type, each a docker run on the pinned image tag:

• rails-web.service — the Rails/puma web process, port published on the box's private IP
• rails-worker.service — background workers (one unit per worker class if the k8s deployment splits them)
• rails-cron.service — clock/scheduler process, if the app has one
• all Restart=always, logs to journald

The env file carries DATABASEURL / REDISURL / MEMCACHED / SECRETKEYBASE etc., populated once by hand from the k8s staging secrets (values never in tofu, never in the repo).

3. Images

The Buildkite-built amd64 images are used unmodified, pulled from the same registry the k8s deployment pulls from. The boxes track the same tags k8s staging deploys, so the two deployments stay comparable during the parallel run. No rebuilds of any kind for the PoC.

4. Dependencies

Staging Postgres/Redis/Memcached are reached by adding ingress rules on their security groups admitting the new boxes' SGs. Preflight before apply: confirm the staging RDS/Elasticache share a VPC with the default VPC these boxes land in (same account/region is confirmed; the svc-pdf pattern uses the default VPC). If EKS keeps its data stores in its own VPC, place the PoC boxes in that VPC instead — a variable on the box resources, not a redesign — or peer.

5. Edge routing

On the IE box (via SSM, as user carlos):

carlos add -kind service -addr <classic-private-ip>:<port> classic.platform.tito.io
carlos add -kind service -addr <pro-private-ip>:<port> pro.platform.tito.io

Gotchas carried from exploration:

• Confirm which registry the edge reads before adding routes — the tito box carries two (/data/registry.db platform, /data/titogo/registry.db titogo's own).
• Never set -hibernate on these routes: hibernation drives the exec/unit backends and their --socket/--db/healthz contract, which these boxes don't speak.
• Repointing an upstream = re-run carlos add (it upserts); there is no carlos route -addr setter.
• Addr-upstream routes are "externally managed by design" (internal/host/socket_advice.go) — no config delivery, no version stamping, no log shipping, no managed restarts. That is expected and is part of what the PoC measures (see §7).

6. Deploys

A small script per app, run over SSM RunCommand from a developer laptop (or CI later): pull the new image tag → restart the units in order → poll the health endpoint → report. No SSH, no new tooling. The tag deployed is whatever CI pushed for k8s staging, so the two deployments stay comparable during the parallel-run phase.

Recorded deviations (execution)

• Box SG ingress uses the IE edge's private IP /32 (172.31.4.137/32) rather than an SG reference — the peering is cross-VPC, and a security group cannot be referenced across the peering boundary the way §2 assumed. Breaks silently if the IE box is ever replaced at a new private IP; re-point the ingress rule then.
• The web port is published on all interfaces, not the private IP only — §2 said "the box's private IP"; what's running binds 0.0.0.0. Exposure is nil in practice (the SG admits only the edge), but the defense-in-depth intent of §2 was dropped.

7. What the PoC deliberately surfaces (the roadmap fallout)

CARLOS gives these routes routing + TLS and nothing else. Every gap met is a named candidate roadmap item, and this list is the actual deliverable of the migration evaluation:

1. No image/bundle release kind — a CARLOS release is one static binary; ExecStart is pinned to /opt/carlos/live/%i (internal/caps/render.go). Running Rails as a first-class app needs a bundle/image release kind or an equivalent. - observed: confirmed — the gap is real, and the PoC boxes hand-roll docker + systemd unit files in its place, not a CARLOS release at all.
2. No attached-database primitive — the platform design spec already sketches attached engines (Postgres provisioned and bound by the platform, docs/superpowers/specs/2026-08-01-carlos-platform-design.md §storage); the objstore primitive (2026-08-10-object-storage-primitive-design.md) is the implementation template. - observed: no such primitive existed to reach for — dependency access was wired by hand (security-group ingress rules on the shared staging DB/cache SGs, plus connection env delivered separately); it worked, but nothing about it would survive a second app onboarding itself.
3. No config/env delivery to addr-upstream routes — the PoC hand-rolls SSM Parameter Store; a platform answer would ride the existing appconfig rails. - observed: confirmed, and it cost a real mistake — the hand-rolled prune rule over-pruned a live key (PROMETHEUS_EXPORTER_PORT) from both apps' env on the first pass; caught by review, not by any safeguard the ad hoc delivery had.
4. No log shipping for external processes — journald-only on the PoC boxes; CARLOS logs stage 2 (unit tenants) is the relevant seam. - observed: confirmed — journald-only, no shipping. Adequate for the PoC's own diagnosis needs; not evidence either way for real ops load.
5. No health/restart/metrics for off-box upstreams — -unit stamping only works for processes on the edge box itself. - observed: confirmed — the deploy script itself carries the whole health gate (on-box HTTP probe with per-app path/header, then a report-only edge check); CARLOS supplies none of it for this route shape.
6. Architecture — the fleet is Graviton; the Rails images are amd64. A first-class Rails-on-CARLOS deployment means arm64 image builds (Buildkite multi-arch). Deferred deliberately: the PoC boxes are disposable, so they run x86 to match the images rather than the other way round. - observed: the x86 boxes ran the unmodified Buildkite amd64 images with no rebuild and no compatibility issue — the deferral cost nothing for the PoC itself; the arm64 gap for a real deployment stands as stated.
7. High availability (added 2026-08-12, Paul's question). The PoC has deliberately little: one box per app, one AZ, edge proxying to one IP; Restart=always covers process death and nothing covers box death. The layers split cleanly into two owners, and nothing Tito-side waits on anything CARLOS-side: - Tito concerns (this migration's own work, plain AWS, no platform changes): EC2 auto-recovery on the boxes; N boxes per app behind an internal NLB with the edge's -addr pointed at the NLB (multi-AZ, health-checked failover, rolling zero-downtime deploys — and networking-trivial in prod, where the existing peering already routes the whole VPC); Classic's supercronic → sidekiq-ent so cron stops being a single-box constraint (Pro's already is, with Redis-backed single-fire); the hostname/host-authorization config. - CARLOS concerns (roadmap candidates, priced by whether Rails-on-CARLOS graduates from migration tactic to product): multi-upstream health-checked addr routes at the edge — the platform-native replacement for the NLB rung (filed as platform #152); and the edge's own HA posture (standby/EIP failover), which is a whole-deployment question every CARLOS tenant shares — the Rails apps widen the blast radius of a risk go.tito.io already carries, they don't create it (filed as platform #153). - Data HA is out of scope by construction: Postgres/MySQL/Valkey/ Memcached stay in RDS/Elasticache with their own multi-AZ story — the apps' state was never on k8s, so the migration never touches the layer where losing things actually hurts.

2026-08-11: staging is live on both apps (see tito/README.md's Rails PoC section in platform-infrastructure for current status). The edge host-authorization gap (classic 403, pro 404) is still open and is what blocks this route shape from serving a real page today — the PoC proved the plumbing, not a finished path to production traffic.

2026-08-11, later — two verification items closed on-box:

• Shared-queue replica behavior (Verification item 4): proven for Pro, organically. The pro box's sidekiq processed 35 real staging jobs in one observed hour (ScheduledTasks::ExpireUnfulfilledAllocationsWorker, real JIDs in journald) — enqueued into the shared staging Redis by Pro's own sidekiq-enterprise cron and won by the box worker in competition with the k8s workers. Classic's staging queues were quiet over the same window, so its half still wants a deliberate enqueue.
• Real-page serving: both apps serve correct full responses from the boxes when addressed by a hostname they already know (curl on-box with Host: + X-Forwarded-Proto: https, exactly what the edge forwards): staging.pro.tito.io → 200 with the real Tito Pro page; staging.tito.io → 301 to canonical staging.ti.to; dashboard.staging.tito.io → 302 into its SSO login flow. The 403/404 on the edge hostnames is therefore ONLY hostname configuration (Classic's host allowlist; a Pro tenant/alias for the hostname), not a serving gap — the remaining decision is which hostnames the parallel deployment should answer to, which is a migration-design question, not an infrastructure one.

2026-08-28 — hostname configuration closed; the edge hostnames serve the real apps. Both turned out to be env-only, applied in the boxes' own SSM params (which the k8s deployment never reads), so the change is box-scoped and reverts by reverting two values:

• Classic's 403 was stock Rails ActionDispatch::HostAuthorization, allowlist fed entirely from *_HOST env vars in config/application.rb. HEALTHCHECK_HOST is the one var with no second meaning anywhere in the app (no AppConfig.host branch, no route constraint, no throttle rule), so the box sets HEALTHCHECK_HOST=classic.platform.tito.io and the deploy script's health probe follows it. ALTERNATIVE_PRIMARY_HOSTS was rejected: it admits a host and then 301s every path to PRIMARY_HOST. Result: classic.platform.tito.io → 302 into Tito ID SSO — Classic's real root behavior. Completing a sign-in through the new hostname would additionally need it in the staging ID service's OAuth redirect allowlist — shared state, deliberately untouched.
• Pro's 404 was the custom-domain catch-all (Custom::BaseController raising RecordNotFound for a host that is neither a configured platform host nor a tenant domain). The primary host is anywayconfig (UrlsConfig), ENV over urls.yml, so the box sets `URLSPRIMARY=pro.platform.tito.io and the whole primary-domain route tree, CSP, ActionCable origins and URL generation follow. URLSSECONDARY was rejected: the app 302s every secondary-host request to primary. Result: pro.platform.tito.io → 200 with the real primary app. Known consequence, accepted as an honest cutover preview: the box's sidekiq shares staging's queue and builds absolute URLs from its own URLSPRIMARY, so staging mails whose jobs run on the box carry working pro.platform.tito.io` links.

The migration-design lesson: neither app needed code or shared-state changes to answer on a new hostname — a real cutover's hostname story is an env rollout, plus the ID service's OAuth redirect allowlist for Classic sign-in. Ops record: teamtito/carlos-infrastructure PR #10.

Out of scope

• Any platform code change (release kinds, primitives) — the PoC is precedent-composition only.
• Cutting any real traffic over from k8s; DNS for the apps' real customer domains is untouched.
• Production apply (written, not applied).
• Any image rebuilds, including arm64/multi-arch (recorded as roadmap fallout in §7 instead).

Verification

1. tofu plan targeted to the new resources — the tito tree carries known drift from open PRs and a full apply would strip live SMTP ingress (recorded in tito/object-store.tf's commit message).
2. Boxes reachable via SSM; units running; env files 0600.
3. https://classic.platform.tito.io and https://pro.platform.tito.io serve the real staging apps with valid certs.
4. A staging background job enqueued through the k8s deployment is observed processed by a PoC-box worker (proves shared-queue replica behavior).
5. One deploy exercised end-to-end over SSM (new tag → restart → healthy).
6. A week running alongside k8s untouched; then the production-apply decision.