| 1 | # The console is an app — retiring the pinfra roll |
| 2 | |
| 3 | **Date:** 2026-08-23 · **Status:** approved (Paul, in-session); reworked after a |
| 4 | Fable review · **Slice:** 1 of 4 |
| 5 | |
| 6 | ## Problem |
| 7 | |
| 8 | Every tenant deploys with one command and no infrastructure access. The |
| 9 | platform deploys itself with a bash script, an AWS profile, an SSM session and |
| 10 | a checkout of a second repository. |
| 11 | |
| 12 | ``` |
| 13 | tenant: carlos deploy -app blog |
| 14 | platform: roll-console.sh <platform-checkout> both # pinfra + AWS + SSM |
| 15 | ``` |
| 16 | |
| 17 | That inversion is the thing CLAUDE.md's own rule complains about ("every |
| 18 | `carlos` command is designed for a user with no access to the underlying |
| 19 | infra"), and it costs more than convenience: a special door is where our own |
| 20 | deficiencies hide. Everything that makes rolling the console feel dangerous — a |
| 21 | version header that lies, a restart nobody trusts, a rollback verified by hand |
| 22 | — is a deficiency **tenants have too**. They just complain less. While the |
| 23 | platform team has its own door, nothing forces us to feel it. |
| 24 | |
| 25 | This slice makes the console an ordinary app, deployed by the ordinary verb. |
| 26 | |
| 27 | Three of the five fixes below (honest build verification §3, the dirty-tree |
| 28 | guard §7, and the restart-recovery fix §4b) are **tenant** fixes that this work |
| 29 | merely forces us to make. |
| 30 | |
| 31 | ### Not in this slice |
| 32 | |
| 33 | `carlos system` is **deferred, not rejected** (Paul, 2026-08-23). It earns its |
| 34 | place for what genuinely cannot be an app: the root `carlos` binary, the edge |
| 35 | router, the activator, the provisioning reconciler. Those have no route, no |
| 36 | channel to be adopted from and no upstream — they are the substrate that does |
| 37 | the adopting and routing. `update-platform-binary.sh` is 85 lines of exactly |
| 38 | that work, none of which has an app-shaped equivalent. That is slice 3. |
| 39 | |
| 40 | The dividing line, stated once so later work can apply it: **`carlos system` is |
| 41 | for what is not an app. Anything adoptable through a route is an app.** |
| 42 | |
| 43 | ## The test the design must pass |
| 44 | |
| 45 | `roll-console.sh` and `verify-console.sh` encode failure modes this project has |
| 46 | already paid for. A replacement that drops any of them is a regression wearing |
| 47 | better clothes. |
| 48 | |
| 49 | | # | Failure mode | Covered by | |
| 50 | |---|---|---| |
| 51 | | 1 | amd64 artifact shipped to Graviton boxes | §5 ELF refusal (`-target` default only helps the careful caller) | |
| 52 | | 2 | Adoption swaps the symlink; the old process keeps serving | §4a — the *right* route stamped | |
| 53 | | 3 | SSM shells missing `set -a; source host.env` | No SSM at all | |
| 54 | | 4 | "Verified" reported from a stale binary | §3 build probe | |
| 55 | | 5 | A dirty tree stamping a version naming no real commit | §7 | |
| 56 | | 6 | sysusers before tmpfiles | Not reached — root-binary concern, slice 3 | |
| 57 | | 7 | **The console artifact is `cmd/console`, not the root binary** (8-minute outage, 2026-08-08) | §5 embedded-build refusal | |
| 58 | |
| 59 | ## 1. What is actually in the way |
| 60 | |
| 61 | Verified against the live estate on 2026-08-23. See the appendix for the raw |
| 62 | readings. |
| 63 | |
| 64 | **The console is already structurally an app.** Releases at |
| 65 | `apps/bab/console/`, properly account-qualified under `ops`, on channel |
| 66 | `canary/console`, adopted by the same pass as every tenant, served through |
| 67 | routes, one route already naming `console.service`. |
| 68 | |
| 69 | **But it is not an app *record*.** `ops` on flagship holds two app rows: |
| 70 | `deploy-probe` and `platform-site`. There is no `console` row, because |
| 71 | `ship-app.sh` ships direct-to-bucket and direct mode deliberately skips the |
| 72 | claim. Without a row the console API cannot see the app at all. |
| 73 | |
| 74 | **Claiming the name is refused today** by `claimApp` |
| 75 | (`internal/console/server.go:2426`) because the prefix has objects — issue |
| 76 | **#79**. |
| 77 | |
| 78 | **The other three pinfra apps need no claim.** Contrary to CLAUDE.md's framing |
| 79 | of four equivalent apps, `kass`, `keymail` and `platform-site` are already |
| 80 | claimed rows with live channels. They are on `ship-app.sh` out of habit. |
| 81 | |
| 82 | ## 2. Claiming an app whose objects already exist (#79) |
| 83 | |
| 84 | `claimApp`'s bucket guard is right for a stranger and wrong for the account |
| 85 | that owns the prefix. `apps/<sqid>/<app>/` is namespaced **by account**, so |
| 86 | objects under `apps/bab/` can only be `ops`'s. An owner of `bab` claiming a |
| 87 | name under `bab`'s own prefix is adopting their own objects. |
| 88 | |
| 89 | ```go |
| 90 | // today — always refuses |
| 91 | if len(keys) > 0 { return claimTaken, "" } |
| 92 | |
| 93 | // after — unless the claimer owns the account the prefix belongs to |
| 94 | if len(keys) > 0 && !ownerClaim { return claimTaken, "" } |
| 95 | ``` |
| 96 | |
| 97 | `ownerClaim` is passed by the caller. Five constraints, each of which a naive |
| 98 | implementation gets wrong: |
| 99 | |
| 100 | - **Owner, not member.** Adopting existing release history is an ownership act. |
| 101 | - **`ownerClaim` is FALSE for every service principal.** A service has no role |
| 102 | at all, and an account-scoped `PrincipalService` with `App == ""` |
| 103 | legitimately reaches `claimApp` (`apicli.go:346–350` blocks only app-scoped |
| 104 | ones). Without an explicit pin, the exemption silently hands every CI |
| 105 | credential the power to adopt release history — exactly the widening the |
| 106 | svcrole work exists to prevent. |
| 107 | - **`handleNewApp` (the browser door) must be changed deliberately, not |
| 108 | incidentally.** It currently *discards* the role |
| 109 | (`addr, raw, acct, _, ok := s.account(…)`, `server.go:2524`) and its doc |
| 110 | comment says "Any member may claim." The exemption changes that door's stated |
| 111 | policy; the comment changes with it. |
| 112 | - **The legacy prefix still blocks, always.** For `ops`, `claimApp` also checks |
| 113 | `release.LegacyAppPrefix(name)` — `apps/<app>/`, unqualified. Those objects |
| 114 | predate account scoping and are owed an `accounts migrate -from legacy`. |
| 115 | The exemption applies **only** to the account-qualified prefix. |
| 116 | - **`HasOwners(name)` still blocks.** A CLI-granted app in `ops` stays |
| 117 | claim-proof. |
| 118 | |
| 119 | **Refusals stay byte-identical for non-members.** The new branch runs after the |
| 120 | owner gate, so only someone who has already proved ownership can reach it. |
| 121 | |
| 122 | **What this does and does not unblock.** Verified: the `owners` table holds |
| 123 | grants for `derp`, `meep` and `platform-site` only. So the exemption unblocks |
| 124 | `console`, `carloku`, `carlosframework`, `keymail` (the `bab` prefix), |
| 125 | `hello` and `apt-carlosframework`; `derp` and `meep` stay claim-proof by |
| 126 | design. An earlier draft claimed otherwise and was wrong. |
| 127 | |
| 128 | **One nuance to absorb:** a *trashed* app passes `HasAppIn` (which filters |
| 129 | `deleted_at = 0`) and is caught later by `CreateAppIn`'s |
| 130 | `ON CONFLICT DO NOTHING` (`store.go:1120–1128`) → `claimTaken`. No hole, but |
| 131 | after the exemption a trashed-name claim burns a bucket listing before hitting |
| 132 | that refusal. Acceptable; noted so it is not mistaken for a bug. |
| 133 | |
| 134 | **Test:** an owner claims a name whose account-qualified prefix has objects and |
| 135 | succeeds, release history immediately visible through `channels`; a non-owner |
| 136 | member gets `claimTaken`; **a service principal gets `claimTaken` whatever its |
| 137 | scope**; a non-member gets the byte-identical 404; a name whose *legacy* prefix |
| 138 | has objects is refused even for an owner; a name with an `owners` row is |
| 139 | refused even for an owner. |
| 140 | |
| 141 | ## 3. Honest verification: the console reports its own build |
| 142 | |
| 143 | Both cheap answers lie. `X-Carlos-Version` is stamped by the edge |
| 144 | (`edge.go:1336–1341`) from the **route row**, so adoption sets it while the old |
| 145 | process keeps serving. `systemctl is-active` reports "active" for a process |
| 146 | started from the previous version just as happily — on 2026-08-09 tito said |
| 147 | "active" while `/proc` said 9668d30, seconds after adopt and restart both |
| 148 | reported success. |
| 149 | |
| 150 | A `carlos deploy -app console` waiting on `X-Carlos-Version` would be **less |
| 151 | trustworthy than the bash it retires**. So the console reports its own build, |
| 152 | from the process, compiled in: |
| 153 | |
| 154 | ``` |
| 155 | GET /healthz → ok <build> # today: "ok" |
| 156 | ``` |
| 157 | |
| 158 | At least as strong as the `/proc` check and arguably stronger: `/proc` proves |
| 159 | which *path* is executing, this proves which *code* is, from |
| 160 | `internal/version.Build` set by ldflags in the same build. Nothing but the |
| 161 | running process can answer it. |
| 162 | |
| 163 | `/healthz` is served by the console's own mux (`server.go:794`), not the edge. |
| 164 | Existing consumers (`internal/vet`, the edge health probe) check **status |
| 165 | only**, so appending the build is additive-safe. The build id is not a secret — |
| 166 | `X-Carlos-Version` already exposes it on every response. |
| 167 | |
| 168 | ### 3a. Where the probe is declared, and who parses it |
| 169 | |
| 170 | `carlos deploy`'s watch gains a second exit criterion: when the app declares a |
| 171 | **build probe**, poll it instead of the header. |
| 172 | |
| 173 | ```json |
| 174 | {"build_probe": {"path": "/healthz"}} |
| 175 | ``` |
| 176 | |
| 177 | The **path** is the only thing that varies. The contract is fixed: a 2xx whose |
| 178 | body contains the build id as a **whitespace-delimited token**. Not a regex, |
| 179 | not a JSON pointer, not a header name — each is a knob whose wrong setting |
| 180 | produces a watch that passes while proving nothing. Token-matching rather than |
| 181 | substring is load-bearing: a substring match lets build `abc123` satisfy a |
| 182 | watch waiting for `abc1234`, and short git shas make that a real collision. |
| 183 | |
| 184 | **Storage, and the consumer list** — CLAUDE.md's rule is that when a storage |
| 185 | shape changes, the consumer list is everything that PARSES the shape: |
| 186 | |
| 187 | The probe is **app-level, not per-instance**. The health probe it resembles |
| 188 | lives in `provision.Record` (`internal/provision/provision.go:33`, wire key |
| 189 | `"probe"`, at `apps/<sqid>/<app>/instances/<host>.json`) and that is the wrong |
| 190 | home for this: `handleAPIInstanceUpdate` is a **read-modify-write** of the |
| 191 | record, so an older console rewriting a newer record would silently drop the |
| 192 | field. The build probe describes the *app*, not one instance of it, and every |
| 193 | instance would carry an identical copy. |
| 194 | |
| 195 | It therefore lives on the **app row**, served to the CLI on the existing |
| 196 | per-app API the deploy path already calls, and is not a bucket object at all. |
| 197 | Parsers: the app row's own query (which names columns explicitly — verified, no |
| 198 | `SELECT *`), the API handler that serves it, and `carlos deploy`'s watch. Three, |
| 199 | all in-repo, none of them a wire shape a box parses. That is the point of |
| 200 | choosing the row over the record. |
| 201 | |
| 202 | ### 3b. Two ways the watch can still lie, and what closes them |
| 203 | |
| 204 | **The ldflags trap.** `carlos deploy` ships a binary the *operator* built. |
| 205 | `-version` defaults to `gitShortHead()` (`deploy.go:52`) but `version.Build` |
| 206 | comes from ldflags the operator must remember (default `"dev"`, |
| 207 | `internal/version/version.go:6`). Build without the flag and the roll |
| 208 | **succeeds** while the probe reports `ok dev` forever — deploy reports failure |
| 209 | on a healthy deploy, the exact inverse of the lie §3 exists to kill. Closed |
| 210 | twice: §5's ship-time refusal when the embedded build ≠ `-version`, and a |
| 211 | watch timeout that **names what the probe actually said** (a body of `dev` is |
| 212 | then self-diagnosing). |
| 213 | |
| 214 | **The edge cache.** `watchVersion` cache-busts every poll because the edge |
| 215 | caches proxied 2xx responses up to five minutes (`deploy.go:1013–1022`, |
| 216 | `edge.go:933`). The console's `/healthz` sends no `Cache-Control` so it is safe |
| 217 | today, but §3 is a tenant feature and a tenant's probe path may not be. The |
| 218 | probe leg inherits the buster, and reads the body through the default transport |
| 219 | so gzip is transparently decoded. |
| 220 | |
| 221 | **Test:** the watch reports live only once the probe body carries the shipped |
| 222 | build as a whole token; `abc123` does not satisfy a watch for `abc1234`; a |
| 223 | probe stuck on the previous build times out with a diagnosis naming both the |
| 224 | expected build and the body received; an app with no probe keeps today's header |
| 225 | behaviour byte for byte; the probe leg sends a cache-buster. |
| 226 | |
| 227 | ## 4. Restart, rollback, and the recovery hole |
| 228 | |
| 229 | `makeRestartRoute` arm 1 (`internal/host/agent.go:2041–2050`) try-restarts a |
| 230 | route that names its unit and **returns** the failure, so `adoptHost` poisons |
| 231 | the build, repoints the symlink back and skips `SetVersions` |
| 232 | (`adopt.go:779–818`). That much is built, and it is what makes a lying header |
| 233 | impossible *in that case*. |
| 234 | |
| 235 | **It is a much narrower guarantee than it looks, and an earlier draft of this |
| 236 | spec overclaimed it.** Two holes, both verified. |
| 237 | |
| 238 | ### 4a. The guarantee only covers systemd-visible start failure |
| 239 | |
| 240 | `systemctl try-restart` on a `Type=simple` unit succeeds the moment the process |
| 241 | is exec'd. A console that starts and then panics on its first request, exits |
| 242 | after a half-applied migration, or listens and 500s, **exits 0**. Then |
| 243 | `SetVersions` stamps the new build fleet-wide, the §3 probe correctly times |
| 244 | out — and nothing rolls back. The symlink stays on the broken build, the next |
| 245 | pass early-returns on `cur == ver` (`adopt.go:637`), and the repair door is the |
| 246 | thing that is down. |
| 247 | |
| 248 | **Requirement:** `console.service` gains a **readiness gate** so that "started |
| 249 | but not serving" becomes a failed start. `Type=notify` with the console |
| 250 | signalling readiness *after* migrations have applied and the listener is up is |
| 251 | the precise form; an `ExecStartPost` readiness poll is the cheap form. |
| 252 | |
| 253 | **This is a pinfra change plus a `cmd/console` change**, and it is a |
| 254 | cross-repo dependency this spec owns. It cannot be assumed: `console.service` |
| 255 | is defined in pinfra, not in `internal/host/units/`, and its current `Type=` is |
| 256 | not recorded anywhere in this repo. Step 0 reads it. |
| 257 | |
| 258 | The residual after this — a console that starts, serves, passes readiness and |
| 259 | is *behaviourally* wrong — has no automatic recovery and is break-glass |
| 260 | (§9). Say so out loud rather than implying the rollback covers it. |
| 261 | |
| 262 | ### 4b. Rollback repoints the symlink but leaves the console DOWN — and the next pass then lies |
| 263 | |
| 264 | This is the more serious of the two and it is **not** in the review that |
| 265 | prompted this rework; it was found verifying that review. |
| 266 | |
| 267 | On restart failure `adoptHost` repoints the symlink to `cur` and returns |
| 268 | (`adopt.go:781–815`). It does **not** start the unit again. For a `try-restart` |
| 269 | that failed to bring the service back, systemd leaves it **failed** — so the |
| 270 | box is left with a correct symlink and no console. |
| 271 | |
| 272 | It gets worse on the next pass. `try-restart` is by definition a **no-op on a |
| 273 | unit that is not running** (the property `makeRestartRoute`'s comment relies on |
| 274 | so that a deliberately-stopped console stays stopped). So: |
| 275 | |
| 276 | ``` |
| 277 | pass 1: PointLive(new) → try-restart → FAILS → poison, PointLive(old), no stamp |
| 278 | unit: failed. console: DOWN. header: honest (old). |
| 279 | pass 2: PointLive(new) → try-restart on a NOT-RUNNING unit → no-op, exit 0 |
| 280 | → SetVersions(new) |
| 281 | unit: still failed. console: STILL DOWN. header: claims the new build. |
| 282 | ``` |
| 283 | |
| 284 | The second pass converts an honest failure into the exact lie the whole |
| 285 | mechanism exists to prevent, over a console that is not running at all. |
| 286 | |
| 287 | **Requirement:** when arm 1's try-restart fails and the symlink has been rolled |
| 288 | back, adoption must **start** the named unit on the restored version — a |
| 289 | `systemctl start`, not a try-restart, because the whole point is that the unit |
| 290 | is down. If that start also fails, the version stays poisoned and unstamped and |
| 291 | the failure is printed; the box is then genuinely broken and break-glass is |
| 292 | correct. |
| 293 | |
| 294 | This is a **tenant** fix. Any bespoke unit-backed route has this hole today. |
| 295 | It is the strongest argument in the whole slice for making the platform eat its |
| 296 | own food: the hole has presumably been there since #110 and nobody found it, |
| 297 | because the pinfra script's explicit post-pass `systemctl restart` has been |
| 298 | papering over it on the only unit-stamped route in the fleet. |
| 299 | |
| 300 | **Test:** a failed named-unit restart leaves the unit RUNNING on the previous |
| 301 | version, the symlink on the previous version, and the row unstamped; a second |
| 302 | pass over the same still-broken build does not stamp; a failed rollback start |
| 303 | is reported and still does not stamp. |
| 304 | |
| 305 | ### 4c. One process, several hostnames — the stamped route is not a free choice |
| 306 | |
| 307 | The console is one `console.service` behind several hostnames. Adoption is |
| 308 | **per host** (`reg.List()`, `ORDER BY host`, `registry.go:604`), and each |
| 309 | `adoptHost` repoints `live/<that host>` (`PointLive(optDir, rt.Host, …)`) and |
| 310 | *then* fires the restart callback. |
| 311 | |
| 312 | So the restart fires inside **one** host's adoption, while the other hosts' |
| 313 | symlinks have not moved yet. Which means: |
| 314 | |
| 315 | > **The route that must carry the `Unit` stamp is the host whose symlink the |
| 316 | > unit's `ExecStart` resolves — not the canonical hostname, not the first |
| 317 | > alphabetically.** |
| 318 | |
| 319 | Flagship today gets this **wrong**, and the pinfra script has been hiding it: |
| 320 | |
| 321 | ``` |
| 322 | ExecStart = /opt/carlos/live/platform.carlosframework.com serve |
| 323 | stamped = console.carloku.com ← sorts BEFORE platform.… |
| 324 | ``` |
| 325 | |
| 326 | The pass restarts at `console.carloku.com` (2nd of 4 by host order), when |
| 327 | `live/platform.carlosframework.com` still points at the **old** build; that |
| 328 | symlink moves last, under an already-running process. Every roll to date has |
| 329 | worked only because `roll-console.sh` runs an explicit `systemctl restart |
| 330 | console` **after** the whole pass. Remove that line — which is precisely what |
| 331 | "the console is an ordinary app" means — and the console keeps serving the old |
| 332 | build while all its routes advertise the new one. |
| 333 | |
| 334 | **Requirement:** step 0 verifies, per box, that the `Unit`-stamped route is the |
| 335 | host named in `ExecStart`, and restamps if not. Exactly one route is stamped: |
| 336 | one is sufficient to cycle the process, and stamping several would try-restart |
| 337 | the same unit several times in one pass. |
| 338 | |
| 339 | The unstamped rows' `X-Carlos-Version` then becomes cosmetic, which §3 makes |
| 340 | harmless. One honesty note: on restart failure the poison map only stops |
| 341 | **later** hosts in the pass from stamping (`adopt.go:648`), so whether the |
| 342 | cosmetic headers stay honest depends on alphabetical order relative to the |
| 343 | stamped host — protection by accident, not by design. The real fix is the |
| 344 | follow-up below. |
| 345 | |
| 346 | **Follow-up to file, explicitly out of scope:** teach the pass to collect |
| 347 | named-unit restarts and run them **once, after every host's symlink has |
| 348 | moved**. That makes the stamp/ExecStart coupling and the alphabetical accident |
| 349 | both irrelevant, and it is the general fix for any bespoke service behind |
| 350 | several hostnames. |
| 351 | |
| 352 | ## 5. Ship-time refusals for a control-plane app |
| 353 | |
| 354 | Two refusals, both at the ship door, both scoped to apps carrying the §6 flag. |
| 355 | |
| 356 | **Architecture.** Read the artifact's ELF header and refuse a machine type that |
| 357 | does not match the declared `-target`. `-target linux-arm64` being `carlos |
| 358 | deploy`'s default only protects the careful caller; this closes the path a |
| 359 | hand-run `carlos ship`, a CI job or a future script takes into the same channel. |
| 360 | |
| 361 | **Embedded build.** Refuse a binary whose embedded `internal/version.Build` |
| 362 | does not equal the `-version` being shipped. This closes §3b's ldflags trap at |
| 363 | the door instead of at a timeout, and it incidentally closes failure mode #7: |
| 364 | the root `carlos` binary and `cmd/console` stamp the same `version.Build`, so |
| 365 | this does not catch a wrong-artifact ship on its own — but combined with the |
| 366 | probe (only `cmd/console` serves `/healthz`) the pair does. |
| 367 | |
| 368 | Scoped to control-plane apps deliberately: a tenant may legitimately ship an |
| 369 | artifact we would guess wrong about, and refusing their ship on our inference |
| 370 | is not our call. For the app that *is* the control plane, on boxes we own and |
| 371 | know are arm64, it is. |
| 372 | |
| 373 | **Test:** an amd64 ELF with `-target linux-arm64` is refused naming both the |
| 374 | declared target and what was found; a binary whose embedded build ≠ `-version` |
| 375 | is refused naming both; a matching artifact ships; a non-control-plane app is |
| 376 | unaffected; a non-ELF artifact (a static tarball) is unaffected. |
| 377 | |
| 378 | ## 6. The control-plane flag gates DESTRUCTION, not promotion — and narrows a service principal's authority |
| 379 | |
| 380 | ```sql |
| 381 | ALTER TABLE apps ADD COLUMN control_plane INTEGER NOT NULL DEFAULT 0; |
| 382 | ``` |
| 383 | |
| 384 | An earlier draft had this flag demand fresh auth to **promote**, "exactly as |
| 385 | stable already does, at every rung." That was wrong, and wrong in a way that |
| 386 | defeated the spec's own purpose. `stable`'s API gate is a bare 403 with a |
| 387 | `console_url` and **no API-side step-up** (`apicli.go:632–638`), and |
| 388 | `carlos deploy` `log.Fatalf`s on a failed promote *after the ship has already |
| 389 | landed* (`deploy.go:206–208`). Extending it to every rung would mean |
| 390 | `carlos deploy -app console` ships the artifact and then dies, every time — |
| 391 | the exact command this spec exists to create. |
| 392 | |
| 393 | **Invert it.** Deploying the console should be easy; destroying it should be |
| 394 | hard. That is also where the real danger is: `handlePurgeApp` (`ownership.go:293`) |
| 395 | can delete `apps/bab/console/**` — the release history the fleet is *currently |
| 396 | running* — behind an owner check, a typed name and fresh auth, none of which |
| 397 | knows this app is the control plane. |
| 398 | |
| 399 | An app carrying the flag: |
| 400 | |
| 401 | - **refuses trash, purge, rename and transfer outright.** Not "with fresh |
| 402 | auth" — refused, with a message saying the flag must be cleared first. The |
| 403 | console's release history is not something to delete behind a confirmation |
| 404 | dialog. |
| 405 | - **refuses promotion by a service principal.** A person with ops membership |
| 406 | can roll the console; a CI credential cannot. This closes the hole §10 step |
| 407 | ordering would otherwise leave open, without a step-up flow. |
| 408 | - **refuses an instance-record DELETE outright** (added mid-rollout, 2026-08-28). |
| 409 | This is the eighth door and the shortest fuse of the set. `carlos deploy` |
| 410 | resolves hosts through `apps/<sqid>/<app>/instances/<host>.json`, so making |
| 411 | the console deployable means giving it a record — and a record is an |
| 412 | authority transfer: once it exists and the route is `Provisioned`, the |
| 413 | provisioning reconciler's removal phase retires the route the moment the |
| 414 | record goes. `DELETE /api/cli/apps/{account}/{app}/instances/{host}` is |
| 415 | `svcrole.Allow(svcrole.Instance)`, so one call from a person *or* a CI |
| 416 | credential holding that role took the control plane off the air. Same class |
| 417 | as trash and purge; same refusal. **The console must not become |
| 418 | instance-backed until this guard is in place.** |
| 419 | - **refuses an instance CREATE or PATCH by a service principal.** A person must |
| 420 | be able to write the console's record — that is the runbook step the whole |
| 421 | §10 sequence turns on — but a CI credential must not acquire the reconciler's |
| 422 | authority over the console's route, and PATCH is a repoint: the reconciler |
| 423 | converges a route's Channel and Environment *from* this record on every pass. |
| 424 | Same person/service split as promote, for the same reason. |
| 425 | - **refuses a domain DETACH outright, and a domain ATTACH by a service |
| 426 | principal.** `DELETE /api/cli/apps/{account}/{app}/domains/{host}`'s |
| 427 | first-route branch calls the same `provision.DeleteRecord` the instance |
| 428 | delete does, and `POST .../domains`'s first-route branch *writes* a declared |
| 429 | record — both registered `svcrole.Allow(svcrole.Publish, svcrole.Instance)`, |
| 430 | so a service credential reaches either. Guarding the instance family and |
| 431 | leaving these open would be a guard with a published way round it. Detach's |
| 432 | browser twin (`detachDomain`) carries the same outright refusal; attach's |
| 433 | needs none, because a browser session's principal is always a person. |
| 434 | Neither guard is conditional on the record-touching branch: a conditional |
| 435 | would have to re-derive "does this claim mint the app's route" ahead of the |
| 436 | code that derives it, and the two would drift. |
| 437 | - gets §5's two ship-time refusals. |
| 438 | |
| 439 | **The closure principle — read this before adding a tenth door.** The set above |
| 440 | is not a list to extend by taste; it is the **`provision.Record` lifecycle**, |
| 441 | closed. A record is what grants the provisioning reconciler authority over a |
| 442 | route, so every door that CREATES, MUTATES or REMOVES one for a flagged app |
| 443 | carries a guard: create (`POST .../instances`, `POST .../domains`) and mutate |
| 444 | (`PATCH .../instances/{host}`) refuse a **service principal**, because a person |
| 445 | performing them is a legitimate operator act; remove |
| 446 | (`DELETE .../instances/{host}`, `DELETE .../domains/{host}`) refuses |
| 447 | **outright**, because removal is destruction and belongs with trash and purge. |
| 448 | A future door that touches that lifecycle needs the same guard by the same |
| 449 | argument, and one that does not, does not. |
| 450 | |
| 451 | Two lifecycle sites carry no `control_plane` call and are still covered, which |
| 452 | is worth writing down so the next audit does not read them as gaps. |
| 453 | `POST .../instances/{host}/channel` (`channelmove.go`) MUTATES a record, and is |
| 454 | registered with the **zero `svcrole.Set{}`** — no service principal reaches it |
| 455 | at all, admin included, which enforces the service half more strictly than a |
| 456 | refusal would; a person repointing the control plane's channel is the same |
| 457 | legitimate operator act as promoting it. And `rollbackAttach` |
| 458 | (`domainattach.go`) deletes a record, but it is not a door: it is the |
| 459 | compensating delete inside the attach path, reachable only once that path's own |
| 460 | guard has passed, and it removes only the record the same call just tried to |
| 461 | write — guarding it would strand half-written attaches instead of protecting |
| 462 | anything. |
| 463 | |
| 464 | Promotion by a **person** is gated by ops membership, which is already the |
| 465 | operator boundary, and by nothing else. That is what keeps |
| 466 | `carlos deploy -app console` working as an ordinary command. |
| 467 | |
| 468 | **How the flag is set.** Not by a box-local sqlite write — that contradicts the |
| 469 | no-box-commands corollary. It is set at claim time: `carlos apps create` gains |
| 470 | `-control-plane`, owner-only, and the console's app settings page can clear it |
| 471 | (clearing is the dangerous direction, so clearing requires fresh auth; setting |
| 472 | does not). |
| 473 | |
| 474 | **Test:** trash/purge/rename/transfer on a flagged app are refused naming the |
| 475 | flag; the instance-record DELETE is refused for a person AND for a service, and |
| 476 | the record survives; a domain detach is refused for a person and a service at |
| 477 | the API door and for the owner at the browser door, and the claim survives; a |
| 478 | service principal's promote, retention change, instance create, instance patch |
| 479 | and domain attach are each refused — the attach in BOTH request shapes, alias |
| 480 | and first-route, since the guard is not conditional on the branch — and each |
| 481 | for both service roles the domains routes admit; a person's promote at any rung succeeds with ordinary |
| 482 | membership, and a person can still CREATE and PATCH the flagged app's instance |
| 483 | record and ATTACH a domain to it — asserted on the stored record and the stored |
| 484 | claim, not on a status code, because that half is the runbook step the flag must |
| 485 | not break; on every service-refused and outright-refused door, clearing the flag |
| 486 | lets the IDENTICAL call through, which is what proves the flag is the only |
| 487 | variable; clearing the flag requires fresh |
| 488 | auth; setting it requires owner; an unflagged app is unaffected in every case; |
| 489 | and every refusal sits behind the existing 404, so a non-member probing any of |
| 490 | these doors gets the byte-identical not-found and never learns the flag's |
| 491 | existence. |
| 492 | |
| 493 | ## 7. The version stamp and the dirty tree |
| 494 | |
| 495 | `roll-console.sh` refuses a dirty tree because the version stamp is the git |
| 496 | short sha, and a stamp naming a commit the binary does not match is how a false |
| 497 | "verified" happens. `carlos deploy` already defaults `-version` to |
| 498 | `gitShortHead()` and inherits the convention — but not the guard. |
| 499 | |
| 500 | `carlos deploy` refuses to ship when `-version` was **derived from git** and |
| 501 | the tree is dirty. An explicit `-version` is the operator saying they mean it, |
| 502 | and is not second-guessed. |
| 503 | |
| 504 | A general fix: any tenant using the default stamp today can ship a binary whose |
| 505 | sha names a commit it was not built from. |
| 506 | |
| 507 | **Test:** dirty tree + derived version refuses before anything ships; dirty |
| 508 | tree + explicit `-version` ships; clean tree ships. |
| 509 | |
| 510 | ## 8. The migration has an ordering trap |
| 511 | |
| 512 | `store.go`'s `migrate()` runs its `ALTER TABLE apps` block (lines 198–219) |
| 513 | **before** `migrateAppsKey()` (line 231), whose rebuild copies an **enumerated |
| 514 | column list** into `apps_new` (271–288) — and the code already carries a |
| 515 | comment about this hazard at line 280. |
| 516 | |
| 517 | Add `control_plane` to that ALTER block and any database that has not yet been |
| 518 | through the rebuild — a self-hoster jumping versions — has the column |
| 519 | **silently dropped**, after which every `SELECT control_plane` errors. |
| 520 | |
| 521 | **Requirement:** either add the ALTER *after* `migrateAppsKey()`, or add the |
| 522 | column to `apps_new` and its copy list. The plan picks one and tests the |
| 523 | jumping-a-version path explicitly. |
| 524 | |
| 525 | Otherwise the column is safe: every apps query names its columns (verified — |
| 526 | no `SELECT *`), so no row parser breaks. |
| 527 | |
| 528 | ## 9. The pinfra scripts stay |
| 529 | |
| 530 | `roll-console.sh` and `verify-console.sh` **stay**, reclassified in their |
| 531 | header comments as **break-glass**: the path that works when the console is |
| 532 | down. That is the honest answer to the bootstrap question — the console |
| 533 | mediates its own replacement on the happy path, §4 covers a build that fails |
| 534 | readiness, and §4a's residual (starts, serves, behaves wrong) has no automatic |
| 535 | recovery at all. A console that is already down cannot be the door to its own |
| 536 | repair. |
| 537 | |
| 538 | `ship-app.sh` and `promote-app.sh` lose their console/kass/keymail/ |
| 539 | platform-site callers but are not deleted; `promote-app.sh` still serves |
| 540 | direct-bucket promotes for apps outside any console's knowledge. |
| 541 | |
| 542 | ## 10. Order of work |
| 543 | |
| 544 | Reordered after review: the flag must exist **before** the claim, and the code |
| 545 | must reach the boxes **before** the first self-deploy. |
| 546 | |
| 547 | **0. Survey both boxes** (read-only, no changes). Per box: `console.service`'s |
| 548 | `ExecStart` path and its `Type=`; which route carries the `Unit` stamp; |
| 549 | every console route and channel (tito is entirely unsurveyed — it has |
| 550 | `console.platform.tito.io` and the still-serving `carlos.tito.io`); |
| 551 | `HasOwners("console")`; any legacy `apps/console/` prefix; the bucket |
| 552 | production flag on `console`. Flagship's readings are in the appendix; |
| 553 | **tito's are not, and step 7 deploys there first.** |
| 554 | |
| 555 | 1. §2 owner exemption, with the service-principal pin and the `handleNewApp` |
| 556 | policy change. |
| 557 | 2. §8 migration + §6 flag, its destructive-verb refusals, its service-principal |
| 558 | promote refusal, and `-control-plane` on `apps create`. |
| 559 | 3. §3 `/healthz` carries the build; the app-level build probe; the watch's |
| 560 | second exit criterion with cache-busting and a naming timeout. |
| 561 | 4. §4b restart-recovery: start the unit on the restored version after a rolled |
| 562 | back adoption. **Tenant fix; lands independently of everything else here.** |
| 563 | 5. §5 ship-time refusals. §7 dirty-tree guard. |
| 564 | 6. **`cmd/console` readiness signalling** (§4a) — the code half of the |
| 565 | `Type=notify` change. |
| 566 | 7. **Delivery roll.** One last `roll-console.sh` on both boxes carrying steps |
| 567 | 1–6, plus a CLI release carrying the new watch. Without this, step 9 runs |
| 568 | against a console that predates its own probe and fails. |
| 569 | 8. **pinfra:** `console.service` gains the readiness gate (§4a). Applied to |
| 570 | both boxes. |
| 571 | 9. **Stamp correctly** (§4c): on each box, move the `Unit` stamp to the host |
| 572 | named in `ExecStart`, exactly one route. Verify with a no-op adopt. |
| 573 | 10. **Claim `ops/console` with the flag set, in one step**, on both |
| 574 | deployments. Not two: a row without the flag is a window in which any ops |
| 575 | member or `Publish` service principal can roll the production console. |
| 576 | 11. **First `carlos deploy -app console` on tito**, verified against the build |
| 577 | probe. Tito first: smaller blast radius, and its console going down does |
| 578 | not take carloku with it. |
| 579 | 12. The same on flagship. |
| 580 | 13. Prove `carlos deploy` on `kass`, `keymail`, `platform-site` — each with |
| 581 | step 9's precondition first. `kass` and `keymail` are addr-backed bespoke |
| 582 | services whose `Unit` stamps are unsurveyed; an unstamped `keymail` proof |
| 583 | run is the #110 shape again (symlink swapped, nothing restarted, header |
| 584 | lies) and would "prove" a deploy that did not happen. |
| 585 | 14. Update CLAUDE.md's "Deploy reality", which currently sends the reader to |
| 586 | the pinfra scripts for all four apps. Reclassify the scripts as break-glass. |
| 587 | 15. File the two follow-ups: the restart-dedupe pass (§4c), and how a **fresh |
| 588 | deployment** gets its `ops/console` row (§11). |
| 589 | |
| 590 | ## 11. Open questions |
| 591 | |
| 592 | - **How does a fresh deployment or a self-hoster get the `ops/console` row?** |
| 593 | Nothing creates it — not provisioning, not `materializeQuietly`, not |
| 594 | `BackfillOpsMemberships`. Steps 10 backfills our two boxes by hand. The |
| 595 | general answer belongs to slice 3 or to #135, and is filed at step 15, but it |
| 596 | should be named as an unsolved product gap rather than left implicit. |
| 597 | - **What alarms on a console that is down?** §4a's residual and §4b's |
| 598 | rollback both print into the root pass's journal — box-only. The CLI's |
| 599 | timeout tells the operator who ran the deploy and nobody else. Out of scope |
| 600 | to build here; named so it is not mistaken for covered. |
| 601 | - **`carlos deploy -app console` run against the console being deployed** is |
| 602 | expected to work — the watch tolerates the restart — but its *diagnostics* |
| 603 | leg reads the platform log stream through the console API, so it will error |
| 604 | precisely when it is most wanted. Confirm at step 11 and degrade the |
| 605 | diagnostics gracefully rather than failing the deploy. |
| 606 | |
| 607 | ## Appendix — flagship readings, 2026-08-23 |
| 608 | |
| 609 | ``` |
| 610 | apps rows in ops (bab): deploy-probe, platform-site (no console) |
| 611 | bucket prefixes under apps/bab/: apt-carlosframework, carloku, carlosframework, |
| 612 | console, deploy-probe, derp, hello, keymail, |
| 613 | meep, platform-site |
| 614 | owners table (grants): derp, meep, platform-site (no console) |
| 615 | legacy apps/console/: absent |
| 616 | |
| 617 | console routes: |
| 618 | platform.carlosframework.com | console | canary/console | (no unit) |
| 619 | console.bab.oncarlos.com | console | (alias) | (no unit) |
| 620 | dashboard.carloku.com | console | canary/console | (no unit) |
| 621 | console.carloku.com | console | canary/console | console.service |
| 622 | |
| 623 | console.service ExecStart: /opt/carlos/live/platform.carlosframework.com serve |
| 624 | console.service Type: not read — step 0 |
| 625 | CONSOLE_ORIGIN: https://console.carloku.com |
| 626 | CONSOLE_DB: /data/console/console.db |
| 627 | ``` |
| 628 | |
| 629 | The `ExecStart` host and the stamped route disagree. That is §4c. |
| 630 | |
amadan