| 1 | # Console release defences |
| 2 | |
| 3 | Status: draft rev 5, 2026-10-06, for Paul's review. Rev 5 applies Astra |
| 4 | round 4 (`…-r4.md`, **accept with fixes**): rev 4's three fixes were judged |
| 5 | sufficient, and two new P2s are fixed. 14 (a non-canary route waiting for a |
| 6 | proof stays in the fast loop) is in §4.5, and 15 (readiness follows the |
| 7 | effective upstream, Unix socket first) is in §2.4. Rev 4 answered Astra round 3 (`…-r3.md`, |
| 8 | reject). Of the six round-2 findings it re-checked, four were fixed. Two |
| 9 | needed more: findings 10 and 11, now in §2.3, where groups key on a non-empty |
| 10 | `Socket` or `Addr` and every group route is stamped to the bound route's |
| 11 | restored version. New finding 13, a `promote --retry`, is in §2.5. All |
| 12 | accepted. Branch `design-console-release-defences`. |
| 13 | Rev 2 answered Astra round 1 (`CODEX-FINDINGS-console-release-defences.md`, |
| 14 | verdict reject, nine findings). All nine were accepted, and each fix is marked |
| 15 | **(codex N)** in the text: 1 §2.3, 2 §2.2, 3 §4.2, 4 §2.3, 5 §4.3, 6 §4.5, |
| 16 | 7 §3.2, 8 §5.2 and §6, 9 §6. Rev 3 answers round 2 (`…-r2.md`, reject: 6 of |
| 17 | the 9 fixes sufficient, 3 new findings). All six are accepted. Round 1's 6, |
| 18 | 7 and 9 are redone: a separate world-readable policy file instead of |
| 19 | host.env, a per-route program history for every route kind, and |
| 20 | direct-bucket mode for the rehearsal. The new findings 10 and 11 (rollback |
| 21 | scoped to the process group, registry stamps restored, refusal for every host |
| 22 | in the group) are in §2.3, and 12 (a binding mismatch keeps propagating the |
| 23 | restart's error) is in §2.2. |
| 24 | |
| 25 | ## 0. Why |
| 26 | |
| 27 | On 2026-10-06 the flagship console (console.carloku.com, dashboard.carloku.com, |
| 28 | platform.carlosframework.com) answered 502 for 24 minutes, 09:30 to 09:54 UTC. |
| 29 | |
| 30 | What happened, confirmed from the box journal and the bucket: |
| 31 | |
| 32 | - At 09:30:15 a console release named `v0.24.2` was shipped to |
| 33 | `apps/bab/console/releases/v0.24.2/` and promoted to `canary/console` at |
| 34 | 09:30:30. Its one artifact was the public `carlos-linux-arm64` from the |
| 35 | v0.24.2 GitHub release (sha256 `bc53cfc5…`, matching that release's |
| 36 | `SHA256SUMS.txt`): the CLI, not `./cmd/console`. The manifest says |
| 37 | `built_by: carlos v0.24.2`. It was a hand step, not `roll-console.sh`, which |
| 38 | builds `./cmd/console` and names versions by git sha. |
| 39 | - The README's release recipe never mentions the console, and the public |
| 40 | release carries no console artifact, so whoever rolled v0.24.2 had nothing |
| 41 | correct to ship. |
| 42 | - The root stage pass staged it, repointed all three live links, and ran |
| 43 | `systemctl try-restart console.service`. `console serve` is not a `carlos` |
| 44 | verb, so the new process printed the CLI's usage and exited 2 after 55 ms. |
| 45 | `try-restart` had already returned 0 (the unit is `Type=simple`), so |
| 46 | `adoptHost`'s existing rollback branch (`internal/release/adopt.go`, the |
| 47 | `rerr != nil` arm: poison, repoint back, start) never ran. |
| 48 | - systemd restarted the unit five times in one second, hit its start limit and |
| 49 | left it `failed`. The live links named the bad build; nothing looked again. |
| 50 | - Nothing alerted. A person noticed. |
| 51 | |
| 52 | Recovery was a promote of the previous build `f4478b15` back to |
| 53 | `canary/console`, then `systemctl reset-failed` and `start` over SSM: the |
| 54 | repoint alone does not revive a unit systemd has given up on. |
| 55 | |
| 56 | ## 1. Rulings |
| 57 | |
| 58 | These were decided with Paul on 2026-10-06 and are not reopened here. |
| 59 | |
| 60 | 1. **Four defences**, in this order of value: (D1) a box rolls back a named |
| 61 | unit that will not come up; (D2) `carlos ship` and adoption refuse a binary |
| 62 | whose Go main package differs from the app's previous release; (D3) the |
| 63 | flagship console adopts only builds proven on dev; (D4) process: the release |
| 64 | publishes a console artifact and the recipe rolls it dev first. |
| 65 | 2. **No new AWS account.** Dev (account 640012953448, |
| 66 | platform.dev.carlosframework.com) already is the separate test deployment. |
| 67 | The failure was that it was skipped. D3 makes skipping it impossible for the |
| 68 | flagship console without root on the console box. |
| 69 | 3. **D1 and D2 are on for every deployment** and add no steps. **D3 is off |
| 70 | unless a deployment turns it on.** An operator who does not want the |
| 71 | testing ceremony (Tito, a self-hoster) configures nothing and adopts |
| 72 | exactly as before, still covered by D1 and D2. There is no per-promote |
| 73 | bypass flag: the bypass is local and explicit, as for bake windows |
| 74 | (`internal/release/bake.go`). |
| 75 | 4. **Enforcement lives on the box that adopts**, not in the `carlos` client. |
| 76 | Anyone holding a deployment's release key can promote anything, and |
| 77 | today's promote was a hand step, so a client-side check would not have |
| 78 | fired. |
| 79 | 5. **Uptime alerting from outside the box** (would have paged within a |
| 80 | minute) belongs to #192, not this spec. |
| 81 | 6. **Scope is named-unit routes** (`registry.Route.Unit`, today only |
| 82 | `console.service`) for D1 and D3. Box binaries keep `carlos system update`'s |
| 83 | own gate and rollback. Exec children and `carlos-app@` tenants are out of |
| 84 | scope for D1; D2 applies to every binary app. |
| 85 | |
| 86 | **Threat model.** These are defences against mistakes. A compromised release |
| 87 | key or bucket is the signing chain's and the bake window's problem. Where a |
| 88 | mechanism below trusts an unsigned manifest field, that is why. |
| 89 | |
| 90 | ## 2. D1: roll back a named unit that will not come up |
| 91 | |
| 92 | ### 2.1 The gap |
| 93 | |
| 94 | `makeRestartRoute`'s first arm (`internal/host/agent.go`) returns |
| 95 | `systemctlTryRestartUnit(rt.Unit)`, whose error is only systemctl's. For a |
| 96 | `Type=simple` unit that is "the start job was queued", which says nothing about |
| 97 | whether the new binary runs. `adoptHost` already does the right thing when the |
| 98 | restart callback returns an error: it poisons the version for the pass, |
| 99 | repoints back to `cur`, restores capability drop-ins, and starts the unit on |
| 100 | the restored build (`WithStartUnit`). D1 makes the named-unit arm return an |
| 101 | honest error. |
| 102 | |
| 103 | ### 2.2 The binding comes first (codex 2) |
| 104 | |
| 105 | The console is one unit behind several hostnames, and adoption is per host in |
| 106 | `ORDER BY host` (`internal/registry/registry.go:800`). Each `adoptHost` |
| 107 | repoints its own `live/<host>` and then restarts. The unit runs whichever link |
| 108 | its `ExecStart` names: on flagship that is `live/platform.carlosframework.com`, |
| 109 | which sorts **last** of the three. A restart fired from an earlier host runs |
| 110 | the old build. The console-as-an-app spec already requires that the one |
| 111 | `Unit`-stamped route be the host `ExecStart` names (2026-08-23 design §4c), and |
| 112 | `roll-console.sh`'s final `systemctl restart` is what has hidden it when it |
| 113 | isn't. |
| 114 | |
| 115 | So D1 judges a restart only when the binding is verified. Before the |
| 116 | settle check, the named-unit arm reads `ExecStart` (`systemctl show -p |
| 117 | ExecStart <unit>`). Its executable path must be `live/<rt.Host>` for this |
| 118 | route. If it is not, the arm logs `binding: <unit> runs live/<X>, but the unit |
| 119 | stamp is on <rt.Host>; restart not checked`, does today's plain try-restart, |
| 120 | and returns **its error unchanged**, as the arm does today |
| 121 | (`internal/host/agent.go:3107`), so a failed `try-restart` still drives the |
| 122 | existing rollback (codex 12). Only the settle check is skipped. That is no |
| 123 | worse than today. Rollout step 1 (§6) verifies the |
| 124 | binding on every console box and restamps where it is wrong, and |
| 125 | `roll-console.sh` keeps its final restart until the binding check passes on |
| 126 | that box. |
| 127 | |
| 128 | ### 2.3 The settle check |
| 129 | |
| 130 | After `systemctl try-restart <unit>` returns 0 on a verified binding, the |
| 131 | named-unit arm waits for the unit to **settle**: |
| 132 | |
| 133 | - Poll every 500 ms, up to **30 s** (`CARLOS_UNIT_SETTLE`, a duration, floor |
| 134 | 5 s), reading `ActiveState`, `InvocationID`, `MainPID` and `NRestarts` from |
| 135 | `systemctl show`. |
| 136 | - The first poll that finds the unit `active` with an `InvocationID` |
| 137 | different from the one before the restart fixes **the new invocation**. |
| 138 | Everything after is judged against it, never against the previous |
| 139 | invocation's counters: systemd resets `NRestarts` on a manual start, so a |
| 140 | before-and-after comparison misreads healthy restarts (codex 4). |
| 141 | - **Healthy** when, within the new invocation, `ActiveState=active`, the |
| 142 | MainPID's `/proc/<pid>/exe` resolves into the new version's directory, and |
| 143 | the route's readiness check passes (§2.4). Return nil immediately. The |
| 144 | console takes about 8 s, so a healthy restart costs about 8 s, not 30. |
| 145 | - **Failed** as soon as `ActiveState` is `failed`, or the `InvocationID` |
| 146 | changes again (systemd restarted it), or `NRestarts` within the new |
| 147 | invocation goes above zero. Return an error carrying the state and the last |
| 148 | 20 lines of `journalctl -u <unit> -o cat`. |
| 149 | - **Failed** at the deadline if the unit never satisfied "healthy". That |
| 150 | includes a process that is still running but never became ready. |
| 151 | |
| 152 | A unit that was **not running** before the restart (an operator stopped it) |
| 153 | is not checked: `try-restart` started nothing, and that stays a no-op. |
| 154 | |
| 155 | The check runs inside the root stage pass and makes it up to 30 s longer |
| 156 | (about 8 s for a healthy console). A stage request written while that pass |
| 157 | runs past the relay's 15 s window waits for the next trigger: the known gap |
| 158 | recorded with the deploy-pickup fix. Only the bound route (§2.2) carries the |
| 159 | `Unit` stamp, so there is one checked restart per unit per pass. Hosts that |
| 160 | sort after it skip a rejected version through `poisoned`, and hosts before it |
| 161 | are repointed by the rollback below. |
| 162 | |
| 163 | On a failure the existing `adoptHost` branch then repoints to `cur`. The |
| 164 | rollback must **replace** whatever is running, not merely start (codex 1). A |
| 165 | deadline failure can leave the rejected build running and unready, and |
| 166 | `systemctl start` does nothing to a running unit (`internal/host/agent.go:3208`). |
| 167 | So for a named unit that D1 rejected, the rollback runs `systemctl |
| 168 | reset-failed <unit>`, which clears a start limit, then `systemctl restart |
| 169 | <unit>`. Every other caller of `makeStartRoute` keeps `start`. The rollback |
| 170 | gets the same settle check against `cur`'s directory. Its failure is logged |
| 171 | as "rollback did not come up either", the one case that still needs a person |
| 172 | (§2.6 alerts it). |
| 173 | |
| 174 | **The process group (codex 10, 11).** Hosts that sort before the bound host |
| 175 | may already have repointed their links to the rejected version and had their |
| 176 | registry versions stamped (`SetVersions`, `internal/release/adopt.go:1203`), |
| 177 | and the edge's `X-Carlos-Version` comes from that stamp |
| 178 | (`internal/edge/edge.go:752`). The rollback covers exactly the routes that |
| 179 | share the rejected **process**. A route's upstream is its `Socket` if set, |
| 180 | else its `Addr` (`Socket` wins, `internal/registry/registry.go:40`). The group |
| 181 | is the bound route plus every unstamped route of the same app with the **same |
| 182 | non-empty upstream**. A bound route with no upstream at all is a group of one, |
| 183 | and two empty upstreams never match (codex 10, round 3). A route of the same |
| 184 | app with a different upstream is a different process, and the rollback does |
| 185 | not touch it, whatever version it runs. |
| 186 | |
| 187 | Every route in the group is served by the one process, so after a rollback |
| 188 | each of them is serving whatever the bound route rolled back to. On rollback, |
| 189 | every group route whose live link names the rejected version is repointed to |
| 190 | the **bound route's restored version**, and its registry versions are set to |
| 191 | match with `SetVersions`. This applies whether the route moved in this pass |
| 192 | or in an earlier, interrupted one (codex 11, round 3). The stamp then names |
| 193 | what is actually answering. The refusal record (§2.5) is written for **every |
| 194 | host in the group**, so no sibling re-adopts the rejected build on the next |
| 195 | pass. |
| 196 | |
| 197 | ### 2.4 Readiness |
| 198 | |
| 199 | In order: |
| 200 | |
| 201 | Readiness follows the route's **effective upstream**: `Socket` if set, else |
| 202 | `Addr`. `Socket` wins, as it does at the edge |
| 203 | (`internal/registry/registry.go:40`), and named units may be socket-backed |
| 204 | (`unitRail`, `cli.go:1624`) (codex 15, round 4). In order: |
| 205 | |
| 206 | 1. The route's own health probe (`registry.Route.Health`, the probe-config |
| 207 | JSON instances already use), sent as HTTP to the effective upstream: over |
| 208 | the Unix socket for a socket-backed route, over TCP to `Addr` otherwise. |
| 209 | 2. Else, for a route with an upstream: a connect to it succeeds (Unix or TCP). |
| 210 | 3. Else: the unit stays `active` within one invocation for 5 s. |
| 211 | |
| 212 | The three console routes get `{"path":"/healthz","status":200}` as part of |
| 213 | rollout (§6). The console's `/healthz` (`internal/console/server.go`) is |
| 214 | unauthenticated and answers as soon as the listener is up. |
| 215 | |
| 216 | ### 2.5 Refusing the build until something changes |
| 217 | |
| 218 | `poisoned` lives for one pass. Without more, the next pass, and the agent's |
| 219 | 2 s fast loop asking for a root stage pass every 15 s, would re-adopt the same |
| 220 | bad build and bounce the console each time. |
| 221 | |
| 222 | When D1 rolls a route back, the root pass writes a **refusal record** to the |
| 223 | box's bake directory (`BakeDir`, under the data dir, which the sandboxed agent |
| 224 | can read): |
| 225 | |
| 226 | ```json |
| 227 | {"schema":1,"host":"console.carloku.com","app":"console","version":"v0.24.2", |
| 228 | "sha256":"bc53cfc5…","platform":"linux-arm64", |
| 229 | "pointer_promoted_at":"2026-10-06T09:30:30Z","refused_at":"…","reason":"…"} |
| 230 | ``` |
| 231 | |
| 232 | `adoptHost`, on every pass, root or agent, before `ensureStaged` and before |
| 233 | `stageBlocked` can relay: if the channel pointer names this sha256 for this |
| 234 | platform **and** the pointer's `PromotedAt` equals the record's, print |
| 235 | `refused: <version> did not come up on this box at <time> (<reason>); retry |
| 236 | with: carlos promote --retry …` and return. A pointer that moves to different |
| 237 | bytes clears it, and so does a pointer re-signed with a new `PromotedAt`. |
| 238 | |
| 239 | **The retry needs a new flag (codex 13, round 3).** Promoting the version a |
| 240 | channel already names is a successful no-op that does not rewrite the pointer |
| 241 | (`PromoteOnLadder`, `internal/release/release.go:910`), so a plain re-promote |
| 242 | leaves `PromotedAt` unchanged and the refusal standing. |
| 243 | `carlos promote --retry <version> <channel>` re-signs the pointer with a fresh |
| 244 | `PromotedAt` even when it already names `<version>`. It refuses unless it does |
| 245 | already name `<version>`, so it can only renew a promotion, never change one. |
| 246 | The API promote door takes the same `retry` field. Both reach it through one |
| 247 | `PromoteOnLadder` option, so the ladder, scope and authorization checks run |
| 248 | exactly as for any promote. |
| 249 | |
| 250 | The record keys on the artifact sha256 read from the manifest, not on the |
| 251 | version name, because names get reused. |
| 252 | |
| 253 | ### 2.6 Being seen |
| 254 | |
| 255 | - A rollback writes a line naming both versions, the reason and the journal |
| 256 | tail to the root pass's output (journal). |
| 257 | - The agent raises an alert through its existing sinks (`alertsink.go`: SNS, |
| 258 | which works on flagship, and `CARLOS_ALERT_WEBHOOK`) the first time it reads |
| 259 | a new refusal record: subject `console.carloku.com: v0.24.2 did not start, |
| 260 | rolled back to f4478b15`. "Rollback did not come up either" alerts with a |
| 261 | louder subject. The plan decides whether this joins the metrics `Event` |
| 262 | shape or a small sibling; it must not wait on #192. |
| 263 | - `carlos status --live` shows a refused host with the refusal's version and |
| 264 | reason. |
| 265 | |
| 266 | ## 3. D2: refuse the wrong program |
| 267 | |
| 268 | ### 3.1 At ship |
| 269 | |
| 270 | For a `binary` artifact, `carlos ship` reads the Go build info with |
| 271 | `debug/buildinfo.ReadFile`. This works on a stripped (`-s -w`) binary built for |
| 272 | another architecture without running it: a stripped arm64 `carlos` read on |
| 273 | amd64 gives `path github.com/carlosframework/platform`, and a console build |
| 274 | gives `github.com/carlosframework/platform/cmd/console`. |
| 275 | |
| 276 | - The manifest gains `program` (string, the build info's main package path), |
| 277 | additive to schema 1 like `capabilities` and `label`: omitted when empty, |
| 278 | ignored by old binaries. A non-Go binary has no build info and no `program`. |
| 279 | With several platform artifacts, every one must report the same path, or |
| 280 | ship refuses. |
| 281 | - Ship reads the app's **latest** release's manifest. If both name a |
| 282 | `program` and they differ, ship refuses: |
| 283 | |
| 284 | ``` |
| 285 | this artifact is github.com/carlosframework/platform; console's releases |
| 286 | are github.com/carlosframework/platform/cmd/console. If the app really is |
| 287 | changing program, ship again with --program-change. |
| 288 | ``` |
| 289 | |
| 290 | - `--program-change` ships anyway and records `"program_change": true` in the |
| 291 | manifest. |
| 292 | - No previous release, a previous release without `program`, or a non-Go |
| 293 | artifact: ship as today. The check starts with the next release of each app. |
| 294 | - The console's API ship door (`internal/console/apicli.go`, the |
| 295 | `release.Ship` caller) makes the same check server-side, so API-mode ships |
| 296 | cannot skip it and direct-bucket ships (`cli.go`) do it client-side. Both |
| 297 | call one function in `internal/release`. |
| 298 | |
| 299 | "Latest" is the newest release by manifest `built_at` under |
| 300 | `apps/<acct>/<app>/releases/`. The plan may use the channel heads instead if |
| 301 | listing every release is too slow; either catches today's case. |
| 302 | |
| 303 | ### 3.2 At adoption |
| 304 | |
| 305 | `program` is an unsigned manifest field, and an older client writes none. So |
| 306 | the box checks the bytes themselves. In `adoptHost`, after `ensureStaged` and |
| 307 | before the capability converge and `PointLive`: read build info from the |
| 308 | staged new binary and from the binary `cur` names. If both are readable and |
| 309 | their main package paths differ, and the new manifest does not carry |
| 310 | `program_change: true`, refuse: |
| 311 | |
| 312 | ``` |
| 313 | console.carloku.com: v0.24.2 is github.com/carlosframework/platform, the live |
| 314 | build is …/cmd/console — not pointed live (ship with --program-change if this |
| 315 | is intended) |
| 316 | ``` |
| 317 | |
| 318 | and write a refusal record (§2.5) so the agent does not relay it again. This |
| 319 | alone would have stopped today's outage on the box, with no service |
| 320 | interruption, even with the bad manifest already promoted. |
| 321 | |
| 322 | **Going back across a deliberate change (codex 7).** After an intended A→B |
| 323 | change, promoting an old A release back would be refused, because A's |
| 324 | immutable manifest has no `program_change`. So each route keeps a **program |
| 325 | history** in the bake directory: every main package path the box has pointed |
| 326 | live for that route, recorded at `PointLive` for every route kind (exec, |
| 327 | `carlos-app@`, named unit). The check passes when the staged binary's program |
| 328 | is in the route's history. Going back to A is therefore accepted on any route |
| 329 | that ran A here, and a program the route has never run still needs |
| 330 | `--program-change`. A route with no history yet, such as a box upgraded to |
| 331 | this release, seeds it from the live binary on its first pass. |
| 332 | |
| 333 | ## 4. D3: the flagship console adopts only builds proven on dev |
| 334 | |
| 335 | ### 4.1 Shape |
| 336 | |
| 337 | - A **proving deployment** (dev) mints a signed **proof** once a build has |
| 338 | run healthily on its console for a bake period. |
| 339 | - An operator **carries** the proof to the target deployment's bucket. It is |
| 340 | signed, so carrying needs no trust and no access between the AWS accounts. |
| 341 | - A **requiring deployment** (flagship) adopts a build of a listed app only |
| 342 | with a proof for those exact bytes from a prover it trusts, or a build it |
| 343 | has itself run healthily before. |
| 344 | |
| 345 | ### 4.2 The proof |
| 346 | |
| 347 | ```json |
| 348 | {"schema":1,"app":"console","platform":"linux-arm64", |
| 349 | "sha256":"e47fcc1e…","version":"f4478b15", |
| 350 | "prover":"dev","active_since":"…","proven_at":"…", |
| 351 | "sig":"<base64 ed25519 over the canonical JSON of every other field>"} |
| 352 | ``` |
| 353 | |
| 354 | It binds app, platform and artifact sha256. `version` is informational. The |
| 355 | target compares sha256, so flagship must run **the same bytes** dev ran |
| 356 | (§5.2), and a version name reused for different bytes has no proof. |
| 357 | |
| 358 | **Every sha256 in D3 is hashed from bytes, never copied from a manifest |
| 359 | (codex 3).** A version name can be reused after pruning |
| 360 | (`internal/release/release.go:632`). An already-staged binary skips the |
| 361 | download check (`internal/release/adopt.go:1317`), and `backfillManifest` |
| 362 | fetches whatever manifest the bucket now holds for that name without |
| 363 | comparing it to the staged file (`internal/release/adopt.go:1402`). So a |
| 364 | manifest's sha256 can name bytes the box is not running. Minting hashes the |
| 365 | running process's executable (`/proc/<MainPID>/exe`). The requiring box |
| 366 | hashes the staged file it is about to point live, on a cache hit too, and the |
| 367 | healthy history records the hash of the executable that passed the settle |
| 368 | check. |
| 369 | |
| 370 | The proving key is a fresh ed25519 key, separate from both release keys, held |
| 371 | only by the proving console box as a root-only file. `carlos proof keygen` |
| 372 | writes the private half and prints the public half. |
| 373 | |
| 374 | ### 4.3 Minting on dev |
| 375 | |
| 376 | Dev's console box gets a `prove` block in its proof policy file (§4.5): |
| 377 | apps `console`, prover `dev`, `after` 15m, and the private key at |
| 378 | `/etc/carlos/proving.key` (root 0600). |
| 379 | |
| 380 | A root pass mints a proof for a named-unit route of a listed app when all of |
| 381 | these hold at once: |
| 382 | |
| 383 | - the route's binding is verified (§2.2), and the unit is `active` with an |
| 384 | `ActiveEnterTimestamp` at least `after` ago and the same |
| 385 | `InvocationID` throughout; |
| 386 | - S is the sha256 of the bytes of `/proc/<MainPID>/exe`, and the live link |
| 387 | resolves to that same file; |
| 388 | - the route's readiness check (§2.4) passes now; |
| 389 | - no refusal record names S; |
| 390 | - no proof for S exists yet in dev's bucket. |
| 391 | |
| 392 | It writes `apps/<acct>/<app>/proofs/<sha256>/<prover>.json` to dev's own |
| 393 | bucket: the same key shape the target uses (§4.4), so `proof get` and `proof |
| 394 | put` move one object between two buckets under one name. |
| 395 | |
| 396 | Minting needs root (the key), and root passes run hourly or when the agent |
| 397 | asks. So the agent asks: on its fast loop, when a listed app's route has been |
| 398 | live and active for `after` with no proof in the bucket, it |
| 399 | requests a root pass through the existing stage-request relay. It **keeps |
| 400 | asking every 2 minutes until it sees the proof in the bucket** (codex 5). The |
| 401 | relay can miss a request written while `carlos-stage.service` is already |
| 402 | running (`internal/host/agent.go:1203`), and a mint write can fail. A proof is |
| 403 | then normally minted within a minute or two of the bake period ending, and at |
| 404 | worst within the hourly pass. |
| 405 | |
| 406 | Dev's bucket layout and IAM already let the box write under its own prefix; |
| 407 | the plan confirms the exact prefix the box role may write. |
| 408 | |
| 409 | ### 4.4 Carrying it |
| 410 | |
| 411 | - `carlos proof get --app console --sha256 <S>`, against the proving |
| 412 | deployment (dev env), prints the proof JSON, or fails "not proven yet: |
| 413 | <S> has been live on dev since <t>, proven after <15m>". |
| 414 | - `carlos proof put --app console <file>`, against the target (flagship env), |
| 415 | checks the JSON's shape and writes it to |
| 416 | `apps/<acct>/<app>/proofs/<sha256>/<prover>.json`. It cannot check the |
| 417 | signature without the target's trust config, and need not: the box does. |
| 418 | |
| 419 | `roll-console.sh` runs both (§5.2). |
| 420 | |
| 421 | ### 4.5 Requiring on flagship |
| 422 | |
| 423 | Flagship's console box gets a `require` block in its proof policy file, |
| 424 | naming app `console`, prover `dev` and dev's public key. The file's shape and |
| 425 | how it is read are below. |
| 426 | |
| 427 | In `adoptHost`, for a route whose app is listed, the check runs twice: |
| 428 | |
| 429 | 1. **A cheap gate, after reading the new manifest and before |
| 430 | `ensureStaged`,** so an unproven build is neither downloaded nor relayed to |
| 431 | root. The box takes the manifest's sha256 for its platform and looks for |
| 432 | `apps/<acct>/<app>/proofs/<sha256>/<prover>.json`. It continues if the |
| 433 | proof parses, its signature verifies against the configured key, and its |
| 434 | `app`, `platform` and `sha256` match. |
| 435 | 2. **The binding check, after `ensureStaged` and before `PointLive`.** The box |
| 436 | hashes the staged file and requires that hash to equal the proof's |
| 437 | sha256, or to be in the healthy history. A mismatch means the staged bytes |
| 438 | are not the proven ones (a reused name, a stale cache). The box refuses |
| 439 | the version with a refusal record (§2.5) naming both hashes. |
| 440 | |
| 441 | It also proceeds, with no proof, if the sha256 is in the box's own **healthy |
| 442 | history**: a record in the bake directory of builds this box has run well. A |
| 443 | root pass adds a sha256 when D1's settle check passes for it, and also when it |
| 444 | finds any named-unit route's live build meeting §4.3's first three mint |
| 445 | conditions (bound, active in one invocation for the policy's `after`, the |
| 446 | hashed exe, ready; 15m when there is no policy file). The second rule is what puts the build already live when |
| 447 | the requirement is switched on into the history without a restart. So rolling |
| 448 | back to a build flagship ran before, as this morning's recovery did, never |
| 449 | waits on dev. |
| 450 | |
| 451 | Otherwise it prints `console.carloku.com: v… (sha256 …) is not proven on dev; |
| 452 | staying on <cur>` and returns. This is not a refusal record: the box looks |
| 453 | again, so the build adopts soon after its proof arrives. The agent's fast |
| 454 | loop runs every 2 s, so a missing proof is cached per (route, sha256) for |
| 455 | 30 s: at most two bucket GETs a minute per listed route while waiting, none |
| 456 | once the build is live. |
| 457 | |
| 458 | **A waiting route stays in the fast loop (codex 14, round 4).** `fastMembers` |
| 459 | (`internal/host/agent.go:1550`) includes canary routes on every tick, but |
| 460 | drops a non-canary route as soon as it has a live version. So a non-canary |
| 461 | route waiting for a proof would be rechecked only by the next serial bump or |
| 462 | the hourly pass. While a listed route has an outstanding "not proven" verdict |
| 463 | for its channel head, `fastMembers` keeps it as a member, and the 30 s cache |
| 464 | above bounds what that costs. The console is on `canary/console` and is |
| 465 | always a member; this is for any other listed app. |
| 466 | |
| 467 | `carlos status --live` shows "waiting for a dev proof" for that host. |
| 468 | |
| 469 | **Where the setting lives (codex 6).** The agent gets host.env through |
| 470 | `EnvironmentFile` only when it starts (`internal/host/units/carlos-edge.service:18`), |
| 471 | while each root pass starts fresh (`carlos-stage.service:30`). An edit would |
| 472 | reach one process and not the other, and restarting `carlos-edge` takes the |
| 473 | box's edge down. host.env is also root-only 0600 |
| 474 | (`infra/modules/region-host/README.md:23`), so the agent cannot read it |
| 475 | itself. |
| 476 | |
| 477 | So the policy is **not in host.env**. It lives in its own file, |
| 478 | `/etc/carlos/proof-policy.json`, root-owned, mode 0644. It holds no secrets: |
| 479 | app names, the prover name, public keys and durations: |
| 480 | |
| 481 | ```json |
| 482 | {"schema":1, |
| 483 | "prove":{"apps":["console"],"prover":"dev","after":"15m", |
| 484 | "key_file":"/etc/carlos/proving.key"}, |
| 485 | "require":[{"app":"console","prover":"dev","key":"<base64 ed25519 public key>"}]} |
| 486 | ``` |
| 487 | |
| 488 | Both the agent and root read it at the start of every pass. An absent file |
| 489 | means D3 is off. An unreadable or unparseable file means D3 is **on and |
| 490 | waiting** for every named-unit route, since the box cannot tell which apps |
| 491 | were listed. It logs this every pass, because a typo must not open the gate. `key_file` names the root-only |
| 492 | private key, and only root opens it. |
| 493 | |
| 494 | **Break glass:** delete the `require` entry (or the file) as root, |
| 495 | then `touch` the stage-request path (or run `carlos adopt --stage-only`) so a |
| 496 | root pass adopts at once. The agent's next tick also sees the line gone. |
| 497 | While the requirement is off for an app on a box that has required a proof |
| 498 | for it before (a marker the root pass writes when it reads a `require` |
| 499 | entry), the box logs once an hour that the requirement is off. A box that |
| 500 | never had the requirement, such as Tito's or a self-hoster's, stays quiet. Nothing in the bucket |
| 501 | or the CLI can switch it off. |
| 502 | |
| 503 | ### 4.6 Who sees none of this |
| 504 | |
| 505 | A deployment with no proof policy file: no proof is minted, read or required, |
| 506 | and adoption is unchanged except for D1 and D2. Tito, self-hosters and every |
| 507 | customer app on carloku.com are in this position. |
| 508 | |
| 509 | ## 5. D4: process |
| 510 | |
| 511 | ### 5.1 The release carries a console |
| 512 | |
| 513 | The release recipe (README "Releasing the platform") builds |
| 514 | `console-linux-arm64` and `console-linux-amd64` from the tag with |
| 515 | `go build -trimpath -ldflags "-s -w -X …version.Build=vX.Y.Z" ./cmd/console`, |
| 516 | lists them in `SHA256SUMS.txt` (signed as today), and publishes them with the |
| 517 | other files. Step 4's "eight files" becomes ten. |
| 518 | |
| 519 | ### 5.2 The console roll recipe |
| 520 | |
| 521 | `carloku-infrastructure/scripts/roll-console.sh` changes: |
| 522 | |
| 523 | - **A `dev` target** with the dev coordinates (profile `kass`, bucket |
| 524 | `carlos-dev-640012953448`, console box `i-05498eada6f0afd23`, |
| 525 | platform.dev.carlosframework.com). The dev coordinates move here from |
| 526 | platform-infrastructure, because flagship's rule now depends on dev. |
| 527 | - **`--artifact <file>`**: roll exactly this file instead of building. A |
| 528 | release's console roll uses the public `console-linux-arm64`, verified |
| 529 | against the signed `SHA256SUMS.txt`, so dev and flagship run the bytes the |
| 530 | public got. Without `--artifact` it builds once with `-trimpath` and keeps |
| 531 | the file for the next target. |
| 532 | - **`flagship` runs `proof get` against dev first** and refuses to ship if |
| 533 | there is none, naming how long dev has had the build. With a proof it ships, |
| 534 | promotes, runs `proof put` and verifies as today. |
| 535 | - **The adopt and restart SSM steps go once the box's binding is verified** |
| 536 | (§2.2): the script reads the binding line from the box first, and keeps its |
| 537 | final `systemctl restart` on any box where the binding is wrong. Otherwise |
| 538 | adoption restarts the unit and D1 checks it. The `/proc/<pid>/exe` |
| 539 | verification stays. |
| 540 | - **A `--bootstrap` flag for the first flagship roll only** skips the proof |
| 541 | step, for the one roll that installs D3 before dev can mint (§6). It prints |
| 542 | that it is skipping. It does not exist on the box: flagship's requirement is |
| 543 | switched on only after that roll. |
| 544 | |
| 545 | The recipe's console step reads: roll the release's console to dev, wait for |
| 546 | the proof (15 minutes), roll it to flagship. It says in so many words that |
| 547 | the public `carlos-linux-*` binary is never a console artifact. D2 enforces |
| 548 | that now, but the recipe should not lean on it. |
| 549 | |
| 550 | ## 6. Rollout |
| 551 | |
| 552 | Order matters: the box binary is what checks, so it goes first. |
| 553 | |
| 554 | 1. **Fix the bindings.** On dev's and flagship's console boxes, check that the |
| 555 | one `Unit`-stamped route is the host `ExecStart` names, and restamp where |
| 556 | it isn't (2026-08-23 design §4c). Flagship was stamped on |
| 557 | `console.carloku.com` while `ExecStart` names |
| 558 | `live/platform.carlosframework.com`; confirm the current state on the box. |
| 559 | 2. Ship D1, D2 and D3 in one platform release. With no config, D3 is inert. |
| 560 | 3. Set the health probe on the console routes on dev and flagship. |
| 561 | 4. Roll the release's boxes (`carlos system update`) on dev and flagship. |
| 562 | 5. **On dev:** run `carlos proof keygen` on the console box, write its proof |
| 563 | policy file with the `prove` block, and roll the console to dev with the |
| 564 | new recipe. That |
| 565 | roll is the first live check of D1: watch the journal for the settle line. |
| 566 | 6. **Rehearse on dev** (codex 9): |
| 567 | - Ship the public `carlos-linux-arm64` as a console release using a |
| 568 | **v0.24.1 or older client in direct-bucket mode** (`CARLOS_DEPLOYMENT_BUCKET` |
| 569 | set, no `--console`), which is how today's bad ship went out. That |
| 570 | client has no ship-side check and writes no `program`. API mode would |
| 571 | hit the console's server-side check instead (codex 9, round 2). Promote |
| 572 | it and confirm that the box's D2 check refuses it, with no restart. |
| 573 | - Ship a console built to exit 2 on start, from a real `./cmd/console` so |
| 574 | D2 passes. Confirm D1 rolls back within 30 s, writes a refusal, raises |
| 575 | the alert, stays refused after a plain re-promote, and retries after |
| 576 | `carlos promote --retry`. |
| 577 | - Ship a console that runs but never answers `/healthz`. Confirm the |
| 578 | deadline rollback replaces the running process. |
| 579 | - Delete all three releases. |
| 580 | 7. **Wait for dev's proof** of the build rolled in step 5. |
| 581 | 8. **Roll that build to flagship.** The recipe carries the proof (codex 8). |
| 582 | If the flagship console must move before dev can mint, use `--bootstrap` |
| 583 | once. |
| 584 | 9. **On flagship:** write the proof policy file with the `require` block and |
| 585 | dev's public key. The live build is unaffected, because the requirement |
| 586 | gates moves only, and it enters the healthy history 15 minutes later |
| 587 | (§4.5). |
| 588 | 10. Every later console roll goes through dev, with the proof carried. |
| 589 | |
| 590 | The proof policy files are not delivered by tofu `user_data`, because a |
| 591 | whole-root apply restarts every box (memory: carloku-infra-split). Write them |
| 592 | over SSM and record their contents in carloku-infrastructure's runbook (for |
| 593 | flagship) and platform-infrastructure's (for dev). The plan checks whether |
| 594 | they have a managed home they should go in instead. |
| 595 | |
| 596 | ## 7. Testing |
| 597 | |
| 598 | - **D1:** `makeRestartRoute`'s named-unit arm with a fake `systemctl show` |
| 599 | sequence: healthy after N polls; `failed`; `InvocationID` changed after |
| 600 | the new invocation was fixed; `NRestarts` reset to 0 by the manual restart |
| 601 | and the unit healthy (must pass); deadline with the process still running; |
| 602 | not running before (no check); `ExecStart` naming another host's link (no |
| 603 | check, plain try-restart). A D1 rollback runs `reset-failed` then |
| 604 | `restart`, and replaces a still-running rejected process. Other start |
| 605 | callers keep `start`. A binding mismatch whose `try-restart` fails still |
| 606 | returns the error and rolls back. `adoptHost` with a failing settle: |
| 607 | every group route whose link names the rejected version is repointed and |
| 608 | stamped to the bound route's restored version, including one advanced by |
| 609 | an earlier interrupted pass; groups key on `Socket` before `Addr`; two |
| 610 | routes with empty upstreams are never grouped; a same-app route on a |
| 611 | different upstream is left alone; the refusal record is written for every |
| 612 | host in the group; the next pass, agent or root, skips without relaying. |
| 613 | - **Readiness:** a socket-backed named unit with a configured probe passes |
| 614 | over its Unix socket; a route with both `Socket` and `Addr` is probed on the |
| 615 | socket only. |
| 616 | - **Retry:** a plain re-promote of the refused version is still a no-op and |
| 617 | the refusal stands; `promote --retry` through the CLI (direct mode) and the |
| 618 | API door re-signs with a fresh `PromotedAt` and clears it; `--retry` against |
| 619 | a channel naming a different version is refused. |
| 620 | - **D2:** `buildinfo` read from tiny stripped binaries with two different |
| 621 | main packages, cross-built for linux-arm64 by a test helper at test time |
| 622 | (no binaries checked in). Ship refuses, accepts with |
| 623 | `--program-change`, accepts non-Go, accepts a first release. The API door |
| 624 | and the CLI call the same function. `adoptHost` refuses a mismatched staged |
| 625 | binary before `PointLive` and writes a refusal. It accepts a program in the |
| 626 | route's program history (rollback across a deliberate change) on an exec |
| 627 | route, a `carlos-app@` route and a named-unit route, and seeds the history |
| 628 | from the live binary on a route that has none. |
| 629 | - **D3:** proof canonical encoding round-trip and signature; mint conditions |
| 630 | each false in turn, including a live link and a running exe that hash |
| 631 | differently; `adoptHost` with the requirement set: no proof, bad signature, |
| 632 | wrong sha256, wrong app, wrong prover, valid proof, healthy history; a |
| 633 | valid proof for the manifest's sha256 with a staged cache hit whose bytes |
| 634 | hash differently (refused at the binding check). The policy file changes |
| 635 | between passes and the agent sees the change without a restart; an |
| 636 | unparseable policy file keeps the gate closed. The agent re-requests a mint |
| 637 | until the proof appears. A non-canary listed route with a live version |
| 638 | stays in `fastMembers` while it waits for a proof, and leaves once it |
| 639 | adopts. No requirement set: no bucket GET is made (assert on the fake |
| 640 | store). |
| 641 | - **The dev rehearsal in §6 step 6** is the end-to-end test. Mocks cannot show |
| 642 | that systemd's start-limit state and `reset-failed` behave as written. |
| 643 | |
| 644 | ## 8. Not in this spec |
| 645 | |
| 646 | - External uptime alerting: #192. |
| 647 | - D1 for exec children and `carlos-app@` units. Those restarts already have |
| 648 | other failure handling, and a wrong-program ship to them is D2's job. |
| 649 | - Making `carlos system update` require dev proofs for box binaries. Its own |
| 650 | gate covers box binaries, and the proof shape extends to it if wanted later. |
| 651 | - Deleting the bad `apps/bab/console/releases/v0.24.2` release. That is an |
| 652 | operator clean-up, independent of this work; D2's box check makes it inert |
| 653 | once rolled. |
| 654 | |
amadan