| 1 | # `carlos system update` — the platform rolls itself |
| 2 | |
| 3 | **Date:** 2026-09-06 · **Status:** draft, two review rounds (Fable, Codex), |
| 4 | awaiting Paul · **Requested by:** Paul |
| 5 | |
| 6 | **Builds on:** discussion #100 (boxes should self-update via a channel |
| 7 | subscription), 2026-08-23-console-as-an-app-design.md (slice 3, deferred |
| 8 | there; the dividing line this spec applies), the root-relay pattern |
| 9 | (`carlos-stage.path` → `carlos adopt --stage-only`), and pinfra's |
| 10 | `update-platform-binary.sh`, whose 85 lines are the procedure this spec |
| 11 | moves into the binary and then retires. |
| 12 | |
| 13 | ## 0. How this spec was arrived at |
| 14 | |
| 15 | One brainstorm with Paul on 2026-09-05/06, after a merge of `main` stalled |
| 16 | on "and deploy": the only way to roll a platform binary today is an |
| 17 | operator with SSM access to every box, running a script per box, in an |
| 18 | order held in operator memory. Paul's framing, which is the requirement: |
| 19 | |
| 20 | > Ultimately, a fleet operator should be able to do `carlos system update` |
| 21 | > and the whole platform updates to the latest version based on what we |
| 22 | > just released. […] And I guess each piece should be updatable |
| 23 | > individually, `carlos system router update` etc. |
| 24 | |
| 25 | Three questions settled the shape before any design was written: |
| 26 | |
| 27 | - **Trigger: promote-driven.** A fleet operator says when. Boxes do not |
| 28 | poll the public release feed and roll on their own; a new release exists |
| 29 | in the world for as long as the operator wants before any box sees it. |
| 30 | - **Rollout: two channels, two promotes.** `canary` and `stable`, one |
| 31 | operator action each, exactly like an app. No auto-promotion in v1. |
| 32 | - **Third-party deployments in v1.** Tito's operator rolls Tito's boxes |
| 33 | from Tito's console. The artifact comes from the public signed release, |
| 34 | so the vendor never needs access to a customer box (the trust shape #100 |
| 35 | was filed about). |
| 36 | |
| 37 | Three approaches were compared; this document is the first. The other two |
| 38 | are recorded in §12. The draft went through two rounds of review by two |
| 39 | independent readers checking claims against the code (§13); everything |
| 40 | below is the post-review shape, and §13 says what the drafts got wrong. |
| 41 | |
| 42 | One ruling stated up front because it changes what "each piece" means: |
| 43 | the edge, the activator and the adoption loop are **one process** |
| 44 | (`carlos agent`, unit `carlos-edge.service`). There is nothing to update |
| 45 | "individually" inside it without splitting the process, which is a |
| 46 | separate project. So the honest component list today is one: the |
| 47 | **router**, which is the substrate binary. The console is already an app |
| 48 | and is updated as one (`carlos deploy --app console`). §3 keys everything |
| 49 | per component so a split later adds a name, not a schema. |
| 50 | |
| 51 | --- |
| 52 | |
| 53 | ## 1. What exists, and what is missing |
| 54 | |
| 55 | The platform already has most of this machinery, built for apps: |
| 56 | |
| 57 | | Have | Where | Reused as | |
| 58 | |---|---|---| |
| 59 | | Signed channel pointer, CAS-written, single parse site | `release.ChannelPointer`, `signPointer`/`verifyPointer` (`release.go:102,350,397`) | The platform pointer's wrapper, signing and verification | |
| 60 | | The one unsigned "something changed" signal, polled at 2s | `release.BumpSerial` on `control/serial` (`serial.go:51`), `fastConvergeLoop` (`agent.go:1412`) | How a box learns there is a new platform pointer | |
| 61 | | Root relay from a sandboxed agent | `/data/stage-requested` → `carlos-stage.path` → `carlos-stage.service` | Same shape, new marker, new oneshot | |
| 62 | | Fleet-box bucket proxy with an explicit grant for the platform binary | `storedoor.go:383` `aclRead`: exact `bootstrap/carlos`, "the sanctioned self-update path (F7)" | Boxes behind the door fetch the artifact with no public egress | |
| 63 | | Verified download of a public release | `update.go` `selfReplace`: tag-pinned URL, `SHA256SUMS.txt.sig` over the sums file, per-asset sha, lie-detector `version` run | The console's ingest of a release | |
| 64 | | A box's running build, published to the bucket | `version.Build` → `BoxMetrics.Release` in `status/boxes/<label>.json` every 5 min (`agent.go:2318`) | `carlos system status` and the fleet-side half of the gate | |
| 65 | | The unit list a box must have armed, and the exact `systemctl` sequence | `enabledUnits`, `systemctlPlan` (`bootstrap.go:72,96`): enable all, start timers and paths, **never start a service** | What apply arms, so apply and bootstrap never disagree and apply never starts itself | |
| 66 | | The apply procedure | `update-platform-binary.sh` (pinfra) | Moved into `carlos system apply`, line for line, then retired | |
| 67 | |
| 68 | What is missing, and is the whole of this spec: |
| 69 | |
| 70 | 1. A **pointer for something that is not an app** — no account, no route, |
| 71 | no upstream, and a version that is a platform tag, not a release |
| 72 | serial. |
| 73 | 2. A **box-side leg** that notices the pointer, fetches and verifies the |
| 74 | artifact, and asks root to apply it. |
| 75 | 3. A **root apply** that does the script's dance from inside the binary, |
| 76 | with the three things the script does not have: a **journal** so an |
| 77 | interrupted apply is undone rather than left half-done, a **health |
| 78 | gate**, and a **rollback**. Today `carlos-edge.service` is |
| 79 | `Restart=always`, `RestartSec=1`, with no `StartLimit*` of its own, so |
| 80 | a binary that dies on start hits systemd's default limit (5 starts in |
| 81 | 10 s) and the unit goes to `failed` with start-limit-hit; nothing puts |
| 82 | the old binary back, and there is no `.prev` anywhere. |
| 83 | 4. A **verb** an operator runs, and a **status** they read. |
| 84 | |
| 85 | --- |
| 86 | |
| 87 | ## 2. The model: the platform is a subscriber of its own releases |
| 88 | |
| 89 | One sentence: **a platform pointer names a version and its artifacts; a |
| 90 | box whose channel pointer names a version other than the one it is |
| 91 | running fetches, verifies, and asks root to swap and restart; root |
| 92 | journals every step, verifies everything again from a copy it owns, and |
| 93 | puts the old binary back if the new one does not come up and stay up.** |
| 94 | |
| 95 | Four things the model deliberately does *not* do: |
| 96 | |
| 97 | - **A box never decides to update.** It reacts to a pointer the deployment |
| 98 | key signed, placed by an operator's promote. No box reads the public |
| 99 | release feed. |
| 100 | - **A box never publishes the binary.** `aclWrite` already refuses |
| 101 | `bootstrap/**` for boxes (`storedoor_test.go:81`), and stays that way. |
| 102 | Only the console, holding the bucket and the signing key, writes the |
| 103 | artifact and the pointer. |
| 104 | - **Root takes exactly one input from the sandboxed user, and treats it |
| 105 | as a hint.** `/data` is `chown -R carlos:carlos` (`bootstrap.go:390`): |
| 106 | any path under it can be a symlink, any file can be swapped between |
| 107 | check and use. So root reads one file at one fixed path, walking to it |
| 108 | component by component with no symlinks followed, and everything root |
| 109 | then trusts — which channel, which version, which sha, which key — |
| 110 | comes from a pointer root fetched and verified itself and from root's |
| 111 | own `host.env`. The candidate binary is copied into a directory only |
| 112 | root can enter, verified there, run there, and installed from there. |
| 113 | A compromised `carlos` user can make the box install exactly what the |
| 114 | deployment key signed for the box's configured channel, and nothing |
| 115 | else — the trust the stage relay already grants, made true by |
| 116 | construction. |
| 117 | - **Root never leaves the box between two builds.** Every step that |
| 118 | changes the box is journaled before it runs; a killed oneshot, a |
| 119 | timeout, or a reboot mid-apply is recovered — always backward, to the |
| 120 | build that was known good — before anything else happens (§6). |
| 121 | |
| 122 | --- |
| 123 | |
| 124 | ## 3. The pointer: `control/system/<component>/<channel>.json` |
| 125 | |
| 126 | Placed under `control/`, beside `control/serial`, because that is where the |
| 127 | deployment's platform-wide signals live, and explicitly **not** under |
| 128 | `apps/` — the console-as-an-app spec's line is *anything adoptable through |
| 129 | a route is an app*, and this is not. |
| 130 | |
| 131 | ```go |
| 132 | // SystemPointer is the platform's channel pointer. It is what a |
| 133 | // ChannelPointer is for an app, for the thing that does the adopting. |
| 134 | type SystemPointer struct { |
| 135 | Schema int `json:"schema"` // 1 |
| 136 | Component string `json:"component"` // "router" |
| 137 | Channel string `json:"channel"` // "canary" | "stable" |
| 138 | Current SystemRelease `json:"current"` |
| 139 | History []SystemRelease `json:"history"` // newest first; rollback targets |
| 140 | Spread string `json:"spread"` // "5m": boxes jitter their start over this |
| 141 | PromotedAt time.Time `json:"promoted_at"` // of THIS write; the replay guard reads this one |
| 142 | Audit []Promotion `json:"audit"` // reused from release.go |
| 143 | } |
| 144 | |
| 145 | // SystemRelease is everything a box needs to install one version. It |
| 146 | // travels whole into History so a rollback re-installs what was there, |
| 147 | // not a version string with someone else's artifacts. |
| 148 | type SystemRelease struct { |
| 149 | Version string `json:"version"` // what `carlos version` must print |
| 150 | Artifacts map[string]Artifact `json:"artifacts"` // "linux-arm64", "linux-amd64" |
| 151 | Source string `json:"source"` // "release:v0.18.0" | "file" |
| 152 | ReleasedAt time.Time `json:"released_at"` // when first mirrored; audit only |
| 153 | } |
| 154 | |
| 155 | type Artifact struct { |
| 156 | Key string `json:"key"` // bootstrap/router/<version>/carlos-linux-arm64 |
| 157 | SHA256 string `json:"sha256"` // hex, of the object at Key |
| 158 | } |
| 159 | ``` |
| 160 | |
| 161 | - **Signed with the deployment's release key**, using the same |
| 162 | `signedWrapper` (payload, sig, optional grant) and the same root-key |
| 163 | anchor `CARLOS_RELEASE_PUBKEY` a box already holds. **Root key only in |
| 164 | v1**: a scoped grant's rungs name app channels, and a grant that could |
| 165 | roll the platform is a bigger key than any app promoter should hold. |
| 166 | `verifyPointer` grows a sibling `verifySystemPointer`; both share the |
| 167 | wrapper parse, which stays the single parse site. |
| 168 | - **CAS-written** with If-Match like `ChannelKey`, so two operators cannot |
| 169 | race each other into a lost promote. |
| 170 | - **Every write sets `PromotedAt` to now** — promote and rollback alike. |
| 171 | The box's replay guard (§6) reads this field and only this field. A |
| 172 | rollback that carried the old release's timestamp would be refused by |
| 173 | every box as a replay of itself; hence the two differently named |
| 174 | fields. |
| 175 | - **`Version` is a string the box compares for equality** with |
| 176 | `version.Build`. For a public release that is the tag (`v0.18.0`, which |
| 177 | release builds put in `Build`); for a `--file` promote (§5) it is whatever |
| 178 | the operator names, which `carlos system apply` checks by running the |
| 179 | staged binary's `version` before installing it. Dev deployments that roll |
| 180 | git-sha builds keep working; nothing assumes semver. |
| 181 | - **A version is immutable.** The console refuses to mirror different |
| 182 | bytes under a `bootstrap/router/<version>/` that already exists |
| 183 | (§5 step 3). Without that, a reused `--file --version` could swap the |
| 184 | artifact under a pointer stable boxes are still on, and boxes already |
| 185 | reporting that version would never take the replacement. |
| 186 | - **`Artifacts` is per platform** because BYO boxes are not all Graviton. |
| 187 | A box picks `runtime.GOOS-runtime.GOARCH`; a pointer with no entry for |
| 188 | the box's platform is "no update for me", logged once, not an error. |
| 189 | - **`Spread`** is the one fleet-policy knob. A box that sees a new pointer |
| 190 | waits `rand(0, Spread)` before requesting the apply, so a stable promote |
| 191 | to nine boxes is nine edge restarts over five minutes, not nine in the |
| 192 | same two seconds. `--spread 0` exists for a hotfix. Home-before-region |
| 193 | ordering (#300) is *not* in v1: the canary channel, with home boxes on |
| 194 | it, is the ordering an operator needs today. **The console's own box |
| 195 | goes on canary by convention** — it hosts the store door, and a fleet |
| 196 | box mid-apply that finds the door restarting gets a transient store |
| 197 | error (§4 says what it does with one). |
| 198 | - **`History`** holds whole releases, `historyMax = 10`, and is what |
| 199 | `carlos system rollback` swaps with (§5). |
| 200 | |
| 201 | The artifact lives at **`bootstrap/router/<version>/carlos-<os>-<arch>`**. |
| 202 | `bootstrap/` because the door already means "the platform's own binary" |
| 203 | by that name; per-version because canary and stable boxes must be able to |
| 204 | run different binaries at the same time, which a single `bootstrap/carlos` |
| 205 | cannot express. The existing exact key stays as it is until §10 retires the |
| 206 | script. |
| 207 | |
| 208 | **Store-door changes, both reads, in `aclRead`:** |
| 209 | |
| 210 | 1. `control/system/<component>/<channel>.json` — exact three-segment shape |
| 211 | under `control/system/`. Without it, a door-backed box gets 404, which |
| 212 | `fleetapi.Get` maps to `os.ErrNotExist` (`fleetapi.go:150`), and the |
| 213 | fleet #100 exists for silently never updates. |
| 214 | 2. `bootstrap/router/<version>/<asset>` — a *shape*, three segments, no |
| 215 | listing (`aclDir` does not change), and still nothing else under |
| 216 | `bootstrap/`. The existing pin that `bootstrap/carlos.sig` is refused |
| 217 | stays, and gains siblings: `bootstrap/router/../../x` and |
| 218 | `control/system/../serial` are refused. |
| 219 | |
| 220 | Boxes need no signature file because the sha in the signed pointer *is* |
| 221 | the artifact's signature. `aclWrite` does not change: a box still cannot |
| 222 | write under `control/` or `bootstrap/`. |
| 223 | |
| 224 | **A streaming read on the store.** `release.Store.Get` returns `[]byte` |
| 225 | and every implementation buffers (`store.go:43`; `fleetapi.go:133`); |
| 226 | `maxStoreBodyBytes` bounds PUT bodies only (`storedoor.go:180`), the |
| 227 | door's GET is uncapped and in memory. The box must not hold a binary in |
| 228 | memory under `MemoryHigh=70%`, so the interface gains |
| 229 | `Open(ctx, key) (io.ReadCloser, etag string, err error)` with all four |
| 230 | implementations (`S3Store`, `DirStore`, `fleetapi`, `prefixStore`). |
| 231 | The door's own GET stays in memory in v1 (§9, §11). |
| 232 | |
| 233 | --- |
| 234 | |
| 235 | ## 4. The box: notice, fetch, verify, ask |
| 236 | |
| 237 | A new leg in the agent, **`systemLeg`**, with two entry points so the |
| 238 | spread has something to run it: |
| 239 | |
| 240 | - `observe(ctx)` — called from `fastTick` on a serial move and from |
| 241 | `adoptLoop` on its hourly pass. Reads the pointer, decides, and either |
| 242 | does nothing or records a *pending* request with a deadline. |
| 243 | - `tick(now)` — called on **every** 2 s fast tick, reads no store. Fires |
| 244 | a pending request when its deadline passes, and **re-touches the marker |
| 245 | once a minute while a request is outstanding** — a request root has not |
| 246 | answered in `result.json`. `PathChanged` does not replay a change made |
| 247 | before the unit was armed, and drops one that lands while the service |
| 248 | is mid-run (`carlos-restart.path`'s own comment); the stage relay |
| 249 | survives that because the unstaged route is re-observed every 2 s, and |
| 250 | this leg needs the same persistence by hand. |
| 251 | |
| 252 | It is sandboxed (`User=carlos`, `ProtectSystem=strict`) and can write only |
| 253 | `/data` and `/run/carlos`, which is exactly enough. |
| 254 | |
| 255 | 1. **Read** `control/system/router/<channel>.json` through the box's normal |
| 256 | store (S3, dir, or the fleet door — `openDeployStore` already picks). |
| 257 | The channel is **`CARLOS_SYSTEM_CHANNEL` in `host.env`**, default |
| 258 | `stable`; an operator puts a box on canary by editing one line and |
| 259 | restarting the edge, which is how every other per-box setting works. |
| 260 | Missing pointer → nothing to do; this is the state of every deployment |
| 261 | until its first `carlos system update`. |
| 262 | 2. **Verify** the pointer with the release pubkey. A pointer that does not |
| 263 | verify is logged and ignored, like a bad channel pointer. |
| 264 | 3. **Decide.** Read root's `seen.json` (§6; root-owned, world-readable) |
| 265 | for this channel's watermark. Then: |
| 266 | - pointer older than the watermark → ignore (root would refuse it); |
| 267 | - `Current.Version == version.Build` and pointer not newer than the |
| 268 | watermark → nothing to do; |
| 269 | - `Current.Version == version.Build` and pointer newer → a **no-op |
| 270 | request** (step 7 with `noop: true`, no artifact fetched), so root |
| 271 | sees every promote and the watermark moves without a 25 MiB download |
| 272 | on every box; |
| 273 | - otherwise → an update. A dev build (`Build == "dev"`) never matches, |
| 274 | so a hand-built binary on a dev box is never rolled over by accident; |
| 275 | it *is* rolled when the operator promotes something, which is the |
| 276 | point. |
| 277 | 4. **Refuse to repeat a failure.** If `result.json` names this exact |
| 278 | `(channel, version, promoted_at)` with outcome `rolled-back`, |
| 279 | `rollback-failed` or `refused`, do nothing until the pointer changes. |
| 280 | A pointer that broke one box must not have that box firing a root |
| 281 | oneshot at every serial bump. `deferred` (a transient store error on |
| 282 | root's side) is retried with the per-host backoff the unstaged-route |
| 283 | sweep uses. A terminal outcome is also when the leg **deletes its own |
| 284 | cache** under `/data/system/router/<version>/` — root never deletes |
| 285 | anything under `/data`. |
| 286 | 5. **Jitter** `rand(0, Spread)` from the first sighting, as the pending |
| 287 | deadline `tick` watches. |
| 288 | 6. **Fetch** the artifact for this box's platform with `Store.Open`, |
| 289 | reading at most 96 MiB + 1 byte — the extra byte is how an object |
| 290 | exactly at the limit is told from one over it — with a running sha, |
| 291 | into `/data/system/router/<version>/carlos.tmp` then `rename`. The |
| 292 | object is never in memory on the box; an oversized object from a |
| 293 | bucket writer is cut off at the limit, not at the OOM killer. |
| 294 | **Verify** the sha against the pointer; mismatch → delete, log, retry |
| 295 | with backoff. |
| 296 | 7. **Write the request** `/data/system/router/requested.json` |
| 297 | `{version, promoted_at, noop, seen_at}` — **no channel, no key, no |
| 298 | sha, no path**: root takes the channel from its own `host.env`, the |
| 299 | key and sha from the pointer, and derives the cache path from the |
| 300 | version — and **touch `/data/system-update-requested`**, throttled to |
| 301 | once a minute like `stageRequester`. |
| 302 | 8. **A transient store error** anywhere above (the door restarting because |
| 303 | the console box is rolling, a network blip) is a retry, not a refusal: |
| 304 | nothing is recorded, the next `observe` starts over. |
| 305 | |
| 306 | The marker fires **`carlos-system-update.path`** (`PathChanged`, the same |
| 307 | form as `carlos-stage.path`), whose unit is **`carlos-system-update.service`**: |
| 308 | `Type=oneshot`, root, `StartLimitIntervalSec=0`, `TimeoutStartSec=15min` |
| 309 | (a hung apply is a failed apply), `OnFailure=carlos-system-recover.service` |
| 310 | (a killed or timed-out apply is recovered without waiting for anyone to |
| 311 | touch anything), the same hardening set as `carlos-stage.service` — and |
| 312 | no `ProtectSystem`, because it must write `/opt/carlos` and |
| 313 | `/etc/systemd/system`. A third unit, **`carlos-system-recover.service`** |
| 314 | (`Type=oneshot`, `Before=carlos-edge.service`, `WantedBy=multi-user.target`, |
| 315 | `TimeoutStartSec=10min`), runs once per boot and on the update unit's |
| 316 | failure. |
| 317 | |
| 318 | **Both units exec through one line of shell:** when a journal is open and |
| 319 | `carlos.prev` exists, exec `carlos.prev system apply …`; otherwise exec |
| 320 | `carlos system apply …`. After the rename the binary at |
| 321 | `/opt/carlos/carlos` is the *candidate*, and the one thing known about a |
| 322 | candidate that needs recovering is that it did not work; the recovery |
| 323 | must not depend on it. (§11 names the residual: a `.prev` that cannot |
| 324 | run `system apply` either, which is a box that was already broken before |
| 325 | this roll.) |
| 326 | |
| 327 | `enabledUnits` gains all three, and `systemctlPlan` does with them what |
| 328 | it does with everything: enable all, `start` only the `.path` — never a |
| 329 | service, which is why apply can call it from inside itself without |
| 330 | starting a second apply. |
| 331 | |
| 332 | --- |
| 333 | |
| 334 | ## 5. The verbs |
| 335 | |
| 336 | All under `carlos system`, a new top-level dispatch beside `ops`. Every |
| 337 | verb runs in **API mode against the console** (`--console`, |
| 338 | `requireDeployAPISession`), because the console holds the bucket, the |
| 339 | signing key, and the egress; the workstation holds none of them. |
| 340 | |
| 341 | **Authorization, stated once:** the endpoints |
| 342 | (`PUT /api/system/upload`, `POST /api/system/update`, |
| 343 | `GET /api/system/jobs/<id>`, `POST /api/system/rollback`, |
| 344 | `GET /api/system/status`) require a **person session** (`apiPerson`, |
| 345 | `apicli.go:298`) whose member is an **owner of the ops account** |
| 346 | (`OpsAccountID`, `store.go:1158`), checked server-side on every call. |
| 347 | Service credentials are refused, and so is every other account: a |
| 348 | deployment has one ops account, and rolling the root binary of every box |
| 349 | is that account's owners' act and no one else's. `requireDeployAPISession` |
| 350 | selects console mode; it grants nothing (`deploy.go:1151`), which is why |
| 351 | this paragraph exists. |
| 352 | |
| 353 | **The update endpoint is a job**, not a request: it returns a job id, the |
| 354 | console does the fetch-verify-mirror-sign work in a goroutine, and the |
| 355 | CLI polls the job and prints each step as it completes. Fetching two |
| 356 | 25 MiB assets from GitHub inside one HTTP request is how a promote times |
| 357 | out halfway through a mirror. Jobs live in console memory; a console |
| 358 | restart mid-job loses it, the CLI reports that, and re-running is safe |
| 359 | because every step is idempotent (mirror skips what is present and |
| 360 | equal; the pointer write is CAS). `--file` is a separate |
| 361 | `PUT /api/system/upload` (body limit 128 MiB, `--platform` required |
| 362 | because the console cannot run a foreign-arch binary to ask it) that |
| 363 | returns a handle the update job then consumes. |
| 364 | |
| 365 | **`carlos system update [--channel stable] [--version vX.Y.Z] [--file path --platform linux-arm64 --version v] [--spread 5m] [--yes]`** |
| 366 | |
| 367 | The fleet operator's one command. It is a promote for the platform: |
| 368 | |
| 369 | 1. **Resolve the version.** Default: `https://apt.carlosframework.com/latest.txt` |
| 370 | (the `carlos update` discovery, `releaseTag`-shaped). `--version` |
| 371 | pins one. `--file` skips step 2. |
| 372 | 2. **Ingest the release, exactly as `carlos update` does.** The console |
| 373 | fetches `SHA256SUMS.txt` and `.sig` from the tag-pinned |
| 374 | `updateDownloadBase`, verifies the signature with the embedded |
| 375 | `cliUpdatePubKey`, and fetches each asset the deployment needs, checking |
| 376 | its sha against the sums. The asset set is the distinct platforms in |
| 377 | every box heartbeat — `status/boxes/*.json` **and** |
| 378 | `fleets/*/status/boxes/*.json` — via a new `BoxMetrics.Platform` field |
| 379 | written alongside `Release`; a box that has not reported since the |
| 380 | field shipped counts as `linux-arm64`. `selfReplace`'s fetch-and-verify |
| 381 | is factored out into a function both callers share; it does not move. |
| 382 | 3. **Mirror** each asset to `bootstrap/router/<version>/<asset>`. Present |
| 383 | with the same sha → skip (a second promote of the same version to |
| 384 | another channel copies nothing). Present with a different sha → |
| 385 | **refuse the whole promote**: versions are immutable (§3). |
| 386 | 4. **Write the pointer** for the channel, signed, CAS, `PromotedAt` now. |
| 387 | `History` gets the previous `Current` whole; `Audit` gets a |
| 388 | `Promotion{From, To, Actor, At}`. |
| 389 | 5. **Bump the serial.** Boxes see it within two seconds and start their |
| 390 | spread. |
| 391 | 6. **Print what will happen:** which boxes are on this channel (from a new |
| 392 | `BoxMetrics.SystemChannel`, also in the heartbeat — without it the |
| 393 | console cannot know which boxes a promote reaches), running what, and |
| 394 | the spread. Then return. |
| 395 | |
| 396 | Without `--yes` it confirms once, showing the box list. The channel ladder |
| 397 | is advisory in v1: promoting to `stable` a version that was never on |
| 398 | `canary` prints a warning and proceeds, because a hotfix needs to. |
| 399 | |
| 400 | **`carlos system router update …`** is the same verb scoped to one |
| 401 | component. With one component today it is an alias; it exists so the |
| 402 | command Paul named works and so a future split needs no new verb. |
| 403 | `carlos system update` with no component updates every component the |
| 404 | pointer schema knows. |
| 405 | |
| 406 | **`carlos system status [--channel …]`** |
| 407 | |
| 408 | Reads the pointers and every heartbeat under both prefixes and prints one |
| 409 | row per box: label, platform, channel, running `Release`, channel target, |
| 410 | and the last apply outcome (§6, published in the heartbeat as |
| 411 | `BoxMetrics.System`). This is the fleet-side half of the health gate: a |
| 412 | box whose heartbeat still shows the old `Release` ten minutes after a |
| 413 | promote is the row an operator is looking for. Heartbeats are |
| 414 | five-minutely, so this is also the honest answer to "is it done yet": |
| 415 | *not yet* for up to five minutes after the last box restarts. A `noted` |
| 416 | outcome (a no-op request, §6) is shown as such, not as an apply. |
| 417 | |
| 418 | **`carlos system rollback [--channel …]`** |
| 419 | |
| 420 | Pops `History[0]` into `Current` — whole releases, so the previous |
| 421 | artifacts and hashes come back with the version — re-signs with |
| 422 | `PromotedAt` now, bumps the serial. The outgoing release is not pushed |
| 423 | back, so a second rollback goes one further back rather than undoing the |
| 424 | first. Nothing GCs `bootstrap/router/` in v1, so the objects are there. Every box rolls back through the same path |
| 425 | it rolled forward on; the box-side `.prev` (§6) is for the box that |
| 426 | cannot come up, not for this. |
| 427 | |
| 428 | **`carlos system apply [--recover]`** is the root oneshot, §6. It is not |
| 429 | for operators and says so if run interactively. |
| 430 | |
| 431 | What is deliberately **not** a verb: `carlos system console update`. The |
| 432 | console is an app; `carlos deploy --app console` is its verb and already |
| 433 | works. Giving it a second name under `system` is the confusion the |
| 434 | dividing line exists to prevent. |
| 435 | |
| 436 | --- |
| 437 | |
| 438 | ## 6. `carlos system apply`: the journal, the dance, the gate, the way back |
| 439 | |
| 440 | Runs as root from `carlos-system-update.service` (or, with `--recover`, |
| 441 | from `carlos-system-recover.service`). Every step is one the pinfra script |
| 442 | does today, plus the ones it lacks. Root state lives in |
| 443 | **`/opt/carlos/system/router/`** (0755, root-owned, never writable by |
| 444 | `carlos`): |
| 445 | |
| 446 | - `lock` — an `flock` every apply takes first. The path-triggered apply |
| 447 | takes it non-blocking and exits 0 with "another apply is running" if it |
| 448 | cannot; `--recover` waits for it. Two processes never touch |
| 449 | `/opt/carlos/carlos` at once, whatever an operator's hand-run |
| 450 | `systemctl start` does. |
| 451 | - `staging/` — **0700**. The root-owned copy of the candidate and the two |
| 452 | staged unit roots. Removed on every exit, success or not. |
| 453 | - `journal.json` — the open transaction, if any: `{phase, target, |
| 454 | prev_build, units_added, units_removed, started_at}`. Written before |
| 455 | each box-changing step, removed on completion. |
| 456 | - `seen.json` — `{<channel>: promoted_at}`: the newest pointer root has |
| 457 | *verified* per channel. The replay guard, and nothing else. |
| 458 | - `applied.json` — `{version, channel, at}`: the last build that **passed |
| 459 | a gate** on this box, forward or rollback. The `.prev` authority, and |
| 460 | nothing else. One record, not per channel: it is a fact about the box. |
| 461 | - `result.json` — `{channel, version, promoted_at, outcome, reason, at}` |
| 462 | of the last run, keyed by the **request's** tuple (two promotes inside |
| 463 | one spread must not let a refusal of the first suppress the second). |
| 464 | What the agent publishes and §4 step 4 reads. |
| 465 | |
| 466 | Every file is written temp-then-`fsync`-then-`rename`-then-**`fsync` of |
| 467 | the directory**, so the rename itself is durable before the next step |
| 468 | depends on it; a leftover `*.tmp` at start is deleted unread. |
| 469 | `carlos.prev` is made the same way (`carlos.prev.tmp`, fsync, rename, |
| 470 | directory fsync), so a `.prev` is either whole or absent. |
| 471 | |
| 472 | `/opt/carlos/carlos` is installed **`root:root 0755`** from here on, not |
| 473 | `carlos:carlos` as the script did. Every root oneshot on the box execs |
| 474 | that file; the sandboxed user has no reason to own it. |
| 475 | |
| 476 | **Every run begins with recovery.** If `journal.json` exists, a previous |
| 477 | run was cut off (SIGKILL, OOM, the timeout, a reboot). Which way it goes |
| 478 | depends on one thing: whether the gate had passed. |
| 479 | |
| 480 | - **Phase `gated`** (step 10 passed, steps 11–12 not finished): the box |
| 481 | is on the new build and proved it. Recovery goes **forward**: finish |
| 482 | steps 11–12. Undoing a roll that passed its gate because the process |
| 483 | that recorded it was killed would be a rollback with no reason. |
| 484 | - **Any earlier phase**: recovery goes **back** — an apply that did not |
| 485 | reach its gate is not one to trust, and `.prev` passed one. It is |
| 486 | **files only**: stop the quiesce set of step 7 (an operator or the |
| 487 | timer may have started a pass since); restore `carlos.prev` over |
| 488 | `carlos` (keeping `.prev`); `disable` and delete `units_added`; restore |
| 489 | `units_removed` and the rest of the previous unit set from |
| 490 | `staging/prev/` or, if `staging/` is gone, from a fresh `carlos.prev |
| 491 | bootstrap -root`; sysusers, tmpfiles, `daemon-reload`; `enable` (not |
| 492 | start) the previous plan's units so they come up at boot. Then, only if |
| 493 | `carlos-edge` has started at least once this boot |
| 494 | (`ActiveEnterTimestampMonotonic != 0`): `reset-failed`, `restart |
| 495 | --no-block carlos-edge`, and `start` the previous plan's timers and |
| 496 | paths. At boot the edge is ordered after this unit and its own start |
| 497 | *is* the restart, and the enabled timers and paths start with the |
| 498 | target; a recovery that restarted and gated the edge from inside a |
| 499 | unit the edge waits on would wait on itself forever. **Only now** write |
| 500 | `result.json` `{outcome: "rolled-back", reason: "interrupted at |
| 501 | <phase>"}` and remove the journal: a kill before this point leaves the |
| 502 | journal in place and the next run recovers again, which is idempotent; |
| 503 | a kill after it leaves nothing to do. |
| 504 | |
| 505 | `--recover` ends here. A path-triggered run continues below if there is |
| 506 | also a request. |
| 507 | |
| 508 | Then the run proper. Failures in steps 1–6 change nothing on the box and |
| 509 | are `refused` (or `deferred`, for a store error); from step 7 on, any |
| 510 | failure goes to step 13. |
| 511 | |
| 512 | 1. **Read the request.** One fixed path, `/data/system/router/requested.json`, |
| 513 | reached by opening `/data`, then each component with |
| 514 | `O_DIRECTORY|O_NOFOLLOW`, then the file with `O_NOFOLLOW`; must be a |
| 515 | regular file under 4 KiB. It yields `{version, promoted_at, noop}` and |
| 516 | nothing root will act on without checking. `version` must match |
| 517 | `^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`, or the run is `refused` — this is |
| 518 | what makes the cache path derivable and un-traversable. |
| 519 | 2. **The channel is root's.** `CARLOS_SYSTEM_CHANNEL` from the unit's own |
| 520 | `EnvironmentFile`, default `stable`. The request does not carry one and |
| 521 | would not be believed if it did: a signed canary pointer authorizes |
| 522 | canary boxes, not a stable box whose sandboxed user would like to be |
| 523 | one. |
| 524 | 3. **Fetch and verify the pointer from the bucket** with |
| 525 | `CARLOS_RELEASE_PUBKEY`. A store error is `deferred`. The request's |
| 526 | `version` and `promoted_at` must equal the pointer's `Current.Version` |
| 527 | and `PromotedAt`, or `refused` (a request from a spread that started |
| 528 | under an earlier promote). **Replay guard:** the pointer's `PromotedAt` |
| 529 | must not be *older* than `seen.json[channel]`; equal is allowed, so an |
| 530 | apply that was interrupted or refused can be retried for the same |
| 531 | promote. Then advance `seen.json[channel]`. A box with no entry for the |
| 532 | channel accepts the first pointer it sees — a fresh, hand-rolled or |
| 533 | channel-moved box has nothing to protect yet (§11). |
| 534 | 4. **A no-op request ends here:** `result.json` `{outcome: "noted"}`, |
| 535 | exit 0. Root has seen the promote; nothing was downloaded, nothing |
| 536 | changes. |
| 537 | 5. **Copy the candidate into `staging/`.** The path is *derived*: |
| 538 | `/data/system/router/<version>/carlos`, walked like step 1, regular |
| 539 | file only. Copy to `staging/carlos` as **0600**; **verify the copy's |
| 540 | sha** against the pointer's artifact for this platform; then `chmod |
| 541 | 0755` and run it: `staging/carlos version`, 30-second deadline, must |
| 542 | print `carlos <version>`. The lie detector from `update.go:350`, on a |
| 543 | file only root can reach, made executable only after it is known to |
| 544 | be what the key signed. Nothing under `/data` is read again. |
| 545 | 6. **Decide `.prev`, then stage both unit sets.** Run |
| 546 | `/opt/carlos/carlos version` (30 s deadline). If it equals |
| 547 | `applied.json.version`, or `applied.json` is absent and `.prev` is |
| 548 | absent (a hand-rolled box: the running binary passed the script's own |
| 549 | post-condition), the on-disk binary is known good: **journal |
| 550 | `backing-up`** and copy it to `carlos.prev`. Otherwise — after a |
| 551 | `rollback-failed`, when the disk holds a bad binary and `.prev` the |
| 552 | good one — leave `.prev` alone. Then stage units from **both |
| 553 | binaries**: `staging/carlos bootstrap -root staging/new -plan json` |
| 554 | and `carlos.prev bootstrap -root staging/prev -plan json` (120 s |
| 555 | deadline each; `-plan json` is new and prints `enabledUnits` and the |
| 556 | inert list as JSON, so apply arms what *that* binary's bootstrap would |
| 557 | arm). `units_added = new − prev`, `units_removed = prev − new`, both |
| 558 | into the journal. Staging from `.prev`, not from the on-disk file, |
| 559 | because after a `rollback-failed` those are different builds and a |
| 560 | rollback must restore the units of the binary it restores. |
| 561 | 7. **Journal `installing`**, then **quiesce**: `systemctl stop` the root |
| 562 | timers and paths *and their services* — `carlos-adopt.timer`, |
| 563 | `carlos-adopt.path`, `carlos-stage.path`, `carlos-restart.path`, |
| 564 | `carlos-adopt.service`, `carlos-config-sync.service`, |
| 565 | `carlos-stage.service`, `carlos-restart.service`. Stopping a timer does |
| 566 | not stop the pass it already started, and a root pass running the new |
| 567 | binary against the old edge and registry is the window this exists to |
| 568 | close. Those passes are idempotent; killing one costs a retry. Then |
| 569 | **install**: `rename` `staging/carlos` over `/opt/carlos/carlos` |
| 570 | (same filesystem, atomic). **Journal `installed`.** |
| 571 | 8. **Install units and configs**: `disable --now` and delete |
| 572 | `units_removed` (a unit the new version dropped must not stay armed |
| 573 | and exec a command the new binary no longer has); copy |
| 574 | `staging/new/etc/systemd/system/*` into place; then, in this order and |
| 575 | never the other, **sysusers** (`cp` + `systemd-sysusers`), then |
| 576 | **tmpfiles** (`cp` + `systemd-tmpfiles --create`), then |
| 577 | `daemon-reload`. The ordering is failure-mode row 6 of the |
| 578 | console-as-an-app spec, and the reason is in the script's comment: |
| 579 | tmpfiles silently drops a line whose group does not resolve, and then |
| 580 | `carlos-edge` (`ReadWritePaths=/run/carlos`, no `-`) can never start |
| 581 | after the next boot. Never full `carlos bootstrap`: it re-chowns |
| 582 | `/data` to 0750, which breaks static-user tenants on the Tito box (the |
| 583 | script's opening comment). And never copy the staging root's `data/` — |
| 584 | it contains a 0-byte `registry.db` (`bootstrap.go:310`). **Journal |
| 585 | `units`.** |
| 586 | 9. **Restart the edge**: `systemctl reset-failed carlos-edge` first — |
| 587 | always, because a unit that hit its start limit refuses `restart` |
| 588 | with "start request repeated too quickly", and the rollback in step 13 |
| 589 | would be refused the same way — then `systemctl restart carlos-edge`. |
| 590 | The one tenant-visible step; on a healthy box it is a one-second blip. |
| 591 | **Journal `restarted`.** |
| 592 | 10. **The gate**, 20 seconds to pass and then a **15-second soak**: |
| 593 | - `carlos-edge` is `active`; across the soak `NRestarts` stays 0 (a |
| 594 | manual restart resets it, so it counts only crash restarts since |
| 595 | step 9) and `ActiveEnterTimestampMonotonic` does not move. A binary |
| 596 | that starts, answers, and dies a second later is caught here, not |
| 597 | recorded as a success. |
| 598 | - The edge accepts a TCP connection on its listener and answers a |
| 599 | request on loopback with *any* HTTP status. Not a tenant probe: a |
| 600 | box with no route yet must still pass. |
| 601 | - `/opt/carlos/carlos version` prints the target. |
| 602 | **Journal `gated`.** From here recovery finishes forward. |
| 603 | 11. **Arm**, now that the box is on the new build and the gate has said |
| 604 | so: run `systemctlPlan(new plan's enabledUnits)` verbatim — enable |
| 605 | all, `start` timers and paths — which re-arms what step 7 stopped and |
| 606 | arms this spec's own path unit for the next roll. Not before the |
| 607 | gate: a rolled-back box is armed once, on the previous plan. |
| 608 | 12. **Record.** `applied.json` `{version, channel, at}`, `result.json` |
| 609 | `{outcome: "applied"}`, remove the journal, remove `staging/`. Done. |
| 610 | The agent deletes its `/data` cache when it sees the outcome. |
| 611 | 13. **Roll back.** `rename` `carlos.prev` to `carlos` keeping a copy as |
| 612 | `.prev`; `disable --now` and delete `units_added`; restore |
| 613 | `staging/prev/etc/systemd/system/*` (which brings `units_removed` |
| 614 | back); sysusers, tmpfiles, `daemon-reload`; `reset-failed`, restart |
| 615 | the edge, run the gate; then `systemctlPlan(prev plan's |
| 616 | enabledUnits)`. `result.json` `{outcome: "rolled-back", reason}`, |
| 617 | `applied.json` unchanged (the `.prev` build is what passed a gate, |
| 618 | and it is what `applied.json` already names), journal removed, exit |
| 619 | non-zero so systemd's journal shows it. If the rollback fails *its* |
| 620 | gate: `outcome: "rollback-failed"`, journal removed, exit non-zero — |
| 621 | `applied.json` still names the `.prev` build, so the next run's step |
| 622 | 6 leaves `.prev` alone. At every exit the installed binary and the |
| 623 | installed units came from the same build, *including* exits by kill, |
| 624 | timeout or reboot, because the next run recovers first and |
| 625 | `OnFailure=` makes sure there is a next run. |
| 626 | |
| 627 | The agent, whichever binary it now is, reads `result.json` on its next |
| 628 | heartbeat and publishes it as **`BoxMetrics.System{Channel, Version, |
| 629 | Outcome, Reason, At}`**, which is what `carlos system status` shows. A |
| 630 | rolled-back box is therefore visible fleet-wide within five minutes, with |
| 631 | the reason, and (§4 step 4) will not try that pointer again. |
| 632 | |
| 633 | --- |
| 634 | |
| 635 | ## 7. The first roll, and compatibility |
| 636 | |
| 637 | The first binary that contains this spec reaches a fleet by the old road: |
| 638 | `update-platform-binary.sh`, one last time per box. That run must **arm |
| 639 | `carlos-system-update.path` and enable `carlos-system-recover.service`**, |
| 640 | or the first promote does nothing anywhere: a staged bootstrap runs no |
| 641 | `systemctl` (`bootstrap.go:327`), and the path unit cannot arm itself |
| 642 | before it works. Two belts: |
| 643 | |
| 644 | 1. The script (pinfra, and Tito's copy in `tito/carlos-infrastructure`) |
| 645 | gains the two lines, guarded on the files existing, the way |
| 646 | `carlos-restart.path` was added. |
| 647 | 2. The root hourly pass (`RunAdopt`, from `carlos-adopt.timer`) gains a |
| 648 | **new, non-fatal first step**, placed before its registry and pubkey |
| 649 | `log.Fatalf` gates: if the embedded system-update units are missing |
| 650 | from `/etc/systemd/system`, or the path unit is inactive, or the |
| 651 | recover unit is not enabled, install and arm them via `systemctlPlan`. |
| 652 | New code — `RunAdopt`'s existing `capabilityPass` converges tenant |
| 653 | `carlos-app@<host>.service.d` drop-ins through `caps.Converge` and |
| 654 | installs no platform unit. So a box the script forgot is armed within |
| 655 | the hour, and the first promote is not lost: the leg re-touches its |
| 656 | marker every minute until root answers (§4), and `PathChanged` sees |
| 657 | the next touch after arming. |
| 658 | |
| 659 | From then on the script is never needed, because that binary contains the |
| 660 | leg, the units and the apply. There is no flag day and no dual-running: an |
| 661 | old agent does not read `control/system/`, so writing a pointer into a |
| 662 | deployment whose boxes are old does nothing until they are rolled by hand |
| 663 | once. |
| 664 | |
| 665 | Within one deployment, canary and stable boxes run different binaries by |
| 666 | design, for as long as the operator leaves them that way. Nothing in the |
| 667 | bucket schema they share changes with this spec (the pointer is a new key; |
| 668 | `status/boxes` gains three optional fields), so a two-version fleet is |
| 669 | exactly as safe as it is today after a partial script run. |
| 670 | |
| 671 | `carlos update` (the workstation CLI) is untouched except for the factored |
| 672 | fetch-and-verify. Its header sentence — *a self-updating box binary would |
| 673 | fight that machinery* — becomes true in the other direction: the box |
| 674 | machinery is now the sanctioned one, and `runUpdate`'s refusal to run on a |
| 675 | box stays. |
| 676 | |
| 677 | --- |
| 678 | |
| 679 | ## 8. Third-party deployments |
| 680 | |
| 681 | Tito's operator, on a laptop with a Tito console session, as an owner of |
| 682 | Tito's ops account: |
| 683 | |
| 684 | ``` |
| 685 | carlos system update --console https://console.tito.example --channel canary |
| 686 | carlos system status --console https://console.tito.example |
| 687 | carlos system update --console https://console.tito.example --channel stable |
| 688 | ``` |
| 689 | |
| 690 | The Tito console fetches the public release from GitHub, verifies the |
| 691 | vendor's signature, mirrors it into Tito's bucket, signs the pointer with |
| 692 | Tito's key. Tito's boxes, behind Tito's store door, fetch from Tito's |
| 693 | bucket. At no point does a CARLOS developer hold anything of Tito's, and |
| 694 | at no point does a Tito box reach anything but Tito's console. What the |
| 695 | vendor ships is a signed public release and nothing else, which is what |
| 696 | #100 asked for. |
| 697 | |
| 698 | Egress: the **console** needs to reach `apt.carlosframework.com` and |
| 699 | `github.com`. Boxes need nothing new. A deployment whose console cannot |
| 700 | reach GitHub uses `--file`. |
| 701 | |
| 702 | --- |
| 703 | |
| 704 | ## 9. What this does NOT do |
| 705 | |
| 706 | - **Split the process.** `router` is the one binary. Updating the edge |
| 707 | without restarting the activator is not possible until they are separate |
| 708 | units, and that is not this spec. |
| 709 | - **Order boxes within a channel** beyond the spread jitter. Two channels |
| 710 | are the ordering primitive in v1. |
| 711 | - **Auto-promote canary to stable**, or bake-time enforcement. The |
| 712 | ladder is advisory. |
| 713 | - **GC `bootstrap/router/`.** Old versions stay; they are ~25 MiB each and |
| 714 | the rollback verb needs them. A later slice can keep `History`'s depth. |
| 715 | - **A console page.** `carlos system status` is the read surface in v1; |
| 716 | the fleet page's box rows already show `Release`, and a target column |
| 717 | there is a small follow-up. |
| 718 | - **Scoped-grant signing** for platform pointers. |
| 719 | - **Update the console** — it is an app. |
| 720 | - **Stream the store door's GET.** It holds the object in memory today |
| 721 | for `bootstrap/carlos`; it will for `bootstrap/router/…`. Named in §11. |
| 722 | - **Durable update jobs.** A console restart mid-promote loses the job; |
| 723 | re-running is safe. |
| 724 | - **Any change to `carlos bootstrap` for a fresh box** beyond `-plan |
| 725 | json` and the three new units in `enabledUnits`. New boxes still |
| 726 | bootstrap and get their first binary by hand (`infra/remote-host.md`); |
| 727 | from the first heartbeat on they are on `stable` and roll with the |
| 728 | fleet. |
| 729 | |
| 730 | --- |
| 731 | |
| 732 | ## 10. What this retires |
| 733 | |
| 734 | - `update-platform-binary.sh` and the per-deployment roll runbooks, after |
| 735 | each deployment has been hand-rolled once onto a binary that carries |
| 736 | this spec. The script's sysusers/tmpfiles guard for *pre-#104* binaries |
| 737 | is not carried: nothing that old is rolled through this path. |
| 738 | - The exact `bootstrap/carlos` grant in `aclRead`, and the key itself, one |
| 739 | release after the script is gone. Until then it is harmless. |
| 740 | - The vendor needing SSM on a third-party deployment for a routine roll. |
| 741 | - The "blocked on deploy scope" state this spec was born from: a merged |
| 742 | `main` becomes a release, and a release becomes a fleet with one |
| 743 | command per channel. |
| 744 | |
| 745 | --- |
| 746 | |
| 747 | ## 11. Proof, and live checks owed |
| 748 | |
| 749 | **Unit-level, in this repo:** |
| 750 | |
| 751 | - Pointer sign/verify round trip; scoped grant refused; wrong component |
| 752 | or channel refused; rollback pointer carries a new `PromotedAt`. The |
| 753 | door: both new shapes granted, both traversal siblings refused, |
| 754 | `bootstrap/carlos.sig` still refused, listing under either prefix still |
| 755 | refused, PUT under either still refused. `Store.Open` on all four |
| 756 | implementations, with the limit+1 read telling exact-limit from over. |
| 757 | - The leg against a `DirStore`: no pointer → no request; pointer for another |
| 758 | platform → no request; version equal, pointer not newer than `seen` → |
| 759 | nothing; version equal, pointer newer → no-op request and no fetch; |
| 760 | pointer older than `seen` → nothing; sha mismatch → no request and file |
| 761 | deleted; oversized object → cut at the limit, no request; a |
| 762 | `result.json` naming this tuple as any terminal outcome → no request |
| 763 | and cache deleted; `deferred` → retried; a good pointer → |
| 764 | `requested.json` with no channel/key/path and the marker; the marker |
| 765 | re-touched every minute until a result names the tuple; the spread → |
| 766 | `tick` fires after the deadline and not before, with no store read in |
| 767 | between. |
| 768 | - `carlos system apply` against a fake root and a fake `systemctl` that |
| 769 | records every call in order: `lock` held → path-triggered run exits 0 |
| 770 | untouched, `--recover` waits; request read refuses a symlinked parent, |
| 771 | a symlinked file, an oversized file, a version that fails the pattern; |
| 772 | channel taken from the environment, not the request (a request under a |
| 773 | canary pointer on a stable box → `refused`); a stale tuple → `refused` |
| 774 | recorded against the *request's* tuple; replay: older → `refused`, |
| 775 | equal → proceeds, `seen` advanced only after verification; no-op → |
| 776 | `noted`, no `staging/`; the candidate copied 0600, sha-checked, chmod |
| 777 | after; a swap of the cache file after the copy → no effect; `version` |
| 778 | over its deadline → `refused`, `staging/` removed; `.prev` refreshed |
| 779 | when disk `version` equals `applied.json`, on a fresh box, and **not** |
| 780 | after `rollback-failed`; `staging/prev` from `.prev`, not from disk; |
| 781 | `units_added`/`units_removed` computed and journaled; quiesce stops |
| 782 | the services too, before the rename; `units_removed` disabled and |
| 783 | deleted forward; sysusers before tmpfiles by *call order*; |
| 784 | `reset-failed` before every restart; **no `start` of any `.service` in |
| 785 | any run** (the self-start deadlock); arming after the gate, forward and |
| 786 | back, from the right plan; a gate that passes and then fails in the |
| 787 | soak → rolled back; every journal phase before `gated` → recovery rolls |
| 788 | back files only, restores `units_removed`, restarts the edge only when |
| 789 | it has started this boot, and removes the journal last; phase `gated` |
| 790 | → recovery finishes forward and `applied.json` names the candidate; the exec shim picks `.prev` when a journal is open; |
| 791 | `rollback-failed` leaves `applied.json` and `.prev` alone; every write |
| 792 | is temp-fsync-rename and a leftover `.tmp` is discarded. |
| 793 | - `carlos system update` against a stub release host: non-owner or service |
| 794 | credential → 403 before any fetch; bad sums signature refused; asset sha |
| 795 | mismatch refused; mirror skip when present and equal; mirror refused |
| 796 | when present and different; pointer CAS conflict retried; serial bumped |
| 797 | last; upload without `--platform` refused; job polled to completion; |
| 798 | job lost on restart reported, re-run idempotent. |
| 799 | |
| 800 | **Live, on the dev deployment first, then flagship canary:** |
| 801 | |
| 802 | 1. Hand-roll the dev box with the script onto a build carrying this spec, |
| 803 | and confirm afterwards: `carlos-system-update.path` active, |
| 804 | `carlos-system-recover.service` enabled. That is the last script run |
| 805 | for that box. |
| 806 | 2. `carlos system update --file <next build> --version <sha> --channel |
| 807 | canary` with the dev box on canary. Watch `status` show the new |
| 808 | `Release`. Re-promote the same version: `status` shows `noted`, no |
| 809 | download in the edge log. |
| 810 | 3. Promote a deliberately broken binary (one that exits at start) the same |
| 811 | way. Confirm the edge went start-limit-hit and came back on `.prev` |
| 812 | regardless, `status` shows `rolled-back` with the reason, and a second |
| 813 | serial bump does not re-attempt it. |
| 814 | 4. Promote a binary that starts and exits after five seconds. Confirm the |
| 815 | soak catches it. |
| 816 | 5. Kill the oneshot between `installed` and `restarted` (a `sleep` build |
| 817 | with a `kill -9`). Confirm `OnFailure=` recovers to `.prev` without a |
| 818 | touch. Then reboot the box in the same state and confirm the boot unit |
| 819 | does, and that the edge starts after it without a hang. |
| 820 | 6. Move the box from canary to stable in `host.env`. Confirm a stable |
| 821 | pointer older than the canary one is applied, not refused. |
| 822 | 7. `carlos system rollback --channel canary`. Confirm a forward roll to the |
| 823 | previous version through the normal path, artifacts and all. |
| 824 | 8. On flagship: the console box on canary; promote a real tag; then stable |
| 825 | with the default spread; watch the tenant hostnames answer 200 |
| 826 | throughout (the plan doc's rule at `console-as-an-app.md:1910`, now |
| 827 | observed rather than performed). |
| 828 | 9. Tito: Tito's operator runs §8 against a Tito console; the CARLOS side |
| 829 | watches nothing and touches nothing. |
| 830 | |
| 831 | **Known gaps to name so they are not discovered:** |
| 832 | |
| 833 | - The gate's loopback probe proves the process serves HTTP, not that a |
| 834 | tenant's route works. A binary whose routing is broken but whose |
| 835 | listener is fine passes the gate; `status` and the tenant's own |
| 836 | monitoring are the backstop, as they are today. |
| 837 | - The replay guard has no baseline on a box that has never seen a pointer |
| 838 | for its channel. A bucket writer without the key can put an old signed |
| 839 | pointer in front of such a box once. That is the exposure app channel |
| 840 | pointers already have on every box, and the same actor could point |
| 841 | `control/serial` at nothing at all; the guard narrows it, it does not |
| 842 | close it. |
| 843 | - Recovery depends on `carlos.prev` being able to run `system apply |
| 844 | --recover`. `.prev` passed a gate on this box, so the case is a `.prev` |
| 845 | that has since been damaged on disk — a box that was broken before this |
| 846 | roll, and is now a hand job. |
| 847 | - The store door's GET holds the artifact in memory inside the console |
| 848 | box's edge (`MemoryHigh=70%`). Nine fleet boxes fetching 25 MiB inside |
| 849 | the same spread is not a problem; ninety would be, and streaming the |
| 850 | door's GET is the fix when that day comes. |
| 851 | - `Spread` jitters starts, it does not serialize them; two boxes can still |
| 852 | restart in the same second. With nine boxes and a five-minute spread the |
| 853 | odds are small and the cost is two one-second blips. |
| 854 | - A box that is asleep or dark during the promote rolls when it next |
| 855 | ticks. `status` shows it on the old version until then, which is the |
| 856 | truth. |
| 857 | - A permanently failing recovery re-runs every minute (the agent re-touches |
| 858 | the marker while its request is unanswered) and never writes |
| 859 | `result.json`, so the fleet sees a stale result; that is the needs-a-human |
| 860 | state, and it is fleet-invisible. |
| 861 | |
| 862 | --- |
| 863 | |
| 864 | ## 12. Approaches not taken |
| 865 | |
| 866 | **Boxes fetch from the public release host directly.** No mirroring, no |
| 867 | console egress. But every box then needs egress to GitHub, which a fleet |
| 868 | box behind the store door may not have — it excludes exactly the |
| 869 | deployment shape #100 exists for, and leaves the door's `bootstrap/` grant |
| 870 | unused. |
| 871 | |
| 872 | **The platform as a reserved app (`ops/platform`).** `carlos promote |
| 873 | --app platform` would just work and the console would show it. It is the |
| 874 | shape the console-as-an-app spec rejected: the substrate has no route, no |
| 875 | upstream, no instance, and every app assumption — health probe, |
| 876 | hibernation, the adoption pass moving a `live/<host>` symlink — has to be |
| 877 | special-cased for the one "app" that is not one. Cheap on day one and |
| 878 | expensive on every day after. |
| 879 | |
| 880 | --- |
| 881 | |
| 882 | ## 13. Review findings, and where each landed |
| 883 | |
| 884 | Two rounds, two independent readers each, checking claims against the |
| 885 | code. The table is so a reader can see what the drafts got wrong and how, |
| 886 | rather than only that the text is now different. |
| 887 | |
| 888 | **Round one** (both readers unless noted): |
| 889 | |
| 890 | | Finding | Landed in | |
| 891 | |---|---| |
| 892 | | Root hashed and exec'd a file the sandboxed user owns — TOCTOU to root | §2, §6 steps 1, 5 | |
| 893 | | No authorization rule; `requireDeployAPISession` grants nothing | §5 | |
| 894 | | Door-backed boxes cannot read `control/system/` — silent never-update on the Tito shape | §3 | |
| 895 | | Interrupted apply strands the box; on-disk `Build` equality then says "done" forever | §6 journal, recovery-first, boot unit, `OnFailure=` | |
| 896 | | Rollback restored the binary but not the unit set | §6 steps 6, 8, 13 | |
| 897 | | Edge start limit (default 5/10 s) refuses the rollback's restart | §1, §6 step 9 | |
| 898 | | Gate had a deadline but no observation window | §6 step 10 | |
| 899 | | `History` of version strings cannot reconstruct artifacts for rollback | §3 `SystemRelease` | |
| 900 | | Root units exec the new binary against the old edge during the swap | §6 steps 7, 11 | |
| 901 | | Generic "arm every timer" contradicts `enabledUnits` | §6 `-plan json`, `systemctlPlan` | |
| 902 | | First-roll arming was circular | §7 | |
| 903 | | `.prev` overwritten by a later run after `rollback-failed` | §6 step 6 | |
| 904 | | Replay watermark missed no-op promotes; blocked canary→stable moves | §4 step 3, §6 `seen.json` | |
| 905 | | Spread had nothing to run it between hourly passes | §4 `observe`/`tick` | |
| 906 | | No exec deadlines; oneshot could hang before the gate | §4 units, §6 steps 5–6 | |
| 907 | | 32 MiB cap is on PUT, not GET | §3 `Store.Open`, §4 step 6 | |
| 908 | | Versions not immutable (Codex) | §3, §5 step 3 | |
| 909 | | Console cannot know channel membership or fleet-prefixed heartbeats (Codex) | §5 | |
| 910 | | Suppression covered only `rolled-back` (Fable) | §4 step 4 | |
| 911 | | Store error during the console box's own roll read as a refusal (Fable) | §4 step 8, §6 `deferred` | |
| 912 | | Binary owned `carlos:carlos` for no reason (Fable) | §6 | |
| 913 | |
| 914 | **Round two**, on the fixes above: |
| 915 | |
| 916 | | Finding | Landed in | |
| 917 | |---|---| |
| 918 | | `enable --now` on the plan starts the update oneshot from inside itself, and the recover unit mid-apply (both) | §4, §6 step 11: `systemctlPlan` verbatim, never starts a service; `lock` | |
| 919 | | Boot recovery ordered before the edge restarts and gates the edge, waiting on itself (both) | §6 recovery: files only, restart only if the edge has started this boot | |
| 920 | | One watermark doing replay guard *and* `.prev` authority breaks both (both) | §6 `seen.json` vs `applied.json` | |
| 921 | | Recovery execs the candidate after the rename (both) | §4 exec shim prefers `.prev` under an open journal | |
| 922 | | Stopping timers does not stop running passes; re-arm before restart reopened the window (both) | §6 steps 7, 11 | |
| 923 | | `enable --now` would start a start-limited edge before `reset-failed` (Codex) | §6 step 11: no service is ever started by the plan | |
| 924 | | `staging/prev` built from the on-disk binary, which after `rollback-failed` is not `.prev` (Codex) | §6 step 6 | |
| 925 | | Crash durability of journal/`.prev` unspecified (Codex) | §6 temp-fsync-rename | |
| 926 | | Request could name another channel's pointer (Codex) | §6 step 2 | |
| 927 | | Request `path` still a root input; `O_NOFOLLOW` covers the last component only; copy before check exposed secrets in a world-readable dir (Fable) | §6 steps 1, 5: derived path, per-component walk, 0700 staging, 0600 until verified | |
| 928 | | `LimitReader` cannot bound a `[]byte` `Get` (both) | §3 `Store.Open` | |
| 929 | | Recover unit had no timeout (both) | §4 | |
| 930 | | Inactive oneshots read as broken by the hourly belt and §11 (Codex) | §7, §11: path active, recover enabled | |
| 931 | | Nothing re-touched the marker; `carlos adopt` does not converge platform units (both) | §4 `tick`, §7 belt 2 | |
| 932 | | Rollback pointer's `PromotedAt` refused as a replay (both) | §3, §5 | |
| 933 | | Forward install left removed units armed (Codex) | §6 steps 6, 8 | |
| 934 | | No-op promote downloaded 25 MiB on every box (Fable) | §4 step 3, §6 step 4 | |
| 935 | | `applied.json` shape undecided; canary→stable move skipped the `.prev` refresh (Fable) | §6: one record, compared to disk `version` | |
| 936 | | `refused` recorded against the wrong tuple (Fable) | §6 `result.json` | |
| 937 | | Update job lifecycle undecided (Fable) | §5, §9 | |
| 938 | |
| 939 | **Round three** (Codex closure check; 16 of 20 closed, four residuals): |
| 940 | |
| 941 | | Finding | Landed in | |
| 942 | |---|---| |
| 943 | | Interruption between `applied.json` and journal removal undid a roll that passed its gate | §6 step 10 `gated`; recovery goes forward from it | |
| 944 | | Recovery removed its journal before issuing the restart; a kill in the gap left the candidate running with no journal | §6 recovery: result and journal removal last | |
| 945 | | Recovery did not quiesce passes or restore the previous plan's armed state | §6 recovery: stops the quiesce set, `enable` at boot, `start` at runtime | |
| 946 | | Directory fsync unspecified after renames | §6 durability | |
| 947 | |
amadan