amadan
spec: carlos system update — the platform rolls itself
Design for fleet self-rolling of the platform binary from a signed channel pointer (control/system/router/<channel>.json), mirrored into the deployment bucket by the console from the public signed release, applied per box by a root oneshot with a journal, health gate and rollback to carlos.prev. Two channels, two promotes; third-party deployments in v1. Retires update-platform-binary.sh. Reviewed in three rounds by two independent readers against the code; §13 records every finding and where it landed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015QCUCpfxbCyEAVQ2m51dH5
1 file changed,
+941
−0
docs/superpowers/specs/2026-09-06-carlos-system-update-design.md+941 −0
diff --git a/docs/superpowers/specs/2026-09-06-carlos-system-update-design.md b/docs/superpowers/specs/2026-09-06-carlos-system-update-design.md| new file mode 100644 |
| index 000000000..447bd811b |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-06-carlos-system-update-design.md |
| @@ -0,0 +1,941 @@ |
| +# `carlos system update` — the platform rolls itself |
| + |
| +**Date:** 2026-09-06 · **Status:** draft, two review rounds (Fable, Codex), |
| +awaiting Paul · **Requested by:** Paul |
| + |
| +**Builds on:** discussion #100 (boxes should self-update via a channel |
| +subscription), 2026-08-23-console-as-an-app-design.md (slice 3, deferred |
| +there; the dividing line this spec applies), the root-relay pattern |
| +(`carlos-stage.path` → `carlos adopt --stage-only`), and pinfra's |
| +`update-platform-binary.sh`, whose 85 lines are the procedure this spec |
| +moves into the binary and then retires. |
| + |
| +## 0. How this spec was arrived at |
| + |
| +One brainstorm with Paul on 2026-09-05/06, after a merge of `main` stalled |
| +on "and deploy": the only way to roll a platform binary today is an |
| +operator with SSM access to every box, running a script per box, in an |
| +order held in operator memory. Paul's framing, which is the requirement: |
| + |
| +> Ultimately, a fleet operator should be able to do `carlos system update` |
| +> and the whole platform updates to the latest version based on what we |
| +> just released. […] And I guess each piece should be updatable |
| +> individually, `carlos system router update` etc. |
| + |
| +Three questions settled the shape before any design was written: |
| + |
| +- **Trigger: promote-driven.** A fleet operator says when. Boxes do not |
| + poll the public release feed and roll on their own; a new release exists |
| + in the world for as long as the operator wants before any box sees it. |
| +- **Rollout: two channels, two promotes.** `canary` and `stable`, one |
| + operator action each, exactly like an app. No auto-promotion in v1. |
| +- **Third-party deployments in v1.** Tito's operator rolls Tito's boxes |
| + from Tito's console. The artifact comes from the public signed release, |
| + so the vendor never needs access to a customer box (the trust shape #100 |
| + was filed about). |
| + |
| +Three approaches were compared; this document is the first. The other two |
| +are recorded in §12. The draft went through two rounds of review by two |
| +independent readers checking claims against the code (§13); everything |
| +below is the post-review shape, and §13 says what the drafts got wrong. |
| + |
| +One ruling stated up front because it changes what "each piece" means: |
| +the edge, the activator and the adoption loop are **one process** |
| +(`carlos agent`, unit `carlos-edge.service`). There is nothing to update |
| +"individually" inside it without splitting the process, which is a |
| +separate project. So the honest component list today is one: the |
| +**router**, which is the substrate binary. The console is already an app |
| +and is updated as one (`carlos deploy --app console`). §3 keys everything |
| +per component so a split later adds a name, not a schema. |
| + |
| +--- |
| + |
| +## 1. What exists, and what is missing |
| + |
| +The platform already has most of this machinery, built for apps: |
| + |
| +| Have | Where | Reused as | |
| +|---|---|---| |
| +| Signed channel pointer, CAS-written, single parse site | `release.ChannelPointer`, `signPointer`/`verifyPointer` (`release.go:102,350,397`) | The platform pointer's wrapper, signing and verification | |
| +| The one unsigned "something changed" signal, polled at 2s | `release.BumpSerial` on `control/serial` (`serial.go:51`), `fastConvergeLoop` (`agent.go:1412`) | How a box learns there is a new platform pointer | |
| +| Root relay from a sandboxed agent | `/data/stage-requested` → `carlos-stage.path` → `carlos-stage.service` | Same shape, new marker, new oneshot | |
| +| Fleet-box bucket proxy with an explicit grant for the platform binary | `storedoor.go:383` `aclRead`: exact `bootstrap/carlos`, "the sanctioned self-update path (F7)" | Boxes behind the door fetch the artifact with no public egress | |
| +| Verified download of a public release | `update.go` `selfReplace`: tag-pinned URL, `SHA256SUMS.txt.sig` over the sums file, per-asset sha, lie-detector `version` run | The console's ingest of a release | |
| +| A box's running build, published to the bucket | `version.Build` → `BoxMetrics.Release` in `status/boxes/<label>.json` every 5 min (`agent.go:2318`) | `carlos system status` and the fleet-side half of the gate | |
| +| The unit list a box must have armed, and the exact `systemctl` sequence | `enabledUnits`, `systemctlPlan` (`bootstrap.go:72,96`): enable all, start timers and paths, **never start a service** | What apply arms, so apply and bootstrap never disagree and apply never starts itself | |
| +| The apply procedure | `update-platform-binary.sh` (pinfra) | Moved into `carlos system apply`, line for line, then retired | |
| + |
| +What is missing, and is the whole of this spec: |
| + |
| +1. A **pointer for something that is not an app** — no account, no route, |
| + no upstream, and a version that is a platform tag, not a release |
| + serial. |
| +2. A **box-side leg** that notices the pointer, fetches and verifies the |
| + artifact, and asks root to apply it. |
| +3. A **root apply** that does the script's dance from inside the binary, |
| + with the three things the script does not have: a **journal** so an |
| + interrupted apply is undone rather than left half-done, a **health |
| + gate**, and a **rollback**. Today `carlos-edge.service` is |
| + `Restart=always`, `RestartSec=1`, with no `StartLimit*` of its own, so |
| + a binary that dies on start hits systemd's default limit (5 starts in |
| + 10 s) and the unit goes to `failed` with start-limit-hit; nothing puts |
| + the old binary back, and there is no `.prev` anywhere. |
| +4. A **verb** an operator runs, and a **status** they read. |
| + |
| +--- |
| + |
| +## 2. The model: the platform is a subscriber of its own releases |
| + |
| +One sentence: **a platform pointer names a version and its artifacts; a |
| +box whose channel pointer names a version other than the one it is |
| +running fetches, verifies, and asks root to swap and restart; root |
| +journals every step, verifies everything again from a copy it owns, and |
| +puts the old binary back if the new one does not come up and stay up.** |
| + |
| +Four things the model deliberately does *not* do: |
| + |
| +- **A box never decides to update.** It reacts to a pointer the deployment |
| + key signed, placed by an operator's promote. No box reads the public |
| + release feed. |
| +- **A box never publishes the binary.** `aclWrite` already refuses |
| + `bootstrap/**` for boxes (`storedoor_test.go:81`), and stays that way. |
| + Only the console, holding the bucket and the signing key, writes the |
| + artifact and the pointer. |
| +- **Root takes exactly one input from the sandboxed user, and treats it |
| + as a hint.** `/data` is `chown -R carlos:carlos` (`bootstrap.go:390`): |
| + any path under it can be a symlink, any file can be swapped between |
| + check and use. So root reads one file at one fixed path, walking to it |
| + component by component with no symlinks followed, and everything root |
| + then trusts — which channel, which version, which sha, which key — |
| + comes from a pointer root fetched and verified itself and from root's |
| + own `host.env`. The candidate binary is copied into a directory only |
| + root can enter, verified there, run there, and installed from there. |
| + A compromised `carlos` user can make the box install exactly what the |
| + deployment key signed for the box's configured channel, and nothing |
| + else — the trust the stage relay already grants, made true by |
| + construction. |
| +- **Root never leaves the box between two builds.** Every step that |
| + changes the box is journaled before it runs; a killed oneshot, a |
| + timeout, or a reboot mid-apply is recovered — always backward, to the |
| + build that was known good — before anything else happens (§6). |
| + |
| +--- |
| + |
| +## 3. The pointer: `control/system/<component>/<channel>.json` |
| + |
| +Placed under `control/`, beside `control/serial`, because that is where the |
| +deployment's platform-wide signals live, and explicitly **not** under |
| +`apps/` — the console-as-an-app spec's line is *anything adoptable through |
| +a route is an app*, and this is not. |
| + |
| +```go |
| +// SystemPointer is the platform's channel pointer. It is what a |
| +// ChannelPointer is for an app, for the thing that does the adopting. |
| +type SystemPointer struct { |
| + Schema int `json:"schema"` // 1 |
| + Component string `json:"component"` // "router" |
| + Channel string `json:"channel"` // "canary" | "stable" |
| + Current SystemRelease `json:"current"` |
| + History []SystemRelease `json:"history"` // newest first; rollback targets |
| + Spread string `json:"spread"` // "5m": boxes jitter their start over this |
| + PromotedAt time.Time `json:"promoted_at"` // of THIS write; the replay guard reads this one |
| + Audit []Promotion `json:"audit"` // reused from release.go |
| +} |
| + |
| +// SystemRelease is everything a box needs to install one version. It |
| +// travels whole into History so a rollback re-installs what was there, |
| +// not a version string with someone else's artifacts. |
| +type SystemRelease struct { |
| + Version string `json:"version"` // what `carlos version` must print |
| + Artifacts map[string]Artifact `json:"artifacts"` // "linux-arm64", "linux-amd64" |
| + Source string `json:"source"` // "release:v0.18.0" | "file" |
| + ReleasedAt time.Time `json:"released_at"` // when first mirrored; audit only |
| +} |
| + |
| +type Artifact struct { |
| + Key string `json:"key"` // bootstrap/router/<version>/carlos-linux-arm64 |
| + SHA256 string `json:"sha256"` // hex, of the object at Key |
| +} |
| +``` |
| + |
| +- **Signed with the deployment's release key**, using the same |
| + `signedWrapper` (payload, sig, optional grant) and the same root-key |
| + anchor `CARLOS_RELEASE_PUBKEY` a box already holds. **Root key only in |
| + v1**: a scoped grant's rungs name app channels, and a grant that could |
| + roll the platform is a bigger key than any app promoter should hold. |
| + `verifyPointer` grows a sibling `verifySystemPointer`; both share the |
| + wrapper parse, which stays the single parse site. |
| +- **CAS-written** with If-Match like `ChannelKey`, so two operators cannot |
| + race each other into a lost promote. |
| +- **Every write sets `PromotedAt` to now** — promote and rollback alike. |
| + The box's replay guard (§6) reads this field and only this field. A |
| + rollback that carried the old release's timestamp would be refused by |
| + every box as a replay of itself; hence the two differently named |
| + fields. |
| +- **`Version` is a string the box compares for equality** with |
| + `version.Build`. For a public release that is the tag (`v0.18.0`, which |
| + release builds put in `Build`); for a `--file` promote (§5) it is whatever |
| + the operator names, which `carlos system apply` checks by running the |
| + staged binary's `version` before installing it. Dev deployments that roll |
| + git-sha builds keep working; nothing assumes semver. |
| +- **A version is immutable.** The console refuses to mirror different |
| + bytes under a `bootstrap/router/<version>/` that already exists |
| + (§5 step 3). Without that, a reused `--file --version` could swap the |
| + artifact under a pointer stable boxes are still on, and boxes already |
| + reporting that version would never take the replacement. |
| +- **`Artifacts` is per platform** because BYO boxes are not all Graviton. |
| + A box picks `runtime.GOOS-runtime.GOARCH`; a pointer with no entry for |
| + the box's platform is "no update for me", logged once, not an error. |
| +- **`Spread`** is the one fleet-policy knob. A box that sees a new pointer |
| + waits `rand(0, Spread)` before requesting the apply, so a stable promote |
| + to nine boxes is nine edge restarts over five minutes, not nine in the |
| + same two seconds. `--spread 0` exists for a hotfix. Home-before-region |
| + ordering (#300) is *not* in v1: the canary channel, with home boxes on |
| + it, is the ordering an operator needs today. **The console's own box |
| + goes on canary by convention** — it hosts the store door, and a fleet |
| + box mid-apply that finds the door restarting gets a transient store |
| + error (§4 says what it does with one). |
| +- **`History`** holds whole releases, `historyMax = 10`, and is what |
| + `carlos system rollback` swaps with (§5). |
| + |
| +The artifact lives at **`bootstrap/router/<version>/carlos-<os>-<arch>`**. |
| +`bootstrap/` because the door already means "the platform's own binary" |
| +by that name; per-version because canary and stable boxes must be able to |
| +run different binaries at the same time, which a single `bootstrap/carlos` |
| +cannot express. The existing exact key stays as it is until §10 retires the |
| +script. |
| + |
| +**Store-door changes, both reads, in `aclRead`:** |
| + |
| +1. `control/system/<component>/<channel>.json` — exact three-segment shape |
| + under `control/system/`. Without it, a door-backed box gets 404, which |
| + `fleetapi.Get` maps to `os.ErrNotExist` (`fleetapi.go:150`), and the |
| + fleet #100 exists for silently never updates. |
| +2. `bootstrap/router/<version>/<asset>` — a *shape*, three segments, no |
| + listing (`aclDir` does not change), and still nothing else under |
| + `bootstrap/`. The existing pin that `bootstrap/carlos.sig` is refused |
| + stays, and gains siblings: `bootstrap/router/../../x` and |
| + `control/system/../serial` are refused. |
| + |
| +Boxes need no signature file because the sha in the signed pointer *is* |
| +the artifact's signature. `aclWrite` does not change: a box still cannot |
| +write under `control/` or `bootstrap/`. |
| + |
| +**A streaming read on the store.** `release.Store.Get` returns `[]byte` |
| +and every implementation buffers (`store.go:43`; `fleetapi.go:133`); |
| +`maxStoreBodyBytes` bounds PUT bodies only (`storedoor.go:180`), the |
| +door's GET is uncapped and in memory. The box must not hold a binary in |
| +memory under `MemoryHigh=70%`, so the interface gains |
| +`Open(ctx, key) (io.ReadCloser, etag string, err error)` with all four |
| +implementations (`S3Store`, `DirStore`, `fleetapi`, `prefixStore`). |
| +The door's own GET stays in memory in v1 (§9, §11). |
| + |
| +--- |
| + |
| +## 4. The box: notice, fetch, verify, ask |
| + |
| +A new leg in the agent, **`systemLeg`**, with two entry points so the |
| +spread has something to run it: |
| + |
| +- `observe(ctx)` — called from `fastTick` on a serial move and from |
| + `adoptLoop` on its hourly pass. Reads the pointer, decides, and either |
| + does nothing or records a *pending* request with a deadline. |
| +- `tick(now)` — called on **every** 2 s fast tick, reads no store. Fires |
| + a pending request when its deadline passes, and **re-touches the marker |
| + once a minute while a request is outstanding** — a request root has not |
| + answered in `result.json`. `PathChanged` does not replay a change made |
| + before the unit was armed, and drops one that lands while the service |
| + is mid-run (`carlos-restart.path`'s own comment); the stage relay |
| + survives that because the unstaged route is re-observed every 2 s, and |
| + this leg needs the same persistence by hand. |
| + |
| +It is sandboxed (`User=carlos`, `ProtectSystem=strict`) and can write only |
| +`/data` and `/run/carlos`, which is exactly enough. |
| + |
| +1. **Read** `control/system/router/<channel>.json` through the box's normal |
| + store (S3, dir, or the fleet door — `openDeployStore` already picks). |
| + The channel is **`CARLOS_SYSTEM_CHANNEL` in `host.env`**, default |
| + `stable`; an operator puts a box on canary by editing one line and |
| + restarting the edge, which is how every other per-box setting works. |
| + Missing pointer → nothing to do; this is the state of every deployment |
| + until its first `carlos system update`. |
| +2. **Verify** the pointer with the release pubkey. A pointer that does not |
| + verify is logged and ignored, like a bad channel pointer. |
| +3. **Decide.** Read root's `seen.json` (§6; root-owned, world-readable) |
| + for this channel's watermark. Then: |
| + - pointer older than the watermark → ignore (root would refuse it); |
| + - `Current.Version == version.Build` and pointer not newer than the |
| + watermark → nothing to do; |
| + - `Current.Version == version.Build` and pointer newer → a **no-op |
| + request** (step 7 with `noop: true`, no artifact fetched), so root |
| + sees every promote and the watermark moves without a 25 MiB download |
| + on every box; |
| + - otherwise → an update. A dev build (`Build == "dev"`) never matches, |
| + so a hand-built binary on a dev box is never rolled over by accident; |
| + it *is* rolled when the operator promotes something, which is the |
| + point. |
| +4. **Refuse to repeat a failure.** If `result.json` names this exact |
| + `(channel, version, promoted_at)` with outcome `rolled-back`, |
| + `rollback-failed` or `refused`, do nothing until the pointer changes. |
| + A pointer that broke one box must not have that box firing a root |
| + oneshot at every serial bump. `deferred` (a transient store error on |
| + root's side) is retried with the per-host backoff the unstaged-route |
| + sweep uses. A terminal outcome is also when the leg **deletes its own |
| + cache** under `/data/system/router/<version>/` — root never deletes |
| + anything under `/data`. |
| +5. **Jitter** `rand(0, Spread)` from the first sighting, as the pending |
| + deadline `tick` watches. |
| +6. **Fetch** the artifact for this box's platform with `Store.Open`, |
| + reading at most 96 MiB + 1 byte — the extra byte is how an object |
| + exactly at the limit is told from one over it — with a running sha, |
| + into `/data/system/router/<version>/carlos.tmp` then `rename`. The |
| + object is never in memory on the box; an oversized object from a |
| + bucket writer is cut off at the limit, not at the OOM killer. |
| + **Verify** the sha against the pointer; mismatch → delete, log, retry |
| + with backoff. |
| +7. **Write the request** `/data/system/router/requested.json` |
| + `{version, promoted_at, noop, seen_at}` — **no channel, no key, no |
| + sha, no path**: root takes the channel from its own `host.env`, the |
| + key and sha from the pointer, and derives the cache path from the |
| + version — and **touch `/data/system-update-requested`**, throttled to |
| + once a minute like `stageRequester`. |
| +8. **A transient store error** anywhere above (the door restarting because |
| + the console box is rolling, a network blip) is a retry, not a refusal: |
| + nothing is recorded, the next `observe` starts over. |
| + |
| +The marker fires **`carlos-system-update.path`** (`PathChanged`, the same |
| +form as `carlos-stage.path`), whose unit is **`carlos-system-update.service`**: |
| +`Type=oneshot`, root, `StartLimitIntervalSec=0`, `TimeoutStartSec=15min` |
| +(a hung apply is a failed apply), `OnFailure=carlos-system-recover.service` |
| +(a killed or timed-out apply is recovered without waiting for anyone to |
| +touch anything), the same hardening set as `carlos-stage.service` — and |
| +no `ProtectSystem`, because it must write `/opt/carlos` and |
| +`/etc/systemd/system`. A third unit, **`carlos-system-recover.service`** |
| +(`Type=oneshot`, `Before=carlos-edge.service`, `WantedBy=multi-user.target`, |
| +`TimeoutStartSec=10min`), runs once per boot and on the update unit's |
| +failure. |
| + |
| +**Both units exec through one line of shell:** when a journal is open and |
| +`carlos.prev` exists, exec `carlos.prev system apply …`; otherwise exec |
| +`carlos system apply …`. After the rename the binary at |
| +`/opt/carlos/carlos` is the *candidate*, and the one thing known about a |
| +candidate that needs recovering is that it did not work; the recovery |
| +must not depend on it. (§11 names the residual: a `.prev` that cannot |
| +run `system apply` either, which is a box that was already broken before |
| +this roll.) |
| + |
| +`enabledUnits` gains all three, and `systemctlPlan` does with them what |
| +it does with everything: enable all, `start` only the `.path` — never a |
| +service, which is why apply can call it from inside itself without |
| +starting a second apply. |
| + |
| +--- |
| + |
| +## 5. The verbs |
| + |
| +All under `carlos system`, a new top-level dispatch beside `ops`. Every |
| +verb runs in **API mode against the console** (`--console`, |
| +`requireDeployAPISession`), because the console holds the bucket, the |
| +signing key, and the egress; the workstation holds none of them. |
| + |
| +**Authorization, stated once:** the endpoints |
| +(`PUT /api/system/upload`, `POST /api/system/update`, |
| +`GET /api/system/jobs/<id>`, `POST /api/system/rollback`, |
| +`GET /api/system/status`) require a **person session** (`apiPerson`, |
| +`apicli.go:298`) whose member is an **owner of the ops account** |
| +(`OpsAccountID`, `store.go:1158`), checked server-side on every call. |
| +Service credentials are refused, and so is every other account: a |
| +deployment has one ops account, and rolling the root binary of every box |
| +is that account's owners' act and no one else's. `requireDeployAPISession` |
| +selects console mode; it grants nothing (`deploy.go:1151`), which is why |
| +this paragraph exists. |
| + |
| +**The update endpoint is a job**, not a request: it returns a job id, the |
| +console does the fetch-verify-mirror-sign work in a goroutine, and the |
| +CLI polls the job and prints each step as it completes. Fetching two |
| +25 MiB assets from GitHub inside one HTTP request is how a promote times |
| +out halfway through a mirror. Jobs live in console memory; a console |
| +restart mid-job loses it, the CLI reports that, and re-running is safe |
| +because every step is idempotent (mirror skips what is present and |
| +equal; the pointer write is CAS). `--file` is a separate |
| +`PUT /api/system/upload` (body limit 128 MiB, `--platform` required |
| +because the console cannot run a foreign-arch binary to ask it) that |
| +returns a handle the update job then consumes. |
| + |
| +**`carlos system update [--channel stable] [--version vX.Y.Z] [--file path --platform linux-arm64 --version v] [--spread 5m] [--yes]`** |
| + |
| +The fleet operator's one command. It is a promote for the platform: |
| + |
| +1. **Resolve the version.** Default: `https://apt.carlosframework.com/latest.txt` |
| + (the `carlos update` discovery, `releaseTag`-shaped). `--version` |
| + pins one. `--file` skips step 2. |
| +2. **Ingest the release, exactly as `carlos update` does.** The console |
| + fetches `SHA256SUMS.txt` and `.sig` from the tag-pinned |
| + `updateDownloadBase`, verifies the signature with the embedded |
| + `cliUpdatePubKey`, and fetches each asset the deployment needs, checking |
| + its sha against the sums. The asset set is the distinct platforms in |
| + every box heartbeat — `status/boxes/*.json` **and** |
| + `fleets/*/status/boxes/*.json` — via a new `BoxMetrics.Platform` field |
| + written alongside `Release`; a box that has not reported since the |
| + field shipped counts as `linux-arm64`. `selfReplace`'s fetch-and-verify |
| + is factored out into a function both callers share; it does not move. |
| +3. **Mirror** each asset to `bootstrap/router/<version>/<asset>`. Present |
| + with the same sha → skip (a second promote of the same version to |
| + another channel copies nothing). Present with a different sha → |
| + **refuse the whole promote**: versions are immutable (§3). |
| +4. **Write the pointer** for the channel, signed, CAS, `PromotedAt` now. |
| + `History` gets the previous `Current` whole; `Audit` gets a |
| + `Promotion{From, To, Actor, At}`. |
| +5. **Bump the serial.** Boxes see it within two seconds and start their |
| + spread. |
| +6. **Print what will happen:** which boxes are on this channel (from a new |
| + `BoxMetrics.SystemChannel`, also in the heartbeat — without it the |
| + console cannot know which boxes a promote reaches), running what, and |
| + the spread. Then return. |
| + |
| +Without `--yes` it confirms once, showing the box list. The channel ladder |
| +is advisory in v1: promoting to `stable` a version that was never on |
| +`canary` prints a warning and proceeds, because a hotfix needs to. |
| + |
| +**`carlos system router update …`** is the same verb scoped to one |
| +component. With one component today it is an alias; it exists so the |
| +command Paul named works and so a future split needs no new verb. |
| +`carlos system update` with no component updates every component the |
| +pointer schema knows. |
| + |
| +**`carlos system status [--channel …]`** |
| + |
| +Reads the pointers and every heartbeat under both prefixes and prints one |
| +row per box: label, platform, channel, running `Release`, channel target, |
| +and the last apply outcome (§6, published in the heartbeat as |
| +`BoxMetrics.System`). This is the fleet-side half of the health gate: a |
| +box whose heartbeat still shows the old `Release` ten minutes after a |
| +promote is the row an operator is looking for. Heartbeats are |
| +five-minutely, so this is also the honest answer to "is it done yet": |
| +*not yet* for up to five minutes after the last box restarts. A `noted` |
| +outcome (a no-op request, §6) is shown as such, not as an apply. |
| + |
| +**`carlos system rollback [--channel …]`** |
| + |
| +Swaps `Current` with `History[0]` — whole releases, so the previous |
| +artifacts and hashes come back with the version — re-signs with |
| +`PromotedAt` now, bumps the serial. Nothing GCs `bootstrap/router/` in |
| +v1, so the objects are there. Every box rolls back through the same path |
| +it rolled forward on; the box-side `.prev` (§6) is for the box that |
| +cannot come up, not for this. |
| + |
| +**`carlos system apply [--recover]`** is the root oneshot, §6. It is not |
| +for operators and says so if run interactively. |
| + |
| +What is deliberately **not** a verb: `carlos system console update`. The |
| +console is an app; `carlos deploy --app console` is its verb and already |
| +works. Giving it a second name under `system` is the confusion the |
| +dividing line exists to prevent. |
| + |
| +--- |
| + |
| +## 6. `carlos system apply`: the journal, the dance, the gate, the way back |
| + |
| +Runs as root from `carlos-system-update.service` (or, with `--recover`, |
| +from `carlos-system-recover.service`). Every step is one the pinfra script |
| +does today, plus the ones it lacks. Root state lives in |
| +**`/opt/carlos/system/router/`** (0755, root-owned, never writable by |
| +`carlos`): |
| + |
| +- `lock` — an `flock` every apply takes first. The path-triggered apply |
| + takes it non-blocking and exits 0 with "another apply is running" if it |
| + cannot; `--recover` waits for it. Two processes never touch |
| + `/opt/carlos/carlos` at once, whatever an operator's hand-run |
| + `systemctl start` does. |
| +- `staging/` — **0700**. The root-owned copy of the candidate and the two |
| + staged unit roots. Removed on every exit, success or not. |
| +- `journal.json` — the open transaction, if any: `{phase, target, |
| + prev_build, units_added, units_removed, started_at}`. Written before |
| + each box-changing step, removed on completion. |
| +- `seen.json` — `{<channel>: promoted_at}`: the newest pointer root has |
| + *verified* per channel. The replay guard, and nothing else. |
| +- `applied.json` — `{version, channel, at}`: the last build that **passed |
| + a gate** on this box, forward or rollback. The `.prev` authority, and |
| + nothing else. One record, not per channel: it is a fact about the box. |
| +- `result.json` — `{channel, version, promoted_at, outcome, reason, at}` |
| + of the last run, keyed by the **request's** tuple (two promotes inside |
| + one spread must not let a refusal of the first suppress the second). |
| + What the agent publishes and §4 step 4 reads. |
| + |
| +Every file is written temp-then-`fsync`-then-`rename`-then-**`fsync` of |
| +the directory**, so the rename itself is durable before the next step |
| +depends on it; a leftover `*.tmp` at start is deleted unread. |
| +`carlos.prev` is made the same way (`carlos.prev.tmp`, fsync, rename, |
| +directory fsync), so a `.prev` is either whole or absent. |
| + |
| +`/opt/carlos/carlos` is installed **`root:root 0755`** from here on, not |
| +`carlos:carlos` as the script did. Every root oneshot on the box execs |
| +that file; the sandboxed user has no reason to own it. |
| + |
| +**Every run begins with recovery.** If `journal.json` exists, a previous |
| +run was cut off (SIGKILL, OOM, the timeout, a reboot). Which way it goes |
| +depends on one thing: whether the gate had passed. |
| + |
| +- **Phase `gated`** (step 10 passed, steps 11–12 not finished): the box |
| + is on the new build and proved it. Recovery goes **forward**: finish |
| + steps 11–12. Undoing a roll that passed its gate because the process |
| + that recorded it was killed would be a rollback with no reason. |
| +- **Any earlier phase**: recovery goes **back** — an apply that did not |
| + reach its gate is not one to trust, and `.prev` passed one. It is |
| + **files only**: stop the quiesce set of step 7 (an operator or the |
| + timer may have started a pass since); restore `carlos.prev` over |
| + `carlos` (keeping `.prev`); `disable` and delete `units_added`; restore |
| + `units_removed` and the rest of the previous unit set from |
| + `staging/prev/` or, if `staging/` is gone, from a fresh `carlos.prev |
| + bootstrap -root`; sysusers, tmpfiles, `daemon-reload`; `enable` (not |
| + start) the previous plan's units so they come up at boot. Then, only if |
| + `carlos-edge` has started at least once this boot |
| + (`ActiveEnterTimestampMonotonic != 0`): `reset-failed`, `restart |
| + --no-block carlos-edge`, and `start` the previous plan's timers and |
| + paths. At boot the edge is ordered after this unit and its own start |
| + *is* the restart, and the enabled timers and paths start with the |
| + target; a recovery that restarted and gated the edge from inside a |
| + unit the edge waits on would wait on itself forever. **Only now** write |
| + `result.json` `{outcome: "rolled-back", reason: "interrupted at |
| + <phase>"}` and remove the journal: a kill before this point leaves the |
| + journal in place and the next run recovers again, which is idempotent; |
| + a kill after it leaves nothing to do. |
| + |
| +`--recover` ends here. A path-triggered run continues below if there is |
| +also a request. |
| + |
| +Then the run proper. Failures in steps 1–6 change nothing on the box and |
| +are `refused` (or `deferred`, for a store error); from step 7 on, any |
| +failure goes to step 13. |
| + |
| +1. **Read the request.** One fixed path, `/data/system/router/requested.json`, |
| + reached by opening `/data`, then each component with |
| + `O_DIRECTORY|O_NOFOLLOW`, then the file with `O_NOFOLLOW`; must be a |
| + regular file under 4 KiB. It yields `{version, promoted_at, noop}` and |
| + nothing root will act on without checking. `version` must match |
| + `^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`, or the run is `refused` — this is |
| + what makes the cache path derivable and un-traversable. |
| +2. **The channel is root's.** `CARLOS_SYSTEM_CHANNEL` from the unit's own |
| + `EnvironmentFile`, default `stable`. The request does not carry one and |
| + would not be believed if it did: a signed canary pointer authorizes |
| + canary boxes, not a stable box whose sandboxed user would like to be |
| + one. |
| +3. **Fetch and verify the pointer from the bucket** with |
| + `CARLOS_RELEASE_PUBKEY`. A store error is `deferred`. The request's |
| + `version` and `promoted_at` must equal the pointer's `Current.Version` |
| + and `PromotedAt`, or `refused` (a request from a spread that started |
| + under an earlier promote). **Replay guard:** the pointer's `PromotedAt` |
| + must not be *older* than `seen.json[channel]`; equal is allowed, so an |
| + apply that was interrupted or refused can be retried for the same |
| + promote. Then advance `seen.json[channel]`. A box with no entry for the |
| + channel accepts the first pointer it sees — a fresh, hand-rolled or |
| + channel-moved box has nothing to protect yet (§11). |
| +4. **A no-op request ends here:** `result.json` `{outcome: "noted"}`, |
| + exit 0. Root has seen the promote; nothing was downloaded, nothing |
| + changes. |
| +5. **Copy the candidate into `staging/`.** The path is *derived*: |
| + `/data/system/router/<version>/carlos`, walked like step 1, regular |
| + file only. Copy to `staging/carlos` as **0600**; **verify the copy's |
| + sha** against the pointer's artifact for this platform; then `chmod |
| + 0755` and run it: `staging/carlos version`, 30-second deadline, must |
| + print `carlos <version>`. The lie detector from `update.go:350`, on a |
| + file only root can reach, made executable only after it is known to |
| + be what the key signed. Nothing under `/data` is read again. |
| +6. **Decide `.prev`, then stage both unit sets.** Run |
| + `/opt/carlos/carlos version` (30 s deadline). If it equals |
| + `applied.json.version`, or `applied.json` is absent and `.prev` is |
| + absent (a hand-rolled box: the running binary passed the script's own |
| + post-condition), the on-disk binary is known good: **journal |
| + `backing-up`** and copy it to `carlos.prev`. Otherwise — after a |
| + `rollback-failed`, when the disk holds a bad binary and `.prev` the |
| + good one — leave `.prev` alone. Then stage units from **both |
| + binaries**: `staging/carlos bootstrap -root staging/new -plan json` |
| + and `carlos.prev bootstrap -root staging/prev -plan json` (120 s |
| + deadline each; `-plan json` is new and prints `enabledUnits` and the |
| + inert list as JSON, so apply arms what *that* binary's bootstrap would |
| + arm). `units_added = new − prev`, `units_removed = prev − new`, both |
| + into the journal. Staging from `.prev`, not from the on-disk file, |
| + because after a `rollback-failed` those are different builds and a |
| + rollback must restore the units of the binary it restores. |
| +7. **Journal `installing`**, then **quiesce**: `systemctl stop` the root |
| + timers and paths *and their services* — `carlos-adopt.timer`, |
| + `carlos-adopt.path`, `carlos-stage.path`, `carlos-restart.path`, |
| + `carlos-adopt.service`, `carlos-config-sync.service`, |
| + `carlos-stage.service`, `carlos-restart.service`. Stopping a timer does |
| + not stop the pass it already started, and a root pass running the new |
| + binary against the old edge and registry is the window this exists to |
| + close. Those passes are idempotent; killing one costs a retry. Then |
| + **install**: `rename` `staging/carlos` over `/opt/carlos/carlos` |
| + (same filesystem, atomic). **Journal `installed`.** |
| +8. **Install units and configs**: `disable --now` and delete |
| + `units_removed` (a unit the new version dropped must not stay armed |
| + and exec a command the new binary no longer has); copy |
| + `staging/new/etc/systemd/system/*` into place; then, in this order and |
| + never the other, **sysusers** (`cp` + `systemd-sysusers`), then |
| + **tmpfiles** (`cp` + `systemd-tmpfiles --create`), then |
| + `daemon-reload`. The ordering is failure-mode row 6 of the |
| + console-as-an-app spec, and the reason is in the script's comment: |
| + tmpfiles silently drops a line whose group does not resolve, and then |
| + `carlos-edge` (`ReadWritePaths=/run/carlos`, no `-`) can never start |
| + after the next boot. Never full `carlos bootstrap`: it re-chowns |
| + `/data` to 0750, which breaks static-user tenants on the Tito box (the |
| + script's opening comment). And never copy the staging root's `data/` — |
| + it contains a 0-byte `registry.db` (`bootstrap.go:310`). **Journal |
| + `units`.** |
| +9. **Restart the edge**: `systemctl reset-failed carlos-edge` first — |
| + always, because a unit that hit its start limit refuses `restart` |
| + with "start request repeated too quickly", and the rollback in step 13 |
| + would be refused the same way — then `systemctl restart carlos-edge`. |
| + The one tenant-visible step; on a healthy box it is a one-second blip. |
| + **Journal `restarted`.** |
| +10. **The gate**, 20 seconds to pass and then a **15-second soak**: |
| + - `carlos-edge` is `active`; across the soak `NRestarts` stays 0 (a |
| + manual restart resets it, so it counts only crash restarts since |
| + step 9) and `ActiveEnterTimestampMonotonic` does not move. A binary |
| + that starts, answers, and dies a second later is caught here, not |
| + recorded as a success. |
| + - The edge accepts a TCP connection on its listener and answers a |
| + request on loopback with *any* HTTP status. Not a tenant probe: a |
| + box with no route yet must still pass. |
| + - `/opt/carlos/carlos version` prints the target. |
| + **Journal `gated`.** From here recovery finishes forward. |
| +11. **Arm**, now that the box is on the new build and the gate has said |
| + so: run `systemctlPlan(new plan's enabledUnits)` verbatim — enable |
| + all, `start` timers and paths — which re-arms what step 7 stopped and |
| + arms this spec's own path unit for the next roll. Not before the |
| + gate: a rolled-back box is armed once, on the previous plan. |
| +12. **Record.** `applied.json` `{version, channel, at}`, `result.json` |
| + `{outcome: "applied"}`, remove the journal, remove `staging/`. Done. |
| + The agent deletes its `/data` cache when it sees the outcome. |
| +13. **Roll back.** `rename` `carlos.prev` to `carlos` keeping a copy as |
| + `.prev`; `disable --now` and delete `units_added`; restore |
| + `staging/prev/etc/systemd/system/*` (which brings `units_removed` |
| + back); sysusers, tmpfiles, `daemon-reload`; `reset-failed`, restart |
| + the edge, run the gate; then `systemctlPlan(prev plan's |
| + enabledUnits)`. `result.json` `{outcome: "rolled-back", reason}`, |
| + `applied.json` unchanged (the `.prev` build is what passed a gate, |
| + and it is what `applied.json` already names), journal removed, exit |
| + non-zero so systemd's journal shows it. If the rollback fails *its* |
| + gate: `outcome: "rollback-failed"`, journal removed, exit non-zero — |
| + `applied.json` still names the `.prev` build, so the next run's step |
| + 6 leaves `.prev` alone. At every exit the installed binary and the |
| + installed units came from the same build, *including* exits by kill, |
| + timeout or reboot, because the next run recovers first and |
| + `OnFailure=` makes sure there is a next run. |
| + |
| +The agent, whichever binary it now is, reads `result.json` on its next |
| +heartbeat and publishes it as **`BoxMetrics.System{Channel, Version, |
| +Outcome, Reason, At}`**, which is what `carlos system status` shows. A |
| +rolled-back box is therefore visible fleet-wide within five minutes, with |
| +the reason, and (§4 step 4) will not try that pointer again. |
| + |
| +--- |
| + |
| +## 7. The first roll, and compatibility |
| + |
| +The first binary that contains this spec reaches a fleet by the old road: |
| +`update-platform-binary.sh`, one last time per box. That run must **arm |
| +`carlos-system-update.path` and enable `carlos-system-recover.service`**, |
| +or the first promote does nothing anywhere: a staged bootstrap runs no |
| +`systemctl` (`bootstrap.go:327`), and the path unit cannot arm itself |
| +before it works. Two belts: |
| + |
| +1. The script (pinfra, and Tito's copy in `tito/carlos-infrastructure`) |
| + gains the two lines, guarded on the files existing, the way |
| + `carlos-restart.path` was added. |
| +2. The root hourly pass (`RunAdopt`, from `carlos-adopt.timer`) gains a |
| + **new, non-fatal first step**, placed before its registry and pubkey |
| + `log.Fatalf` gates: if the embedded system-update units are missing |
| + from `/etc/systemd/system`, or the path unit is inactive, or the |
| + recover unit is not enabled, install and arm them via `systemctlPlan`. |
| + New code — `RunAdopt`'s existing `capabilityPass` converges tenant |
| + `carlos-app@<host>.service.d` drop-ins through `caps.Converge` and |
| + installs no platform unit. So a box the script forgot is armed within |
| + the hour, and the first promote is not lost: the leg re-touches its |
| + marker every minute until root answers (§4), and `PathChanged` sees |
| + the next touch after arming. |
| + |
| +From then on the script is never needed, because that binary contains the |
| +leg, the units and the apply. There is no flag day and no dual-running: an |
| +old agent does not read `control/system/`, so writing a pointer into a |
| +deployment whose boxes are old does nothing until they are rolled by hand |
| +once. |
| + |
| +Within one deployment, canary and stable boxes run different binaries by |
| +design, for as long as the operator leaves them that way. Nothing in the |
| +bucket schema they share changes with this spec (the pointer is a new key; |
| +`status/boxes` gains three optional fields), so a two-version fleet is |
| +exactly as safe as it is today after a partial script run. |
| + |
| +`carlos update` (the workstation CLI) is untouched except for the factored |
| +fetch-and-verify. Its header sentence — *a self-updating box binary would |
| +fight that machinery* — becomes true in the other direction: the box |
| +machinery is now the sanctioned one, and `runUpdate`'s refusal to run on a |
| +box stays. |
| + |
| +--- |
| + |
| +## 8. Third-party deployments |
| + |
| +Tito's operator, on a laptop with a Tito console session, as an owner of |
| +Tito's ops account: |
| + |
| +``` |
| +carlos system update --console https://console.tito.example --channel canary |
| +carlos system status --console https://console.tito.example |
| +carlos system update --console https://console.tito.example --channel stable |
| +``` |
| + |
| +The Tito console fetches the public release from GitHub, verifies the |
| +vendor's signature, mirrors it into Tito's bucket, signs the pointer with |
| +Tito's key. Tito's boxes, behind Tito's store door, fetch from Tito's |
| +bucket. At no point does a CARLOS developer hold anything of Tito's, and |
| +at no point does a Tito box reach anything but Tito's console. What the |
| +vendor ships is a signed public release and nothing else, which is what |
| +#100 asked for. |
| + |
| +Egress: the **console** needs to reach `apt.carlosframework.com` and |
| +`github.com`. Boxes need nothing new. A deployment whose console cannot |
| +reach GitHub uses `--file`. |
| + |
| +--- |
| + |
| +## 9. What this does NOT do |
| + |
| +- **Split the process.** `router` is the one binary. Updating the edge |
| + without restarting the activator is not possible until they are separate |
| + units, and that is not this spec. |
| +- **Order boxes within a channel** beyond the spread jitter. Two channels |
| + are the ordering primitive in v1. |
| +- **Auto-promote canary to stable**, or bake-time enforcement. The |
| + ladder is advisory. |
| +- **GC `bootstrap/router/`.** Old versions stay; they are ~25 MiB each and |
| + the rollback verb needs them. A later slice can keep `History`'s depth. |
| +- **A console page.** `carlos system status` is the read surface in v1; |
| + the fleet page's box rows already show `Release`, and a target column |
| + there is a small follow-up. |
| +- **Scoped-grant signing** for platform pointers. |
| +- **Update the console** — it is an app. |
| +- **Stream the store door's GET.** It holds the object in memory today |
| + for `bootstrap/carlos`; it will for `bootstrap/router/…`. Named in §11. |
| +- **Durable update jobs.** A console restart mid-promote loses the job; |
| + re-running is safe. |
| +- **Any change to `carlos bootstrap` for a fresh box** beyond `-plan |
| + json` and the three new units in `enabledUnits`. New boxes still |
| + bootstrap and get their first binary by hand (`infra/remote-host.md`); |
| + from the first heartbeat on they are on `stable` and roll with the |
| + fleet. |
| + |
| +--- |
| + |
| +## 10. What this retires |
| + |
| +- `update-platform-binary.sh` and the per-deployment roll runbooks, after |
| + each deployment has been hand-rolled once onto a binary that carries |
| + this spec. The script's sysusers/tmpfiles guard for *pre-#104* binaries |
| + is not carried: nothing that old is rolled through this path. |
| +- The exact `bootstrap/carlos` grant in `aclRead`, and the key itself, one |
| + release after the script is gone. Until then it is harmless. |
| +- The vendor needing SSM on a third-party deployment for a routine roll. |
| +- The "blocked on deploy scope" state this spec was born from: a merged |
| + `main` becomes a release, and a release becomes a fleet with one |
| + command per channel. |
| + |
| +--- |
| + |
| +## 11. Proof, and live checks owed |
| + |
| +**Unit-level, in this repo:** |
| + |
| +- Pointer sign/verify round trip; scoped grant refused; wrong component |
| + or channel refused; rollback pointer carries a new `PromotedAt`. The |
| + door: both new shapes granted, both traversal siblings refused, |
| + `bootstrap/carlos.sig` still refused, listing under either prefix still |
| + refused, PUT under either still refused. `Store.Open` on all four |
| + implementations, with the limit+1 read telling exact-limit from over. |
| +- The leg against a `DirStore`: no pointer → no request; pointer for another |
| + platform → no request; version equal, pointer not newer than `seen` → |
| + nothing; version equal, pointer newer → no-op request and no fetch; |
| + pointer older than `seen` → nothing; sha mismatch → no request and file |
| + deleted; oversized object → cut at the limit, no request; a |
| + `result.json` naming this tuple as any terminal outcome → no request |
| + and cache deleted; `deferred` → retried; a good pointer → |
| + `requested.json` with no channel/key/path and the marker; the marker |
| + re-touched every minute until a result names the tuple; the spread → |
| + `tick` fires after the deadline and not before, with no store read in |
| + between. |
| +- `carlos system apply` against a fake root and a fake `systemctl` that |
| + records every call in order: `lock` held → path-triggered run exits 0 |
| + untouched, `--recover` waits; request read refuses a symlinked parent, |
| + a symlinked file, an oversized file, a version that fails the pattern; |
| + channel taken from the environment, not the request (a request under a |
| + canary pointer on a stable box → `refused`); a stale tuple → `refused` |
| + recorded against the *request's* tuple; replay: older → `refused`, |
| + equal → proceeds, `seen` advanced only after verification; no-op → |
| + `noted`, no `staging/`; the candidate copied 0600, sha-checked, chmod |
| + after; a swap of the cache file after the copy → no effect; `version` |
| + over its deadline → `refused`, `staging/` removed; `.prev` refreshed |
| + when disk `version` equals `applied.json`, on a fresh box, and **not** |
| + after `rollback-failed`; `staging/prev` from `.prev`, not from disk; |
| + `units_added`/`units_removed` computed and journaled; quiesce stops |
| + the services too, before the rename; `units_removed` disabled and |
| + deleted forward; sysusers before tmpfiles by *call order*; |
| + `reset-failed` before every restart; **no `start` of any `.service` in |
| + any run** (the self-start deadlock); arming after the gate, forward and |
| + back, from the right plan; a gate that passes and then fails in the |
| + soak → rolled back; every journal phase before `gated` → recovery rolls |
| + back files only, restores `units_removed`, restarts the edge only when |
| + it has started this boot, and removes the journal last; phase `gated` |
| + → recovery finishes forward and `applied.json` names the candidate; the exec shim picks `.prev` when a journal is open; |
| + `rollback-failed` leaves `applied.json` and `.prev` alone; every write |
| + is temp-fsync-rename and a leftover `.tmp` is discarded. |
| +- `carlos system update` against a stub release host: non-owner or service |
| + credential → 403 before any fetch; bad sums signature refused; asset sha |
| + mismatch refused; mirror skip when present and equal; mirror refused |
| + when present and different; pointer CAS conflict retried; serial bumped |
| + last; upload without `--platform` refused; job polled to completion; |
| + job lost on restart reported, re-run idempotent. |
| + |
| +**Live, on the dev deployment first, then flagship canary:** |
| + |
| +1. Hand-roll the dev box with the script onto a build carrying this spec, |
| + and confirm afterwards: `carlos-system-update.path` active, |
| + `carlos-system-recover.service` enabled. That is the last script run |
| + for that box. |
| +2. `carlos system update --file <next build> --version <sha> --channel |
| + canary` with the dev box on canary. Watch `status` show the new |
| + `Release`. Re-promote the same version: `status` shows `noted`, no |
| + download in the edge log. |
| +3. Promote a deliberately broken binary (one that exits at start) the same |
| + way. Confirm the edge went start-limit-hit and came back on `.prev` |
| + regardless, `status` shows `rolled-back` with the reason, and a second |
| + serial bump does not re-attempt it. |
| +4. Promote a binary that starts and exits after five seconds. Confirm the |
| + soak catches it. |
| +5. Kill the oneshot between `installed` and `restarted` (a `sleep` build |
| + with a `kill -9`). Confirm `OnFailure=` recovers to `.prev` without a |
| + touch. Then reboot the box in the same state and confirm the boot unit |
| + does, and that the edge starts after it without a hang. |
| +6. Move the box from canary to stable in `host.env`. Confirm a stable |
| + pointer older than the canary one is applied, not refused. |
| +7. `carlos system rollback --channel canary`. Confirm a forward roll to the |
| + previous version through the normal path, artifacts and all. |
| +8. On flagship: the console box on canary; promote a real tag; then stable |
| + with the default spread; watch the tenant hostnames answer 200 |
| + throughout (the plan doc's rule at `console-as-an-app.md:1910`, now |
| + observed rather than performed). |
| +9. Tito: Tito's operator runs §8 against a Tito console; the CARLOS side |
| + watches nothing and touches nothing. |
| + |
| +**Known gaps to name so they are not discovered:** |
| + |
| +- The gate's loopback probe proves the process serves HTTP, not that a |
| + tenant's route works. A binary whose routing is broken but whose |
| + listener is fine passes the gate; `status` and the tenant's own |
| + monitoring are the backstop, as they are today. |
| +- The replay guard has no baseline on a box that has never seen a pointer |
| + for its channel. A bucket writer without the key can put an old signed |
| + pointer in front of such a box once. That is the exposure app channel |
| + pointers already have on every box, and the same actor could point |
| + `control/serial` at nothing at all; the guard narrows it, it does not |
| + close it. |
| +- Recovery depends on `carlos.prev` being able to run `system apply |
| + --recover`. `.prev` passed a gate on this box, so the case is a `.prev` |
| + that has since been damaged on disk — a box that was broken before this |
| + roll, and is now a hand job. |
| +- The store door's GET holds the artifact in memory inside the console |
| + box's edge (`MemoryHigh=70%`). Nine fleet boxes fetching 25 MiB inside |
| + the same spread is not a problem; ninety would be, and streaming the |
| + door's GET is the fix when that day comes. |
| +- `Spread` jitters starts, it does not serialize them; two boxes can still |
| + restart in the same second. With nine boxes and a five-minute spread the |
| + odds are small and the cost is two one-second blips. |
| +- A box that is asleep or dark during the promote rolls when it next |
| + ticks. `status` shows it on the old version until then, which is the |
| + truth. |
| + |
| +--- |
| + |
| +## 12. Approaches not taken |
| + |
| +**Boxes fetch from the public release host directly.** No mirroring, no |
| +console egress. But every box then needs egress to GitHub, which a fleet |
| +box behind the store door may not have — it excludes exactly the |
| +deployment shape #100 exists for, and leaves the door's `bootstrap/` grant |
| +unused. |
| + |
| +**The platform as a reserved app (`ops/platform`).** `carlos promote |
| +--app platform` would just work and the console would show it. It is the |
| +shape the console-as-an-app spec rejected: the substrate has no route, no |
| +upstream, no instance, and every app assumption — health probe, |
| +hibernation, the adoption pass moving a `live/<host>` symlink — has to be |
| +special-cased for the one "app" that is not one. Cheap on day one and |
| +expensive on every day after. |
| + |
| +--- |
| + |
| +## 13. Review findings, and where each landed |
| + |
| +Two rounds, two independent readers each, checking claims against the |
| +code. The table is so a reader can see what the drafts got wrong and how, |
| +rather than only that the text is now different. |
| + |
| +**Round one** (both readers unless noted): |
| + |
| +| Finding | Landed in | |
| +|---|---| |
| +| Root hashed and exec'd a file the sandboxed user owns — TOCTOU to root | §2, §6 steps 1, 5 | |
| +| No authorization rule; `requireDeployAPISession` grants nothing | §5 | |
| +| Door-backed boxes cannot read `control/system/` — silent never-update on the Tito shape | §3 | |
| +| Interrupted apply strands the box; on-disk `Build` equality then says "done" forever | §6 journal, recovery-first, boot unit, `OnFailure=` | |
| +| Rollback restored the binary but not the unit set | §6 steps 6, 8, 13 | |
| +| Edge start limit (default 5/10 s) refuses the rollback's restart | §1, §6 step 9 | |
| +| Gate had a deadline but no observation window | §6 step 10 | |
| +| `History` of version strings cannot reconstruct artifacts for rollback | §3 `SystemRelease` | |
| +| Root units exec the new binary against the old edge during the swap | §6 steps 7, 11 | |
| +| Generic "arm every timer" contradicts `enabledUnits` | §6 `-plan json`, `systemctlPlan` | |
| +| First-roll arming was circular | §7 | |
| +| `.prev` overwritten by a later run after `rollback-failed` | §6 step 6 | |
| +| Replay watermark missed no-op promotes; blocked canary→stable moves | §4 step 3, §6 `seen.json` | |
| +| Spread had nothing to run it between hourly passes | §4 `observe`/`tick` | |
| +| No exec deadlines; oneshot could hang before the gate | §4 units, §6 steps 5–6 | |
| +| 32 MiB cap is on PUT, not GET | §3 `Store.Open`, §4 step 6 | |
| +| Versions not immutable (Codex) | §3, §5 step 3 | |
| +| Console cannot know channel membership or fleet-prefixed heartbeats (Codex) | §5 | |
| +| Suppression covered only `rolled-back` (Fable) | §4 step 4 | |
| +| Store error during the console box's own roll read as a refusal (Fable) | §4 step 8, §6 `deferred` | |
| +| Binary owned `carlos:carlos` for no reason (Fable) | §6 | |
| + |
| +**Round two**, on the fixes above: |
| + |
| +| Finding | Landed in | |
| +|---|---| |
| +| `enable --now` on the plan starts the update oneshot from inside itself, and the recover unit mid-apply (both) | §4, §6 step 11: `systemctlPlan` verbatim, never starts a service; `lock` | |
| +| Boot recovery ordered before the edge restarts and gates the edge, waiting on itself (both) | §6 recovery: files only, restart only if the edge has started this boot | |
| +| One watermark doing replay guard *and* `.prev` authority breaks both (both) | §6 `seen.json` vs `applied.json` | |
| +| Recovery execs the candidate after the rename (both) | §4 exec shim prefers `.prev` under an open journal | |
| +| Stopping timers does not stop running passes; re-arm before restart reopened the window (both) | §6 steps 7, 11 | |
| +| `enable --now` would start a start-limited edge before `reset-failed` (Codex) | §6 step 11: no service is ever started by the plan | |
| +| `staging/prev` built from the on-disk binary, which after `rollback-failed` is not `.prev` (Codex) | §6 step 6 | |
| +| Crash durability of journal/`.prev` unspecified (Codex) | §6 temp-fsync-rename | |
| +| Request could name another channel's pointer (Codex) | §6 step 2 | |
| +| Request `path` still a root input; `O_NOFOLLOW` covers the last component only; copy before check exposed secrets in a world-readable dir (Fable) | §6 steps 1, 5: derived path, per-component walk, 0700 staging, 0600 until verified | |
| +| `LimitReader` cannot bound a `[]byte` `Get` (both) | §3 `Store.Open` | |
| +| Recover unit had no timeout (both) | §4 | |
| +| Inactive oneshots read as broken by the hourly belt and §11 (Codex) | §7, §11: path active, recover enabled | |
| +| Nothing re-touched the marker; `carlos adopt` does not converge platform units (both) | §4 `tick`, §7 belt 2 | |
| +| Rollback pointer's `PromotedAt` refused as a replay (both) | §3, §5 | |
| +| Forward install left removed units armed (Codex) | §6 steps 6, 8 | |
| +| No-op promote downloaded 25 MiB on every box (Fable) | §4 step 3, §6 step 4 | |
| +| `applied.json` shape undecided; canary→stable move skipped the `.prev` refresh (Fable) | §6: one record, compared to disk `version` | |
| +| `refused` recorded against the wrong tuple (Fable) | §6 `result.json` | |
| +| Update job lifecycle undecided (Fable) | §5, §9 | |
| + |
| +**Round three** (Codex closure check; 16 of 20 closed, four residuals): |
| + |
| +| Finding | Landed in | |
| +|---|---| |
| +| Interruption between `applied.json` and journal removal undid a roll that passed its gate | §6 step 10 `gated`; recovery goes forward from it | |
| +| Recovery removed its journal before issuing the restart; a kill in the gap left the candidate running with no journal | §6 recovery: result and journal removal last | |
| +| Recovery did not quiesce passes or restore the previous plan's armed state | §6 recovery: stops the quiesce set, `enable` at boot, `start` at runtime | |
| +| Directory fsync unspecified after renames | §6 durability | |