carlos system update — the platform rolls itself

Date: 2026-09-06 · Status: draft, two review rounds (Fable, Codex), awaiting Paul · Requested by: Paul

Builds on: discussion #100 (boxes should self-update via a channel subscription), 2026-08-23-console-as-an-app-design.md (slice 3, deferred there; the dividing line this spec applies), the root-relay pattern (carlos-stage.path → carlos adopt --stage-only), and pinfra's update-platform-binary.sh, whose 85 lines are the procedure this spec moves into the binary and then retires.

0. How this spec was arrived at

One brainstorm with Paul on 2026-09-05/06, after a merge of main stalled on "and deploy": the only way to roll a platform binary today is an operator with SSM access to every box, running a script per box, in an order held in operator memory. Paul's framing, which is the requirement:

> Ultimately, a fleet operator should be able to do carlos system update > and the whole platform updates to the latest version based on what we > just released. […] And I guess each piece should be updatable > individually, carlos system router update etc.

Three questions settled the shape before any design was written:

• Trigger: promote-driven. A fleet operator says when. Boxes do not poll the public release feed and roll on their own; a new release exists in the world for as long as the operator wants before any box sees it.
• Rollout: two channels, two promotes. canary and stable, one operator action each, exactly like an app. No auto-promotion in v1.
• Third-party deployments in v1. Tito's operator rolls Tito's boxes from Tito's console. The artifact comes from the public signed release, so the vendor never needs access to a customer box (the trust shape #100 was filed about).

Three approaches were compared; this document is the first. The other two are recorded in §12. The draft went through two rounds of review by two independent readers checking claims against the code (§13); everything below is the post-review shape, and §13 says what the drafts got wrong.

One ruling stated up front because it changes what "each piece" means: the edge, the activator and the adoption loop are one process (carlos agent, unit carlos-edge.service). There is nothing to update "individually" inside it without splitting the process, which is a separate project. So the honest component list today is one: the router, which is the substrate binary. The console is already an app and is updated as one (carlos deploy --app console). §3 keys everything per component so a split later adds a name, not a schema.

---

1. What exists, and what is missing

The platform already has most of this machinery, built for apps:

| Have | Where | Reused as | |---|---|---| | Signed channel pointer, CAS-written, single parse site | release.ChannelPointer, signPointer/verifyPointer (release.go:102,350,397) | The platform pointer's wrapper, signing and verification | | The one unsigned "something changed" signal, polled at 2s | release.BumpSerial on control/serial (serial.go:51), fastConvergeLoop (agent.go:1412) | How a box learns there is a new platform pointer | | Root relay from a sandboxed agent | /data/stage-requested → carlos-stage.path → carlos-stage.service | Same shape, new marker, new oneshot | | Fleet-box bucket proxy with an explicit grant for the platform binary | storedoor.go:383 aclRead: exact bootstrap/carlos, "the sanctioned self-update path (F7)" | Boxes behind the door fetch the artifact with no public egress | | Verified download of a public release | update.go selfReplace: tag-pinned URL, SHA256SUMS.txt.sig over the sums file, per-asset sha, lie-detector version run | The console's ingest of a release | | A box's running build, published to the bucket | version.Build → BoxMetrics.Release in status/boxes/<label>.json every 5 min (agent.go:2318) | carlos system status and the fleet-side half of the gate | | The unit list a box must have armed, and the exact systemctl sequence | enabledUnits, systemctlPlan (bootstrap.go:72,96): enable all, start timers and paths, never start a service | What apply arms, so apply and bootstrap never disagree and apply never starts itself | | The apply procedure | update-platform-binary.sh (pinfra) | Moved into carlos system apply, line for line, then retired |

What is missing, and is the whole of this spec:

1. A pointer for something that is not an app — no account, no route, no upstream, and a version that is a platform tag, not a release serial.
2. A box-side leg that notices the pointer, fetches and verifies the artifact, and asks root to apply it.
3. A root apply that does the script's dance from inside the binary, with the three things the script does not have: a journal so an interrupted apply is undone rather than left half-done, a health gate, and a rollback. Today carlos-edge.service is Restart=always, RestartSec=1, with no StartLimit* of its own, so a binary that dies on start hits systemd's default limit (5 starts in 10 s) and the unit goes to failed with start-limit-hit; nothing puts the old binary back, and there is no .prev anywhere.
4. A verb an operator runs, and a status they read.

---

2. The model: the platform is a subscriber of its own releases

One sentence: a platform pointer names a version and its artifacts; a box whose channel pointer names a version other than the one it is running fetches, verifies, and asks root to swap and restart; root journals every step, verifies everything again from a copy it owns, and puts the old binary back if the new one does not come up and stay up.

Four things the model deliberately does not do:

• A box never decides to update. It reacts to a pointer the deployment key signed, placed by an operator's promote. No box reads the public release feed.
• A box never publishes the binary. aclWrite already refuses bootstrap/** for boxes (storedoor_test.go:81), and stays that way. Only the console, holding the bucket and the signing key, writes the artifact and the pointer.
• Root takes exactly one input from the sandboxed user, and treats it as a hint. /data is chown -R carlos:carlos (bootstrap.go:390): any path under it can be a symlink, any file can be swapped between check and use. So root reads one file at one fixed path, walking to it component by component with no symlinks followed, and everything root then trusts — which channel, which version, which sha, which key — comes from a pointer root fetched and verified itself and from root's own host.env. The candidate binary is copied into a directory only root can enter, verified there, run there, and installed from there. A compromised carlos user can make the box install exactly what the deployment key signed for the box's configured channel, and nothing else — the trust the stage relay already grants, made true by construction.
• Root never leaves the box between two builds. Every step that changes the box is journaled before it runs; a killed oneshot, a timeout, or a reboot mid-apply is recovered — always backward, to the build that was known good — before anything else happens (§6).

---

3. The pointer: control/system/<component>/<channel>.json

Placed under control/, beside control/serial, because that is where the deployment's platform-wide signals live, and explicitly not under apps/ — the console-as-an-app spec's line is anything adoptable through a route is an app, and this is not.

// SystemPointer is the platform's channel pointer. It is what a
// ChannelPointer is for an app, for the thing that does the adopting.
type SystemPointer struct {
    Schema     int             `json:"schema"`    // 1
    Component  string          `json:"component"` // "router"
    Channel    string          `json:"channel"`   // "canary" | "stable"
    Current    SystemRelease   `json:"current"`
    History    []SystemRelease `json:"history"`   // newest first; rollback targets
    Spread     string          `json:"spread"`    // "5m": boxes jitter their start over this
    PromotedAt time.Time       `json:"promoted_at"` // of THIS write; the replay guard reads this one
    Audit      []Promotion     `json:"audit"`     // reused from release.go
}

// SystemRelease is everything a box needs to install one version. It
// travels whole into History so a rollback re-installs what was there,
// not a version string with someone else's artifacts.
type SystemRelease struct {
    Version    string              `json:"version"`   // what `carlos version` must print
    Artifacts  map[string]Artifact `json:"artifacts"` // "linux-arm64", "linux-amd64"
    Source     string              `json:"source"`    // "release:v0.18.0" | "file"
    ReleasedAt time.Time           `json:"released_at"` // when first mirrored; audit only
}

type Artifact struct {
    Key    string `json:"key"`    // bootstrap/router/<version>/carlos-linux-arm64
    SHA256 string `json:"sha256"` // hex, of the object at Key
}

• Signed with the deployment's release key, using the same signedWrapper (payload, sig, optional grant) and the same root-key anchor CARLOS_RELEASE_PUBKEY a box already holds. Root key only in v1: a scoped grant's rungs name app channels, and a grant that could roll the platform is a bigger key than any app promoter should hold. verifyPointer grows a sibling verifySystemPointer; both share the wrapper parse, which stays the single parse site.
• CAS-written with If-Match like ChannelKey, so two operators cannot race each other into a lost promote.
• Every write sets PromotedAt to now — promote and rollback alike. The box's replay guard (§6) reads this field and only this field. A rollback that carried the old release's timestamp would be refused by every box as a replay of itself; hence the two differently named fields.
• Version is a string the box compares for equality with version.Build. For a public release that is the tag (v0.18.0, which release builds put in Build); for a --file promote (§5) it is whatever the operator names, which carlos system apply checks by running the staged binary's version before installing it. Dev deployments that roll git-sha builds keep working; nothing assumes semver.
• A version is immutable. The console refuses to mirror different bytes under a bootstrap/router/<version>/ that already exists (§5 step 3). Without that, a reused --file --version could swap the artifact under a pointer stable boxes are still on, and boxes already reporting that version would never take the replacement.
• Artifacts is per platform because BYO boxes are not all Graviton. A box picks runtime.GOOS-runtime.GOARCH; a pointer with no entry for the box's platform is "no update for me", logged once, not an error.
• Spread is the one fleet-policy knob. A box that sees a new pointer waits rand(0, Spread) before requesting the apply, so a stable promote to nine boxes is nine edge restarts over five minutes, not nine in the same two seconds. --spread 0 exists for a hotfix. Home-before-region ordering (#300) is not in v1: the canary channel, with home boxes on it, is the ordering an operator needs today. The console's own box goes on canary by convention — it hosts the store door, and a fleet box mid-apply that finds the door restarting gets a transient store error (§4 says what it does with one).
• History holds whole releases, historyMax = 10, and is what carlos system rollback swaps with (§5).

The artifact lives at bootstrap/router/<version>/carlos-<os>-<arch>. bootstrap/ because the door already means "the platform's own binary" by that name; per-version because canary and stable boxes must be able to run different binaries at the same time, which a single bootstrap/carlos cannot express. The existing exact key stays as it is until §10 retires the script.

Store-door changes, both reads, in aclRead:

1. control/system/<component>/<channel>.json — exact three-segment shape under control/system/. Without it, a door-backed box gets 404, which fleetapi.Get maps to os.ErrNotExist (fleetapi.go:150), and the fleet #100 exists for silently never updates.
2. bootstrap/router/<version>/<asset> — a shape, three segments, no listing (aclDir does not change), and still nothing else under bootstrap/. The existing pin that bootstrap/carlos.sig is refused stays, and gains siblings: bootstrap/router/../../x and control/system/../serial are refused.

Boxes need no signature file because the sha in the signed pointer is the artifact's signature. aclWrite does not change: a box still cannot write under control/ or bootstrap/.

A streaming read on the store. release.Store.Get returns []byte and every implementation buffers (store.go:43; fleetapi.go:133); maxStoreBodyBytes bounds PUT bodies only (storedoor.go:180), the door's GET is uncapped and in memory. The box must not hold a binary in memory under MemoryHigh=70%, so the interface gains Open(ctx, key) (io.ReadCloser, etag string, err error) with all four implementations (S3Store, DirStore, fleetapi, prefixStore). The door's own GET stays in memory in v1 (§9, §11).

---

4. The box: notice, fetch, verify, ask

A new leg in the agent, systemLeg, with two entry points so the spread has something to run it:

• observe(ctx) — called from fastTick on a serial move and from adoptLoop on its hourly pass. Reads the pointer, decides, and either does nothing or records a pending request with a deadline.
• tick(now) — called on every 2 s fast tick, reads no store. Fires a pending request when its deadline passes, and re-touches the marker once a minute while a request is outstanding — a request root has not answered in result.json. PathChanged does not replay a change made before the unit was armed, and drops one that lands while the service is mid-run (carlos-restart.path's own comment); the stage relay survives that because the unstaged route is re-observed every 2 s, and this leg needs the same persistence by hand.

It is sandboxed (User=carlos, ProtectSystem=strict) and can write only /data and /run/carlos, which is exactly enough.

1. Read control/system/router/<channel>.json through the box's normal store (S3, dir, or the fleet door — openDeployStore already picks). The channel is CARLOS_SYSTEM_CHANNEL in host.env, default stable; an operator puts a box on canary by editing one line and restarting the edge, which is how every other per-box setting works. Missing pointer → nothing to do; this is the state of every deployment until its first carlos system update.
2. Verify the pointer with the release pubkey. A pointer that does not verify is logged and ignored, like a bad channel pointer.
3. Decide. Read root's seen.json (§6; root-owned, world-readable) for this channel's watermark. Then: - pointer older than the watermark → ignore (root would refuse it); - Current.Version == version.Build and pointer not newer than the watermark → nothing to do; - Current.Version == version.Build and pointer newer → a no-op request (step 7 with noop: true, no artifact fetched), so root sees every promote and the watermark moves without a 25 MiB download on every box; - otherwise → an update. A dev build (Build == "dev") never matches, so a hand-built binary on a dev box is never rolled over by accident; it is rolled when the operator promotes something, which is the point.
4. Refuse to repeat a failure. If result.json names this exact (channel, version, promoted_at) with outcome rolled-back, rollback-failed or refused, do nothing until the pointer changes. A pointer that broke one box must not have that box firing a root oneshot at every serial bump. deferred (a transient store error on root's side) is retried with the per-host backoff the unstaged-route sweep uses. A terminal outcome is also when the leg deletes its own cache under /data/system/router/<version>/ — root never deletes anything under /data.
5. Jitter rand(0, Spread) from the first sighting, as the pending deadline tick watches.
6. Fetch the artifact for this box's platform with Store.Open, reading at most 96 MiB + 1 byte — the extra byte is how an object exactly at the limit is told from one over it — with a running sha, into /data/system/router/<version>/carlos.tmp then rename. The object is never in memory on the box; an oversized object from a bucket writer is cut off at the limit, not at the OOM killer. Verify the sha against the pointer; mismatch → delete, log, retry with backoff.
7. Write the request /data/system/router/requested.json {version, promoted_at, noop, seen_at} — no channel, no key, no sha, no path: root takes the channel from its own host.env, the key and sha from the pointer, and derives the cache path from the version — and touch /data/system-update-requested, throttled to once a minute like stageRequester.
8. A transient store error anywhere above (the door restarting because the console box is rolling, a network blip) is a retry, not a refusal: nothing is recorded, the next observe starts over.

The marker fires carlos-system-update.path (PathChanged, the same form as carlos-stage.path), whose unit is carlos-system-update.service: Type=oneshot, root, StartLimitIntervalSec=0, TimeoutStartSec=15min (a hung apply is a failed apply), OnFailure=carlos-system-recover.service (a killed or timed-out apply is recovered without waiting for anyone to touch anything), the same hardening set as carlos-stage.service — and no ProtectSystem, because it must write /opt/carlos and /etc/systemd/system. A third unit, carlos-system-recover.service (Type=oneshot, Before=carlos-edge.service, WantedBy=multi-user.target, TimeoutStartSec=10min), runs once per boot and on the update unit's failure.

Both units exec through one line of shell: when a journal is open and carlos.prev exists, exec carlos.prev system apply …; otherwise exec carlos system apply …. After the rename the binary at /opt/carlos/carlos is the candidate, and the one thing known about a candidate that needs recovering is that it did not work; the recovery must not depend on it. (§11 names the residual: a .prev that cannot run system apply either, which is a box that was already broken before this roll.)

enabledUnits gains all three, and systemctlPlan does with them what it does with everything: enable all, start only the .path — never a service, which is why apply can call it from inside itself without starting a second apply.

---

5. The verbs

All under carlos system, a new top-level dispatch beside ops. Every verb runs in API mode against the console (--console, requireDeployAPISession), because the console holds the bucket, the signing key, and the egress; the workstation holds none of them.

Authorization, stated once: the endpoints (PUT /api/system/upload, POST /api/system/update, GET /api/system/jobs/<id>, POST /api/system/rollback, GET /api/system/status) require a person session (apiPerson, apicli.go:298) whose member is an owner of the ops account (OpsAccountID, store.go:1158), checked server-side on every call. Service credentials are refused, and so is every other account: a deployment has one ops account, and rolling the root binary of every box is that account's owners' act and no one else's. requireDeployAPISession selects console mode; it grants nothing (deploy.go:1151), which is why this paragraph exists.

The update endpoint is a job, not a request: it returns a job id, the console does the fetch-verify-mirror-sign work in a goroutine, and the CLI polls the job and prints each step as it completes. Fetching two 25 MiB assets from GitHub inside one HTTP request is how a promote times out halfway through a mirror. Jobs live in console memory; a console restart mid-job loses it, the CLI reports that, and re-running is safe because every step is idempotent (mirror skips what is present and equal; the pointer write is CAS). --file is a separate PUT /api/system/upload (body limit 128 MiB, --platform required because the console cannot run a foreign-arch binary to ask it) that returns a handle the update job then consumes.

carlos system update [--channel stable] [--version vX.Y.Z] [--file path --platform linux-arm64 --version v] [--spread 5m] [--yes]

The fleet operator's one command. It is a promote for the platform:

1. Resolve the version. Default: https://apt.carlosframework.com/latest.txt (the carlos update discovery, releaseTag-shaped). --version pins one. --file skips step 2.
2. Ingest the release, exactly as carlos update does. The console fetches SHA256SUMS.txt and .sig from the tag-pinned updateDownloadBase, verifies the signature with the embedded cliUpdatePubKey, and fetches each asset the deployment needs, checking its sha against the sums. The asset set is the distinct platforms in every box heartbeat — status/boxes/*.json and fleets/*/status/boxes/*.json — via a new BoxMetrics.Platform field written alongside Release; a box that has not reported since the field shipped counts as linux-arm64. selfReplace's fetch-and-verify is factored out into a function both callers share; it does not move.
3. Mirror each asset to bootstrap/router/<version>/<asset>. Present with the same sha → skip (a second promote of the same version to another channel copies nothing). Present with a different sha → refuse the whole promote: versions are immutable (§3).
4. Write the pointer for the channel, signed, CAS, PromotedAt now. History gets the previous Current whole; Audit gets a Promotion{From, To, Actor, At}.
5. Bump the serial. Boxes see it within two seconds and start their spread.
6. Print what will happen: which boxes are on this channel (from a new BoxMetrics.SystemChannel, also in the heartbeat — without it the console cannot know which boxes a promote reaches), running what, and the spread. Then return.

Without --yes it confirms once, showing the box list. The channel ladder is advisory in v1: promoting to stable a version that was never on canary prints a warning and proceeds, because a hotfix needs to.

carlos system router update … is the same verb scoped to one component. With one component today it is an alias; it exists so the command Paul named works and so a future split needs no new verb. carlos system update with no component updates every component the pointer schema knows.

carlos system status [--channel …]

Reads the pointers and every heartbeat under both prefixes and prints one row per box: label, platform, channel, running Release, channel target, and the last apply outcome (§6, published in the heartbeat as BoxMetrics.System). This is the fleet-side half of the health gate: a box whose heartbeat still shows the old Release ten minutes after a promote is the row an operator is looking for. Heartbeats are five-minutely, so this is also the honest answer to "is it done yet": not yet for up to five minutes after the last box restarts. A noted outcome (a no-op request, §6) is shown as such, not as an apply.

carlos system rollback [--channel …]

Pops History[0] into Current — whole releases, so the previous artifacts and hashes come back with the version — re-signs with PromotedAt now, bumps the serial. The outgoing release is not pushed back, so a second rollback goes one further back rather than undoing the first. Nothing GCs bootstrap/router/ in v1, so the objects are there. Every box rolls back through the same path it rolled forward on; the box-side .prev (§6) is for the box that cannot come up, not for this.

carlos system apply [--recover] is the root oneshot, §6. It is not for operators and says so if run interactively.

What is deliberately not a verb: carlos system console update. The console is an app; carlos deploy --app console is its verb and already works. Giving it a second name under system is the confusion the dividing line exists to prevent.

---

6. carlos system apply: the journal, the dance, the gate, the way back

Runs as root from carlos-system-update.service (or, with --recover, from carlos-system-recover.service). Every step is one the pinfra script does today, plus the ones it lacks. Root state lives in /opt/carlos/system/router/ (0755, root-owned, never writable by carlos):

• lock — an flock every apply takes first. The path-triggered apply takes it non-blocking and exits 0 with "another apply is running" if it cannot; --recover waits for it. Two processes never touch /opt/carlos/carlos at once, whatever an operator's hand-run systemctl start does.
• staging/ — 0700. The root-owned copy of the candidate and the two staged unit roots. Removed on every exit, success or not.
• journal.json — the open transaction, if any: {phase, target, prev_build, units_added, units_removed, started_at}. Written before each box-changing step, removed on completion.
• seen.json — {<channel>: promoted_at}: the newest pointer root has verified per channel. The replay guard, and nothing else.
• applied.json — {version, channel, at}: the last build that passed a gate on this box, forward or rollback. The .prev authority, and nothing else. One record, not per channel: it is a fact about the box.
• result.json — {channel, version, promoted_at, outcome, reason, at} of the last run, keyed by the request's tuple (two promotes inside one spread must not let a refusal of the first suppress the second). What the agent publishes and §4 step 4 reads.

Every file is written temp-then-fsync-then-rename-then-fsync of the directory, so the rename itself is durable before the next step depends on it; a leftover *.tmp at start is deleted unread. carlos.prev is made the same way (carlos.prev.tmp, fsync, rename, directory fsync), so a .prev is either whole or absent.

/opt/carlos/carlos is installed root:root 0755 from here on, not carlos:carlos as the script did. Every root oneshot on the box execs that file; the sandboxed user has no reason to own it.

Every run begins with recovery. If journal.json exists, a previous run was cut off (SIGKILL, OOM, the timeout, a reboot). Which way it goes depends on one thing: whether the gate had passed.

• Phase gated (step 10 passed, steps 11–12 not finished): the box is on the new build and proved it. Recovery goes forward: finish steps 11–12. Undoing a roll that passed its gate because the process that recorded it was killed would be a rollback with no reason.
• Any earlier phase: recovery goes back — an apply that did not reach its gate is not one to trust, and .prev passed one. It is files only: stop the quiesce set of step 7 (an operator or the timer may have started a pass since); restore carlos.prev over carlos (keeping .prev); disable and delete units_added; restore units_removed and the rest of the previous unit set from staging/prev/ or, if staging/ is gone, from a fresh carlos.prev bootstrap -root; sysusers, tmpfiles, daemon-reload; enable (not start) the previous plan's units so they come up at boot. Then, only if carlos-edge has started at least once this boot (ActiveEnterTimestampMonotonic != 0): reset-failed, restart --no-block carlos-edge, and start the previous plan's timers and paths. At boot the edge is ordered after this unit and its own start is the restart, and the enabled timers and paths start with the target; a recovery that restarted and gated the edge from inside a unit the edge waits on would wait on itself forever. Only now write result.json {outcome: "rolled-back", reason: "interrupted at <phase>"} and remove the journal: a kill before this point leaves the journal in place and the next run recovers again, which is idempotent; a kill after it leaves nothing to do.

--recover ends here. A path-triggered run continues below if there is also a request.

Then the run proper. Failures in steps 1–6 change nothing on the box and are refused (or deferred, for a store error); from step 7 on, any failure goes to step 13.

1. Read the request. One fixed path, /data/system/router/requested.json, reached by opening /data, then each component with O_DIRECTORY|O_NOFOLLOW, then the file with O_NOFOLLOW; must be a regular file under 4 KiB. It yields {version, promoted_at, noop} and nothing root will act on without checking. version must match ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$, or the run is refused — this is what makes the cache path derivable and un-traversable.
2. The channel is root's. CARLOS_SYSTEM_CHANNEL from the unit's own EnvironmentFile, default stable. The request does not carry one and would not be believed if it did: a signed canary pointer authorizes canary boxes, not a stable box whose sandboxed user would like to be one.
3. Fetch and verify the pointer from the bucket with CARLOS_RELEASE_PUBKEY. A store error is deferred. The request's version and promoted_at must equal the pointer's Current.Version and PromotedAt, or refused (a request from a spread that started under an earlier promote). Replay guard: the pointer's PromotedAt must not be older than seen.json[channel]; equal is allowed, so an apply that was interrupted or refused can be retried for the same promote. Then advance seen.json[channel]. A box with no entry for the channel accepts the first pointer it sees — a fresh, hand-rolled or channel-moved box has nothing to protect yet (§11).
4. A no-op request ends here: result.json {outcome: "noted"}, exit 0. Root has seen the promote; nothing was downloaded, nothing changes.
5. Copy the candidate into staging/. The path is derived: /data/system/router/<version>/carlos, walked like step 1, regular file only. Copy to staging/carlos as 0600; verify the copy's sha against the pointer's artifact for this platform; then chmod 0755 and run it: staging/carlos version, 30-second deadline, must print carlos <version>. The lie detector from update.go:350, on a file only root can reach, made executable only after it is known to be what the key signed. Nothing under /data is read again.
6. Decide .prev, then stage both unit sets. Run /opt/carlos/carlos version (30 s deadline). If it equals applied.json.version, or applied.json is absent and .prev is absent (a hand-rolled box: the running binary passed the script's own post-condition), the on-disk binary is known good: journal backing-up and copy it to carlos.prev. Otherwise — after a rollback-failed, when the disk holds a bad binary and .prev the good one — leave .prev alone. Then stage units from both binaries: staging/carlos bootstrap -root staging/new -plan json and carlos.prev bootstrap -root staging/prev -plan json (120 s deadline each; -plan json is new and prints enabledUnits and the inert list as JSON, so apply arms what that binary's bootstrap would arm). units_added = new − prev, units_removed = prev − new, both into the journal. Staging from .prev, not from the on-disk file, because after a rollback-failed those are different builds and a rollback must restore the units of the binary it restores.
7. Journal installing, then quiesce: systemctl stop the root timers and paths and their services — carlos-adopt.timer, carlos-adopt.path, carlos-stage.path, carlos-restart.path, carlos-adopt.service, carlos-config-sync.service, carlos-stage.service, carlos-restart.service. Stopping a timer does not stop the pass it already started, and a root pass running the new binary against the old edge and registry is the window this exists to close. Those passes are idempotent; killing one costs a retry. Then install: rename staging/carlos over /opt/carlos/carlos (same filesystem, atomic). Journal installed.
8. Install units and configs: disable --now and delete units_removed (a unit the new version dropped must not stay armed and exec a command the new binary no longer has); copy staging/new/etc/systemd/system/* into place; then, in this order and never the other, sysusers (cp + systemd-sysusers), then tmpfiles (cp + systemd-tmpfiles --create), then daemon-reload. The ordering is failure-mode row 6 of the console-as-an-app spec, and the reason is in the script's comment: tmpfiles silently drops a line whose group does not resolve, and then carlos-edge (ReadWritePaths=/run/carlos, no -) can never start after the next boot. Never full carlos bootstrap: it re-chowns /data to 0750, which breaks static-user tenants on the Tito box (the script's opening comment). And never copy the staging root's data/ — it contains a 0-byte registry.db (bootstrap.go:310). Journal units.
9. Restart the edge: systemctl reset-failed carlos-edge first — always, because a unit that hit its start limit refuses restart with "start request repeated too quickly", and the rollback in step 13 would be refused the same way — then systemctl restart carlos-edge. The one tenant-visible step; on a healthy box it is a one-second blip. Journal restarted.
10. The gate, 20 seconds to pass and then a 15-second soak: - carlos-edge is active; across the soak NRestarts stays 0 (a manual restart resets it, so it counts only crash restarts since step 9) and ActiveEnterTimestampMonotonic does not move. A binary that starts, answers, and dies a second later is caught here, not recorded as a success. - The edge accepts a TCP connection on its listener and answers a request on loopback with any HTTP status. Not a tenant probe: a box with no route yet must still pass. - /opt/carlos/carlos version prints the target. Journal gated. From here recovery finishes forward.
11. Arm, now that the box is on the new build and the gate has said so: run systemctlPlan(new plan's enabledUnits) verbatim — enable all, start timers and paths — which re-arms what step 7 stopped and arms this spec's own path unit for the next roll. Not before the gate: a rolled-back box is armed once, on the previous plan.
12. Record. applied.json {version, channel, at}, result.json {outcome: "applied"}, remove the journal, remove staging/. Done. The agent deletes its /data cache when it sees the outcome.
13. Roll back. rename carlos.prev to carlos keeping a copy as .prev; disable --now and delete units_added; restore staging/prev/etc/systemd/system/* (which brings units_removed back); sysusers, tmpfiles, daemon-reload; reset-failed, restart the edge, run the gate; then systemctlPlan(prev plan's enabledUnits). result.json {outcome: "rolled-back", reason}, applied.json unchanged (the .prev build is what passed a gate, and it is what applied.json already names), journal removed, exit non-zero so systemd's journal shows it. If the rollback fails its gate: outcome: "rollback-failed", journal removed, exit non-zero — applied.json still names the .prev build, so the next run's step 6 leaves .prev alone. At every exit the installed binary and the installed units came from the same build, including exits by kill, timeout or reboot, because the next run recovers first and OnFailure= makes sure there is a next run.

The agent, whichever binary it now is, reads result.json on its next heartbeat and publishes it as BoxMetrics.System{Channel, Version, Outcome, Reason, At}, which is what carlos system status shows. A rolled-back box is therefore visible fleet-wide within five minutes, with the reason, and (§4 step 4) will not try that pointer again.

---

7. The first roll, and compatibility

The first binary that contains this spec reaches a fleet by the old road: update-platform-binary.sh, one last time per box. That run must arm carlos-system-update.path and enable carlos-system-recover.service, or the first promote does nothing anywhere: a staged bootstrap runs no systemctl (bootstrap.go:327), and the path unit cannot arm itself before it works. Two belts:

1. The script (pinfra, and Tito's copy in tito/carlos-infrastructure) gains the two lines, guarded on the files existing, the way carlos-restart.path was added.
2. The root hourly pass (RunAdopt, from carlos-adopt.timer) gains a new, non-fatal first step, placed before its registry and pubkey log.Fatalf gates: if the embedded system-update units are missing from /etc/systemd/system, or the path unit is inactive, or the recover unit is not enabled, install and arm them via systemctlPlan. New code — RunAdopt's existing capabilityPass converges tenant carlos-app@<host>.service.d drop-ins through caps.Converge and installs no platform unit. So a box the script forgot is armed within the hour, and the first promote is not lost: the leg re-touches its marker every minute until root answers (§4), and PathChanged sees the next touch after arming.

From then on the script is never needed, because that binary contains the leg, the units and the apply. There is no flag day and no dual-running: an old agent does not read control/system/, so writing a pointer into a deployment whose boxes are old does nothing until they are rolled by hand once.

Within one deployment, canary and stable boxes run different binaries by design, for as long as the operator leaves them that way. Nothing in the bucket schema they share changes with this spec (the pointer is a new key; status/boxes gains three optional fields), so a two-version fleet is exactly as safe as it is today after a partial script run.

carlos update (the workstation CLI) is untouched except for the factored fetch-and-verify. Its header sentence — a self-updating box binary would fight that machinery — becomes true in the other direction: the box machinery is now the sanctioned one, and runUpdate's refusal to run on a box stays.

---

8. Third-party deployments

Tito's operator, on a laptop with a Tito console session, as an owner of Tito's ops account:

carlos system update --console https://console.tito.example --channel canary
carlos system status  --console https://console.tito.example
carlos system update --console https://console.tito.example --channel stable

The Tito console fetches the public release from GitHub, verifies the vendor's signature, mirrors it into Tito's bucket, signs the pointer with Tito's key. Tito's boxes, behind Tito's store door, fetch from Tito's bucket. At no point does a CARLOS developer hold anything of Tito's, and at no point does a Tito box reach anything but Tito's console. What the vendor ships is a signed public release and nothing else, which is what #100 asked for.

Egress: the console needs to reach apt.carlosframework.com and github.com. Boxes need nothing new. A deployment whose console cannot reach GitHub uses --file.

---

9. What this does NOT do

• Split the process. router is the one binary. Updating the edge without restarting the activator is not possible until they are separate units, and that is not this spec.
• Order boxes within a channel beyond the spread jitter. Two channels are the ordering primitive in v1.
• Auto-promote canary to stable, or bake-time enforcement. The ladder is advisory.
• GC bootstrap/router/. Old versions stay; they are ~25 MiB each and the rollback verb needs them. A later slice can keep History's depth.
• A console page. carlos system status is the read surface in v1; the fleet page's box rows already show Release, and a target column there is a small follow-up.
• Scoped-grant signing for platform pointers.
• Update the console — it is an app.
• Stream the store door's GET. It holds the object in memory today for bootstrap/carlos; it will for bootstrap/router/…. Named in §11.
• Durable update jobs. A console restart mid-promote loses the job; re-running is safe.
• Any change to carlos bootstrap for a fresh box beyond -plan json and the three new units in enabledUnits. New boxes still bootstrap and get their first binary by hand (infra/remote-host.md); from the first heartbeat on they are on stable and roll with the fleet.

---

10. What this retires

• update-platform-binary.sh and the per-deployment roll runbooks, after each deployment has been hand-rolled once onto a binary that carries this spec. The script's sysusers/tmpfiles guard for pre-#104 binaries is not carried: nothing that old is rolled through this path.
• The exact bootstrap/carlos grant in aclRead, and the key itself, one release after the script is gone. Until then it is harmless.
• The vendor needing SSM on a third-party deployment for a routine roll.
• The "blocked on deploy scope" state this spec was born from: a merged main becomes a release, and a release becomes a fleet with one command per channel.

---

11. Proof, and live checks owed

Unit-level, in this repo:

• Pointer sign/verify round trip; scoped grant refused; wrong component or channel refused; rollback pointer carries a new PromotedAt. The door: both new shapes granted, both traversal siblings refused, bootstrap/carlos.sig still refused, listing under either prefix still refused, PUT under either still refused. Store.Open on all four implementations, with the limit+1 read telling exact-limit from over.
• The leg against a DirStore: no pointer → no request; pointer for another platform → no request; version equal, pointer not newer than seen → nothing; version equal, pointer newer → no-op request and no fetch; pointer older than seen → nothing; sha mismatch → no request and file deleted; oversized object → cut at the limit, no request; a result.json naming this tuple as any terminal outcome → no request and cache deleted; deferred → retried; a good pointer → requested.json with no channel/key/path and the marker; the marker re-touched every minute until a result names the tuple; the spread → tick fires after the deadline and not before, with no store read in between.
• carlos system apply against a fake root and a fake systemctl that records every call in order: lock held → path-triggered run exits 0 untouched, --recover waits; request read refuses a symlinked parent, a symlinked file, an oversized file, a version that fails the pattern; channel taken from the environment, not the request (a request under a canary pointer on a stable box → refused); a stale tuple → refused recorded against the request's tuple; replay: older → refused, equal → proceeds, seen advanced only after verification; no-op → noted, no staging/; the candidate copied 0600, sha-checked, chmod after; a swap of the cache file after the copy → no effect; version over its deadline → refused, staging/ removed; .prev refreshed when disk version equals applied.json, on a fresh box, and not after rollback-failed; staging/prev from .prev, not from disk; units_added/units_removed computed and journaled; quiesce stops the services too, before the rename; units_removed disabled and deleted forward; sysusers before tmpfiles by call order; reset-failed before every restart; no start of any .service in any run (the self-start deadlock); arming after the gate, forward and back, from the right plan; a gate that passes and then fails in the soak → rolled back; every journal phase before gated → recovery rolls back files only, restores units_removed, restarts the edge only when it has started this boot, and removes the journal last; phase gated → recovery finishes forward and applied.json names the candidate; the exec shim picks .prev when a journal is open; rollback-failed leaves applied.json and .prev alone; every write is temp-fsync-rename and a leftover .tmp is discarded.
• carlos system update against a stub release host: non-owner or service credential → 403 before any fetch; bad sums signature refused; asset sha mismatch refused; mirror skip when present and equal; mirror refused when present and different; pointer CAS conflict retried; serial bumped last; upload without --platform refused; job polled to completion; job lost on restart reported, re-run idempotent.

Live, on the dev deployment first, then flagship canary:

1. Hand-roll the dev box with the script onto a build carrying this spec, and confirm afterwards: carlos-system-update.path active, carlos-system-recover.service enabled. That is the last script run for that box.
2. carlos system update --file <next build> --version <sha> --channel canary with the dev box on canary. Watch status show the new Release. Re-promote the same version: status shows noted, no download in the edge log.
3. Promote a deliberately broken binary (one that exits at start) the same way. Confirm the edge went start-limit-hit and came back on .prev regardless, status shows rolled-back with the reason, and a second serial bump does not re-attempt it.
4. Promote a binary that starts and exits after five seconds. Confirm the soak catches it.
5. Kill the oneshot between installed and restarted (a sleep build with a kill -9). Confirm OnFailure= recovers to .prev without a touch. Then reboot the box in the same state and confirm the boot unit does, and that the edge starts after it without a hang.
6. Move the box from canary to stable in host.env. Confirm a stable pointer older than the canary one is applied, not refused.
7. carlos system rollback --channel canary. Confirm a forward roll to the previous version through the normal path, artifacts and all.
8. On flagship: the console box on canary; promote a real tag; then stable with the default spread; watch the tenant hostnames answer 200 throughout (the plan doc's rule at console-as-an-app.md:1910, now observed rather than performed).
9. Tito: Tito's operator runs §8 against a Tito console; the CARLOS side watches nothing and touches nothing.

Known gaps to name so they are not discovered:

• The gate's loopback probe proves the process serves HTTP, not that a tenant's route works. A binary whose routing is broken but whose listener is fine passes the gate; status and the tenant's own monitoring are the backstop, as they are today.
• The replay guard has no baseline on a box that has never seen a pointer for its channel. A bucket writer without the key can put an old signed pointer in front of such a box once. That is the exposure app channel pointers already have on every box, and the same actor could point control/serial at nothing at all; the guard narrows it, it does not close it.
• Recovery depends on carlos.prev being able to run system apply --recover. .prev passed a gate on this box, so the case is a .prev that has since been damaged on disk — a box that was broken before this roll, and is now a hand job.
• The store door's GET holds the artifact in memory inside the console box's edge (MemoryHigh=70%). Nine fleet boxes fetching 25 MiB inside the same spread is not a problem; ninety would be, and streaming the door's GET is the fix when that day comes.
• Spread jitters starts, it does not serialize them; two boxes can still restart in the same second. With nine boxes and a five-minute spread the odds are small and the cost is two one-second blips.
• A box that is asleep or dark during the promote rolls when it next ticks. status shows it on the old version until then, which is the truth.
• A permanently failing recovery re-runs every minute (the agent re-touches the marker while its request is unanswered) and never writes result.json, so the fleet sees a stale result; that is the needs-a-human state, and it is fleet-invisible.

---

12. Approaches not taken

Boxes fetch from the public release host directly. No mirroring, no console egress. But every box then needs egress to GitHub, which a fleet box behind the store door may not have — it excludes exactly the deployment shape #100 exists for, and leaves the door's bootstrap/ grant unused.

The platform as a reserved app (ops/platform). carlos promote --app platform would just work and the console would show it. It is the shape the console-as-an-app spec rejected: the substrate has no route, no upstream, no instance, and every app assumption — health probe, hibernation, the adoption pass moving a live/<host> symlink — has to be special-cased for the one "app" that is not one. Cheap on day one and expensive on every day after.

---

13. Review findings, and where each landed

Two rounds, two independent readers each, checking claims against the code. The table is so a reader can see what the drafts got wrong and how, rather than only that the text is now different.

Round one (both readers unless noted):

| Finding | Landed in | |---|---| | Root hashed and exec'd a file the sandboxed user owns — TOCTOU to root | §2, §6 steps 1, 5 | | No authorization rule; requireDeployAPISession grants nothing | §5 | | Door-backed boxes cannot read control/system/ — silent never-update on the Tito shape | §3 | | Interrupted apply strands the box; on-disk Build equality then says "done" forever | §6 journal, recovery-first, boot unit, OnFailure= | | Rollback restored the binary but not the unit set | §6 steps 6, 8, 13 | | Edge start limit (default 5/10 s) refuses the rollback's restart | §1, §6 step 9 | | Gate had a deadline but no observation window | §6 step 10 | | History of version strings cannot reconstruct artifacts for rollback | §3 SystemRelease | | Root units exec the new binary against the old edge during the swap | §6 steps 7, 11 | | Generic "arm every timer" contradicts enabledUnits | §6 -plan json, systemctlPlan | | First-roll arming was circular | §7 | | .prev overwritten by a later run after rollback-failed | §6 step 6 | | Replay watermark missed no-op promotes; blocked canary→stable moves | §4 step 3, §6 seen.json | | Spread had nothing to run it between hourly passes | §4 observe/tick | | No exec deadlines; oneshot could hang before the gate | §4 units, §6 steps 5–6 | | 32 MiB cap is on PUT, not GET | §3 Store.Open, §4 step 6 | | Versions not immutable (Codex) | §3, §5 step 3 | | Console cannot know channel membership or fleet-prefixed heartbeats (Codex) | §5 | | Suppression covered only rolled-back (Fable) | §4 step 4 | | Store error during the console box's own roll read as a refusal (Fable) | §4 step 8, §6 deferred | | Binary owned carlos:carlos for no reason (Fable) | §6 |

Round two, on the fixes above:

| Finding | Landed in | |---|---| | enable --now on the plan starts the update oneshot from inside itself, and the recover unit mid-apply (both) | §4, §6 step 11: systemctlPlan verbatim, never starts a service; lock | | Boot recovery ordered before the edge restarts and gates the edge, waiting on itself (both) | §6 recovery: files only, restart only if the edge has started this boot | | One watermark doing replay guard and .prev authority breaks both (both) | §6 seen.json vs applied.json | | Recovery execs the candidate after the rename (both) | §4 exec shim prefers .prev under an open journal | | Stopping timers does not stop running passes; re-arm before restart reopened the window (both) | §6 steps 7, 11 | | enable --now would start a start-limited edge before reset-failed (Codex) | §6 step 11: no service is ever started by the plan | | staging/prev built from the on-disk binary, which after rollback-failed is not .prev (Codex) | §6 step 6 | | Crash durability of journal/.prev unspecified (Codex) | §6 temp-fsync-rename | | Request could name another channel's pointer (Codex) | §6 step 2 | | Request path still a root input; O_NOFOLLOW covers the last component only; copy before check exposed secrets in a world-readable dir (Fable) | §6 steps 1, 5: derived path, per-component walk, 0700 staging, 0600 until verified | | LimitReader cannot bound a []byte Get (both) | §3 Store.Open | | Recover unit had no timeout (both) | §4 | | Inactive oneshots read as broken by the hourly belt and §11 (Codex) | §7, §11: path active, recover enabled | | Nothing re-touched the marker; carlos adopt does not converge platform units (both) | §4 tick, §7 belt 2 | | Rollback pointer's PromotedAt refused as a replay (both) | §3, §5 | | Forward install left removed units armed (Codex) | §6 steps 6, 8 | | No-op promote downloaded 25 MiB on every box (Fable) | §4 step 3, §6 step 4 | | applied.json shape undecided; canary→stable move skipped the .prev refresh (Fable) | §6: one record, compared to disk version | | refused recorded against the wrong tuple (Fable) | §6 result.json | | Update job lifecycle undecided (Fable) | §5, §9 |

Round three (Codex closure check; 16 of 20 closed, four residuals):

| Finding | Landed in | |---|---| | Interruption between applied.json and journal removal undid a roll that passed its gate | §6 step 10 gated; recovery goes forward from it | | Recovery removed its journal before issuing the restart; a kill in the gap left the candidate running with no journal | §6 recovery: result and journal removal last | | Recovery did not quiesce passes or restore the previous plan's armed state | §6 recovery: stops the quiesce set, enable at boot, start at runtime | | Directory fsync unspecified after renames | §6 durability |