carlos / platform Public

Keep bootstrap/carlos in step with the fleet: system update mirrors releases but new boxes still boot a stale binary Feature Request open

carlos system update mirrors each release to bootstrap/router/<version>/carlos-linux-arm64 but never updates bootstrap/carlos, which is what a new, rebuilt or failover box downloads (Tito's bootstrap-box.sh and failover runbook use it).

On 2026-10-18, while Tito's fleet moved from v0.22.2 to v0.28.0:

  • production's bootstrap/carlos was an unidentified 2026-09-24 build (33,423,522 bytes);
  • staging's was still v0.22.2.

A failover box would have booted that build and then had to update itself across several releases in the middle of an incident: an untested path, and one that leaves carlos-instance-control stale, because update code older than v0.26.0 doesn't restart it.

We uploaded the signature-verified v0.28.0 carlos-linux-arm64 to bootstrap/carlos in both buckets by hand. That's a step that's easy to forget.

Request, either of:

  • the bootstrap path reads the channel's mirror (bootstrap/router/<stable's version>/…), so a new box boots what its channel runs; or
  • carlos system update refreshes bootstrap/carlos once every box on stable reports current, and system status warns when it lags.

First step pushed on branch bootstrap-carlos-follows-316 (ce50cbbb), per Paul's ruling: a warning first.

carlos system status now compares bootstrap/carlos with stable's signed linux-arm64 artifact (new bootstrap field on /api/cli/system/status: current, behind or unreadable; null when there is nothing to compare). When it lags, it prints the mirrored key to copy over it. Otherwise it stays quiet. The read is bounded to 10 s so a slow store can't time out status or the system update preview. Copy approved. Astra reviewed it twice: round 1 found one P2 (the unbounded read), which is fixed, and round 2 found nothing new.

Not done, and why:

  • Booting new boxes from the channel (install-channel) is the real fix, but it refuses until a host floor is published, so it can't help Tito or flagship yet.
  • Having the console overwrite bootstrap/carlos was declined. It needs a non-create-only write family and a new IAM grant, and it never fires while a box is unreachable.

Census ruling for its envelope-only overlay asked on #280.

The status warning is merged to main as cd2dd6e (squash). Not released or rolled. Leaving this open for the real fix: new boxes boot the channel's build. That waits on published host floors (install-channel) and needs a change to bootstrap-box.sh. Releasing my claim. The census overlay ruling is still asked on #280.