claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 1b69e34c · claude-code/2.1.292 · DOCS-T1
What happened (2026-10-07, flagship, v0.24.2 → v0.25.1 via carlos system update, 13:34–13:51 UTC)
13:36–13:44: every migrated instance on IE failed to start (Failed to set up mount namespacing: /opt/carlos/app: No such file or directory, status 226/NAMESPACE), and visitors got 503. The 104 instances moved to nested at 13:36:59, but carlos-instance-control.service was still the binary started on 2026-09-16. carlos system apply restarts the edge but not the control service. The old control service doesn't write the per-instance exe links (/run/carlos-control/exe/<iid>/) that v0.25.x's templates bind, and the probe unit failed the same way, so ready=false. Fixed by hand with systemctl restart carlos-instance-control.service on all four boxes at 13:44, plus reset-failed.
13:44–13:50: bdn/ips (canada, ips, kpmg) crashed at start. The app derives its hostname from the activation socket's file name, and /run/carlos/instances/<host>/app.sock no longer names it. Fixed by setting IPS_ORIGIN=https://<host> in each environment (Paul's call; Tony to be told).
13:48–13:51: the same three instances minted fresh instance.key files. The migration doesn't move directories an app creates itself in /data: /data/<host>/instance.key and report.token were left behind. Fixed: originals copied into /data/instances/<host>/<host>/, and the minted keys kept beside them as instance.key.minted-after-migration-*. Anything those three sealed during that window used the minted key. /data/proving.bdr.oncarlos.com-blobs was also left behind, but it was empty.
Every box now reports ready, none has a failed unit, and IE has 104 instances nested.
Platform bugs to fix before any further roll:
A (critical):carlos system apply (and update-platform-binary.sh) must restart carlos-instance-control.service when the binary changes, and the probe's readiness should check the control service runs the current binary.
B (high): the layout pass moves only its planned names. Any other entry an app made under /data for its host (/data/<host>/, /data/<host>-blobs, ...) must be moved into the instance tree too, or the move refused and reported.
C (medium): the platform should hand every instance its hostname explicitly (an env var such as CARLOS_HOST), so apps don't have to parse socket paths. Document it in the activation contract.
Tracked under #247.
claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 1b69e34c · claude-code/2.1.292 · DOCS-T1
Update 2026-10-08 07:40 UTC: a fourth break, found and fixed.
D (high): the working directory.bdn/sharingangels had been down since the migration (about 17 hours; its scheduled bank-sync job failed too). It writes ./data/blobs relative to its working directory. Flat children ran with cwd /, so that meant /data/blobs. A nested unit has no WorkingDirectory, so cwd is / on a read-only tmpfs, and the app exits 1 with mkdir ./data/blobs: read-only file system.
Emergency fix: a drop-in on the IE box only, /etc/systemd/system/carlos-instance@sharingangels.bdn.oncarlos.com.service.d/50-workdir-incident-256.conf with WorkingDirectory=/data/instances/%I. The app is back (200).
It has 0 documents, so nothing was lost.
Remove the drop-in once the platform sets WorkingDirectory for every instance.
Platform fix: every nested unit gets WorkingDirectory=/data/instances/%I, and seed and sidecar match. Also decide what flat-relative paths should move with the instance.
Related, legacy, cross-tenant:/data/blobs on IE holds 7,766 content-addressed blobs (1.1 GB), written by any app with cwd /. No nested app's database references them by hex hash. Who owns them needs establishing before anything is deleted; under the flat layout every app could read them.
Sweep:carlos logs for all 121 IE apps since the roll. Every other failed wake falls in the 13:37–13:43 window (the stale control service). Nothing else is failing now.
claude agent, operated by paul@keymail.devoperated by paul@keymail.devsession 1b69e34c · claude-code/2.1.292 · DOCS-T1
Update 2026-10-08: two more left-behind keys, both regenerated. A hand scan of IE for flat leftovers of already-nested hosts found:
/data/jb-book.beb.oncarlos.com.db.key (from 2025-09-25). After the move the app minted a new db.key (2026-10-07 14:00) and has written to its database since. jb-book's source isn't available here, so what the key protects is unknown. If it encrypts data, rows from before the move need the old key and rows since need the new one, so swapping either way could make data unreadable. Not swapped: the original is now beside the new one as .db.key.pre-migration in the instance dir. The owner of account beb needs to decide (Paul isn't a member).
/data/proving.bdr.oncarlos.com.vapid (from 2025-09-05): elevenmessenger's VAPID key. The app minted a new one (2026-10-07 16:27), so web push subscriptions made before then stop working, and restoring the old key would break any made since. Not swapped: the original is beside it as .vapid.pre-migration. Owner: eleven (bdr).
proving.bdr.oncarlos.com-blobs was empty; the ips keys were handled earlier.
This is what fix B (and the nested-host leftover report the review asked for) is meant to prevent.
amadan