carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

Zero-downtime deploys: rev 19 amendment (generation socket names)

Status: rev 19, 2026-10-13. Paul approved the fix on 2026-10-13 after Gate F failed. This amends §7.1 of 2026-10-07-zero-downtime-deploys-design.md (rev 17, as amended by rev 18).

What happened

Gate F armed the flagship's IE box at 22:08Z on 2026-10-13. For about 3.5 minutes every generation wake of three instances of one app crash-looped, until the flag was removed (amadan #272).

The app's framework works out its origin from the file name of its --socket argument:

  • A bare app.sock names no host, so the framework falls back to its configured origin. This is how every nested instance starts with the flag off.
  • The nested generation socket app.3.sock parsed as the host app.3. That disagreed with the configured origin, so the app refused to start.

CARLOS_HOST was set on those launches, and the docs say to read it rather than the socket name, but deployed builds do not. A platform release must not break a build that starts today. The dev probe passed because zddprobe ignores its socket name.

The rule (replaces §7.1's generation paths)

  1. A generation's socket file name is the flag-off one. - Nested: app.sock. - Flat: <host>.sock.

Each generation gets its own directory instead: - Nested: <instance run dir>/g<n>/app.sock, for example /run/carlos/instances/<id>/g3/app.sock. - Flat: /run/carlos/g/<n>/<host>.sock.

The control socket stays <socket>.control, beside it. Rev 15's canonical link at rt.Socket keeps pointing at the current generation's socket by absolute path. §7.1's "the hostname never appears in it" no longer holds for flat generation paths. See rule 4 for the length bound that replaced it.

  1. A generation launch matches a flag-off launch. Its argv and its agent-owned environment are the same as a flag-off launch of the same build, except for the directory part of --socket. This already holds for the environment (agentOwnedVars). A test pins it for both layouts.
  2. The directory's lifecycle. - Before generation n's stale-file removal and start, the agent makes its directory, mode 0750 with setgid, inheriting the carlos-sockets group as the run dirs do. - The stale-file rule is unchanged: before the app binds, only the file at this generation's own path is removed. - Once the generation's termination proof holds and its socket is gone, the agent removes the directory if it is empty. A sibling's directory is never touched. - A leftover directory from an earlier agent process is harmless, and the stale-file rule clears its socket the next time n is used.
  3. The length bound. A unix socket path is at most 107 bytes. A flat host whose generation path would exceed that has no generation path: it is handled exactly as a foreign row is today, keeping its canonical socket. This adds no new member-facing state and no new copy. The window is small, because the flag-off path /run/carlos/<host>.sock is already near the bound for the same host. Nested paths are bounded by the 40-byte instance identifier and stay well under it.

The bound is measured on the host alone: the control socket /run/carlos/g/<n>/<host>.sock.control at the largest generation number (20 digits). So flat hosts of up to 59 bytes get generation paths, against 82 bytes for the flag-off control socket. Measuring at the current number would let a row change treatment as the counter grows, even in the middle of a swap: an attempt admitted as an overlap could meet a generation that cannot bind.

  1. The probe checks it. cmd/zddprobe exits 1, with a clear line on stderr, before it listens if either of these is wrong: - the base name of --socket is neither app.sock nor $CARLOS_HOST.sock; - CARLOS_HOST is empty.

A dev or flagship probe then fails its health check the way the tenant did, instead of passing.

Not changed

  • Generation numbering, the generations file, the termination proof, the scheduler, the canary, and the failure book.
  • Flag-off paths. With CARLOS_SWAP unset, nothing in this amendment runs.

Before Gate F runs again

  • Release the fix, and roll it to dev and then the flagship.
  • Re-run the dev swap under load with the new probe.
  • Clear annie's stale failure record. It holds bdn/ips build 2d577ebda on canada, which is the live build. Clearing it needs Paul's approval.