| 1 | # Zero-downtime deploys: rev 19 amendment (generation socket names) |
| 2 | |
| 3 | Status: rev 19, 2026-10-13. Paul approved the fix on 2026-10-13 after Gate F |
| 4 | failed. This amends §7.1 of |
| 5 | `2026-10-07-zero-downtime-deploys-design.md` (rev 17, as amended by rev 18). |
| 6 | |
| 7 | ## What happened |
| 8 | |
| 9 | Gate F armed the flagship's IE box at 22:08Z on 2026-10-13. For about |
| 10 | 3.5 minutes every generation wake of three instances of one app crash-looped, |
| 11 | until the flag was removed (amadan #272). |
| 12 | |
| 13 | The app's framework works out its origin from the file name of its `--socket` |
| 14 | argument: |
| 15 | |
| 16 | - A bare `app.sock` names no host, so the framework falls back to its |
| 17 | configured origin. This is how every nested instance starts with the flag |
| 18 | off. |
| 19 | - The nested generation socket `app.3.sock` parsed as the host `app.3`. That |
| 20 | disagreed with the configured origin, so the app refused to start. |
| 21 | |
| 22 | `CARLOS_HOST` was set on those launches, and the docs say to read it rather |
| 23 | than the socket name, but deployed builds do not. A platform release must not |
| 24 | break a build that starts today. The dev probe passed because `zddprobe` |
| 25 | ignores its socket name. |
| 26 | |
| 27 | ## The rule (replaces §7.1's generation paths) |
| 28 | |
| 29 | 1. **A generation's socket file name is the flag-off one.** |
| 30 | - Nested: `app.sock`. |
| 31 | - Flat: `<host>.sock`. |
| 32 | |
| 33 | Each generation gets its own directory instead: |
| 34 | - Nested: `<instance run dir>/g<n>/app.sock`, for example |
| 35 | `/run/carlos/instances/<id>/g3/app.sock`. |
| 36 | - Flat: `/run/carlos/g/<n>/<host>.sock`. |
| 37 | |
| 38 | The control socket stays `<socket>.control`, beside it. Rev 15's canonical |
| 39 | link at `rt.Socket` keeps pointing at the current generation's socket by |
| 40 | absolute path. §7.1's "the hostname never appears in it" no longer holds |
| 41 | for flat generation paths. See rule 4 for the length bound that replaced |
| 42 | it. |
| 43 | 2. **A generation launch matches a flag-off launch.** Its argv and its |
| 44 | agent-owned environment are the same as a flag-off launch of the same |
| 45 | build, except for the directory part of `--socket`. This already holds for |
| 46 | the environment (`agentOwnedVars`). A test pins it for both layouts. |
| 47 | 3. **The directory's lifecycle.** |
| 48 | - Before generation n's stale-file removal and start, the agent makes its |
| 49 | directory, mode 0750 with setgid, inheriting the carlos-sockets group as |
| 50 | the run dirs do. |
| 51 | - The stale-file rule is unchanged: before the app binds, only the file at |
| 52 | this generation's own path is removed. |
| 53 | - Once the generation's termination proof holds and its socket is gone, |
| 54 | the agent removes the directory if it is empty. A sibling's directory is |
| 55 | never touched. |
| 56 | - A leftover directory from an earlier agent process is harmless, and the |
| 57 | stale-file rule clears its socket the next time n is used. |
| 58 | 4. **The length bound.** A unix socket path is at most 107 bytes. A flat host |
| 59 | whose generation path would exceed that has no generation path: it is |
| 60 | handled exactly as a foreign row is today, keeping its canonical socket. |
| 61 | This adds no new member-facing state and no new copy. The window is small, |
| 62 | because the flag-off path `/run/carlos/<host>.sock` is already near the |
| 63 | bound for the same host. Nested paths are bounded by the 40-byte instance |
| 64 | identifier and stay well under it. |
| 65 | |
| 66 | The bound is measured on the host alone: the control socket |
| 67 | `/run/carlos/g/<n>/<host>.sock.control` at the largest generation number |
| 68 | (20 digits). So flat hosts of up to 59 bytes get generation paths, |
| 69 | against 82 bytes for the flag-off control socket. Measuring at the |
| 70 | current number would let a row change treatment as the counter grows, |
| 71 | even in the middle of a swap: an attempt admitted as an overlap could |
| 72 | meet a generation that cannot bind. |
| 73 | 5. **The probe checks it.** `cmd/zddprobe` exits 1, with a clear line on |
| 74 | stderr, before it listens if either of these is wrong: |
| 75 | - the base name of `--socket` is neither `app.sock` nor |
| 76 | `$CARLOS_HOST.sock`; |
| 77 | - `CARLOS_HOST` is empty. |
| 78 | |
| 79 | A dev or flagship probe then fails its health check the way the tenant |
| 80 | did, instead of passing. |
| 81 | |
| 82 | ## Not changed |
| 83 | |
| 84 | - Generation numbering, the generations file, the termination proof, the |
| 85 | scheduler, the canary, and the failure book. |
| 86 | - Flag-off paths. With `CARLOS_SWAP` unset, nothing in this amendment runs. |
| 87 | |
| 88 | ## Before Gate F runs again |
| 89 | |
| 90 | - Release the fix, and roll it to dev and then the flagship. |
| 91 | - Re-run the dev swap under load with the new probe. |
| 92 | - Clear annie's stale failure record. It holds bdn/ips build `2d577ebda` on |
| 93 | canada, which is the live build. Clearing it needs Paul's approval. |
| 94 | |
amadan