| 1 | # Zero-downtime deploys: rev 18 amendment (signed host retries) |
| 2 | |
| 3 | Status: approved by Paul, 2026-10-13. Nothing here is merged, |
| 4 | released or rolled. |
| 5 | |
| 6 | This file amends `2026-10-07-zero-downtime-deploys-design.md` (rev 17), which |
| 7 | lives on the design branch (`platform-design-zero-downtime-deploys`) and is |
| 8 | not edited from this branch. It quotes each line it replaces and gives the |
| 9 | replacement. After sign-off, one copy step puts the text into the spec and |
| 10 | raises the status to rev 18. |
| 11 | |
| 12 | Why a revision at all. Rev 14 dropped a separate signed retry object and let |
| 13 | `carlos promote --retry` be the retry signal. That holds for the app-wide |
| 14 | retry and is unchanged. It never covered `carlos instances retry <host>`, the |
| 15 | one-host retry, which rev 17 describes only as "clears it for that host only" |
| 16 | and which §11 wrongly lists under the re-signed pointer. The built feature |
| 17 | is a console-signed grant, bound to one failure book and to one restriction |
| 18 | state of that book, and the spec has to say so before the flag goes on. |
| 19 | |
| 20 | Built on `zdd-flag-readiness-2`, on top of the grant's first form on |
| 21 | `zdd-flag-readiness` (`d68a1dde`, then `c12c35c3` for the book identity; |
| 22 | the branch is pending merge): |
| 23 | |
| 24 | - the restriction generation and the book rules: `7c7c6141`, `191c12b9` |
| 25 | - the host-retry log, the box's read of it, and the console's report check: |
| 26 | `6e9e84bd`, `9f27cdfa`, `4d8957c1`, `18bde560` |
| 27 | - the console page for a declined box: `6e6ad000`, `65d4e135` |
| 28 | |
| 29 | ## 1. Status line (spec line 3) |
| 30 | |
| 31 | Replace the opening words of the line, "Status: rev 17, 2026-10-07. Rev 17 |
| 32 | adds", with: |
| 33 | |
| 34 | > Status: rev 18, 2026-10-12. Rev 18 makes `carlos instances retry <host>` a |
| 35 | > root-signed grant bound to one failure book and one restriction state, and |
| 36 | > says where it rides and how a box checks it (§8.4, §11). Rev 17 adds |
| 37 | |
| 38 | Replace this sentence (line 3): |
| 39 | |
| 40 | > Rev 14 adopts `carlos promote --retry` from `main` (9e628024) as the retry |
| 41 | > signal, in place of a separate signed retry object. That is a simplification |
| 42 | > found while planning. |
| 43 | |
| 44 | with: |
| 45 | |
| 46 | > Rev 14 adopted `carlos promote --retry` from `main` (9e628024) as the |
| 47 | > app-wide retry signal, in place of a separate signed retry object. That |
| 48 | > stands. Rev 18 adds a signed object back for the one-host retry only, |
| 49 | > because a one-host retry cannot ride a channel pointer (§11). |
| 50 | |
| 51 | The rest of line 3 ("**Accepted by Astra in round 13**" and "Awaiting Paul's |
| 52 | review") stays, with one sentence added after "Awaiting Paul's review": |
| 53 | "Rev 18 has not been through a numbered spec round. The code that builds it |
| 54 | was reviewed task by task on `zdd-flag-readiness-2`, and Astra's whole-branch |
| 55 | review of that code is separate." |
| 56 | |
| 57 | ## 2. §8.4, the one-host retry (spec lines 704 and 705) |
| 58 | |
| 59 | Replace: |
| 60 | |
| 61 | > - **`carlos instances retry <host>`** clears it for that host only, and |
| 62 | > the next attempt runs as a canary. |
| 63 | |
| 64 | with: |
| 65 | |
| 66 | > - **`carlos instances retry <host>`** clears it for that host only, on a |
| 67 | > grant the console signs (§11). On a box that arms the scheduler, the next |
| 68 | > attempt runs as a canary. A box that declined the overlap flag (for |
| 69 | > example with `host-claims-on`) still honours the grant: the failure hold |
| 70 | > is cleared and its next wake starts the build as it does today, with no |
| 71 | > canary, because that box runs none. The console offers the retry only for |
| 72 | > a host whose box reports that it can take the grant, and refuses it with |
| 73 | > a message otherwise (§11). |
| 74 | |
| 75 | Lines 706 onward (`carlos promote --retry <version> <channel>`) are |
| 76 | unchanged. |
| 77 | |
| 78 | ## 3. §11, the retry sentence (spec lines 836 and 837) |
| 79 | |
| 80 | Replace: |
| 81 | |
| 82 | > - A retry is a re-signed channel pointer (`carlos promote --retry`), so one |
| 83 | > box can't forge a retry that clears another box's failure records (§8.4). |
| 84 | |
| 85 | with: |
| 86 | |
| 87 | > - An app-wide retry is a re-signed channel pointer (`carlos promote |
| 88 | > --retry`), so one box can't forge a retry that clears another box's |
| 89 | > failure records (§8.4). A one-host retry (`carlos instances retry |
| 90 | > <host>`) is a grant the console signs with the deployment root key. A box |
| 91 | > clears a failure hold only on such a grant, checked as described below. |
| 92 | |
| 93 | ## 4. §11, new paragraph: the host retry grant |
| 94 | |
| 95 | Add after the sentence above: |
| 96 | |
| 97 | > **The host retry grant.** `carlos instances retry <host>` makes the console |
| 98 | > sign a `HostRetryGrant` and write it to the app's host-retry log. The grant |
| 99 | > is `{v: 1, account, app, host, build, epoch, book, issued_at}`. `build` is |
| 100 | > the build whose restriction the grant lifts, which is not the build the |
| 101 | > host should move to. `epoch` and `book` are what the host's box last |
| 102 | > reported for that restriction (`held_epoch` and `held_book` in its status), |
| 103 | > so the grant names the exact state the member saw. |
| 104 | > |
| 105 | > *Signature.* The envelope is signed with the deployment root key. A scoped |
| 106 | > release key is refused at signing: a channel delegate has no say over |
| 107 | > another instance's failure hold. A box verifies against the root public key |
| 108 | > it already holds for channel pointers, and a box without one drops every |
| 109 | > grant. The log is writable by every box, so an unsigned entry is a request |
| 110 | > anyone with a box could have written, and it clears nothing. |
| 111 | > |
| 112 | > *Time.* A box accepts a grant for 24 hours after `issued_at` and tolerates |
| 113 | > one minute of clock skew into the future. That window is a backstop. What |
| 114 | > binds a grant to one failure is the next two items, not the clock. |
| 115 | > |
| 116 | > *Book and generation.* A box's failure book has a random identity, and it |
| 117 | > keeps one restriction generation per app, a counter that moves on every |
| 118 | > change to any restriction of that app (a record made, changed or removed, a |
| 119 | > host retried, a hold-ran marker set, renewed, replaced or removed). The box |
| 120 | > publishes the generation only once it is on disk, and never hands the same |
| 121 | > value out twice. A grant clears a host's restriction only if the box's book |
| 122 | > is the one the grant names, the book is saved, the app's generation still |
| 123 | > equals the grant's epoch, and the grant's build still restricts the host. |
| 124 | > So a grant clears the state the member saw and nothing that changed after |
| 125 | > it, not even after an agent restart: a reopened book starts past every |
| 126 | > value it ever saved, and a book written by an earlier agent that may have |
| 127 | > published unsaved values is given a new identity when it is adopted, so |
| 128 | > every grant issued before names the old one. Epochs restart per book, which |
| 129 | > is why the grant needs both. A retry spends the grant: it moves the |
| 130 | > generation, so the same grant clears nothing a second time. |
| 131 | > |
| 132 | > The guarantee rests on two assumptions about the book's file that the box |
| 133 | > cannot check from inside. Only an agent that follows this rule writes it |
| 134 | > (the file is written as schema 3, which an earlier agent refuses to open), |
| 135 | > and it is never restored from an older copy. Restoring or cloning an agent |
| 136 | > data dir must give the copy a new identity, by dropping the file's `book` |
| 137 | > key before the agent starts with the flag on. The proof sketch and the |
| 138 | > assumptions are in the doc comment on `FailureBook` |
| 139 | > (`internal/rollout/failures.go`). |
| 140 | > |
| 141 | > *Where it rides.* Grants live in `apps/<account>/<app>/host-retry.json`, a |
| 142 | > CAS-guarded request log of the same shape as the restart-scope log |
| 143 | > (`apps/<account>/<app>/restart-scope.json`, which carries `carlos restart` |
| 144 | > narrowed to a host or an environment), kept to the last 32 entries. They do not ride |
| 145 | > the restart-scope log. A console built before the grant existed |
| 146 | > re-marshals that whole log whenever it appends a restart, and in doing so |
| 147 | > strips the signature from every pending retry, so the member's retry |
| 148 | > silently clears nothing. No released binary reads or writes the host-retry |
| 149 | > log, so no older console can rewrite it and no older box ever sees a grant. |
| 150 | > The restart-scope log no longer accepts a retry kind or a signature when it |
| 151 | > is written. A box still honours a signed grant that an unreleased build left |
| 152 | > in the restart-scope log, and checks it the same way. A box reads the |
| 153 | > host-retry log on the serial move, at the cost of one extra read per app, |
| 154 | > and does not skip the entries it finds at agent start: a grant written |
| 155 | > before an agent restart is checked and, if still current, applied, and a |
| 156 | > spent or stale one is a no-op under the generation rule. |
| 157 | > |
| 158 | > *What a box drops.* An unsigned entry, a forged signature, a grant for |
| 159 | > another account, app or host than its entry names, a grant outside its |
| 160 | > window, one with no book or a zero epoch, one for another book, and one for |
| 161 | > a generation that has moved are each dropped with a log line and clear |
| 162 | > nothing. A box that declined the overlap flag and still keeps a failure |
| 163 | > book honours grants too, with no canary. |
| 164 | > |
| 165 | > *The console.* A console with no signing key answers 503 and writes |
| 166 | > nothing. Before it signs, it reads the box's report for the host and |
| 167 | > refuses, with a message and a 409 on the API, a box whose report carries no |
| 168 | > `retry_v` (a build older than the host-retry log), no `held_book`, or an |
| 169 | > epoch of zero (a book that owes a save). It offers no Retry button for |
| 170 | > those hosts. Nothing is written for a refused retry, so the sequence of box |
| 171 | > and console rolls cannot make a grant apply unsafely: a Retry pressed |
| 172 | > before the box has updated is refused with a message, never lost silently. |
| 173 | > `retry_v` is additive on the wire, and older readers ignore it. |
| 174 | |
| 175 | ## 5. Unchanged |
| 176 | |
| 177 | `carlos promote --retry` stays the app-wide re-signed pointer. The §9.1 list |
| 178 | (spec lines 761 and 762: "`carlos promote --retry` (existing) retries a build |
| 179 | that failed (§8.4)." and "`carlos instances retry <host>` clears one |
| 180 | instance's failure hold.") and the line-998 table row ("§8.4: `carlos deploy |
| 181 | --retry`, and host-scoped clearing") still hold as written. |
| 182 | |
| 183 | ## 6. Rulings this amendment records |
| 184 | |
| 185 | - **Paul, 2026-10-11 (plan 6, Q4).** Host retries would stay on the unsigned |
| 186 | restart-scope log for now, with the trust of `carlos restart`, and signing |
| 187 | them in the console was owed before the flag goes on anywhere hold mode is |
| 188 | used. This amendment is that signing, as built: the console signs, and a |
| 189 | box ignores an unsigned retry. |
| 190 | - **Paul, 2026-10-12 (the crash-epoch hard limit).** A stale grant must never |
| 191 | clear a later failure, including across an agent crash, so a generation is |
| 192 | made durable before it is used and never repeats. The earlier bound (the |
| 193 | sequence advanced by 2^20 on every open, c12c35c3) is replaced by that rule |
| 194 | (`7c7c6141`). The cost accepted: a box whose disk is failing shows a hold |
| 195 | with no Retry until a save lands. |
| 196 | |
| 197 | ## 7. What Paul is asked to sign off |
| 198 | |
| 199 | 1. The §8.4 and §11 text above, including the declined-box sentence (a |
| 200 | declined box clears the hold and runs no canary). |
| 201 | 2. That the one-host retry is a signed object while the app-wide retry stays |
| 202 | a re-signed pointer. |
| 203 | 3. That the hard rule, and the data-dir copy rule that goes with it (drop the |
| 204 | `book` key), are the supported way to keep the guarantee. |
| 205 | |
| 206 | New member-facing strings from the console's retry pages are separate: they |
| 207 | are in `copy-review/strings.json` on the branch and wait for the copy |
| 208 | review. |
| 209 | |
amadan