Zero-downtime deploys: rev 18 amendment (signed host retries)

Status: approved by Paul, 2026-10-13. Nothing here is merged, released or rolled.

This file amends 2026-10-07-zero-downtime-deploys-design.md (rev 17), which lives on the design branch (platform-design-zero-downtime-deploys) and is not edited from this branch. It quotes each line it replaces and gives the replacement. After sign-off, one copy step puts the text into the spec and raises the status to rev 18.

Why a revision at all. Rev 14 dropped a separate signed retry object and let carlos promote --retry be the retry signal. That holds for the app-wide retry and is unchanged. It never covered carlos instances retry <host>, the one-host retry, which rev 17 describes only as "clears it for that host only" and which §11 wrongly lists under the re-signed pointer. The built feature is a console-signed grant, bound to one failure book and to one restriction state of that book, and the spec has to say so before the flag goes on.

Built on zdd-flag-readiness-2, on top of the grant's first form on zdd-flag-readiness (d68a1dde, then c12c35c3 for the book identity; the branch is pending merge):

• the restriction generation and the book rules: 7c7c6141, 191c12b9
• the host-retry log, the box's read of it, and the console's report check: 6e9e84bd, 9f27cdfa, 4d8957c1, 18bde560
• the console page for a declined box: 6e6ad000, 65d4e135

1. Status line (spec line 3)

Replace the opening words of the line, "Status: rev 17, 2026-10-07. Rev 17 adds", with:

> Status: rev 18, 2026-10-12. Rev 18 makes carlos instances retry <host> a > root-signed grant bound to one failure book and one restriction state, and > says where it rides and how a box checks it (§8.4, §11). Rev 17 adds

Replace this sentence (line 3):

> Rev 14 adopts carlos promote --retry from main (9e628024) as the retry > signal, in place of a separate signed retry object. That is a simplification > found while planning.

with:

> Rev 14 adopted carlos promote --retry from main (9e628024) as the > app-wide retry signal, in place of a separate signed retry object. That > stands. Rev 18 adds a signed object back for the one-host retry only, > because a one-host retry cannot ride a channel pointer (§11).

The rest of line 3 ("Accepted by Astra in round 13" and "Awaiting Paul's review") stays, with one sentence added after "Awaiting Paul's review": "Rev 18 has not been through a numbered spec round. The code that builds it was reviewed task by task on zdd-flag-readiness-2, and Astra's whole-branch review of that code is separate."

2. §8.4, the one-host retry (spec lines 704 and 705)

Replace:

> - carlos instances retry <host> clears it for that host only, and > the next attempt runs as a canary.

with:

> - carlos instances retry <host> clears it for that host only, on a > grant the console signs (§11). On a box that arms the scheduler, the next > attempt runs as a canary. A box that declined the overlap flag (for > example with host-claims-on) still honours the grant: the failure hold > is cleared and its next wake starts the build as it does today, with no > canary, because that box runs none. The console offers the retry only for > a host whose box reports that it can take the grant, and refuses it with > a message otherwise (§11).

Lines 706 onward (carlos promote --retry <version> <channel>) are unchanged.

3. §11, the retry sentence (spec lines 836 and 837)

Replace:

> - A retry is a re-signed channel pointer (carlos promote --retry), so one > box can't forge a retry that clears another box's failure records (§8.4).

with:

> - An app-wide retry is a re-signed channel pointer (carlos promote > --retry), so one box can't forge a retry that clears another box's > failure records (§8.4). A one-host retry (carlos instances retry > <host>) is a grant the console signs with the deployment root key. A box > clears a failure hold only on such a grant, checked as described below.

4. §11, new paragraph: the host retry grant

Add after the sentence above:

> The host retry grant. carlos instances retry <host> makes the console > sign a HostRetryGrant and write it to the app's host-retry log. The grant > is {v: 1, account, app, host, build, epoch, book, issued_at}. build is > the build whose restriction the grant lifts, which is not the build the > host should move to. epoch and book are what the host's box last > reported for that restriction (held_epoch and held_book in its status), > so the grant names the exact state the member saw. > > Signature. The envelope is signed with the deployment root key. A scoped > release key is refused at signing: a channel delegate has no say over > another instance's failure hold. A box verifies against the root public key > it already holds for channel pointers, and a box without one drops every > grant. The log is writable by every box, so an unsigned entry is a request > anyone with a box could have written, and it clears nothing. > > Time. A box accepts a grant for 24 hours after issued_at and tolerates > one minute of clock skew into the future. That window is a backstop. What > binds a grant to one failure is the next two items, not the clock. > > Book and generation. A box's failure book has a random identity, and it > keeps one restriction generation per app, a counter that moves on every > change to any restriction of that app (a record made, changed or removed, a > host retried, a hold-ran marker set, renewed, replaced or removed). The box > publishes the generation only once it is on disk, and never hands the same > value out twice. A grant clears a host's restriction only if the box's book > is the one the grant names, the book is saved, the app's generation still > equals the grant's epoch, and the grant's build still restricts the host. > So a grant clears the state the member saw and nothing that changed after > it, not even after an agent restart: a reopened book starts past every > value it ever saved, and a book written by an earlier agent that may have > published unsaved values is given a new identity when it is adopted, so > every grant issued before names the old one. Epochs restart per book, which > is why the grant needs both. A retry spends the grant: it moves the > generation, so the same grant clears nothing a second time. > > The guarantee rests on two assumptions about the book's file that the box > cannot check from inside. Only an agent that follows this rule writes it > (the file is written as schema 3, which an earlier agent refuses to open), > and it is never restored from an older copy. Restoring or cloning an agent > data dir must give the copy a new identity, by dropping the file's book > key before the agent starts with the flag on. The proof sketch and the > assumptions are in the doc comment on FailureBook > (internal/rollout/failures.go). > > Where it rides. Grants live in apps/<account>/<app>/host-retry.json, a > CAS-guarded request log of the same shape as the restart-scope log > (apps/<account>/<app>/restart-scope.json, which carries carlos restart > narrowed to a host or an environment), kept to the last 32 entries. They do not ride > the restart-scope log. A console built before the grant existed > re-marshals that whole log whenever it appends a restart, and in doing so > strips the signature from every pending retry, so the member's retry > silently clears nothing. No released binary reads or writes the host-retry > log, so no older console can rewrite it and no older box ever sees a grant. > The restart-scope log no longer accepts a retry kind or a signature when it > is written. A box still honours a signed grant that an unreleased build left > in the restart-scope log, and checks it the same way. A box reads the > host-retry log on the serial move, at the cost of one extra read per app, > and does not skip the entries it finds at agent start: a grant written > before an agent restart is checked and, if still current, applied, and a > spent or stale one is a no-op under the generation rule. > > What a box drops. An unsigned entry, a forged signature, a grant for > another account, app or host than its entry names, a grant outside its > window, one with no book or a zero epoch, one for another book, and one for > a generation that has moved are each dropped with a log line and clear > nothing. A box that declined the overlap flag and still keeps a failure > book honours grants too, with no canary. > > The console. A console with no signing key answers 503 and writes > nothing. Before it signs, it reads the box's report for the host and > refuses, with a message and a 409 on the API, a box whose report carries no > retry_v (a build older than the host-retry log), no held_book, or an > epoch of zero (a book that owes a save). It offers no Retry button for > those hosts. Nothing is written for a refused retry, so the sequence of box > and console rolls cannot make a grant apply unsafely: a Retry pressed > before the box has updated is refused with a message, never lost silently. > retry_v is additive on the wire, and older readers ignore it.

5. Unchanged

carlos promote --retry stays the app-wide re-signed pointer. The §9.1 list (spec lines 761 and 762: "carlos promote --retry (existing) retries a build that failed (§8.4)." and "carlos instances retry <host> clears one instance's failure hold.") and the line-998 table row ("§8.4: carlos deploy --retry, and host-scoped clearing") still hold as written.

6. Rulings this amendment records

• Paul, 2026-10-11 (plan 6, Q4). Host retries would stay on the unsigned restart-scope log for now, with the trust of carlos restart, and signing them in the console was owed before the flag goes on anywhere hold mode is used. This amendment is that signing, as built: the console signs, and a box ignores an unsigned retry.
• Paul, 2026-10-12 (the crash-epoch hard limit). A stale grant must never clear a later failure, including across an agent crash, so a generation is made durable before it is used and never repeats. The earlier bound (the sequence advanced by 2^20 on every open, c12c35c3) is replaced by that rule (7c7c6141). The cost accepted: a box whose disk is failing shows a hold with no Retry until a save lands.

7. What Paul is asked to sign off

1. The §8.4 and §11 text above, including the declined-box sentence (a declined box clears the hold and runs no canary).
2. That the one-host retry is a signed object while the app-wide retry stays a re-signed pointer.
3. That the hard rule, and the data-dir copy rule that goes with it (drop the book key), are the supported way to keep the guarantee.

New member-facing strings from the console's retry pages are separate: they are in copy-review/strings.json on the branch and wait for the copy review.