amadan

carlos / platform Public

plans reconciled across the set (cross-plan interface ledger in the index); spec rev 16: sleepers acked at delivery with the hibernating marker

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude 6aee9edfd8dd8dcbbee864a65d63be708d65ba00 parent 024b629
8 files changed, +1660 −431
  • docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md +156 −4
  • docs/superpowers/plans/2026-10-07-zero-downtime-deploys-1-generations.md +5 −3
  • docs/superpowers/plans/2026-10-07-zero-downtime-deploys-2-containment.md +165 −46
  • docs/superpowers/plans/2026-10-07-zero-downtime-deploys-3-swap.md +245 −25
  • docs/superpowers/plans/2026-10-07-zero-downtime-deploys-4-release-side.md +6 −6
  • docs/superpowers/plans/2026-10-07-zero-downtime-deploys-5-scheduler.md +863 −267
  • docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-surfaces.md +215 −78
  • docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md +5 −2

CI passed — run details

diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md
index 017a11f2c..5cd28bc2e 100644
--- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md
+++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md
@@ -5,7 +5,7 @@
> read the spec before both.
**Spec:** `docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md`
-(rev 13, accepted by Astra in round 13, approved by Paul on 2026-10-07).
+(rev 15; rev 13 was accepted by Astra in round 13; approved by Paul on 2026-10-07).
The spec is the authority. Where a plan and the spec disagree, the spec
wins and the plan is the bug. §16 of the spec records every review
finding; none may be reopened.
@@ -20,10 +20,10 @@ names each one). The design branch only holds docs.
| # | Plan | Lands | Depends on |
|---|------|-------|------------|
| 1 | `…-1-generations.md`: generation identity, bounded generation sockets, generation handles, `AppSocket`, every app-socket consumer, the per-connection edge dial, per-socket idle pools, the cache tagged with serving route and generation, the version header from the serving generation | dormant: flag off keeps canonical paths and one generation per entry | none |
-| 2 | `…-2-containment.md`: per-generation child cgroups, parent limits and their raise and restore, per-generation strict reservation, per-child `KillAndEmpty`, parent stop evidence, the nested generation template `carlos-instance-gen@<id>:<n>` and the control service's generation argument | dormant | 1 |
-| 3 | `…-3-swap.md`: the entry's operation set and barrier, `Shutdown` covering waking and swapping entries, admission through `hostwrite` (claims off and on), the overlap swap (§7.5), the `hold` swap (§7.8), canary settle observation, intentional-stop cancellation, scheduled-work rules (§7.10), legacy sidecar binding (§7.5), and exec of an attempt's staged path | dormant: reachable only through `Service.Swap`, which nothing calls until plan 4 | 1, 2 |
+| 2 | `…-2-containment.md`: per-generation child cgroups, parent limits and their raise and restore, per-generation strict reservation, per-child `KillAndEmpty` and `KillGen` (with its `SplitBackend` forward), parent stop evidence, the nested generation template `carlos-instance-gen@<id>:<n>` and the control service's generation argument | dormant | 1 |
+| 3 | `…-3-swap.md`: the entry's operation set and barrier, `Shutdown` covering waking and swapping entries, admission through `hostwrite` (claims off and on), the overlap swap (§7.5), the `hold` swap (§7.8), canary settle observation, intentional-stop cancellation, scheduled-work rules (§7.10), legacy sidecar binding (§7.5), exec of an attempt's staged path, and the claimed wake's health cap and one-time fallback that plan 5 drives (`StartPolicy.HealthBy`, `StartPolicy.Fallback`) | dormant: reachable only through `Service.Swap`, which nothing calls until plan 5 | 1, 2 |
| 4 | `…-4-release-side.md`: the manifest `swap` field, raw-byte staging, the canonical-digest check and refusal to restage referenced copies, retained artifact identity, `SidecarAdoptable`, worker recycling onto the captured serving build, and worker identity versus reported version. The retry signal is `carlos promote --retry` (already on `main`), not a new object | additive; the `swap` field is read by nobody until plan 5 | none (can run in parallel with 1–3) |
-| 5 | `…-5-scheduler.md`: the per-app scheduler, attempts and their identity, cohorts, canary gating, `swap_concurrency`, failure records and their clearing, `GoodVersion` with its digest and demand, bootstrap, version-sweep pins, the first-wake budget and fallback, delivered and active config revisions, restart and config attempts, the converge leg and sweep feeding the scheduler, stage-relay retransmission, events | **this plan wires `CARLOS_SWAP`**. Flag off: today's paths. Flag on: everything | 1, 2, 3, 4 |
+| 5 | `…-5-scheduler.md`: the per-app scheduler, attempts and their identity, cohorts, canary gating, `swap_concurrency`, failure records and their clearing, `GoodVersion` with its digest and demand, bootstrap, version-sweep pins, the first-wake budget and fallback (claims off and on), socketless attempts on the same entry (`ReplaceWorker`), delivered and active config revisions, restart and config attempts, the converge leg and sweep feeding the scheduler, stage-relay retransmission, events | **this plan wires `CARLOS_SWAP`** (`SetGenerations` and `ExecBackend.GenContainment`). Flag off: today's paths. Flag on: everything | 1, 2, 3, 4 |
| 6 | `…-6-surfaces.md`: app swap policy (`swap`, `swap_concurrency`) in console and CLI, delivered with app config; `carlos deploy --swap hold`; `carlos instances retry`; deploy-watch reporting of failures, naming `carlos promote --retry`; deploy-watch completion tied to the watched build; `carlos instances list` showing installed, serving and reason; console instance view; event wording (copy review); docs for the migrations rule; roll notes | additive | 4, 5 |
Plans 1, 2 and 3 can merge without plan 5. While the flag is off nothing
@@ -104,6 +104,7 @@ type SwapResult struct {
Outcome SwapOutcome
Gen Gen // the generation now current (on SwapDone)
Reason string
+ Drain time.Duration // added by plan 3: the retiring generation's drain, for the "swapped" event (spec §9.4)
}
// Swap runs one attempt on an awake entry (spec §7.3–§7.5, §7.8).
@@ -230,3 +231,154 @@ type StagedIdentity struct {
3). It must lie under `/opt/carlos/versions/<acct>/<app>/<version>/`
for that host's own app, resolved without following symlinks, and its
digest must match the attempt's.
+
+## Reconciliation (2026-10-07, after all six plans were written)
+
+The plans were drafted in parallel and guessed some of each other's names.
+These are the rulings that made them agree. Each is applied in the plans
+named; the ledger below records the result.
+
+- **The generation number reaches containment as a parameter.** Plan 1's
+ flat and nested spawns (`startChildAt`, `startUnitAt`, `prepareAt`) take
+ a socket, not a `Gen`. Plan 2 adds a trailing `n uint64` (`gen` for
+ `prepareAt`); `StartGen` passes its `n`, `Start` passes 0. Plan 3's
+ `startChildFrom` and `startUnitFrom` carry it through.
+- **The kill is plan 2's.** `KillGen(ctx, rt, n uint64)` on `ExecBackend`,
+ plus a `SplitBackend` forward in plan 2 Task 10. Plan 5 calls it for a
+ failed first wake; plan 3 does not call it.
+- **A planned wake spawns through plan 3's `SwapBackend.StartStaged`**
+ (under `startPinned`), never through plan 1's `GenBackend.StartGen`,
+ which execs the live link.
+- **There is no `Config.Swap`.** Plan 3's `Swap` is armed by plan 1's
+ `SetGenerations` alone. Plan 5 sets `SetGenerations` and
+ `ExecBackend.GenContainment` from `CARLOS_SWAP=overlap`.
+- **Socketless attempts are same-entry** (spec §6.5, §7.9). Plan 5 Task 11's
+ `Service.ReplaceWorker` respawns through plan 4's `respawnWorkerLocked`,
+ keeping the lease and the replicator, observes 10 s, and on failure
+ respawns the build it replaced (`restart`, `config`), the good build
+ (adoption, `overlap`), or stops the entry (adoption, `hold`, or no good
+ build). Plan 5's `CoalesceRecycle` answers true while an attempt runs
+ for the host, so the sweep never recycles under it.
+- **Claimed wakes get the first-wake budget and fallback** (spec §8.2).
+ Plan 3 Task 10 adds `StartPolicy.HealthBy`, `StartPolicy.Fallback` and
+ the entry's `pinFailed` record, and `startAdmitted` falls back once
+ without repeating `Restore`. Plan 5 Task 9 sets them from its wake plan
+ and reports the outcome through `claimedPlanResult`.
+- **Sleepers' config** (spec §8.1). Config sync keeps today's
+ acknowledgement for an instance with no process, marked `Hibernating`,
+ and the scheduler creates no pending config attempt for it.
+- **The stage relay stops at root's outcome.** A (host, version) leaves
+ the retransmission set once root's stamp names it or root's refusal
+ record (`release.ReadRefusal`, `9e628024`) names it.
+- **The "swapped" event carries the drain time** from plan 3's
+ `SwapResult.Drain` (overlap: the retiring generation's stop; hold: the
+ old generation's stop) or, for a socketless attempt, the old child's
+ retire time. The "cohort halted" event names the box (`Deps.Box`,
+ `host.BoxLabel()`).
+- **The deploy watch hears about installs at once.** Plan 5's `observe`
+ and `retryBuild` call `Manager.changed()`, so a box of sleepers
+ republishes its status within seconds of a new install.
+
+## Cross-plan interface ledger
+
+Every type, function, method or field that one plan defines and another
+plan uses. "Defined" names the plan and task that own it; "Used by" lists
+the consuming plans with task and line (line numbers as of this
+reconciliation; `hdr` is the plan's header tables). Earlier plans win: a
+later plan adapts to the definition here. A later plan that needed a new
+interface had the owning plan define it.
+
+### Plan 1 (generations)
+
+| Interface | Defined | Used by |
+|---|---|---|
+| `type Gen struct{ N uint64; Socket, Control, Build, StagedPath, Digest, Demand string }` | P1 T1 | P2 hdr L61; P3 hdr L62, T1 L130, T4 L2208; P5 hdr L42, T1 L158 |
+| `func SetGenerations(on bool)` | P1 T1 | P3 hdr L34, L62; P5 T1 L168, T14 L7115 |
+| `func nextGen() uint64` | P1 T1 | P3 T4 L2208, T6 L3006, T10 L4735, L4831, T11 L5240, L5263 |
+| `func canonicalGen(rt registry.Route, n uint64, build string) Gen` | P1 T1 | P3 hdr L64 |
+| `func GenSocketPath(host string, n uint64) string` | P1 T1 | P3 hdr L58 |
+| `func nestedGenSocketIn(runDir string, n uint64) string` | P1 T1 | P2 T11 L4081, L4128; P3 T9 L4406, T10 L4902 |
+| `func (b *ExecBackend) flatGenSocket(host string, n uint64) string` | P1 T2 | P3 T1 L567, T10 L4588, L4900 |
+| `func clearOwnGenSocket(sock string) error` | P1 T2 | P3 T1 L568 |
+| `type GenBackend interface{ StartGen(ctx, rt, n uint64, perGen bool) (Proc, Gen, error); NextDueVia(ctx, rt, socket string) (int64, error) }` (on `ExecBackend`, `SplitBackend`) | P1 T2 | P2 hdr L54, T10 L3509; P3 hdr L66 (the wake's live-link spawn, which plan 3 leaves alone) |
+| `func (b *ExecBackend) startChildAt(ctx, rt, sock string, n uint64) (Proc, string, error)` (plan 2 adds `n`) | P1 T2, P2 T10 | P3 T1 L453 (`startChildFrom(ctx, rt, sock string, n uint64, exe string)` forwards) |
+| `func (b *ExecBackend) startUnitAt(ctx, rt, sock string, n uint64) (Proc, string, error)` (plan 2 adds `n`) | P1 T2, P2 T11 | P3 T9 L4400 (`startUnitFrom(ctx, rt, sock string, gen uint64, exe *instancectl.StagedExe)` forwards) |
+| `func (b *ExecBackend) prepareAt(rt, role instancectl.Role, sock string, gen uint64) (…)` (plan 2 adds `gen`) | P1 T2, P2 T11 | P3 T9 L4400 |
+| `func (b *ExecBackend) normalizeSocketMode(rt registry.Route, sock string, p Proc)` | P1 T2 | P3 T1 L624 |
+| `func (b *ExecBackend) listenControl(rt registry.Route, control, envPath string) *edge.ControlListener` | P1 T2 | P3 T8 L3586, L3822 |
+| `actEntry.gen Gen`; `wokenInstance.gen Gen` | P1 T3 | P3 T4 L1799, T6 L3048, T11 L5233; P5 T1 L188–189, T9 L4712, L5030 |
+| `Service.gens bool` (read once in `New`); `func (a *Service) GenerationsOn() bool` | P1 T3 | P3 T2 L706, T4 L1761, T10 L4734, T11 L5236; P5 T1 L184, T9 L4340 |
+| `func (a *Service) AppSocket(host string) (socket string, ok bool)` | P1 T3 | P3 T4 L1791, T12 L5318; P5 T1 L159 |
+| `func (a *Service) ServingGen(host string) (edge.GenInfo, bool)` | P1 T3 | P3 T7 L3247 |
+| `func (a *Service) noteCurrent(host string, n uint64)`; `func (a *Service) genRetired(host string, g Gen)` | P1 T3 | P3 T4 L2320, L2333 |
+| `func (a *Service) OnGenRetired(fn func(host, socket string))` | P1 T3 | P1 only (edge wiring); not to be confused with plan 5's `Service.OnGen` |
+| `func linksCanonical(rt registry.Route, g Gen) bool`; `func swapCanonicalLink(canonical, target string) error` | P1 T4 | P3 T1 L613, L616 |
+
+### Plan 2 (containment)
+
+| Interface | Defined | Used by |
+|---|---|---|
+| `ExecBackend.GenContainment bool` | P2 T10 | P3 T1 L314, T9 L4158 (tests); P5 T1 L185, T14 L7116 (arms it) |
+| `func (b *ExecBackend) ParentLimit(ctx, rt registry.Route, l cgroup.Limit) error` | P2 T10 | P3 T1 L586 (`RaiseGenParent`, `RestoreGenParent`) |
+| `func (b *ExecBackend) KillGen(ctx, rt registry.Route, n uint64) error`; `func (b *SplitBackend) KillGen(ctx, rt registry.Route, n uint64) error` | P2 T10 | P5 T1 L169–170, T9 L4174, L4751 (through `genKiller`) |
+| `func cgroup.OverlapLimit(incumbent, successor cgroup.Limit) cgroup.Limit` | P2 T1 | P3 T1 L592, T10 L5003 |
+| `func (m *cgroup.Manager) PrepareStrictParent(host string, tier cgroup.Limit) (*cgroup.Prepared, error)`; `(*Prepared).SetParentLimit(l)`; `(*Prepared).CloseRemove()` | P2 T3 | P3 T10 L4881, L4987, L4994 |
+| `func spawnClaimedGen(ctx, parent *cgroup.Prepared, n uint64, tier cgroup.Limit, selected hostwrite.Work, identity hostwrite.Identity, build func() *exec.Cmd, before func() error) (*exec.Cmd, *cgroup.Prepared, error)` | P2 T12 | P3 T10 L4708, L4771 |
+| `type GenControl interface{ StartGen, StopGen, ResetGen, Parent }` (plan 3 Task 9 adds `StartGenExe`) | P2 T10 | P3 T9 L3934, L4379 |
+| `func (b *ExecBackend) startRole(ctx, rt, role instancectl.Role, gen uint64) (string, error)` (plan 3 Task 9 adds a trailing `exe *instancectl.StagedExe`) | P2 T11 | P3 T9 L3935, L4163 |
+| `instancectl.Client.StartGen(ctx, host string, n uint64, memoryMax int64) (string, error)`; `Server.ensureGenExe(host, n)`; `Server.refuseStart(w, host, unit, exe func() error)`; `genValues(host, n)` | P2 T9 | P3 T9 L3930, L4337, L4360 |
+
+### Plan 3 (swap)
+
+| Interface | Defined | Used by |
+|---|---|---|
+| `Cause`, `SwapMode`, `Attempt`, `SwapOutcome` and their constants (index shared interfaces) | P3 T1 | P5 T1 L158–164, T5, T8 |
+| `type SwapResult struct{ Outcome SwapOutcome; Gen Gen; Reason string; Drain time.Duration }` | P3 T1 (Drain set in T4, T6) | P5 T1 L163, T8 L4113 (`textSwapped(build, res.Drain)`), T11 L5898 |
+| `func (a *Service) Swap(ctx context.Context, at Attempt, newest func(host string) uint64) SwapResult` | P3 T4 | P5 T8 L2595, L3381 (the `Activator` interface), L4085 |
+| `type SwapBackend interface{ StartStaged(ctx, rt, g Gen) (Gen, Proc, error); RaiseGenParent(ctx, rt) error; RestoreGenParent(ctx, rt) error; GenSwitched(rt, g Gen, p Proc) error }` | P3 T1 | P5 T1 L171, T9 L4223–4259 (`planGenBackend` implements it) |
+| `func (a *Service) swapChecksLocked(e *actEntry, at Attempt, newest func(string) uint64) error`; `errSwapNotAwake`, `errSwapBusy`, `errSwapKind`, `errSwapStopped`; `func abandon(err error) SwapResult` | P3 T3, T4 | P5 T1 L172, T11 L5830 (`ReplaceWorker`) |
+| `actEntry.ops opSet` (`members map[uint64]*genMember`, `genMember.gen`); `addMemberLocked`; `leaveSet` | P3 T2 | P5 T1, T12 L6170, L6366, L6419 |
+| `swapOverlap`'s `g, proc, spawnErr := a.spawnSuccessor(mctx, gb, e, rt, work, m.gen)` | P3 T10 | P5 T12 L6419 (persist point) |
+| `Service.genHealth func(ctx context.Context, g Gen) error` (test seam) | P3 T4 | P5 T1 L186, T9 L4344 |
+| `func observeSettle(ctx context.Context, p Proc, window time.Duration) settleResult`; `settleOK`, `settleExited`, `settleCancelled` | P3 T5 | P5 T1 L174, T9 L4859, T11 L5883 |
+| `actEntry.pin *Gen`; `func (a *Service) startPinned(ctx, rt, pin Gen) (Proc, string, Gen, error)` (plan 5 Task 9 adds `healthBy time.Time`) | P3 T6 | P5 T1 L173, T9 L4160, L5043 |
+| `StartPolicy.Pin *Gen` (T6); `StartPolicy.Gen Gen`, `StartPolicy.Fallback *Gen`, `StartPolicy.HealthBy time.Time`, `actEntry.pinFailed *Gen`, `actEntry.pinFailReason string` (T10) | P3 T6, T10 | P5 T1 L177, L191, T9 L4531, L5075 (`doClaimWake`, `claimedPlanResult`) |
+| `Service.RetiredControl(host, control string) bool`; `schedule.MountGated`, `schedule.RetiredWriteMessage` | P3 T8 | P3 only (wired in `agent.go` by plan 3); plan 6 does not reword the 409 (plan 3 Task 8 reviews it) |
+
+### Plan 4 (release side)
+
+| Interface | Defined | Used by |
+|---|---|---|
+| `release.Manifest.Swap string`; `const DemandHold = "hold"`; `func NormalizeDemand(swap string) string` | P4 T1 | P5 T1 L217; P6 T7 L1738, L1776 |
+| `func CanonicalDemand(ctx, s Store, acct, app, version, platform, stagedPath string) (demand, digest string, err error)`; `ErrStagedDigestMismatch` | P4 T4 | P5 T1 L215, L218, T8 L3403, T14 L7015 |
+| `type StagedIdentity struct{ Version, StagedPath, Digest, Demand string }` | P4 T3 | P5 T8, T9 L4167, T10 L5290, T11 L5583 |
+| `Generations`, `GenerationsPath`, `ReadGenerations`, `WriteGenerations`, `GenerationPins` | P4 T5 | P5 T1 L222–225, T12 L6175–6218, T14 L7036 |
+| `type StagedPaths`; `type PinSource`; `func BoxPins(optDir string, routes []registry.Route) (StagedPaths, error)` | P4 T5 | P5 T1 L226, T12 L6171 (adds `GoodVersion`) |
+| `func SidecarAdoptable(rt registry.Route) bool` | P4 T7 | P5 T1 L219 |
+| `func (b *ExecBackend) StartWorkerStaged(ctx, rt, extra map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error)` (and on `SplitBackend`) | P4 T8 | P4 T9 (`startWorker`) |
+| `actEntry.workerServing release.StagedIdentity`; `func (a *Service) startWorker(ctx, rt, env map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error)` | P4 T9 | P5 T1 L178, L195, T9 L4710, L4801, T11 L5842 |
+| `func (a *Service) respawnWorkerLocked(e *actEntry, stopOld Proc, why string, onto *release.StagedIdentity)` | P4 T9 | P3 T11 L5244 (body only); P5 T1 L179, T11 L5579 (`ReplaceWorker`, retire timing), T12 L6420 |
+| `Service.CoalesceRecycle func(host string) bool` | P4 T9 | P5 T1 L187, T11 L6044 (`Manager.CoalesceRecycle`), T14 L7035 |
+| `func (a *Service) WorkerReportedVersion(host string) string` | P4 T9 | display metadata; no other plan calls it yet |
+
+### Plan 5 (scheduler)
+
+| Interface | Defined | Used by |
+|---|---|---|
+| registry `GoodVersion`, `GoodDigest`, `GoodDemand`, `ServingVersion`; `type GoodBuild`; `Route.Good()`; `SetGoodBuild`; `SetServingVersion` | P5 T2 | P6 T13 L4350 (`ServingVersion`) |
+| `appconfig.Manifest.Swap`, `SwapConcurrency`; `SwapOverlap`, `SwapHold`; `ValidSwap`, `ValidSwapConcurrency`; `rollout.PolicyFromManifest` | P5 T3 | P6 T3 L625, L628, L771, L818 |
+| `func (m *rollout.Manager) RetryHost(host string) bool` | P5 T8 | P6 T11 L3444, L3458, L3544 |
+| event texts `textSwapping(build string, cause activator.Cause)`, `textSwapped(build string, drain time.Duration)`, `textSwapFailed(build, serving, reason string)`, `textSwapDown(build string)`, `textCohortHalted(build, box, reason string)`, `textFallback(failed, good string)`, `textConfigPending(reason string, wait time.Duration)`, `textStagingRefused(build string)` | P5 T8 | P6 T14 L4480–4560 (rewords the bodies) |
+| `func bounceText(worker bool, cause Cause) string` (used by `BounceFor` and `ReplaceWorker`) | P5 T9 | P6 T14 L4471–4480 |
+| `func (a *Service) ReplaceWorker(ctx, at Attempt, fallback *release.StagedIdentity, newest func(host string) uint64) SwapResult` | P5 T11 | P5 T8 L2596, L3382 (`Activator`), L2731 (`fakeAct`), T11 L6078 (`executeWorker`) |
+| `status.RolloutReport`, `HostRollout`, `AppRollout`, `RolloutFailure`; `Attempt*` and `Reason*` words; `Probes.Rollout`, `BoxMetrics.Rollout`; `func reasonWord(text string) string`; `Deps.OnChange` | P5 T13 | P6 T2, T8 L2110–2183, T10 L2779, T11 L3670, L4022, T12 L4064 (through `rolloutView`), T14 L4553 |
+| `func armedRollout() *rollout.Manager` | P5 T14 | P6 T11 L3458, L3627, L3632 |
+
+### On `main` (consumed, not defined by any plan)
+
+| Interface | Used by |
+|---|---|
+| `release.ReadRefusal(bakeDir, host string) (*release.Refusal, error)`, `Refusal.Version` (`9e628024`) | P5 T1 L228, T15 L7237, L7411 |
+| `release.ChannelPointer.PromotedAt`, `release.Channel`, `release.FileSHA256` | P5 T1, T8 |
+| `host.BoxLabel() string` (`internal/host/metrics_loop.go`) | P5 T14 |
+| `actEntry.respawnDone`, `actEntry.workerRespawning`, `procCrashed(p Proc) bool` | P5 T11 |
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-1-generations.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-1-generations.md
index b57463d0e..7f2187765 100644
--- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-1-generations.md
+++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-1-generations.md
@@ -36,13 +36,15 @@ git switch -c zdd-1-generations origin/main
## How plan 5 turns this on
-The switch is `activator.SetGenerations(on bool)` (Task 1). A `Service` reads it once, in `activator.New`, and keeps the answer for its life (`Service.GenerationsOn`). The edge, scheduled delivery and the status loop all ask the Service, never the package variable. So plan 5 adds one line to `runAgent` in `internal/host/agent.go`, before `activator.New` (line 373 today):
+The switch is `activator.SetGenerations(on bool)` (Task 1). A `Service` reads it once, in `activator.New`, and keeps the answer for its life (`Service.GenerationsOn`). The edge, scheduled delivery and the status loop all ask the Service, never the package variable. So plan 5 (Task 14) adds these lines to `runAgent` in `internal/host/agent.go`, before `activator.New` (line 373 today), the second for plan 2's containment:
```go
-activator.SetGenerations(os.Getenv("CARLOS_SWAP") == "overlap")
+swapOn := rollout.Enabled(os.Getenv) // CARLOS_SWAP=overlap, trimmed
+activator.SetGenerations(swapOn)
+eb.GenContainment = swapOn
```
-Nothing else needs wiring: `edge.New` finds the Service's `GenSource` through the `Activator` it is already handed.
+Nothing else in this plan needs wiring: `edge.New` finds the Service's `GenSource` through the `Activator` it is already handed.
## Review Focus
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-2-containment.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-2-containment.md
index db8f2dec0..072a47573 100644
--- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-2-containment.md
+++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-2-containment.md
@@ -8,7 +8,7 @@
**Tech Stack:** Go 1.26, `golang.org/x/sys/unix` (already in go.mod), cgroup v2, systemd 257 unit files embedded with `embed.FS`, the repo's test style (`testing`, no assertion library).
-**Spec:** `docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md` (rev 13). Read §3, §7.1, §7.3, §7.6, §10, §12 ("Containment", "Nested") and §15 before starting. Read the index `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md` too. §16 records every review finding. None may be reopened.
+**Spec:** `docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md` (rev 15). Read §3, §7.1, §7.3, §7.6, §10, §12 ("Containment", "Nested") and §15 before starting. Read the index `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md` too. §16 records every review finding. None may be reopened.
## Global Constraints
@@ -48,15 +48,27 @@ Each is flagged to Paul in the plan's hand-off. A reviewer can reject the owning
## Plan 1 handshake
-This plan reads the generation number at four spawn sites. The index fixes the carrier (`activator.Gen`, field `N`) but not how plan 1 hands it to the exec backend. After cutting the branch, run:
+Plan 1 Task 2 leaves the exec backend with these spawn functions, and no `Gen` value is in scope inside the two bodies:
-```bash
-grep -n "Gen\b\|Gen{" internal/activator/backend_exec.go | head -20
+```go
+func (b *ExecBackend) StartGen(ctx context.Context, rt registry.Route, n uint64, perGen bool) (Proc, Gen, error)
+func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock string) (Proc, string, error) // flat and foreign
+func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock string) (Proc, string, error) // nested
+func (b *ExecBackend) prepare(rt registry.Route, role instancectl.Role) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) // calls prepareAt(rt, role, rt.Socket)
+func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock string) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error)
```
-Expected: `startChild` and `startUnit` each have a `Gen` value in scope (plan 1 binds its generation socket from it). The code below calls that value `g`. Use plan 1's variable name wherever this plan writes `g.N`. If plan 1 left the generation out of the exec backend (for example, it rewrote `rt.Socket` before calling `Start` and passed no `Gen`), stop and raise it on the plan-1 branch description: containment needs `N` at the spawn.
+`Start` calls `startChildAt`/`startUnitAt` with `rt.Socket`. `StartGen` calls them with the generation's socket (or `rt.Socket` when `perGen` is false). Plan 1's `Service.startGen` reaches every instance wake through `StartGen` with a fresh `n` from `nextGen()`, which is never 0.
+
+This plan threads the generation number itself, as a trailing parameter:
+
+- Task 10 adds `n uint64` to `startChildAt`: `startChildAt(ctx, rt, sock string, n uint64)`. Task 11 adds `n uint64` to `startUnitAt`: `startUnitAt(ctx, rt, sock string, n uint64)`. `StartGen` passes its `n`. `Start` passes `0`.
+- Task 11 adds `gen uint64` to `prepareAt`: `prepareAt(rt, role, sock string, gen uint64)`. `prepare` passes `rt.Socket, 0`. `startUnitAt` passes `sock, gen`, where `gen` is `unitGen(n)`.
+- Inside the bodies the number is `n`: `b.placeFor(spawnInstance, n)` in `startChildAt` and `b.unitGen(n)` in `startUnitAt`.
-Where this plan's tests call `b.Start(context.Background(), rt)`, they use the signature `main` has today. If plan 1 changed it, write the call exactly as plan 1's own tests in the same file do. That is the only edit allowed to those test bodies.
+With `GenContainment` off, `placeFor` and `unitGen` ignore `n`, so every spawn is byte for byte plan 1's, whichever `n` arrives. With it on, both refuse `n == 0`, so a bare `Start` of an instance refuses. That is deliberate: under the flag every instance wake goes through `StartGen`, and a spawn that could not name its generation must not land in the parent.
+
+Where this plan's tests call `b.Start(context.Background(), rt)`, they use `Start`'s signature, which plan 1 keeps unchanged.
---
@@ -91,10 +103,11 @@ Where this plan's tests call `b.Start(context.Background(), rt)`, they use the s
| `internal/instancectl/server_gen_test.go` (create) | server and client tests |
| `internal/host/units/carlos-instance-control.service` (modify) | `ReadWritePaths=/run/carlos-control /run/systemd/system` |
| `internal/activator/control.go` (modify) | `GenControl` |
-| `internal/activator/containment.go` (create) | `spawnKind`, `placeFor`, `spawnInCgroupAt`, `genControl`, `unitGen`, `ParentLimit`, `KillGen` |
-| `internal/activator/backend_exec.go` (modify) | `GenContainment`; the instance, sidecar and seed spawn sites; `startUnit` |
+| `internal/activator/containment.go` (create) | `spawnKind`, `placeFor`, `spawnInCgroupAt`, `genControl`, `unitGen`, `ParentLimit`, `KillGen`, `startRole` |
+| `internal/activator/backend_exec.go` (modify) | `GenContainment`; the instance, sidecar and seed spawn sites; `startChildAt` and `startUnitAt` take the generation number `n`; `StartGen` passes it, `Start` passes 0 |
+| `internal/activator/backend_unit.go` (modify) | `SplitBackend.KillGen` |
| `internal/activator/containment_test.go` (create) | tests, and the `fakeGenControl` used by Task 11 |
-| `internal/activator/prepare.go`, `unitproc.go` (modify) | generation-keyed launch file, log socket, exit records and unit |
+| `internal/activator/prepare.go`, `unitproc.go` (modify) | `prepareAt` takes the generation; generation-keyed launch file, log socket, exit records and unit |
| `internal/activator/unitproc_gen_test.go` (create) | tests |
| `internal/activator/claimspawn.go` (modify) | `spawnClaimedGen` |
| `internal/activator/claimspawn_test.go` (modify) | test |
@@ -3488,11 +3501,12 @@ git commit -m "zdd plan 2 task 9: control service starts, stops and sweeps gener
**Files:**
- Create: `internal/activator/containment.go`
- Modify: `internal/activator/control.go` (add `GenControl`)
-- Modify: `internal/activator/backend_exec.go` (`ExecBackend` fields near line 100; `seedChild` ~444-520; `startChild` ~825-997; `startSidecarChild` ~1315-1342)
+- Modify: `internal/activator/backend_exec.go` (`ExecBackend` fields near line 100; `seedChild` ~444-520; plan 1's `Start` and `StartGen`; `startChildAt` ~825-997; `startSidecarChild` ~1315-1342)
+- Modify: `internal/activator/backend_unit.go` (after plan 1's `SplitBackend.NextDueVia`)
- Test: `internal/activator/containment_test.go` (create)
**Interfaces:**
-- Consumes: `cgroup.GenChild`, `RoleChild`, `StartChild`, `SetParentLimit`, `RemoveChild`, `OpenCleanupChild`, `ErrNoCgroup`, `ErrNoDelegation` (Tasks 1-2); `instancectl.Client`'s generation methods (Task 9); plan 1's `Gen` in `startChild` (see "Plan 1 handshake").
+- Consumes: `cgroup.GenChild`, `RoleChild`, `StartChild`, `SetParentLimit`, `RemoveChild`, `OpenCleanupChild`, `ErrNoCgroup`, `ErrNoDelegation` (Tasks 1-2); `instancectl.Client`'s generation methods (Task 9); plan 1's `ExecBackend.Start`, `ExecBackend.StartGen(ctx, rt, n uint64, perGen bool)`, `ExecBackend.startChildAt(ctx, rt, sock string)` (see "Plan 1 handshake"). Existing: `SplitBackend` (fields `Exec`, `Unit`), `registry.BackingUnit`, and the test helpers `nestedBackend`, `fakeControl`, `fakeBackend`.
- Produces:
```go
// ExecBackend.GenContainment bool (false: today; set only by plan 5)
@@ -3504,12 +3518,14 @@ git commit -m "zdd plan 2 task 9: control service starts, stops and sweeps gener
}
func (b *ExecBackend) ParentLimit(ctx context.Context, rt registry.Route, l cgroup.Limit) error
func (b *ExecBackend) KillGen(ctx context.Context, rt registry.Route, n uint64) error
+ func (b *SplitBackend) KillGen(ctx context.Context, rt registry.Route, n uint64) error
func (b *ExecBackend) placeFor(kind spawnKind, n uint64) (child string, ownTier bool, err error)
func (b *ExecBackend) spawnInCgroupAt(rt registry.Route, child string, ownTier bool, build func() *exec.Cmd) (*exec.Cmd, error)
func (b *ExecBackend) genControl() (GenControl, error)
func (b *ExecBackend) unitGen(n uint64) (uint64, error)
+ func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock string, n uint64) (Proc, string, error) // plan 1's, plus n
```
- Plan 3 calls `ParentLimit` with `cgroup.OverlapLimit(...)` before the successor spawns and with the tier once the retiring generation's child is empty, and `KillGen` for a successor that fails health.
+ Plan 1's `Start` now calls `b.startChildAt(ctx, rt, rt.Socket, 0)` and its `StartGen` calls `b.startChildAt(ctx, rt, sock, n)`. Plan 3 calls `ParentLimit` with `cgroup.OverlapLimit(...)` before the successor spawns and with the tier once the retiring generation's child is empty. Plan 5 calls `KillGen(ctx, rt, n)` through `SplitBackend` to kill a failed first-wake build (spec §8.2 step 1). `SplitBackend.KillGen` refuses a unit-backed row, and an exec half without `KillGen`, with an error that wraps `errNoGenContainment`.
- [ ] **Step 1: Write the failing tests**
@@ -3710,12 +3726,41 @@ func TestParentLimitAndKillGenNestedAskTheControlService(t *testing.T) {
t.Errorf("ParentLimit through a Control without generations: %v", err)
}
}
+
+// SplitBackend forwards KillGen to the exec half for an exec row. A
+// unit-backed row, and an exec half that cannot kill a generation, refuse:
+// a caller must never believe a kill it could not make.
+func TestSplitBackendKillGen(t *testing.T) {
+ ctl := &fakeControl{}
+ b, rt := nestedBackend(t, ctl, &fakeUnit{})
+ b.GenContainment = true
+ b.Control = fakeGenControl{ctl}
+ sb := &SplitBackend{Exec: b, Unit: &UnitBackend{}}
+ if err := sb.KillGen(context.Background(), rt, 7); err != nil {
+ t.Fatal(err)
+ }
+ if want := "stopgen h.example.com 7 kill=true"; ctl.joined() != want {
+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)
+ }
+ unitRt := rt
+ unitRt.Backing = registry.BackingUnit
+ if err := sb.KillGen(context.Background(), unitRt, 7); !errors.Is(err, errNoGenContainment) {
+ t.Errorf("KillGen of a unit-backed row: %v", err)
+ }
+ if want := "stopgen h.example.com 7 kill=true"; ctl.joined() != want {
+ t.Errorf("a unit-backed KillGen reached the control service:\n%s", ctl.joined())
+ }
+ plain := &SplitBackend{Exec: &fakeBackend{}, Unit: &UnitBackend{}}
+ if err := plain.KillGen(context.Background(), rt, 7); !errors.Is(err, errNoGenContainment) {
+ t.Errorf("KillGen through an exec half without KillGen: %v", err)
+ }
+}
```
- [ ] **Step 2: Run them to verify they fail**
-Run: `mkdir -p /tmp/k && TMPDIR=/tmp/k go test ./internal/activator -run 'TestPlaceFor|TestSpawnInCgroupAt|TestParentLimit|TestKillGen' -count=1`
-Expected: FAIL to compile with `undefined: spawnKind`, `b.GenContainment undefined`, `b.ParentLimit undefined`.
+Run: `mkdir -p /tmp/k && TMPDIR=/tmp/k go test ./internal/activator -run 'TestPlaceFor|TestSpawnInCgroupAt|TestParentLimit|TestKillGen|TestSplitBackendKillGen' -count=1`
+Expected: FAIL to compile with `undefined: spawnKind`, `b.GenContainment undefined`, `b.ParentLimit undefined`, `sb.KillGen undefined`.
- [ ] **Step 3: Add `GenControl` to `internal/activator/control.go`**
@@ -3909,16 +3954,30 @@ Add to `ExecBackend`, after the `Tier` field:
GenContainment bool
```
-**`startChild`.** Directly after `dbPath := paths.dbPath`, add:
+**`startChildAt` takes the generation number.** Change plan 1's signature
+
+```go
+func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock string) (Proc, string, error) {
+```
+
+to
```go
- child, ownTier, err := b.placeFor(spawnInstance, g.N)
+func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock string, n uint64) (Proc, string, error) {
+```
+
+and add to the end of its doc comment: `n is the generation this spawn is (StartGen's n; 0 from Start). It only places the spawn's cgroup, and only under GenContainment (placeFor).` Update its two callers: in `Start`, `return b.startChildAt(ctx, rt, rt.Socket)` becomes `return b.startChildAt(ctx, rt, rt.Socket, 0)`; in `StartGen`, `proc, ver, err = b.startChildAt(ctx, rt, sock)` becomes `proc, ver, err = b.startChildAt(ctx, rt, sock, n)`.
+
+**`startChildAt` places the spawn.** Directly after `dbPath := paths.dbPath`, add:
+
+```go
+ child, ownTier, err := b.placeFor(spawnInstance, n)
if err != nil {
return nil, "", err
}
```
-where `g` is plan 1's `Gen` (see "Plan 1 handshake"). If a later `err :=` in `startChild` now fails to compile with "no new variables", change that `:=` to `=`. Replace `cmd, err := b.spawnInCgroup(rt, func() *exec.Cmd {` with `cmd, err := b.spawnInCgroupAt(rt, child, ownTier, func() *exec.Cmd {`. In its exit goroutine, replace:
+If a later `err :=` in `startChildAt` now fails to compile with "no new variables", change that `:=` to `=`. Replace `cmd, err := b.spawnInCgroup(rt, func() *exec.Cmd {` with `cmd, err := b.spawnInCgroupAt(rt, child, ownTier, func() *exec.Cmd {`. In its exit goroutine, replace:
```go
b.Cgroups.CleanupHost(host)
@@ -3959,21 +4018,52 @@ Change that line to `cmd, err := b.spawnInCgroupAt(rt, child, false, func() *exe
`StartWorker` is not changed (D4).
-- [ ] **Step 6: Run the activator tests**
+- [ ] **Step 6: `SplitBackend` forwards `KillGen`**
+
+In `internal/activator/backend_unit.go`, directly after plan 1's `func (b *SplitBackend) NextDueVia(...)`, add:
+
+```go
+// KillGen dispatches like Start, for an exec row only (zero-downtime spec
+// §8.2 step 1). A unit-backed row has no generation child to kill: systemd
+// owns the unit tenant's one process tree. An exec half without KillGen (a
+// test double) cannot prove a kill either. Both refuse, wrapping
+// errNoGenContainment, so a caller never believes a kill it could not make.
+func (b *SplitBackend) KillGen(ctx context.Context, rt registry.Route, n uint64) error {
+ if rt.Backing == registry.BackingUnit {
+ return fmt.Errorf("%s: a unit-backed row has no generation %d to kill: %w", rt.Host, n, errNoGenContainment)
+ }
+ k, ok := b.Exec.(interface {
+ KillGen(ctx context.Context, rt registry.Route, n uint64) error
+ })
+ if !ok {
+ return fmt.Errorf("%s: this backend cannot kill generation %d: %w", rt.Host, n, errNoGenContainment)
+ }
+ return k.KillGen(ctx, rt, n)
+}
+```
+
+(`backend_unit.go` already imports `context`, `fmt` and `registry`. The capability is an unnamed interface so the file adds no type name a later plan might also declare.)
+
+- [ ] **Step 7: Run the activator tests**
Run: `mkdir -p /tmp/k && TMPDIR=/tmp/k go test ./internal/activator -count=1`
-Expected: `ok`. `TestSpawnInCgroupAppliesTheTier` and every existing flat-wake test pass unchanged, which is the flag-off proof for the flat spawn sites.
+Expected: `ok`. `TestSpawnInCgroupAppliesTheTier`, plan 1's `TestStartGen*` tests and every existing flat-wake test pass unchanged, which is the flag-off proof for the flat spawn sites. `TestSplitBackendKillGen` passes.
Check that every spawn site is placed:
Run: `grep -n "placeFor(" internal/activator/backend_exec.go`
-Expected: exactly three lines (`spawnInstance`, `spawnSidecar`, `spawnSeed`).
+Expected: exactly three lines (`spawnInstance, n`, `spawnSidecar`, `spawnSeed`).
-- [ ] **Step 7: Gate and commit**
+Check that both callers pass a generation:
+
+Run: `grep -n "startChildAt(" internal/activator/backend_exec.go`
+Expected: three lines: the definition (`sock string, n uint64`), `Start`'s call ending `rt.Socket, 0)` and `StartGen`'s call ending `sock, n)`.
+
+- [ ] **Step 8: Gate and commit**
```bash
gofmt -l internal/activator && go vet ./internal/activator && GOOS=darwin GOARCH=arm64 go build ./... && GOOS=windows GOARCH=amd64 go build ./...
-git add internal/activator/containment.go internal/activator/containment_test.go internal/activator/control.go internal/activator/backend_exec.go
+git add internal/activator/containment.go internal/activator/containment_test.go internal/activator/control.go internal/activator/backend_exec.go internal/activator/backend_unit.go
git commit -m "zdd plan 2 task 10: exec backend places spawns in generation children; ParentLimit, KillGen"
```
@@ -3982,16 +4072,17 @@ git commit -m "zdd plan 2 task 10: exec backend places spawns in generation chil
### Task 11: Nested generation units in the exec backend
**Files:**
-- Modify: `internal/activator/prepare.go` (`prepare` 33-102, `logSocket` 131-136, `writeLaunch` 149-155)
+- Modify: `internal/activator/prepare.go` (plan 1's `prepare` wrapper and `prepareAt` 33-102, `logSocket` 131-136, `writeLaunch` 149-155)
- Modify: `internal/activator/unitproc.go` (struct 31-56, `newUnitProc` 58-63, `Stop` 71-104, `observe` 121-225, `readRecord`/`removeRecord` 271-290)
-- Modify: `internal/activator/backend_exec.go` (`startUnit` ~1066-1135)
+- Modify: `internal/activator/backend_exec.go` (plan 1's `Start` and `StartGen`; `startUnitAt` ~1066-1135)
- Test: `internal/activator/unitproc_gen_test.go` (create)
**Interfaces:**
-- Consumes: `unitGen`, `genControl`, `fakeGenControl` (Task 10); Task 6 names; plan 1's `Gen` in `startUnit`.
-- Produces (the existing names stay as `gen=0` wrappers, so every existing call and test is unchanged):
+- Consumes: `unitGen`, `genControl`, `fakeGenControl` (Task 10); Task 6 names; plan 1's `prepareAt(rt, role, sock string)`, `startUnitAt(ctx, rt, sock string)`, `nestedGenSocketIn(runDir string, n uint64) string`, and `StartGen`'s and `Start`'s calls of `startUnitAt` (see "Plan 1 handshake").
+- Produces (`prepare`, `logSocket`, `writeLaunch` and `newUnitProc` stay as `gen=0` wrappers, so every existing call and test is unchanged; `prepareAt` and `startUnitAt` gain a trailing generation parameter and their callers pass it):
```go
- func (b *ExecBackend) prepareFor(rt registry.Route, role instancectl.Role, gen uint64) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error)
+ func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock string, gen uint64) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error)
+ func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock string, n uint64) (Proc, string, error)
func (b *ExecBackend) logSocketFor(host string, role instancectl.Role, gen uint64) string
func (b *ExecBackend) writeLaunchFor(p instancePaths, role instancectl.Role, gen uint64, host string, argv, env []string) error
func (b *ExecBackend) newUnitProcFor(rt registry.Route, role instancectl.Role, gen uint64, p instancePaths, inv string, ctl *edge.ControlListener, logs *logListener) *unitProc
@@ -4021,7 +4112,7 @@ import (
"github.com/carlosframework/platform/internal/provision"
)
-func TestPrepareForAGenerationKeysItsFilesByGeneration(t *testing.T) {
+func TestPrepareAtAGenerationKeysItsFilesByGeneration(t *testing.T) {
ctl := &fakeControl{}
b, rt := nestedBackend(t, ctl, &fakeUnit{})
b.GenContainment = true
@@ -4034,7 +4125,7 @@ func TestPrepareForAGenerationKeysItsFilesByGeneration(t *testing.T) {
t.Fatal(err)
}
}
- p2, ctlL, logs, _, err := b.prepareFor(rt, instancectl.RoleRun, 7)
+ p2, ctlL, logs, _, err := b.prepareAt(rt, instancectl.RoleRun, nestedGenSocketIn(p.runDir, 7), 7)
if err != nil {
t.Fatal(err)
}
@@ -4209,21 +4300,30 @@ func TestStartUnitWithContainmentOffKeepsTheOriginalTemplate(t *testing.T) {
- [ ] **Step 2: Run them to verify they fail**
-Run: `mkdir -p /tmp/k && TMPDIR=/tmp/k go test ./internal/activator -run 'TestPrepareForAGeneration|TestGenerationUnitProc|TestStartRoleDispatches|TestUnitGen|TestStartUnitWithContainmentOff' -count=1`
-Expected: FAIL to compile with `b.prepareFor undefined`, `b.newUnitProcFor undefined`, `up.gen undefined`.
+Run: `mkdir -p /tmp/k && TMPDIR=/tmp/k go test ./internal/activator -run 'TestPrepareAtAGeneration|TestGenerationUnitProc|TestStartRoleDispatches|TestUnitGen|TestStartUnitWithContainmentOff' -count=1`
+Expected: FAIL to compile with `too many arguments in call to b.prepareAt`, `b.logSocketFor undefined`, `b.writeLaunchFor undefined`, `b.newUnitProcFor undefined`, `up.gen undefined`.
- [ ] **Step 3: Generation-keyed files in `prepare.go`**
-Rename `prepare` to `prepareFor` and add the parameter: `func (b *ExecBackend) prepareFor(rt registry.Route, role instancectl.Role, gen uint64) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) {`. Add above it:
+Add the parameter to plan 1's `prepareAt`. Its signature
```go
-// prepare is prepareFor for the original templates (gen 0).
-func (b *ExecBackend) prepare(rt registry.Route, role instancectl.Role) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) {
- return b.prepareFor(rt, role, 0)
-}
+func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock string) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) {
+```
+
+becomes
+
+```go
+func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock string, gen uint64) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) {
```
-Inside `prepareFor`, change both `b.logSocket(rt.Host, role)` calls to `b.logSocketFor(rt.Host, role, gen)`. Replace the stale-record loop's prefix test:
+and add to the end of its doc comment: `gen is the generation unit being prepared (0 for the original templates): it keys the log socket and the exit records cleared (zero-downtime spec §7.6).` Plan 1's `prepare` wrapper's body `return b.prepareAt(rt, role, rt.Socket)` becomes:
+
+```go
+ return b.prepareAt(rt, role, rt.Socket, 0)
+```
+
+Inside `prepareAt`, change both `b.logSocket(rt.Host, role)` calls to `b.logSocketFor(rt.Host, role, gen)`. Replace the stale-record loop's prefix test:
```go
if strings.HasPrefix(e.Name(), "exit."+string(role)+".") {
@@ -4391,7 +4491,7 @@ func (p *unitProc) removeLaunch() {
}
```
-- [ ] **Step 5: `startUnit` starts a generation under containment**
+- [ ] **Step 5: `startUnitAt` starts a generation under containment**
Add to `internal/activator/containment.go`:
@@ -4412,30 +4512,49 @@ func (b *ExecBackend) startRole(ctx context.Context, rt registry.Route, role ins
(add `"github.com/carlosframework/platform/internal/instancectl"` to its imports).
-In `startUnit` in `backend_exec.go`, directly after the `nestedGate` check, add:
+In `backend_exec.go`, change plan 1's signature
+
+```go
+func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock string) (Proc, string, error) {
+```
+
+to
```go
- gen, err := b.unitGen(g.N)
+func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock string, n uint64) (Proc, string, error) {
+```
+
+and add to the end of its doc comment: `n is the generation this spawn is (StartGen's n; 0 from Start). Under GenContainment it selects the carlos-instance-gen@<id>:<n> unit and its generation-keyed files (unitGen); otherwise it is ignored.` Update its two callers: in `Start`, `return b.startUnitAt(ctx, rt, rt.Socket)` becomes `return b.startUnitAt(ctx, rt, rt.Socket, 0)`; in `StartGen`, `proc, ver, err = b.startUnitAt(ctx, rt, sock)` becomes `proc, ver, err = b.startUnitAt(ctx, rt, sock, n)`.
+
+In `startUnitAt`, directly after the `nestedGate` check, add:
+
+```go
+ gen, err := b.unitGen(n)
if err != nil {
return nil, "", err
}
```
-where `g` is plan 1's `Gen` (see "Plan 1 handshake"). If the next `err :=` then fails with "no new variables", change it to `=`. Then make these four replacements in `startUnit`:
+If the next `err :=` then fails with "no new variables", change it to `=`. Then make these four replacements in `startUnitAt`:
-- `b.prepare(rt, instancectl.RoleRun)` becomes `b.prepareFor(rt, instancectl.RoleRun, gen)`.
+- `p, ctl, logs, fileEnv, err := b.prepareAt(rt, instancectl.RoleRun, sock)` becomes `p, ctl, logs, fileEnv, err := b.prepareAt(rt, instancectl.RoleRun, sock, gen)`.
- `b.writeLaunch(p, instancectl.RoleRun, rt.Host, argv, childEnv(childVars))` becomes `b.writeLaunchFor(p, instancectl.RoleRun, gen, rt.Host, argv, childEnv(childVars))`.
- `inv, err := b.Control.Start(ctx, rt.Host, instancectl.RoleRun, b.tier(rt).MemoryMax)` becomes `inv, err := b.startRole(ctx, rt, instancectl.RoleRun, gen)`.
- `b.newUnitProc(rt, instancectl.RoleRun, p, inv, ctl, logs)` becomes `b.newUnitProcFor(rt, instancectl.RoleRun, gen, p, inv, ctl, logs)`.
-Also replace both `instancectl.UnitName(instancectl.RoleRun, rt.Host)` in `startUnit`'s "exited before it was observed" error and its "started … memory.max" log line with `unitName(instancectl.RoleRun, rt.Host, gen)`. With `gen == 0` the strings are byte for byte today's.
+Also replace both `instancectl.UnitName(instancectl.RoleRun, rt.Host)` in `startUnitAt`'s "exited before it was observed" error and its "started … memory.max" log line with `unitName(instancectl.RoleRun, rt.Host, gen)`. With `gen == 0` the strings are byte for byte today's.
`seedUnit` and `startSidecarUnit` keep the original templates (`prepare`, `writeLaunch`, `newUnitProc`, unchanged).
- [ ] **Step 6: Run the activator tests**
Run: `mkdir -p /tmp/k && TMPDIR=/tmp/k go test ./internal/activator -count=1`
-Expected: `ok`. Every existing nested test (`TestStartUnitWritesTheLaunchFileAndReturnsAUnitProc`, `TestPrepareRemovesAStaleSocketFile`, the stop and observer tests) passes unchanged, which is the flag-off proof for the nested path.
+Expected: `ok`. Every existing nested test (`TestStartUnitWritesTheLaunchFileAndReturnsAUnitProc`, `TestPrepareRemovesAStaleSocketFile`, the stop and observer tests) and plan 1's `TestStartGen*` tests pass unchanged, which is the flag-off proof for the nested path.
+
+Check that every caller passes a generation:
+
+Run: `grep -n "startUnitAt(\|prepareAt(" internal/activator/*.go`
+Expected: the two definitions (`sock string, n uint64` and `sock string, gen uint64`); `Start`'s call ending `rt.Socket, 0)`; `StartGen`'s call ending `sock, n)`; `prepare`'s call ending `rt.Socket, 0)`; `startUnitAt`'s call ending `sock, gen)`; and the test call in `unitproc_gen_test.go`.
- [ ] **Step 7: Gate and commit**
@@ -4564,8 +4683,8 @@ Expected: no output. Nothing outside the activator reads or sets the field.
Run: `grep -rn "GenContainment *[:=]" --include=*.go . | grep -v "_test.go"`
Expected: no output. Nothing outside tests sets it.
-Run: `grep -rn "PrepareStrictParent\|spawnClaimedGen\|\.ParentLimit(\|\.KillGen(" --include=*.go . | grep -v "_test.go" | grep -v "^./internal/cgroup/\|containment.go\|claimspawn.go"`
-Expected: no output. The new entry points are reachable only from tests until plan 3.
+Run: `grep -rn "PrepareStrictParent\|spawnClaimedGen\|\.ParentLimit(\|\.KillGen(" --include=*.go . | grep -v "_test.go" | grep -v "^./internal/cgroup/\|containment.go\|claimspawn.go" | grep -v "backend_unit.go:.*return k\.KillGen(ctx, rt, n)"`
+Expected: no output. The new entry points are reachable only from tests until plans 3 and 5 (`SplitBackend.KillGen`'s forward to the exec half is the one call filtered out).
- [ ] **Step 2: The full gate, serially, at nice 19**
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-3-swap.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-3-swap.md
index 30ddf3364..4ef66f0bb 100644
--- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-3-swap.md
+++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-3-swap.md
@@ -63,9 +63,9 @@ The index fixes `Gen`, `GenSocketPath` and `AppSocket`. Plans 1 and 2 were draft
| C2 | plan 1 T3 | `actEntry.gen Gen`; `AppSocket`, `ServingGen` | The current generation. Both read `e.gen` under `a.mu`, for an awake or stopping entry. |
| C3 | plan 1 T3 | `wokenInstance.gen`; `wake()` copies it into `e.gen` and calls `noteCurrent(host, n)` | When `w.gen.N == 0` for a non-worker (the claim path), `wake()` falls back to `canonicalGen`. Task 10 replaces that fallback for claims-on wakes with a real handle. |
| C4 | plan 1 T3 | `func (a *Service) noteCurrent(host string, n uint64)`, `func (a *Service) genRetired(host string, g Gen)` | `noteCurrent` is the cache-generation bump (a `sync.Map`, safe under `a.mu`). `genRetired` closes the edge's idle connections to `g`, and must be called with no lock held. |
-| C5 | plan 1 T2 | `ExecBackend.flatGenSocket(host, n)`, `nestedGenSocketIn(runDir, n)`, `clearOwnGenSocket(sock)`, `startChildAt(ctx, rt, sock)`, `startUnitAt(ctx, rt, sock)`, `normalizeSocketMode(rt, sock, proc)`, `listenControl(rt, control, envPath)` | Plan 1's own `GenBackend.StartGen(ctx, rt, n, perGen)` is the wake's spawn, and it execs the **live link**. Plan 3 adds a staged-path spawn beside it. |
+| C5 | plan 1 T2, plan 2 T10–T11 | `ExecBackend.flatGenSocket(host, n)`, `nestedGenSocketIn(runDir, n)`, `clearOwnGenSocket(sock)`, `startChildAt(ctx, rt, sock string, n uint64)`, `startUnitAt(ctx, rt, sock string, n uint64)`, `prepareAt(rt, role, sock string, gen uint64)`, `normalizeSocketMode(rt, sock, proc)`, `listenControl(rt, control, envPath)` | Plan 1's own `GenBackend.StartGen(ctx, rt, n, perGen)` is the wake's spawn, and it execs the **live link**. Plan 2 adds the trailing generation number to plan 1's `startChildAt`, `startUnitAt` and `prepareAt` (`Start` passes 0). Plan 3 adds a staged-path spawn beside it. |
| C6 | plan 1 (spec rev 15) | `swapCanonicalLink(canonical, target string) error` | Replaces the canonical path with a symlink to `target`, atomically: a temp link, then rename. |
-| C7 | plan 2 T10 | `ExecBackend.GenContainment`, `ParentLimit(ctx, rt, cgroup.Limit) error`, `KillGen(ctx, rt, n) error`, `cgroup.OverlapLimit(inc, succ)`, `ExecBackend.tier(rt)` | `ParentLimit` answers `errNoGenContainment` when `GenContainment` is off. |
+| C7 | plan 2 T10 | `ExecBackend.GenContainment`, `ParentLimit(ctx, rt, cgroup.Limit) error`, `cgroup.OverlapLimit(inc, succ)`, `ExecBackend.tier(rt)` (on `main`) | `ParentLimit` answers `errNoGenContainment` when `GenContainment` is off. Plan 2's `KillGen(ctx, rt, n uint64)` is not used here: a failed successor is stopped through the operation set. Plan 5 calls `KillGen` for a failed first wake. |
| C8 | plan 2 T9, T11 | `instancectl.Client.StartGen(ctx, host, n, memoryMax)`, `activator.GenControl`, `Server.startGen`, `Server.ensureGenExe(host, n)`, `Server.refuseStart(w, host, unit, exe func() error)`, `genValues(host, n)`, the server's `request` struct, `ExecBackend.startRole(ctx, rt, role, gen)` | The nested generation start. Its executable mount targets the live link until Task 9. |
| C9 | plan 2 T12 | `(*cgroup.Manager).PrepareStrictParent(host, tier)`, `(*cgroup.Prepared).SetParentLimit(l)`, `(*cgroup.Prepared).CloseRemove()`, `spawnClaimedGen(ctx, parent, n, tier, work, identity, build, before)` | Plan 2 leaves the entry's strict parent and its life to plan 3 (Task 10). |
@@ -98,7 +98,7 @@ The cross-plan items this plan owns (index review focus 2 and 4) are Task 12.
| `internal/activator/activator.go` (modify) | `actEntry.ops`, `pin`, `holdNext`; `Service.genHealth`; `wake()`; `doWake` (pinned spawn, sidecar); `stop()` barrier; `finishPendingStops`; `finishStop` (hold, parent); `sweepOnce` sidecar; `Shutdown` (waking); `startSidecar`, `replaceSidecar`; `respawnWorkerLocked` (Task 11) |
| `internal/activator/backend_exec.go` (modify) | `ExecBackend.Retired`; `startChildFrom`; `startSidecarChildFrom`; `MountGated` in the flat spawn |
| `internal/activator/prepare.go` (modify) | `MountGated` in `listenControl` |
-| `internal/activator/claimwake.go` (modify) | `StartPolicy.Pin`, `StartPolicy.Gen`; `ClaimStart.Gen`; the checks in `startAdmitted` |
+| `internal/activator/claimwake.go` (modify) | `StartPolicy.Pin`, `StartPolicy.Gen`, `StartPolicy.Fallback`, `StartPolicy.HealthBy`; `ClaimStart.Gen`; the checks, the health cap and the one-time fallback in `startAdmitted` |
| `internal/instancectl/client.go`, `server.go`, `stagedexe.go` (modify, create) | The staged executable on the nested generation start, and root's validation of it (Task 9) |
| `internal/schedule/control.go` (modify) | `MountGated`, `RetiredWriteMessage` |
| `internal/host/agent.go` (modify) | `eb.Retired = svc.RetiredControl` |
@@ -125,7 +125,7 @@ The cross-plan items this plan owns (index review focus 2 and 4) are Task 12.
type SwapMode string; const ModeOverlap, ModeHold SwapMode
type Attempt struct{ Host string; Cause Cause; Build, StagedPath, Digest, Demand string; Mode SwapMode; ConfigRev, RestartID string; SchedRev uint64; Canary bool }
type SwapOutcome string; const SwapDone, SwapFailed, SwapAbandoned, SwapDown SwapOutcome
- type SwapResult struct{ Outcome SwapOutcome; Gen Gen; Reason string }
+ type SwapResult struct{ Outcome SwapOutcome; Gen Gen; Reason string; Drain time.Duration }
type SwapBackend interface {
StartStaged(ctx context.Context, rt registry.Route, g Gen) (Gen, Proc, error)
RaiseGenParent(ctx context.Context, rt registry.Route) error
@@ -134,7 +134,7 @@ The cross-plan items this plan owns (index review focus 2 and 4) are Task 12.
}
var errSwapNoGens, errUnitNoGens, errSwapLayout, errNestedStaged error
func verifyStaged(path, digest string) error
- func (b *ExecBackend) startChildFrom(ctx context.Context, rt registry.Route, sock, exe string) (Proc, string, error)
+ func (b *ExecBackend) startChildFrom(ctx context.Context, rt registry.Route, sock string, n uint64, exe string) (Proc, string, error)
```
`SwapBackend` is new and separate from plan 1's `GenBackend` (`StartGen(ctx, rt, n, perGen)`, the wake's live-link spawn). Plan 3 never changes plan 1's interface.
@@ -338,6 +338,8 @@ Create `internal/activator/swap.go`:
// generations switch off, it refuses every attempt and spawns nothing.
package activator
+import "time"
+
// Cause is why an attempt exists (spec §6.3).
type Cause string
@@ -388,6 +390,10 @@ type SwapResult struct {
Outcome SwapOutcome
Gen Gen // the generation now current (on SwapDone)
Reason string
+ // Drain is how long the retiring generation took from SIGTERM to its
+ // proven exit (an overlap), or the old generation's stop (a hold). Zero
+ // when nothing retired. The "swapped" event reports it (spec §9.4).
+ Drain time.Duration
}
```
@@ -444,21 +450,22 @@ func verifyStaged(path, digest string) error {
- [ ] **Step 5: Give the flat spawn an executable parameter**
-In `backend_exec.go`, rename plan 1's `startChildAt` body to `startChildFrom`, add an `exe string` parameter, and keep `startChildAt` as a forwarder:
+In `backend_exec.go`, rename the body of `startChildAt` (plan 1's, with plan 2's trailing generation number `n`, which places the spawn in its `g<n>` child) to `startChildFrom`, add an `exe string` parameter after `n`, and keep `startChildAt` as a forwarder:
```go
-func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock string) (Proc, string, error) {
- return b.startChildFrom(ctx, rt, sock, "")
+func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock string, n uint64) (Proc, string, error) {
+ return b.startChildFrom(ctx, rt, sock, n, "")
}
// startChildFrom is the flat spawn with the executable named. Empty is the
// live link, which is every wake. A staged path is a swap's successor or a
// hold's pinned wake (zero-downtime spec §7.5 step 3): the bytes the attempt
-// chose, whatever root does to the live link meanwhile.
-func (b *ExecBackend) startChildFrom(ctx context.Context, rt registry.Route, sock, exe string) (Proc, string, error) {
+// chose, whatever root does to the live link meanwhile. n is the generation
+// number plan 2's containment places the child by.
+func (b *ExecBackend) startChildFrom(ctx context.Context, rt registry.Route, sock string, n uint64, exe string) (Proc, string, error) {
```
-Keep any extra parameter plan 2 added for the generation's cgroup child (for example the generation number). `startChildFrom` takes it too, and `startChildAt` passes it through. Inside the body, make three replacements.
+Inside the body, make three replacements.
The live-link lines:
```go
@@ -561,7 +568,7 @@ func (b *ExecBackend) StartStaged(ctx context.Context, rt registry.Route, g Gen)
if err := clearOwnGenSocket(sock); err != nil {
return g, nil, fmt.Errorf("%s: generation socket: %w", rt.Host, err)
}
- proc, _, err := b.startChildFrom(ctx, rt, sock, g.StagedPath)
+ proc, _, err := b.startChildFrom(ctx, rt, sock, g.N, g.StagedPath)
if err != nil {
return g, proc, err
}
@@ -1588,7 +1595,7 @@ git commit -m "zdd plan 3 task 3: swap admission through hostwrite, claims off a
- Create: `internal/activator/swap_fixture_test.go`, `internal/activator/swap_overlap_test.go`
**Interfaces:**
-- Consumes: Tasks 1–3; C2 (`e.gen`), C4 (`nextGen()`), C7 (`AppSocket`); `waitHealthyCtx`, `spawnHealthTimeout` (`backend_exec.go:1477`, `:36`); `emitEvent`.
+- Consumes: Tasks 1–3; C1 (`nextGen()`), C2 (`e.gen`, `AppSocket`), C4 (`noteCurrent`, `genRetired`); `waitHealthyCtx`, `spawnHealthTimeout` (`backend_exec.go:1477`, `:36`); `emitEvent`.
- Produces:
```go
func (a *Service) Swap(ctx context.Context, at Attempt, newest func(host string) uint64) SwapResult
@@ -1892,6 +1899,22 @@ func TestSwapOverlapSwitchesAfterHealthAndRetiresTheIncumbent(t *testing.T) {
}
}
+// The "swapped" event carries the retiring generation's drain time (spec
+// §9.4), so the result measures it: from SIGTERM to the proven exit.
+func TestSwapOverlapReportsTheDrainTime(t *testing.T) {
+ f := newSwapFixture(t)
+ gate := make(chan struct{})
+ f.b.lastProc().setStopGate(gate)
+ time.AfterFunc(80*time.Millisecond, func() { close(gate) })
+ res := f.a.Swap(context.Background(), f.attempt(ModeOverlap), f.newest)
+ if res.Outcome != SwapDone {
+ t.Fatalf("Swap = %+v", res)
+ }
+ if res.Drain < 50*time.Millisecond || res.Drain > 5*time.Second {
+ t.Fatalf("Drain = %v, want the retiring generation's stop (about 80ms)", res.Drain)
+ }
+}
+
// A successor that fails its health check, fails to spawn, or exits before
// the switch leaves the incumbent current and untouched (spec §8.1).
func TestSwapOverlapFailureKeepsTheIncumbent(t *testing.T) {
@@ -2318,7 +2341,12 @@ func (a *Service) swapOverlap(ctx context.Context, gb SwapBackend, at Attempt, n
a.mu.Unlock()
emitEvent(a.Logs, rt, "info", "instance stopping")
res := SwapResult{Outcome: SwapDone, Gen: m.gen}
- if !a.dropMember(e, retiring) {
+ // The drain is timed on the wall clock (not a.now, which tests fake), for
+ // the "swapped" event's drain time (spec §9.4).
+ drainStart := time.Now()
+ proven := a.dropMember(e, retiring)
+ res.Drain = time.Since(drainStart)
+ if !proven {
res.Reason = "retiring generation not yet proven stopped; the sweep finishes it"
}
return res
@@ -2692,6 +2720,9 @@ func TestSwapWitnessHoldExecsTheStagedBuild(t *testing.T) {
if res.Outcome != SwapDone || res.Gen.Build != "v2" || res.Gen.StagedPath != f.staged {
t.Fatalf("Swap = %+v", res)
}
+ if res.Drain < 40*time.Millisecond {
+ t.Fatalf("Drain = %v, want the old generation's gated stop (spec §9.4)", res.Drain)
+ }
if err := <-served; err != nil {
t.Fatalf("the held request was refused: %v", err)
}
@@ -2841,6 +2872,7 @@ import (
"context"
"errors"
"fmt"
+ "time"
"github.com/carlosframework/platform/internal/registry"
)
@@ -2895,6 +2927,7 @@ func (a *Service) swapHold(ctx context.Context, at Attempt, newest func(string)
newest: newest,
woke: make(chan holdWoke, 1),
}
+ var stopAt time.Time // wall clock: the old generation's stop, for the drain time (spec §9.4)
err = a.admit(ctx, work, func() error {
a.mu.Lock()
defer a.mu.Unlock()
@@ -2909,6 +2942,7 @@ func (a *Service) swapHold(ctx context.Context, at Attempt, newest func(string)
return errSwapBusy
}
e.holdNext = h
+ stopAt = time.Now()
a.beginStopLocked(e)
return nil
})
@@ -2921,6 +2955,7 @@ func (a *Service) swapHold(ctx context.Context, at Attempt, newest func(string)
case <-a.lifeDone():
return abandon(errSwapShutdown)
}
+ drain := time.Since(stopAt) // finishStop answers once the old generation is proven gone
if hw.err != nil {
return abandon(hw.err)
}
@@ -2930,7 +2965,7 @@ func (a *Service) swapHold(ctx context.Context, at Attempt, newest func(string)
a.mu.Unlock()
switch {
case werr == nil:
- return SwapResult{Outcome: SwapDone, Gen: g}
+ return SwapResult{Outcome: SwapDone, Gen: g, Drain: drain}
case errors.Is(werr, ErrFenced) || errors.Is(werr, context.Canceled):
// An intentional stop ended the pinned wake (spec §6.4).
return abandon(errSwapStopped)
@@ -4362,7 +4397,7 @@ Give plan 2's `startRole` a trailing `exe *instancectl.StagedExe` parameter, and
return gc.StartGen(ctx, rt.Host, gen, b.tier(rt).MemoryMax)
```
-Rename plan 1's `startUnitAt` body to `startUnitFrom(ctx, rt, sock, gen, exe)`. Keep `startUnitAt` as a forwarder that passes `exe` nil and the generation number it already has. Pass `exe` to the run role's `startRole` call. The launch file's argv is unchanged: inside the namespace the trampoline still execs `/opt/carlos/live/<host>`, which the generation's executable mount now resolves to the staged build.
+Rename the body of `startUnitAt` (plan 1's, with plan 2's trailing `n uint64`) to `startUnitFrom(ctx context.Context, rt registry.Route, sock string, gen uint64, exe *instancectl.StagedExe) (Proc, string, error)`. Keep `func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock string, n uint64) (Proc, string, error)` as a forwarder: `return b.startUnitFrom(ctx, rt, sock, n, nil)`. Inside the body, plan 2's `gen, err := b.unitGen(n)` becomes `ug, err := b.unitGen(gen)`, and every later use of that result (`prepareAt`'s last argument, `writeLaunchFor`, `startRole`, `newUnitProcFor`, `unitName`) takes `ug`. Pass `exe` to the run role's `startRole` call: `b.startRole(ctx, rt, instancectl.RoleRun, ug, exe)`. The launch file's argv is unchanged: inside the namespace the trampoline still execs `/opt/carlos/live/<host>`, which the generation's executable mount now resolves to the staged build.
In `genbackend.go`, the nested arm of `StartStaged` becomes:
@@ -4421,7 +4456,8 @@ git commit -m "zdd plan 3 task 9: nested generations mount the staged build; roo
Claims-on wakes spawn through the claim runtime's provider (`ClaimStart.Build`), and plan 1 left them on the canonical socket with a numbered canonical handle. This task:
- gives provider spawns a generation socket and a real handle;
- holds the entry's strict cgroup parent (plan 2's `PrepareStrictParent`) for the entry's life and spawns each generation under it with `spawnClaimedGen`;
-- spawns a claims-on swap's successor the same way.
+- spawns a claims-on swap's successor the same way;
+- gives a claimed first wake the same health cap and one-time fallback to the good build as an unclaimed one (spec §8.2): `StartPolicy.HealthBy` and `StartPolicy.Fallback`, which plan 5 Task 9 sets from its wake plan. The work before exec (the admission, `Restore`) runs once and is shared.
`main` has no production provider (`productionClaimFactory` returns `ErrUnavailable`). So a runtime opts in with `ClaimRuntime.Generations`, and without it claims-on swaps are refused before anything stops or spawns.
@@ -4440,6 +4476,9 @@ Claims-on wakes spawn through the claim runtime's provider (`ClaimStart.Build`),
- Produces:
```go
// StartPolicy gains Gen Gen; ClaimStart gains Gen Gen; ClaimRuntime gains Generations bool
+ // StartPolicy gains Fallback *Gen; HealthBy time.Time (a claimed first wake's fallback and health cap, spec §8.2; plan 5 sets them)
+ // actEntry gains pinFailed *Gen; pinFailReason string (the pinned build a claimed start saw fail before it was ready)
+ // Service gains: claimWakeSpawn func(ctx context.Context, s ClaimStart, g Gen, before func() error) (*exec.Cmd, *cgroup.Prepared, error) (test seam)
// opSet gains parent *cgroup.Prepared, parentTier cgroup.Limit
var errClaimsNoGenerations error
func (b *ExecBackend) GenPaths(rt registry.Route, n uint64) (Gen, bool)
@@ -4554,12 +4593,81 @@ func TestClaimGenAt(t *testing.T) {
t.Fatalf("foreign: %+v", g)
}
}
+
+// A claims-on first wake falls back as an unclaimed one does (spec §8.2).
+// The pinned build starts and is not ready by HealthBy. Its child is killed
+// (claimProcess.Stop), and the start is prepared once more, for the good
+// build, inside the same admission. Restore runs once, and the entry records
+// which pinned build failed, for plan 5's events.
+func TestStartAdmittedFallsBackToTheGoodBuildOnce(t *testing.T) {
+ obs := hostclaim.Observed{Host: "x.example", ETag: "one", Claim: &hostclaim.Claim{
+ Holder: &hostclaim.Holder{Location: "instances/x.example/inc", Session: 1, Generation: "one"},
+ Op: hostclaim.Op{Kind: hostclaim.KindServing, Exec: "box"},
+ }}
+ work, _ := (hostwrite.Config{}).Select(context.Background(), obs.Host, "spawn")
+ var (
+ mu sync.Mutex
+ restores, prepares int
+ readyBy []time.Time
+ )
+ a := &Service{cfg: Config{HostWrites: hostwrite.Config{Enabled: true}, Claims: &ClaimRuntime{Box: "box",
+ Read: func(context.Context, string) (hostclaim.Observed, error) { return obs, nil },
+ Prepare: func(_ context.Context, _ registry.Route, _ hostclaim.Observed, p startPolicy) (ClaimStart, error) {
+ mu.Lock()
+ prepares++
+ mu.Unlock()
+ ver := p.Pin.Build
+ return ClaimStart{Host: obs.Host, Work: work, Identity: hostwrite.Identity{Config: "c", Release: "r"}, Version: ver,
+ Restore: func(context.Context) error { mu.Lock(); restores++; mu.Unlock(); return nil },
+ Materialize: func(context.Context) error { return nil }, Build: func() *exec.Cmd { return nil },
+ Ready: func(ctx context.Context, _ *exec.Cmd) error {
+ dl, _ := ctx.Deadline()
+ mu.Lock()
+ readyBy = append(readyBy, dl)
+ mu.Unlock()
+ if ver == "v2" {
+ return errors.New("GET / answered 500")
+ }
+ return nil
+ },
+ Reserve: func(context.Context, int) error { return nil }, Failed: func(context.Context, int) error { return nil },
+ Healthy: func(context.Context, int, *exec.Cmd) error { return nil },
+ Replicate: func(context.Context, string) (Rep, error) { return &fakeRep{}, nil },
+ Publish: func(context.Context, hostclaim.Observed) (hostclaim.Observed, error) { return obs, nil },
+ Retain: func(Proc, Rep, error) {}}, nil
+ }}}}
+ // Never started, so the claim process has nothing to kill and its Stop
+ // succeeds: the kernel half is plan 2's claimspawn kernel test.
+ a.claimWakeSpawn = func(context.Context, ClaimStart, Gen, func() error) (*exec.Cmd, *cgroup.Prepared, error) {
+ return exec.Command("true"), nil, nil
+ }
+ e := &actEntry{host: obs.Host}
+ healthBy := time.Now().Add(2 * time.Second)
+ policy := startPolicy{MaxStarts: 5, Window: 30 * time.Second, Pin: &Gen{Build: "v2"}, Fallback: &Gen{Build: "v1"}, HealthBy: healthBy}
+ if err := a.startAdmitted(context.Background(), e, obs, policy); err != nil {
+ t.Fatalf("the good build did not serve: %v", err)
+ }
+ if restores != 1 || prepares != 2 {
+ t.Fatalf("restores %d, prepares %d: want the work before exec once and one more Prepare for the good build", restores, prepares)
+ }
+ if e.claimResult == nil || e.claimResult.spawnVersion != "v1" {
+ t.Fatalf("serving %+v, want the good v1", e.claimResult)
+ }
+ if e.pinFailed == nil || e.pinFailed.Build != "v2" || e.pinFailReason == "" {
+ t.Fatalf("pinFailed = %+v (%q), want v2 and why", e.pinFailed, e.pinFailReason)
+ }
+ if len(readyBy) != 2 || !readyBy[0].Equal(healthBy) || !readyBy[1].After(healthBy) {
+ t.Fatalf("ready deadlines %v: B's window ends at HealthBy, the good build's does not", readyBy)
+ }
+}
```
+Add `"sync"` and `"github.com/carlosframework/platform/internal/cgroup"` to the file's imports.
+
- [ ] **Step 2: Run the tests to verify they fail**
-Run: `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator -run 'TestClaimsOnSwapNeedsGenerations|TestStartAdmittedRefusesAGenerationMismatch|TestClaimGenAt' -count=1`
-Expected: build failure: `undefined: errClaimsNoGenerations`, `unknown field Gen in struct literal of type ClaimStart`, `a.claimGenAt undefined`.
+Run: `TMPDIR=/tmp/k nice -n 19 go test ./internal/activator -run 'TestClaimsOnSwapNeedsGenerations|TestStartAdmittedRefusesAGenerationMismatch|TestClaimGenAt|TestStartAdmittedFallsBack' -count=1`
+Expected: build failure: `undefined: errClaimsNoGenerations`, `unknown field Gen in struct literal of type ClaimStart`, `a.claimGenAt undefined`, `unknown field Fallback in struct literal of type StartPolicy`.
- [ ] **Step 3: Extend the claim types and `startAdmitted`**
@@ -4570,6 +4678,36 @@ Expected: build failure: `undefined: errClaimsNoGenerations`, `unknown field Gen
// and the sockets the provider must bind (zero-downtime plan 3, Task
// 10). Zero is today's start on the canonical socket.
Gen Gen
+ // Fallback is a first wake's good build (zero-downtime spec §8.2), set
+ // only beside Pin, by plan 5's planner. When the pinned start executes
+ // and is not ready, its child is killed outright (claimProcess.Stop is
+ // cgroup.kill, then wait until empty), and the start is prepared once
+ // more for Fallback inside the same admission. Restore is not repeated.
+ Fallback *Gen
+ // HealthBy, when set, caps the pinned start's Ready window: a first
+ // wake's health check ends at the earlier of Window and HealthBy (spec
+ // §8.2: the waiters' deadline minus 25 s). Fallback's start is not capped.
+ HealthBy time.Time
+```
+
+Add to `actEntry` (`activator.go`, after Task 6's `holdNext`):
+
+```go
+ // pinFailed is the pinned build a claims-on start ran and saw fail
+ // before it was ready, and pinFailReason why (zero-downtime spec §8.2).
+ // startAdmitted writes them under a.mu; plan 5 reports them. Nil when
+ // the pinned build served.
+ pinFailed *Gen
+ pinFailReason string
+```
+
+Add to `Service` (beside `genHealth`):
+
+```go
+ // claimWakeSpawn, when set, replaces a claimed wake's spawn
+ // (spawnClaimedGen or spawnClaimedBefore). It is a test seam: a real one
+ // needs a delegated cgroup tree.
+ claimWakeSpawn func(ctx context.Context, s ClaimStart, g Gen, before func() error) (*exec.Cmd, *cgroup.Prepared, error)
```
Add to `ClaimStart`:
@@ -4620,7 +4758,10 @@ with
prepared *cgroup.Prepared
startErr error
)
- if policy.Gen.N != 0 {
+ switch {
+ case a.claimWakeSpawn != nil:
+ cmd, prepared, startErr = a.claimWakeSpawn(ctx, s, policy.Gen, beforeStart)
+ case policy.Gen.N != 0:
// Generations on: a child of the entry's strict parent (plan 2,
// held by plan 3 for the entry's life).
parent, perr := a.entryParent(e, s)
@@ -4628,11 +4769,90 @@ with
return perr
}
cmd, prepared, startErr = spawnClaimedGen(ctx, parent, policy.Gen.N, s.Limit, s.Work, s.Identity, s.Build, beforeStart)
- } else {
+ default:
cmd, prepared, startErr = spawnClaimedBefore(ctx, true, s.Manager, s.Host, s.Limit, s.Work, s.Identity, s.Build, beforeStart)
}
```
+In the same loop, cap the pinned start's health window. After
+
+```go
+ deadline := time.Now().Add(policy.Window)
+ if policy.Probe {
+ deadline = budget.opened.Add(policy.Window)
+ }
+```
+
+add:
+
+```go
+ if !policy.HealthBy.IsZero() && policy.HealthBy.Before(deadline) {
+ // A first wake's B: health until the waiters' deadline
+ // minus 25 s (zero-downtime spec §8.2).
+ deadline = policy.HealthBy
+ }
+```
+
+Then, in the failure path, replace
+
+```go
+ if err := s.Failed(ctx, attempt); err != nil {
+ return errors.Join(startErr, err)
+ }
+ last = startErr
+```
+
+with:
+
+```go
+ if err := s.Failed(ctx, attempt); err != nil {
+ return errors.Join(startErr, err)
+ }
+ last = startErr
+ if policy.Pin != nil && !ready && cmd != nil {
+ // The pinned build ran and failed before it was ready. Recorded
+ // for plan 5's events (a failed first wake, spec §8.2); a later
+ // ready start of the same pin clears it below.
+ a.mu.Lock()
+ if e.pinFailed == nil {
+ failed := *policy.Pin
+ e.pinFailed, e.pinFailReason = &failed, startErr.Error()
+ }
+ a.mu.Unlock()
+ }
+ if policy.Fallback != nil && !ready && cmd != nil && ctx.Err() == nil && attempt+1 < policy.MaxStarts {
+ // A first wake onto a build this instance has never served
+ // (zero-downtime spec §8.2). p.Stop above killed the failed
+ // child outright. The good build gets what remains, with no
+ // backoff. Restore and the admission ran once and are shared:
+ // Prepare has no side effects of its own.
+ policy.Pin, policy.Fallback, policy.HealthBy = policy.Fallback, nil, time.Time{}
+ if policy.Gen.N != 0 {
+ policy.Gen = a.claimGenAt(e.route, nextGen())
+ }
+ next, perr := r.Prepare(ctx, e.route, admitted, policy)
+ if perr != nil {
+ return errors.Join(startErr, perr)
+ }
+ if !next.valid() || next.Host != e.host || next.Version != policy.Pin.Build ||
+ (policy.Gen.N != 0 && (next.Gen.N != policy.Gen.N || next.Gen.Socket != policy.Gen.Socket || next.Gen.Control != policy.Gen.Control)) {
+ return errors.Join(startErr, hostwrite.ErrUnavailable)
+ }
+ s = next
+ continue
+ }
+```
+
+And in the success arm, directly before `e.claimResult = …`, add:
+
+```go
+ a.mu.Lock()
+ if e.pinFailed != nil && policy.Pin != nil && e.pinFailed.Build == policy.Pin.Build {
+ e.pinFailed, e.pinFailReason = nil, "" // a retry of the same pin served
+ }
+ a.mu.Unlock()
+```
+
And the success line:
```go
@@ -4926,7 +5146,7 @@ The kernel path of a claims-on generation spawn (`spawnClaimedGen` under a real
- [ ] **Step 6: Run the tests to verify they pass**
-Run: `go vet ./internal/activator && TMPDIR=/tmp/k nice -n 19 go test ./internal/activator -run 'TestClaimsOnSwapNeedsGenerations|TestStartAdmittedRefuses|TestClaimGenAt|TestSwapClaimsOn|TestSwapHoldClaimsOn|TestClaim' -count=1`
+Run: `go vet ./internal/activator && TMPDIR=/tmp/k nice -n 19 go test ./internal/activator -run 'TestClaimsOnSwapNeedsGenerations|TestStartAdmittedRefuses|TestStartAdmittedFallsBack|TestClaimGenAt|TestSwapClaimsOn|TestSwapHoldClaimsOn|TestClaim' -count=1`
Expected: `ok`.
- [ ] **Step 7: Run the whole package (generations off: claims-on wakes unchanged)**
@@ -5064,11 +5284,11 @@ with
ctx, cancel := context.WithTimeout(spawnParent, workerSpawnTimeout)
```
-In the success arm (`default:`), after `e.spawnVersion = spawnVer`, add:
+In the success arm (`default:`), after the line that sets `e.spawnVersion` (`e.spawnVersion = spawnVer` on `main`; `e.spawnVersion = ident.Version` once plan 4 Task 9 has merged, whichever lands first), add:
```go
if m != nil {
- e.gen = Gen{N: m.gen.N, Build: spawnVer}
+ e.gen = Gen{N: m.gen.N, Build: e.spawnVersion}
}
```
@@ -5095,7 +5315,7 @@ This task covers index review focus 2, a box rolled in the middle of a swap, and
- Create: `internal/activator/swap_review_test.go`
**Interfaces:**
-- Consumes: Tasks 2–7; C7 (`AppSocket`); `socketClient` (`backend_exec.go:1440`); `spawnHealthTimeout`.
+- Consumes: Tasks 2–7; C2 (`AppSocket`); `socketClient` (`backend_exec.go:1440`); `spawnHealthTimeout`.
- Produces: `fakeRep.setOnClose(func())`. Two witnesses: `TestSwapWitnessAgentRestartMidSwap` and `TestSwapNeverHealthyAppKeepsTheIncumbentServing`.
- [ ] **Step 1: Give `fakeRep` a close hook**
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-4-release-side.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-4-release-side.md
index 6b2c94e22..92d3fe217 100644
--- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-4-release-side.md
+++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-4-release-side.md
@@ -10,7 +10,7 @@
**Tech Stack:** Go (stdlib), the repo's `internal/release`, `internal/activator` and `internal/registry` packages. No new dependencies.
-**Spec:** `docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md` (rev 14). Read §3, §4.2, §7.5 step 3, §7.9, §8.5 and §12 before starting any task, and the index's "Amendments found while planning" section. The plan index, `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md`, fixes the shared names `release.CanonicalDemand` and `release.StagedIdentity`. They are used here exactly as the index spells them.
+**Spec:** `docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md` (rev 15). Read §3, §4.2, §7.5 step 3, §7.9, §8.5 and §12 before starting any task, and the index's "Amendments found while planning" section. The plan index, `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md`, fixes the shared names `release.CanonicalDemand` and `release.StagedIdentity`. They are used here exactly as the index spells them.
**Branch:** `zdd-4-release-side`, cut from `origin/main`. Line numbers below are from `origin/main` at `289043db`. If main has moved, find each edit by the function name given beside the line number.
@@ -2640,7 +2640,7 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>"
**Interfaces:**
- Consumes: `stagedWorkerStarter`, `stagedFake`, `fakeIdentity` (Task 8); `release.StagedIdentity`.
- Produces:
- - `Service.CoalesceRecycle func(host string) bool`: **plan 5's coalescing hook.** It is called under `a.mu` before a crash or `max_run` recycle. True means a queued, admissible `release` attempt will make the spawn, and the recycle is skipped.
+ - `Service.CoalesceRecycle func(host string) bool`: **plan 5's coalescing hook** (plan 5 Task 11's `Manager.CoalesceRecycle`). It is called under `a.mu` before a crash or `max_run` recycle. True means an attempt will make the spawn (a queued `release` attempt admissible now, or one already running for the host), and the recycle is skipped.
- `func (a *Service) WorkerReportedVersion(host string) string`: display metadata for plan 6.
- unexported `actEntry.workerServing release.StagedIdentity`, `wokenInstance.spawnIdentity release.StagedIdentity`, `func (a *Service) startWorker(...) (Proc, release.StagedIdentity, error)`, `func (a *Service) recycleWorkerLocked(e *actEntry, stopOld Proc, why string)`.
- `respawnWorkerLocked(e *actEntry, stopOld Proc, why string, onto *release.StagedIdentity)`.
@@ -2947,9 +2947,9 @@ func (a *Service) WorkerReportedVersion(host string) string {
```go
// CoalesceRecycle, when set, is asked before a worker is recycled for a
- // crash or max_run (spec §7.9). True means a queued release attempt for
- // that host is admissible now and will make the one spawn itself, so the
- // recycle is skipped. The sweep asks again next time if the child is still
+ // crash or max_run (spec §7.9). True means an attempt for that host (a
+ // queued release attempt admissible now, or one already running) will
+ // make the one spawn itself, so the recycle is skipped. The sweep asks again next time if the child is still
// due. Nil, the default and the whole of plan 4, recycles every time. It
// is called with a.mu held: it must answer from its own state, never
// block, and never call back into Service. Set before Start, like OnWake.
@@ -3101,7 +3101,7 @@ Expected: no `FAIL` lines.
rm -rf /tmp/k /var/tmp/ck ~/.cache/zdd-4
cd $W && git push origin zdd-4-release-side
amadan branch describe carlos/platform zdd-4-release-side -body - <<'EOF'
-Zero-downtime deploys, plan 4 (release side). Spec: docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md (rev 14) §4.2, §7.9, §8.5. Plan: docs/superpowers/plans/2026-10-07-zero-downtime-deploys-4-release-side.md.
+Zero-downtime deploys, plan 4 (release side). Spec: docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md (rev 15) §4.2, §7.9, §8.5. Plan: docs/superpowers/plans/2026-10-07-zero-downtime-deploys-4-release-side.md.
Additive: Manifest.Swap, lossless staged siblings, CanonicalDemand, the pin set, PinSource, and the generations file format root reads (/data/zdd/generations.json; plan 5 writes it). Nothing reads swap until plan 5.
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-5-scheduler.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-5-scheduler.md
index 29bded50c..3d235f4b2 100644
--- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-5-scheduler.md
+++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-5-scheduler.md
@@ -4,11 +4,11 @@
**Goal:** Wire `CARLOS_SWAP=overlap`: one rollout scheduler per (account, app) on each box that moves every instance to its installed build through paced, canaried attempts, records failed builds, boots the good build instead of a failed one, and retries config until it is active. With the flag unset, every path on the box runs exactly as it does today.
-**Architecture:** A new package, `internal/rollout`, holds the decisions: per-instance desired state and attempt identity (`queue.go`), cohorts, canaries and `swap_concurrency` (`queue.go`), failure records in the agent's data dir (`failures.go`), config revisions (`config.go`), which build a wake boots (`wakeplan.go`), and a `Manager` that runs one goroutine per app and calls plan 3's `Service.Swap` (`manager.go`, `scheduler.go`). The activator gains the first-wake budget, the fallback and the 10 s observation on top of plan 3's pinned wake (`internal/activator/wakeplan.go`), reached only through hooks the manager sets (`Planner`, `OnGen`, `OnDrift`), and it persists every live or pending generation to plan 4's generations file (`internal/activator/genfile.go`). `internal/host` arms all of it when the flag is on: plan 1's `SetGenerations`, plan 3's `Config.Swap`, the manager, and plan 4's `WithPinSource` on every adopt pass. It points the converge leg, the adopt bounce, the restart leg and config sync at the manager instead of at `Bounce`.
+**Architecture:** A new package, `internal/rollout`, holds the decisions: per-instance desired state and attempt identity (`queue.go`), cohorts, canaries and `swap_concurrency` (`queue.go`), failure records in the agent's data dir (`failures.go`), config revisions (`config.go`), which build a wake boots (`wakeplan.go`), and a `Manager` that runs one goroutine per app and calls plan 3's `Service.Swap` (`manager.go`, `scheduler.go`). The activator gains the first-wake budget, the fallback and the 10 s observation on top of plan 3's pinned wake (`internal/activator/wakeplan.go`), reached only through hooks the manager sets (`Planner`, `OnGen`, `OnDrift`), and it persists every live or pending generation to plan 4's generations file (`internal/activator/genfile.go`). `internal/host` arms all of it when the flag is on: plan 1's `SetGenerations` (which also arms plan 3's `Swap`), plan 2's `ExecBackend.GenContainment`, the manager, and the generations file that plan 4's adopt pass reads. It points the converge leg, the adopt bounce, the restart leg and config sync at the manager instead of at `Bounce`.
-**Tech Stack:** Go 1.26. Consumes plans 1, 3 and 4 by the names their plan documents give (see "The surface this plan consumes"), plan 2's generation kill, `internal/registry`, `internal/appconfig`, `internal/release`, `internal/status`.
+**Tech Stack:** Go 1.26. Consumes plans 1 to 4 by the names their plan documents give (see "The surface this plan consumes", and the index's "Cross-plan interface ledger"), `internal/registry`, `internal/appconfig`, `internal/release`, `internal/status`.
-**Spec:** `docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md` (rev 14): §4.1, §4.2, §6 (all), §7.7, §7.9 (the recycle coalescing), §8 (all; §8.4 is rev 14's `carlos promote --retry` clearing), §9.4, §10, §12 (Scheduler, Config activation, Failure-hold scope, Failure without a repoint, Fallback, Bootstrap, Recovery mode, Lifecycle versus settle, Concurrency). The index is `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md`; its shared interfaces and its "Amendments found while planning" are used here as written.
+**Spec:** `docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md` (rev 15): §4.1, §4.2, §6 (all), §7.7, §7.9 (the recycle coalescing), §8 (all; §8.4 is rev 14's `carlos promote --retry` clearing), §9.4, §10, §12 (Scheduler, Config activation, Failure-hold scope, Failure without a repoint, Fallback, Bootstrap, Recovery mode, Lifecycle versus settle, Concurrency). The index is `docs/superpowers/plans/2026-10-07-zero-downtime-deploys-0-index.md`; its shared interfaces and its "Amendments found while planning" are used here as written.
## Global Constraints
@@ -16,16 +16,17 @@
- **Line numbers** are from `origin/main` at `17362eaf` (2026-10-07). Plans 1 to 4 move them again; find every edit site by the quoted code, and re-read line numbers against the branch point before starting (index amendments).
- **Gate before every commit:** `go build ./...`, `go vet ./...`, `test -z "$(gofmt -l .)"`, and the touched packages' tests with `-count=1`. Use the default Go build cache (`~/.cache/go-build`), never one under `/tmp`. Packages that bind sockets (`internal/activator`, `internal/host`) run with `TMPDIR=/tmp/k` (create it with `mkdir -p /tmp/k`), per the index.
- **Cross-compile:** darwin, windows and linux-arm64 must build. Nothing in this plan adds a syscall; `internal/host` is linked into the CLI.
-- **The flag** (spec §15): `CARLOS_SWAP=overlap` arms everything: `activator.SetGenerations(true)` (plan 1), `Config.Swap = true` (plan 3), the generations file, and the manager. Any other value, including unset, leaves every path as it was: generations off, `Config.Swap` false, `svc.Planner`, `svc.OnGen`, `svc.OnDrift`, `svc.CoalesceRecycle` and `svc.GenerationsFile` unset, the converge leg is `convergeAdopted`, the restart leg calls `Bounce`, config sync calls `Bounce` and acks at delivery.
+- **The flag** (spec §15): `CARLOS_SWAP=overlap` arms everything: `activator.SetGenerations(true)` (plan 1; plan 3's `Swap` refuses every attempt while it is off, and adds no switch of its own), `ExecBackend.GenContainment = true` (plan 2), the generations file, and the manager. Any other value, including unset, leaves every path as it was: generations off, `GenContainment` false, `svc.Planner`, `svc.OnGen`, `svc.OnDrift`, `svc.CoalesceRecycle` and `svc.GenerationsFile` unset, the converge leg is `convergeAdopted`, the restart leg calls `Bounce`, config sync calls `Bounce` and acks at delivery.
- **The generations file** (index amendments, plan 4 Task 5): `/data/zdd/generations.json`, written only through `release.WriteGenerations(release.GenerationsPath(dataDir), g)`, before the agent's first spawn and again on every change to an operation set (a generation spawning, becoming current, retiring or ending). An absent file means no generations; an unreadable one is an incomplete pin set, and root then neither restages nor sweeps that pass.
- **Retry** (spec rev 14 §8.4): a failure record keeps the `PromotedAt` of the channel pointer it failed under, and clears when this box sees a pointer naming the same build with a newer `PromotedAt` (`carlos promote --retry`, on `main` since `9e628024`). There is no signed retry object.
- **App policy** (spec §4.1): `swap` is `overlap` or `hold`, default `overlap`; `swap_concurrency` is an integer ≥ 1, default 1. They live in the app's default config bundle (`apps/<acct>/<app>/config.json`) as `"swap"` and `"swap_concurrency"`, additive to schema 1. Plan 6 adds the doors that write them.
- **Swap mode** (spec §4.2, §6.4): an adoption attempt onto B runs in `hold` if the app policy says `hold` or B's canonical demand says `hold`; an unreadable manifest or uncomparable digest gives `hold`. Recovery runs in `hold` if the **retained** demand or the **current** app policy says `hold`. A policy this box has never managed to read gives `hold`.
- **Canary** (spec §6.2): a cohort's first swap must pass the health check and a **10 s settle**; no other swap in that cohort starts until it settles. After that, up to `swap_concurrency` swaps run at once across the app's cohorts on the box. `swap_concurrency` counts canaries too and is never exceeded.
-- **First-wake budget** (spec §8.2): waiting requests get arrival + **55 s**; B's health check runs until the earlier of exec + **30 s** and deadline − **25 s**; killing B takes up to **2 s** with `cgroup.kill`; the good build gets what remains; replication starts once the serving generation is healthy. An **unexpected** exit within **10 s** of healthy is a failure of the build; an intentional stop is not (spec §6.4).
-- **Socketless** (spec §6.5): ready means still running 10 s after start; a clean exit inside those 10 s is success for a one-shot; any other exit is failure.
+- **First-wake budget** (spec §8.2): waiting requests get arrival + **55 s**; B's health check runs until the earlier of exec + **30 s** and deadline − **25 s**; killing B takes up to **2 s** with `cgroup.kill`; the good build gets what remains; replication starts once the serving generation is healthy. An **unexpected** exit within **10 s** of healthy is a failure of the build; an intentional stop is not (spec §6.4). Claimed wakes (claims on) get the same budget and the same fallback, through plan 3's `StartPolicy.HealthBy` and `StartPolicy.Fallback`.
+- **Socketless** (spec §6.5, §7.9): an attempt on a worker or native sidecar member is stop-then-start on the **same entry**, through `respawnWorkerLocked`, so the lease and the replicator stay put. Ready means still running 10 s after start; a clean exit inside those 10 s is success for a one-shot; any other exit is failure.
- **Config** (spec §8.1): delivered and active revisions per instance; a pending config attempt retries with backoff **from 1 minute up to 30 minutes**; a revision is acknowledged only once active.
-- **Stage relay** (spec §7.7): touched again every `stageRequestEvery` (15 s) while any exec route's channel head is not installed.
+- **Stage relay** (spec §7.7): touched again every `stageRequestEvery` (15 s) while any exec route's channel head is not installed. It stops for a (host, version) once root has recorded an outcome for it: an install stamp (registry `Version`), or a refusal record (`release.ReadRefusal`, on `main` since `9e628024`).
+- **Sleeping instances and config** (spec §8.1): a sleeper gets no pending config attempt. Its next wake starts with the delivered file, so config sync's acknowledgement goes out at delivery, as today, marked `Hibernating` (the `ConfigAck` field unit config sync already sets for a route asleep at sync time).
- **Failure records** (spec §8.4): scope (app, build, box), stored at `<dataDir>/swap-failures.json`, reported in box status; they block only instances that have not served the build.
- **Lock order:** `activator.Service.mu` → `rollout.Manager.mu` → `scheduler.mu` → `FailureBook.mu`. Never call the activator while holding `Manager.mu` or a `scheduler.mu`: `Swap` calls back into `newest`, and `CoalesceRecycle` is called under the activator's lock.
- **Copy:** no user-facing string ships without copy review. The event texts in `internal/rollout/events.go` are drafts with stable internal codes; **plan 6 finalizes the wording through copy review**, and the flag stays off outside dev until plan 6 has merged. No em dashes in any string.
@@ -43,7 +44,7 @@ From the index (exact names):
| `Cause`, `CauseRelease/Install/Retry/Restart/Config` | `activator` | 3 |
| `SwapMode`, `ModeOverlap`, `ModeHold` | `activator` | 3 |
| `Attempt{Host, Cause, Build, StagedPath, Digest, Demand, Mode, ConfigRev, RestartID, SchedRev, Canary}` | `activator` | 3 |
-| `SwapOutcome` (`SwapDone/Failed/Abandoned/Down`), `SwapResult{Outcome, Gen, Reason}` | `activator` | 3 |
+| `SwapOutcome` (`SwapDone/Failed/Abandoned/Down`), `SwapResult{Outcome, Gen, Reason, Drain}` (`Drain time.Duration` is plan 3's added field) | `activator` | 3 |
| `(*Service).Swap(ctx, Attempt, newest func(string) uint64) SwapResult` | `activator` | 3 |
| `CanonicalDemand(ctx, Store, acct, app, version, platform, stagedPath) (demand, digest string, err error)` | `release` | 4 |
| `StagedIdentity{Version, StagedPath, Digest, Demand}` | `release` | 4 |
@@ -51,25 +52,26 @@ From the index (exact names):
| `SidecarAdoptable(registry.Route) bool` | `release` | 4 |
| `GoodVersion`, `GoodDigest`, `GoodDemand`, `ServingVersion` registry columns | `registry` | **5 (this plan)** |
-**From the sibling plans' documents** (plans 1, 3 and 4 are written; plan 2 is not). The index does not fix these names; each plan's own "Produces" block does. Task 1 pins every one at compile time. If Task 1 does not compile, **stop**: a plan merged a different name. Reconcile on amadan with that plan's owner, update this table and the tasks that use the name, then resume. Do not bend the other plans' code to match this table.
+**From the sibling plans' documents.** The index does not fix these names; each plan's own "Produces" block does, and the index's "Cross-plan interface ledger" records them. Task 1 pins every one at compile time. If Task 1 does not compile, **stop**: a plan merged a different name. Reconcile on amadan with that plan's owner, update this table and the tasks that use the name, then resume. Do not bend the other plans' code to match this table.
| Name and signature | Package | From | Used for |
|---|---|---|---|
-| `func SetGenerations(on bool)` | `activator` | plan 1 Task 1 | arming generations (Task 14) |
+| `func SetGenerations(on bool)`; `Service.gens` (read once in `New`) | `activator` | plan 1 Task 1, Task 3 | arming generations, and with them plan 3's `Swap` (Task 14) |
| `actEntry.gen Gen`, `wokenInstance.gen Gen` | `activator` | plan 1 Task 3 | the current generation; Task 9's `Current` reads it |
-| `Config.Swap bool` | `activator` | plan 3 Task 1 | arming `Swap` (Task 14) |
+| `ExecBackend.GenContainment bool`; `func (b *ExecBackend) KillGen(ctx context.Context, rt registry.Route, n uint64) error`; `func (b *SplitBackend) KillGen(ctx context.Context, rt registry.Route, n uint64) error` | `activator` | plan 2 Task 10 | arming containment (Task 14); `cgroup.kill` on a failed first wake's generation child, waited until empty (spec §8.2 step 1, Task 9) |
+| `type SwapBackend interface { StartStaged(ctx, rt, g Gen) (Gen, Proc, error); RaiseGenParent(ctx, rt) error; RestoreGenParent(ctx, rt) error; GenSwitched(rt, g Gen, p Proc) error }` | `activator` | plan 3 Task 1 | the staged spawn under `startPinned`; Task 9's test backend implements it |
+| `func (a *Service) swapChecksLocked(e *actEntry, at Attempt, newest func(string) uint64) error`; `errSwapNotAwake`, `errSwapBusy`, `errSwapStopped` | `activator` | plan 3 Tasks 3, 4 | the §7.4 checks before a socketless attempt (Task 11) |
| `actEntry.pin *Gen`; `func (a *Service) startPinned(ctx context.Context, rt registry.Route, pin Gen) (Proc, string, Gen, error)`; the test seam `Service.genHealth func(context.Context, Gen) error` | `activator` | plan 3 Tasks 4, 6 | every planned wake execs a staged build through it; Task 9 adds a health-deadline parameter |
-| `actEntry.ops opSet` with `members map[uint64]*genMember`, `genMember.gen Gen`; `addMemberLocked`, `leaveSet` | `activator` | plan 3 Task 2 | the generations file (Task 12) |
-| `func observeSettle(ctx context.Context, p Proc, window time.Duration) settleResult`; `settleOK`, `settleExited`, `settleCancelled` | `activator` | plan 3 Task 5 | the first wake's 10 s observation (§6.4), as plan 3 intends |
-| `actEntry.workerServing release.StagedIdentity`; `func (a *Service) startWorker(ctx context.Context, rt registry.Route, env map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error)` | `activator` | plan 4 Task 9 | socketless serving build and first wakes |
+| `StartPolicy.Pin *Gen`, `StartPolicy.Fallback *Gen`, `StartPolicy.HealthBy time.Time`; `actEntry.pinFailed *Gen`, `actEntry.pinFailReason string` | `activator` | plan 3 Tasks 6, 10 | a claimed (claims-on) first wake's budget and fallback (Task 9) |
+| `actEntry.ops opSet` with `members map[uint64]*genMember`, `genMember.gen Gen`; `addMemberLocked`, `leaveSet`; `swapOverlap`'s `a.spawnSuccessor(` call | `activator` | plan 3 Tasks 2, 4, 10 | the generations file (Task 12) |
+| `func observeSettle(ctx context.Context, p Proc, window time.Duration) settleResult`; `settleOK`, `settleExited`, `settleCancelled` | `activator` | plan 3 Task 5 | the first wake's 10 s observation (§6.4), as plan 3 intends, and a socketless attempt's readiness (Task 11) |
+| `actEntry.workerServing release.StagedIdentity`; `func (a *Service) startWorker(ctx context.Context, rt registry.Route, env map[string]string, onto *release.StagedIdentity) (Proc, release.StagedIdentity, error)`; `func (a *Service) respawnWorkerLocked(e *actEntry, stopOld Proc, why string, onto *release.StagedIdentity)` | `activator` | plan 4 Task 9 | socketless serving build, first wakes, and socketless attempts on the same entry (Task 11) |
| `Service.CoalesceRecycle func(host string) bool`, called under `a.mu` before a crash or `max_run` recycle; true skips the recycle | `activator` | plan 4 Task 9 | spec §7.9 coalescing (Task 11) |
| `DemandHold`, `ErrStagedDigestMismatch`; `CanonicalDemand` gives `(DemandHold, digest or "", nil)` when it cannot compare | `release` | plan 4 Task 4 | attempt mode; an empty digest is hashed here with `FileSHA256` |
| `Generations{Schema int; Hosts map[string][]string}`, `GenerationsPath(dataDir string) string`, `ReadGenerations(path string) (*Generations, error)`, `WriteGenerations(path string, g Generations) error`, `GenerationPins(path string, rows func() ([]registry.Route, error)) PinSource` | `release` | plan 4 Task 5 | the generations file (Task 12) |
-| `StagedPaths`, `PinSource`, `BoxPins(optDir string, routes []registry.Route) (StagedPaths, error)`, `WithPinSource(src PinSource) PassOption` | `release` | plan 4 Tasks 5, 6 | `GoodVersion` in the pin set; the generation pin source on every pass (Task 12) |
+| `StagedPaths`, `PinSource`, `BoxPins(optDir string, routes []registry.Route) (StagedPaths, error)` | `release` | plan 4 Task 5 | `GoodVersion` in the pin set (Task 12). Plan 4's adopt pass reads the generations file itself, so this plan passes no `WithPinSource`. |
-**Already on `main`:** `release.ChannelPointer.PromotedAt`, `release.Channel(ctx, s, pub, acct, app, channel) (*ChannelPointer, error)`, `release.FileSHA256(path string) (string, error)` (`internal/release/proof.go:169`), and the refusal records whose clearing pattern §8.4 copies (`internal/release/refusal.go`).
-
-**Assumed, because plan 2 is not yet written.** One name is still a guess, and Task 1 pins it too: `func (b *ExecBackend) KillGen(ctx context.Context, rt registry.Route, g Gen) error`, which runs `cgroup.kill` on generation `g`'s child cgroup and waits until it is empty (spec §8.2 step 1). Task 9 forwards it through `SplitBackend` if plan 2 did not.
+**Already on `main`:** `release.ChannelPointer.PromotedAt`, `release.Channel(ctx, s, pub, acct, app, channel) (*ChannelPointer, error)`, `release.FileSHA256(path string) (string, error)` (`internal/release/proof.go:169`), the refusal records whose clearing pattern §8.4 copies (`internal/release/refusal.go`: `type Refusal struct{ …; Host, App, Version string; … }`, `func ReadRefusal(bakeDir, host string) (*Refusal, error)`, read from the agent's data dir, which is root's bake dir), and `host.BoxLabel() string` (`internal/host/metrics_loop.go`).
## Review Focus
@@ -100,12 +102,13 @@ From the index (exact names):
| `internal/registry/registry.go` (modify) | four columns, `GoodBuild`, `SetGoodBuild`, `SetServingVersion` |
| `internal/appconfig/appconfig.go` (modify) | `Manifest.Swap`, `Manifest.SwapConcurrency`, validators |
| `internal/appconfig/claimsync.go`, `syncexec.go` (modify) | `Callbacks.Delivered` |
-| `internal/activator/wakeplan.go` (create) | `WakePlan`, `GenEvent`, `Current`, budget vars, `spawnPlanned`, `spawnPlannedWorker`, `afterWake`, `watchFirstWake`, `BounceFor`, the `KillGen` forward |
+| `internal/activator/wakeplan.go` (create) | `WakePlan`, `GenEvent`, `Current`, budget vars, `genKiller`, `spawnPlanned`, `spawnPlannedWorker`, `afterWake`, `watchFirstWake`, `claimedPlanResult`, `BounceFor`, `bounceText` |
+| `internal/activator/workerswap.go` (create) | `ReplaceWorker`: a socketless attempt on the same entry (Task 11) |
| `internal/activator/genfile.go` (create) | `generationsLocked`, `persistGenerations`: the generations file writer |
-| `internal/activator/activator.go`, `swap.go`, `claimwake.go` (modify) | Service hooks, entry fields, `begin` deadline, `doWake` planning and step 4, `startPinned`'s health deadline, the claimed wake's pin, sweep drift feed, `unexpectedExit`, the generation-change calls |
+| `internal/activator/activator.go`, `swap.go`, `claimwake.go` (modify) | Service hooks, entry fields, `begin` deadline, `doWake` planning and step 4, `startPinned`'s health deadline, the claimed wake's pin, fallback and health cap, sweep drift feed, `unexpectedExit`, `respawnWorkerLocked`'s retire timing, the generation-change calls |
| `internal/release/pins.go` (modify) | `BoxPins` reads `GoodVersion` |
| `internal/status/snapshot.go` (modify) | `RolloutReport` in the private Metrics block |
-| `internal/host/rollout.go` (create) | `armRollout`, `armedRollout`, `rolloutLegFor`, `feedRestarts`, `rolloutStatusProbe`, `generationPinSource` |
+| `internal/host/rollout.go` (create) | `armRollout`, `armedRollout`, `rolloutLegFor`, `feedRestarts`, `rolloutStatusProbe`, `statusNudge` |
| `internal/host/stagerelay.go` (create) | `stageRelay`, retransmission loop |
| `internal/host/agent.go`, `restart.go`, `versionsweep.go` (modify) | wiring |
@@ -148,8 +151,8 @@ import (
)
// Plan 5 of the zero-downtime deploys work consumes these names. The index
-// fixes the first block; plans 1, 3 and 4 fix the rest in their own
-// documents; KillGen is plan 5's guess at plan 2. If this file does not
+// fixes the first block; plans 1 to 4 fix the rest in their own documents
+// (the index's "Cross-plan interface ledger"). If this file does not
// compile, a plan merged a different name, and plan 5 stops until it is
// reconciled with that plan's owner.
var (
@@ -157,30 +160,40 @@ var (
_ func(*Service, context.Context, Attempt, func(string) uint64) SwapResult = (*Service).Swap
_ = Gen{N: 1, Socket: "", Control: "", Build: "", StagedPath: "", Digest: "", Demand: ""}
_ = Attempt{Host: "", Cause: CauseRelease, Build: "", StagedPath: "", Digest: "", Demand: "", Mode: ModeOverlap, ConfigRev: "", RestartID: "", SchedRev: 0, Canary: false}
- _ = SwapResult{Outcome: SwapDone, Gen: Gen{}, Reason: ""}
+ _ = SwapResult{Outcome: SwapDone, Gen: Gen{}, Reason: "", Drain: time.Second}
_ = []SwapOutcome{SwapDone, SwapFailed, SwapAbandoned, SwapDown}
_ = []Cause{CauseRelease, CauseInstall, CauseRetry, CauseRestart, CauseConfig}
_ = []SwapMode{ModeOverlap, ModeHold}
_ func(bool) = SetGenerations // plan 1
+ _ func(*ExecBackend, context.Context, registry.Route, uint64) error = (*ExecBackend).KillGen // plan 2
+ _ func(*SplitBackend, context.Context, registry.Route, uint64) error = (*SplitBackend).KillGen // plan 2
+ _ SwapBackend = (*SplitBackend)(nil) // plan 3
+ _ func(*Service, *actEntry, Attempt, func(string) uint64) error = (*Service).swapChecksLocked // plan 3
_ func(*Service, context.Context, registry.Route, Gen) (Proc, string, Gen, error) = (*Service).startPinned // plan 3
_ func(context.Context, Proc, time.Duration) settleResult = observeSettle // plan 3
_ = []settleResult{settleOK, settleExited, settleCancelled}
+ _ = []error{errSwapNotAwake, errSwapBusy, errSwapStopped} // plan 3
+ _ = StartPolicy{Pin: &Gen{}, Fallback: &Gen{}, HealthBy: time.Time{}} // plan 3
_ func(*Service, context.Context, registry.Route, map[string]string, *release.StagedIdentity) (Proc, release.StagedIdentity, error) = (*Service).startWorker // plan 4
- _ func(*ExecBackend, context.Context, registry.Route, Gen) error = (*ExecBackend).KillGen // plan 2, assumed
+ _ func(*Service, *actEntry, Proc, string, *release.StagedIdentity) = (*Service).respawnWorkerLocked // plan 4
)
// Fields plan 5 reads or sets.
-func _(a *Service, e *actEntry, w *wokenInstance, m *genMember) {
- a.cfg.Swap = true // plan 3
+func _(a *Service, e *actEntry, w *wokenInstance, m *genMember, eb *ExecBackend) {
+ a.gens = true // plan 1
+ eb.GenContainment = true // plan 2
a.genHealth = func(context.Context, Gen) error { return nil } // plan 3, test seam
- a.CoalesceRecycle = func(string) bool { return false } // plan 4
- var _ Gen = e.gen // plan 1
- var _ Gen = w.gen // plan 1
- var _ *Gen = e.pin // plan 3
- var _ map[uint64]*genMember = e.ops.members // plan 3
- var _ Gen = m.gen // plan 3
- var _ release.StagedIdentity = e.workerServing // plan 4
+ a.CoalesceRecycle = func(string) bool { return false } // plan 4
+ var _ Gen = e.gen // plan 1
+ var _ Gen = w.gen // plan 1
+ var _ *Gen = e.pin // plan 3
+ var _ *Gen = e.pinFailed // plan 3
+ var _ string = e.pinFailReason // plan 3
+ var _ map[uint64]*genMember = e.ops.members // plan 3
+ var _ Gen = m.gen // plan 3
+ var _ release.StagedIdentity = e.workerServing // plan 4
+ var _ chan struct{} = e.respawnDone // main
}
```
@@ -211,9 +224,10 @@ var (
_ func(string, Generations) error = WriteGenerations
_ func(string, func() ([]registry.Route, error)) PinSource = GenerationPins
_ func(string, []registry.Route) (StagedPaths, error) = BoxPins
- _ func(PinSource) PassOption = WithPinSource
_ func(string) (string, error) = FileSHA256
+ _ func(string, string) (*Refusal, error) = ReadRefusal // main, 9e628024
_ = ChannelPointer{}.PromotedAt
+ _ = Refusal{}.Version
)
```
@@ -2261,7 +2275,7 @@ git commit -m "zdd-5 task 6: cohorts, canary gating, swap_concurrency and the co
// appconfig
// Callbacks gains: Delivered func(host, rev string)
// rollout
- type configTrack struct { delivered, active, wakeRev string; fails int; lastErr string; ack *appconfig.ConfigAck; ackRev string; ackWrite func(appconfig.ConfigAck) }
+ type configTrack struct { delivered, active, wakeRev string; sleeping bool; fails int; lastErr string; ack *appconfig.ConfigAck; ackRev string; ackWrite func(appconfig.ConfigAck) }
func configBackoff(n int) time.Duration
func (t *configTrack) deferAck(a appconfig.ConfigAck, write func(appconfig.ConfigAck)) func()
func (t *configTrack) activate(rev string) (func(), bool)
@@ -2315,6 +2329,22 @@ func TestAnAckWaitsForItsRevisionToBeActive(t *testing.T) {
t.Fatal("an untracked route's ack must not wait")
}
}
+
+// Spec §8.1: a sleeping instance needs no attempt, because its next wake
+// starts with the delivered file. Its ack goes out at delivery, as today,
+// marked Hibernating like unit config sync's ack for a sleeping route.
+func TestASleepersAckGoesOutAtDeliveryAsHibernating(t *testing.T) {
+ var got []appconfig.ConfigAck
+ tr := &configTrack{delivered: "c2", active: "c2", sleeping: true}
+ run := tr.deferAck(appconfig.ConfigAck{Host: "a", Restarted: true}, func(a appconfig.ConfigAck) { got = append(got, a) })
+ if run == nil {
+ t.Fatal("a sleeper's ack waited for a wake")
+ }
+ run()
+ if len(got) != 1 || !got[0].Hibernating || got[0].Restarted {
+ t.Fatalf("ack = %+v, want Hibernating and not Restarted", got)
+ }
+}
```
- [ ] **Step 2: Write the failing appconfig test**
@@ -2380,7 +2410,7 @@ Add `slices` and `strings` to `syncexec_test.go`'s imports if they are not there
- [ ] **Step 3: Run them to see them fail**
-Run: `go test ./internal/rollout/ ./internal/appconfig/ -run 'TestConfigBackoff|TestAnAckWaits|TestSyncExecReportsTheDeliveredRevision' -count=1`
+Run: `go test ./internal/rollout/ ./internal/appconfig/ -run 'TestConfigBackoff|TestAnAckWaits|TestASleepersAck|TestSyncExecReportsTheDeliveredRevision' -count=1`
Expected: FAIL, `undefined: configBackoff` and `unknown field Delivered`.
- [ ] **Step 4: Add the delivery report to appconfig**
@@ -2475,8 +2505,12 @@ type configTrack struct {
// runs is at least this one: activating wakeRev can only under-claim,
// which costs one more config attempt, never a missed one.
wakeRev string
- fails int
- lastErr string
+ // sleeping: the instance had no process when delivered was written
+ // (spec §8.1). Its next wake starts with that file, so no attempt is
+ // pending and the ack goes out at once, marked Hibernating.
+ sleeping bool
+ fails int
+ lastErr string
ack *appconfig.ConfigAck
ackRev string
@@ -2496,8 +2530,14 @@ func configBackoff(n int) time.Duration {
// deferAck holds a config ack until the revision it was written for is
// active, and returns a func to run now when it already is. A route config
// sync never reported a delivery for (a shape the scheduler does not
-// restart) acknowledges at once, as before.
+// restart) acknowledges at once, as before. So does a sleeping instance,
+// with today's hibernating acknowledgement: the config landed and nothing
+// woke to run it (ConfigAck.Hibernating, "delivered on wake").
func (t *configTrack) deferAck(a appconfig.ConfigAck, write func(appconfig.ConfigAck)) func() {
+ if t.sleeping {
+ a.Hibernating, a.Restarted = true, false
+ return func() { write(a) }
+ }
if t.delivered == "" || t.active == t.delivered {
return func() { write(a) }
}
@@ -2553,15 +2593,14 @@ git commit -m "zdd-5 task 7: delivered and active config revisions; the delivery
Current(host string) (activator.Gen, bool)
Owns(host string) bool
Swap(ctx context.Context, a activator.Attempt, newest func(host string) uint64) activator.SwapResult
- BounceFor(host string, cause activator.Cause) bool
- WakeWorker(rt registry.Route) error
+ ReplaceWorker(ctx context.Context, a activator.Attempt, fallback *release.StagedIdentity, newest func(host string) uint64) activator.SwapResult
}
type Routes interface {
Get(host string) (*registry.Route, error)
SetGoodBuild(host string, expect, next registry.GoodBuild) error
SetServingVersion(host, ver string) error
}
- type Deps struct { Act Activator; Routes Routes; Book *FailureBook; OptDir, DataDir string; Demand func(ctx context.Context, acct, app, version, stagedPath string) (string, string, error); Policy func(ctx context.Context, acct, app string) (*appconfig.Manifest, error); Pointer func(ctx context.Context, acct, app, channel string) (*release.ChannelPointer, error); Events activator.EventSink; Touch func(); Now func() time.Time; Logf func(string, ...any) }
+ type Deps struct { Act Activator; Routes Routes; Book *FailureBook; OptDir, DataDir, Box string; Demand func(ctx context.Context, acct, app, version, stagedPath string) (string, string, error); Policy func(ctx context.Context, acct, app string) (*appconfig.Manifest, error); Pointer func(ctx context.Context, acct, app, channel string) (*release.ChannelPointer, error); Events activator.EventSink; Touch func(); Now func() time.Time; Logf func(string, ...any) }
func NewManager(ctx context.Context, d Deps) *Manager
func (m *Manager) Pass(routes []registry.Route)
func (m *Manager) Bootstrap(routes []registry.Route)
@@ -2574,7 +2613,7 @@ git commit -m "zdd-5 task 7: delivered and active config revisions; the delivery
const pointerReadEvery = 30 * time.Second
type EventCode string // EvSwapping, EvSwapped, EvSwapFailed, EvSwapDown, EvCohortHalted, EvFallbackServed, EvConfigPending, EvStagingRefused
```
- `*activator.Service` satisfies `Activator` once Task 9 adds `Current` and `BounceFor`; `*registry.Registry` satisfies `Routes`.
+ `*activator.Service` satisfies `Activator` once Task 9 adds `Current` and Task 11 adds `ReplaceWorker`; `*registry.Registry` satisfies `Routes`. The event texts (drafts, plan 6 Task 14 rewords them) are `textSwapping(build string, cause activator.Cause)`, `textSwapped(build string, drain time.Duration)`, `textSwapFailed(build, serving, reason string)`, `textSwapDown(build string)`, `textCohortHalted(build, box, reason string)`, `textFallback(failed, good string)`, `textConfigPending(reason string, wait time.Duration)`, `textStagingRefused(build string)`, all returning `string`.
- [ ] **Step 1: Write the fakes**
@@ -2604,6 +2643,11 @@ type swapCall struct {
outcome activator.SwapOutcome
}
+type replaceCall struct {
+ a activator.Attempt
+ fallback *release.StagedIdentity
+}
+
// fakeAct models plan 3's Swap faithfully on the one point this plan relies
// on: the re-check before the switch. A superseded attempt never switches.
type fakeAct struct {
@@ -2617,8 +2661,8 @@ type fakeAct struct {
calls []swapCall
inFlight int
maxFlight int
- bounced []string
- wake func(rt registry.Route) error
+ replaced []replaceCall
+ worker func(a activator.Attempt, fallback *release.StagedIdentity) activator.SwapResult
}
func newFakeAct() *fakeAct {
@@ -2684,20 +2728,42 @@ func (f *fakeAct) Swap(_ context.Context, a activator.Attempt, newest func(strin
return activator.SwapResult{Outcome: out, Gen: g}
}
-func (f *fakeAct) BounceFor(host string, _ activator.Cause) bool {
+// ReplaceWorker models plan 5's same-entry socketless attempt (Task 11):
+// worker decides the outcome, and every call is recorded with the fallback
+// it was handed. The default is a clean replacement onto the attempt's build.
+func (f *fakeAct) ReplaceWorker(_ context.Context, a activator.Attempt, fallback *release.StagedIdentity, newest func(string) uint64) activator.SwapResult {
+ f.mu.Lock()
+ f.replaced = append(f.replaced, replaceCall{a: a, fallback: fallback})
+ decide := f.worker
+ f.mu.Unlock()
+ if newest(a.Host) != a.SchedRev {
+ return activator.SwapResult{Outcome: activator.SwapAbandoned, Reason: "superseded"}
+ }
+ res := activator.SwapResult{Outcome: activator.SwapDone}
+ if decide != nil {
+ res = decide(a, fallback)
+ }
f.mu.Lock()
defer f.mu.Unlock()
- f.bounced = append(f.bounced, host)
- _, ok := f.gens[host]
- delete(f.gens, host)
- return ok
+ switch {
+ case res.Outcome == activator.SwapDone:
+ f.n++
+ res.Gen = activator.Gen{N: f.n, Build: a.Build, StagedPath: a.StagedPath, Digest: a.Digest, Demand: a.Demand}
+ f.gens[a.Host] = res.Gen
+ case res.Outcome == activator.SwapFailed && fallback != nil:
+ f.n++
+ res.Gen = activator.Gen{N: f.n, Build: fallback.Version, StagedPath: fallback.StagedPath, Digest: fallback.Digest, Demand: fallback.Demand}
+ f.gens[a.Host] = res.Gen
+ case res.Outcome == activator.SwapDown:
+ delete(f.gens, a.Host)
+ }
+ return res
}
-func (f *fakeAct) WakeWorker(rt registry.Route) error {
- if f.wake == nil {
- return errors.New("no worker wake in this test")
- }
- return f.wake(rt)
+func (f *fakeAct) replacements() []replaceCall {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return append([]replaceCall(nil), f.replaced...)
}
func (f *fakeAct) snapshot() ([]swapCall, []string, int) {
@@ -2825,7 +2891,7 @@ func newHarness(t *testing.T, hosts ...string) *harness {
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
h.m = NewManager(ctx, Deps{
- Act: h.act, Routes: h.rows, Book: h.book, OptDir: h.opt, DataDir: h.data,
+ Act: h.act, Routes: h.rows, Book: h.book, OptDir: h.opt, DataDir: h.data, Box: "box-1",
Demand: h.demand, Policy: h.policy, Pointer: h.pointerFor, Events: h.events,
Touch: func() { h.touched.Add(1) }, Now: h.clock, Logf: func(string, ...any) {},
})
@@ -3253,7 +3319,10 @@ func textSwapping(build string, cause activator.Cause) string {
return "swapping to build " + build + " (" + string(cause) + ")"
}
-func textSwapped(build string) string { return "swapped to build " + build }
+// textSwapped carries the retiring generation's drain time (spec §9.4).
+func textSwapped(build string, drain time.Duration) string {
+ return "swapped to build " + build + ". the previous build drained in " + drain.Round(100*time.Millisecond).String()
+}
func textSwapFailed(build, serving, reason string) string {
return "swap to build " + build + " failed: " + reason + ". still serving build " + serving
@@ -3263,8 +3332,10 @@ func textSwapDown(build string) string {
return "build " + build + " failed in hold mode. the instance stays down until it is rolled back with carlos rollback"
}
-func textCohortHalted(build, reason string) string {
- return "rollout of build " + build + " halted on this box: " + reason
+// textCohortHalted names the box (spec §9.4); the event sink already
+// carries the account, the app and the host.
+func textCohortHalted(build, box, reason string) string {
+ return "rollout of build " + build + " halted on box " + box + ": " + reason
}
func textFallback(failed, good string) string {
@@ -3301,13 +3372,14 @@ import (
"github.com/carlosframework/platform/internal/release"
)
-// Activator is the part of *activator.Service the scheduler drives.
+// Activator is the part of *activator.Service the scheduler drives. Swap is
+// plan 3's, for instances; ReplaceWorker is Task 11's same-entry
+// stop-then-start for workers and native sidecar members (spec §7.9).
type Activator interface {
Current(host string) (activator.Gen, bool)
Owns(host string) bool
Swap(ctx context.Context, a activator.Attempt, newest func(host string) uint64) activator.SwapResult
- BounceFor(host string, cause activator.Cause) bool
- WakeWorker(rt registry.Route) error
+ ReplaceWorker(ctx context.Context, a activator.Attempt, fallback *release.StagedIdentity, newest func(host string) uint64) activator.SwapResult
}
// Routes is the part of *registry.Registry the scheduler reads and writes.
@@ -3325,6 +3397,9 @@ type Deps struct {
Book *FailureBook
OptDir string
DataDir string
+ // Box is this box's label (host.BoxLabel), named by the cohort-halted
+ // event (spec §9.4: app, box, build, reason).
+ Box string
// Demand is release.CanonicalDemand bound to the deploy store and this
// box's platform (spec §4.2).
Demand func(ctx context.Context, acct, app, version, stagedPath string) (demand, digest string, err error)
@@ -3696,18 +3771,27 @@ func (m *Manager) Restart(host, id string) bool {
// revision rev (spec §8.1). An instance with a process (awake, waking or
// stopping) gets a pending config attempt; if its wake turns out to have read
// rev, the wake's activation clears the attempt before it runs. A sleeping
-// one needs none, because its next wake starts with the file. A host this
+// one needs none, because its next wake starts with the file: rev counts as
+// what that wake will run, and config sync's ack goes out at once with
+// today's hibernating acknowledgement (configTrack.deferAck). A host this
// manager does not manage is not tracked, so its ack goes out as before.
func (m *Manager) ConfigDelivered(host, rev string) {
s := m.schedulerOf(host)
if s == nil {
return
}
+ // Asked before m.mu: the activator is never called under the manager's
+ // lock (the lock order in Global Constraints).
+ awake := m.d.Act.Owns(host)
m.mu.Lock()
t := m.track(host)
t.delivered, t.fails, t.lastErr = rev, 0, ""
+ t.sleeping = !awake
+ if !awake {
+ t.active = rev
+ }
m.mu.Unlock()
- if !m.d.Act.Owns(host) {
+ if !awake {
return
}
s.mu.Lock()
@@ -3871,7 +3955,7 @@ func (m *Manager) cohortFailed(s *scheduler, rt registry.Route, rec FailureRecor
m.d.Logf("rollout: %s/%s build %s failed: superseding attempts past spawn on %v", rec.Account, rec.App, rec.Build, superseded)
}
if added {
- m.emit(rt, EvCohortHalted, "warn", textCohortHalted(rec.Build, rec.Reason))
+ m.emit(rt, EvCohortHalted, "warn", textCohortHalted(rec.Build, m.d.Box, rec.Reason))
}
s.poke()
}
@@ -4026,7 +4110,7 @@ func (s *scheduler) done(d dispatch, res activator.SwapResult, mode activator.Sw
delete(m.retried, d.ID.Host)
}
m.mu.Unlock()
- m.emit(*rt, EvSwapped, "info", textSwapped(d.ID.Build))
+ m.emit(*rt, EvSwapped, "info", textSwapped(d.ID.Build, res.Drain))
case activator.SwapFailed, activator.SwapDown:
if transition {
rec := FailureRecord{Account: s.key.acct, App: s.key.app, Build: d.ID.Build, Host: d.ID.Host,
@@ -4077,7 +4161,7 @@ git commit -m "zdd-5 task 8: the manager and the per-app scheduler"
- Modify: `internal/activator/zdd5_surface_test.go` (the `startPinned` pin)
**Interfaces:**
-- Consumes: plan 3's `startPinned`, `observeSettle`, `genHealth`, `GenBackend`, `e.pin`; plan 1's `e.gen`, `w.gen`; plan 4's `startWorker`, `e.workerServing`; plan 2's `KillGen` (assumed).
+- Consumes: plan 3's `startPinned`, `observeSettle`, `genHealth`, `SwapBackend`, `e.pin`, `StartPolicy.Pin/Fallback/HealthBy`, `e.pinFailed`, `e.pinFailReason` (Tasks 1, 4, 5, 6, 10); plan 1's `e.gen`, `w.gen`, `Service.gens`; plan 4's `startWorker`, `e.workerServing`; plan 2's `(*ExecBackend).KillGen(ctx, rt, n uint64) error` and its `SplitBackend` forward (Task 10).
- Produces:
```go
type WakePlan struct { Boot release.StagedIdentity; Mode SwapMode; FirstWake bool; Fallback *release.StagedIdentity; Refuse string }
@@ -4087,7 +4171,8 @@ git commit -m "zdd-5 task 8: the manager and the per-app scheduler"
func (a *Service) Current(host string) (Gen, bool)
func (a *Service) BounceFor(host string, cause Cause) bool
func bounceText(worker bool, cause Cause) string // plan 6 rewords it
- func (b *SplitBackend) KillGen(ctx context.Context, rt registry.Route, g Gen) error // only if plan 2 did not add it
+ type genKiller interface { KillGen(ctx context.Context, rt registry.Route, n uint64) error } // plan 2's signature
+ func (a *Service) claimedPlanResult(e *actEntry, w *wokenInstance) *wokenInstance
// startPinned gains a last parameter: healthBy time.Time (zero: no cap)
var firstWakeBudget, firstWakeKillReserve, firstWakeKillBound, firstWakeSettle time.Duration // 55 s, 25 s, 2 s, 10 s
```
@@ -4118,8 +4203,9 @@ import (
"github.com/carlosframework/platform/internal/release"
)
-// planGenBackend is plan 3's GenBackend plus plan 2's KillGen over the
-// package's fakeBackend. Its generations are fakeProcs a test can crash.
+// planGenBackend is plan 3's SwapBackend (startPinned's staged spawn) plus
+// plan 2's KillGen over the package's fakeBackend. Its generations are
+// fakeProcs a test can crash.
type planGenBackend struct {
*fakeBackend
gmu sync.Mutex
@@ -4134,7 +4220,7 @@ func newPlanGenBackend() *planGenBackend {
return &planGenBackend{fakeBackend: &fakeBackend{}, procs: map[uint64]*fakeProc{}}
}
-func (b *planGenBackend) StartGen(_ context.Context, _ registry.Route, g Gen) (Gen, Proc, error) {
+func (b *planGenBackend) StartStaged(_ context.Context, _ registry.Route, g Gen) (Gen, Proc, error) {
b.gmu.Lock()
gate := b.gate
b.gmu.Unlock()
@@ -4151,23 +4237,28 @@ func (b *planGenBackend) StartGen(_ context.Context, _ registry.Route, g Gen) (G
return g, p, nil
}
-func (b *planGenBackend) RaiseGenParent(registry.Route) error { return nil }
-func (b *planGenBackend) RestoreGenParent(registry.Route) error { return nil }
-func (b *planGenBackend) GenSwitched(registry.Route, Gen, Proc) {}
+func (b *planGenBackend) RaiseGenParent(context.Context, registry.Route) error { return nil }
+func (b *planGenBackend) RestoreGenParent(context.Context, registry.Route) error { return nil }
+func (b *planGenBackend) GenSwitched(registry.Route, Gen, Proc) error { return nil }
-func (b *planGenBackend) KillGen(_ context.Context, _ registry.Route, g Gen) error {
+func (b *planGenBackend) KillGen(_ context.Context, _ registry.Route, n uint64) error {
b.gmu.Lock()
defer b.gmu.Unlock()
- b.killed = append(b.killed, g.N)
+ b.killed = append(b.killed, n)
if b.killErr != nil {
return b.killErr
}
- if p := b.procs[g.N]; p != nil {
+ if p := b.procs[n]; p != nil {
p.exit()
}
return nil
}
+var (
+ _ SwapBackend = (*planGenBackend)(nil)
+ _ genKiller = (*planGenBackend)(nil)
+)
+
func (b *planGenBackend) started() []Gen {
b.gmu.Lock()
defer b.gmu.Unlock()
@@ -4245,7 +4336,8 @@ func planActivator(t *testing.T, plan WakePlan) (*Service, *planGenBackend, *gen
if err != nil {
t.Fatal(err)
}
- a := New(Config{Bucket: "test-bucket", Idle: time.Hour, Swap: true}, b, reg)
+ a := New(Config{Bucket: "test-bucket", Idle: time.Hour}, b, reg)
+ a.gens = true // plan 1's switch, as CARLOS_SWAP=overlap sets it before New
evs := &genLog{}
a.OnGen = evs.add
a.Planner = func(context.Context, registry.Route) WakePlan { return plan }
@@ -4428,6 +4520,28 @@ func TestARefusedPlanTakesNoLease(t *testing.T) {
}
}
+// Claims on (plan 3 Task 10 runs the fallback inside startAdmitted): a
+// claimed first wake that fell back reports as one, a failed one names the
+// build that failed, and one where nothing of the build ran reports nothing.
+func TestAClaimedFirstWakeReportsLikeAnUnclaimedOne(t *testing.T) {
+ good := release.StagedIdentity{Version: "v1", StagedPath: "/opt/v1", Digest: "d1"}
+ next := release.StagedIdentity{Version: "v2", StagedPath: "/opt/v2", Digest: "d2"}
+ a := &Service{}
+ e := &actEntry{host: "one.example.com", plan: WakePlan{Boot: next, Mode: ModeOverlap, FirstWake: true, Fallback: &good},
+ pinFailed: &Gen{Build: "v2"}, pinFailReason: "GET / answered 500"}
+ w := a.claimedPlanResult(e, &wokenInstance{spawnVersion: "v1", gen: Gen{N: 9}})
+ if !w.fellBack || w.failed == nil || w.failed.Version != "v2" || w.planned.Build != "v1" || w.planned.N != 9 {
+ t.Fatalf("fell back: %+v", w)
+ }
+ if w := a.claimedPlanResult(e, nil); w == nil || w.failed == nil || w.failed.Version != "v2" {
+ t.Fatalf("failed: %+v", w)
+ }
+ e.pinFailed = nil
+ if w := a.claimedPlanResult(e, nil); w != nil {
+ t.Fatalf("nothing of the build ran, yet %+v", w)
+ }
+}
+
// The flag off: no Planner, today's wake, no events.
func TestWithoutAPlannerTheWakeIsTodays(t *testing.T) {
a, b, _, _ := testActivator(t, time.Hour)
@@ -4499,7 +4613,7 @@ func TestBounceForNamesTheCause(t *testing.T) {
- [ ] **Step 2: Run them to see them fail**
-Run: `mkdir -p /tmp/k && TMPDIR=/tmp/k go test ./internal/activator/ -run 'TestFirstWake|TestAHoldFirstWake|TestAnUnprovenKill|TestAnExitInside|TestAnIntentionalStop|TestAFirstWakeThat|TestARefusedPlan|TestWithoutAPlanner|TestDriftFeeds|TestBounceFor' -count=1`
+Run: `mkdir -p /tmp/k && TMPDIR=/tmp/k go test ./internal/activator/ -run 'TestFirstWake|TestAHoldFirstWake|TestAnUnprovenKill|TestAnExitInside|TestAnIntentionalStop|TestAFirstWakeThat|TestARefusedPlan|TestAClaimedFirstWake|TestWithoutAPlanner|TestDriftFeeds|TestBounceFor' -count=1`
Expected: FAIL, `undefined: WakePlan`.
- [ ] **Step 3: Write the plan types and helpers**
@@ -4574,10 +4688,11 @@ var (
firstWakeSettle = 10 * time.Second
)
-// genKiller is plan 2's generation kill: cgroup.kill on one generation's
-// child cgroup, then wait until it is empty.
+// genKiller is plan 2's generation kill (ExecBackend.KillGen and its
+// SplitBackend forward): cgroup.kill on generation n's child cgroup, then
+// wait until it is empty.
type genKiller interface {
- KillGen(ctx context.Context, rt registry.Route, g Gen) error
+ KillGen(ctx context.Context, rt registry.Route, n uint64) error
}
// Current is the generation an awake instance serves, for the scheduler: a
@@ -4633,7 +4748,7 @@ func (a *Service) killFailed(rt registry.Route, g Gen, p Proc, worker bool) erro
if !ok {
return fmt.Errorf("%s: this backend cannot kill a generation", rt.Host)
}
- return k.KillGen(ctx, rt, g)
+ return k.KillGen(ctx, rt, g.N)
}
if err := p.Stop(ctx); err != nil {
return err
@@ -4773,18 +4888,36 @@ func (a *Service) watchFirstWake(e *actEntry, g Gen, plan WakePlan) {
}()
}
-// KillGen forwards plan 2's generation kill for exec rows. Skip this method
-// if plan 2 already gave SplitBackend one.
-func (b *SplitBackend) KillGen(ctx context.Context, rt registry.Route, g Gen) error {
- k, ok := b.pick(rt).(genKiller)
- if !ok {
- return fmt.Errorf("%s: this backend cannot kill a generation", rt.Host)
+// claimedPlanResult carries a claims-on planned wake's outcome to afterWake.
+// Plan 3's startAdmitted records the pinned build it saw fail
+// (e.pinFailed) and, when the plan gave it one, starts the good build in its
+// place (StartPolicy.Fallback). w is the claim's result, nil when the wake
+// failed. The returned value is what doClaimWake hands back to wake().
+func (a *Service) claimedPlanResult(e *actEntry, w *wokenInstance) *wokenInstance {
+ a.mu.Lock()
+ failed, why, plan := e.pinFailed, e.pinFailReason, e.plan
+ a.mu.Unlock()
+ if !plan.planned() {
+ return w
+ }
+ if w == nil {
+ if failed == nil || !plan.FirstWake {
+ return nil // nothing of the planned build ran: no failure of the build
+ }
+ return &wokenInstance{failed: &plan.Boot, failReason: why}
+ }
+ served := plan.Boot
+ if failed != nil && plan.FirstWake && plan.Fallback != nil && w.spawnVersion == plan.Fallback.Version {
+ served = *plan.Fallback
+ w.failed, w.failReason, w.fellBack = &plan.Boot, why, true
}
- return k.KillGen(ctx, rt, g)
+ w.planned = genOf(served)
+ w.planned.N, w.planned.Socket, w.planned.Control = w.gen.N, w.gen.Socket, w.gen.Control
+ return w
}
```
-Before keeping `SplitBackend.KillGen`, run `grep -n 'func (b \*SplitBackend) KillGen' internal/activator/*.go`; if plan 2 defined one, delete this copy.
+Plan 2 Task 10 gives `SplitBackend` its `KillGen` forward, which the agent's backend needs; this task adds none.
- [ ] **Step 4: Add the Service hooks and entry fields**
@@ -4888,35 +5021,61 @@ In `doWake`, after the layout refusal (step 0.5, the `if c, ok := a.backend.(Cla
}
```
-In step 4, plan 3 left the instance spawn as `} else if e.pin != nil { // swapsim:M3` followed by `proc, spawnVer, g, err = a.startPinned(ctx, rt, *e.pin)`. Insert the planned branch before it and pass the new parameter, keeping plan 3's marker on its line:
+In step 4, plan 3 Task 6 left the instance spawn as:
+
+```go
+ } else if e.pin != nil { // swapsim:M3
+ var g Gen
+ proc, spawnVer, g, err = a.startPinned(ctx, rt, *e.pin)
+ w.gen = g
+ } else {
+```
+
+Insert the planned branch before it and pass the new parameter, keeping plan 3's marker on its line. `w.gen` is what plan 1's `wake()` copies into `e.gen`:
```go
} else if plan.planned() {
+ var g Gen
proc, spawnVer, g, err = a.spawnPlanned(ctx, e, rt, plan, w)
- w.planned = g
+ w.gen, w.planned = g, g
} else if e.pin != nil { // swapsim:M3
+ var g Gen
proc, spawnVer, g, err = a.startPinned(ctx, rt, *e.pin, time.Time{})
+ w.gen = g
+ } else {
+```
+
+In the worker branch, plan 4 Task 9 left:
+
+```go
+ var ident release.StagedIdentity
+ proc, ident, err = a.startWorker(ctx, rt, workerEnv(a.agentSocket(), host, token), nil)
+ spawnVer, w.spawnIdentity = ident.Version, ident
```
-In the worker branch, plan 4 calls `a.startWorker(ctx, rt, <env>, nil)`. Wrap it, keeping plan 4's variable names for the returned proc and identity (shown here as `proc` and `served`):
+Replace those three lines with:
```go
+ var ident release.StagedIdentity
+ env := workerEnv(a.agentSocket(), host, token)
if plan.planned() {
- proc, served, err = a.spawnPlannedWorker(ctx, e, rt, plan, env, w)
- w.planned = genOf(served)
+ proc, ident, err = a.spawnPlannedWorker(ctx, e, rt, plan, env, w)
+ w.planned = genOf(ident)
} else {
- proc, served, err = a.startWorker(ctx, rt, env, nil)
+ proc, ident, err = a.startWorker(ctx, rt, env, nil)
}
+ spawnVer, w.spawnIdentity = ident.Version, ident
```
-where `env` is the `workerEnv(a.agentSocket(), host, token)` value plan 4's call passed (bind it to a local first). Then change step 4b's guard from `if e.worker {` to `if e.worker && !plan.planned() {`: a planned worker's settle already ran inside `spawnPlannedWorker`.
+Then change step 4b's guard from `if e.worker {` to `if e.worker && !plan.planned() {`: a planned worker's settle already ran inside `spawnPlannedWorker`.
In `doClaimWake` (claims on), after the route refresh (`rt = *fresh`) and before `policy := startPolicy{...}`, add:
```go
- // Claimed wakes run plan 3's pinned start (StartPolicy.Pin). They boot
- // the planned build, but have no first-wake fallback: see plan 5's
- // report.
+ // Claimed wakes run plan 3's pinned start (StartPolicy.Pin) and get the
+ // same first-wake budget and fallback as an unclaimed wake (spec §8.2):
+ // firstWake extends the waiters' deadline in begin, and the policy below
+ // carries the health cap and the good build.
if a.Planner != nil && e.pin == nil {
plan := a.Planner(ctx, rt)
if plan.Refuse != "" {
@@ -4925,12 +5084,38 @@ In `doClaimWake` (claims on), after the route refresh (`rt = *fresh`) and before
if plan.planned() {
pin := genOf(plan.Boot)
a.mu.Lock()
- e.pin, e.plan = &pin, plan
+ e.pin, e.plan, e.firstWake = &pin, plan, plan.FirstWake
a.mu.Unlock()
}
}
```
+Directly after the `policy := startPolicy{…, Pin: e.pin}` line (and plan 3 Task 10's `policy.Gen` line after it), add:
+
+```go
+ if p := e.plan; p.planned() && p.FirstWake && p.Mode == ModeOverlap && p.Fallback != nil {
+ // The same budget as spawnPlanned: B's health until the waiters'
+ // deadline minus 25 s, then the good build in its place.
+ fb := genOf(*p.Fallback)
+ policy.Fallback = &fb
+ policy.HealthBy = e.wakeStart.Add(firstWakeBudget - firstWakeKillReserve)
+ }
+```
+
+At the end of `doClaimWake`, route both returns through `claimedPlanResult`, so `afterWake` sees the claimed wake's outcome as it sees an unclaimed one's. The failure return
+
+```go
+ return nil, errors.Join(err, r.FailedAdmission(recoveryCtx, admitted, err))
+```
+
+becomes
+
+```go
+ return a.claimedPlanResult(e, nil), errors.Join(err, r.FailedAdmission(recoveryCtx, admitted, err))
+```
+
+and the success return `return e.claimResult, nil` becomes `return a.claimedPlanResult(e, e.claimResult), nil`. A non-nil result with an error carries no `stopOutcome`, so `wake()` records the uncertain stop it records today.
+
- [ ] **Step 7: Give `startPinned` its health deadline**
In plan 3's `swap.go`, change `startPinned`'s signature to `func (a *Service) startPinned(ctx context.Context, rt registry.Route, pin Gen, healthBy time.Time) (Proc, string, Gen, error)` and its health call from `if err := a.genHealthy(ctx, g); err != nil {` to:
@@ -5006,10 +5191,8 @@ git commit -m "zdd-5 task 9: planned wakes, the first-wake budget and fallback,
- Produces:
```go
type wakeBook interface { Blocks(acct, app, build, host string, served bool) bool; HeldDown(acct, app, build, host string) bool }
- type wakeInputs struct { Route registry.Route; OptDir string; Policy AppPolicy; Book wakeBook; Demand func(version, stagedPath string) (string, string, error); Attempt *Identity }
+ type wakeInputs struct { Route registry.Route; OptDir string; Policy AppPolicy; Book wakeBook; Demand func(version, stagedPath string) (string, string, error) }
func decideWake(in wakeInputs) (activator.WakePlan, error)
- type workerWait struct { id Identity; ch chan activator.GenEvent }
- // Manager gains: workers map[string]*workerWait
func (m *Manager) Plan(ctx context.Context, rt registry.Route) activator.WakePlan
func (m *Manager) OnGen(ev activator.GenEvent)
```
@@ -5083,11 +5266,6 @@ func TestDecideWake(t *testing.T) {
if err == nil || p.Boot.Version != "v1" || p.FirstWake {
t.Fatalf("a staging refusal with a good build boots the good build, got %+v, %v", p, err)
}
- in = inputs(wakeRoute("v2", "v1"), &FailureBook{}, overlap)
- in.Attempt = &Identity{Host: "a", Cause: activator.CauseRestart, Build: "v1"}
- if p, _ := decideWake(in); p.Boot.Version != "v1" || p.FirstWake {
- t.Fatalf("a worker attempt names its own build, got %+v", p)
- }
}
// Review focus 3.
@@ -5154,7 +5332,8 @@ func TestAWakeActivatesTheConfigItStartedWith(t *testing.T) {
h := newHarness(t, "a")
h.pass()
h.act.mu.Lock()
- delete(h.act.gens, "a") // asleep
+ delete(h.act.gens, "a")
+ h.act.owned["a"] = true // waking: a process, not yet current
h.act.mu.Unlock()
var acks int
ack := h.m.DeferAck(func(appconfig.ConfigAck) { acks++ })
@@ -5170,18 +5349,41 @@ func TestAWakeActivatesTheConfigItStartedWith(t *testing.T) {
t.Fatalf("acks = %d after the wake, want 1", acks)
}
}
+
+// Spec §8.1: a sleeping instance needs no config attempt. Its next wake
+// starts with the delivered file, and config sync's ack goes out at
+// delivery, as today, marked Hibernating.
+func TestASleepersConfigNeedsNoAttempt(t *testing.T) {
+ h := newHarness(t, "a")
+ h.pass()
+ h.act.mu.Lock()
+ delete(h.act.gens, "a") // asleep: no process at all
+ h.act.mu.Unlock()
+ var acks []appconfig.ConfigAck
+ ack := h.m.DeferAck(func(a appconfig.ConfigAck) { acks = append(acks, a) })
+ h.m.ConfigDelivered("a", "sha256:c1")
+ ack(appconfig.ConfigAck{Host: "a", Restarted: true})
+ if len(acks) != 1 || !acks[0].Hibernating || acks[0].Restarted {
+ t.Fatalf("acks = %+v, want one at delivery, Hibernating", acks)
+ }
+ h.pass()
+ time.Sleep(50 * time.Millisecond)
+ if calls, _, _ := h.act.snapshot(); len(calls) != 0 {
+ t.Fatalf("attempts %+v: a sleeper got a config attempt", calls)
+ }
+}
```
-Add `"github.com/carlosframework/platform/internal/release"` to that file's imports.
+Add `"time"` and `"github.com/carlosframework/platform/internal/release"` to that file's imports.
- [ ] **Step 2: Run them to see them fail**
-Run: `go test ./internal/rollout/ -run 'TestDecideWake|TestPlanBoots|TestAFellBack|TestOnlyASettled|TestAFailedHold|TestAWakeActivates' -count=1`
+Run: `go test ./internal/rollout/ -run 'TestDecideWake|TestPlanBoots|TestAFellBack|TestOnlyASettled|TestAFailedHold|TestAWakeActivates|TestASleepersConfig' -count=1`
Expected: FAIL, `undefined: decideWake`.
- [ ] **Step 3: Write the planner and the outcome handler**
-Add the field `workers map[string]*workerWait` to `Manager` (`manager.go`), and `workers: map[string]*workerWait{}` to `NewManager`'s literal. Create `internal/rollout/wakeplan.go`:
+Create `internal/rollout/wakeplan.go`. (A socketless attempt never goes through a wake: Task 11 runs it on the same entry, so the planner needs no attempt of its own.)
```go
package rollout
@@ -5210,9 +5412,6 @@ type wakeInputs struct {
// Demand is the canonical check for an adoption (spec §4.2), returning a
// digest always.
Demand func(version, stagedPath string) (demand, digest string, err error)
- // Attempt is a socketless attempt in flight for this host: its wake boots
- // the attempt's build (spec §6.5).
- Attempt *Identity
}
// decideWake is spec §8.2's "when the wake picks a build", as a pure
@@ -5226,9 +5425,6 @@ func decideWake(in wakeInputs) (activator.WakePlan, error) {
rt := in.Route
acct, app, host := release.RouteAccount(rt), rt.App, rt.Host
installed := rt.Version
- if in.Attempt != nil {
- installed = in.Attempt.Build
- }
if installed == "" {
return activator.WakePlan{}, nil
}
@@ -5286,10 +5482,6 @@ func (m *Manager) Plan(ctx context.Context, rt registry.Route) activator.WakePla
return demand, digest, err
}}
m.mu.Lock()
- if w := m.workers[rt.Host]; w != nil {
- id := w.id
- in.Attempt = &id
- }
t := m.track(rt.Host)
t.wakeRev = t.delivered
m.mu.Unlock()
@@ -5300,13 +5492,6 @@ func (m *Manager) Plan(ctx context.Context, rt registry.Route) activator.WakePla
return plan
}
-// workerWait is a socketless attempt in flight: the wake that follows its
-// stop boots the attempt's build, and the outcome comes back as events.
-type workerWait struct {
- id Identity
- ch chan activator.GenEvent
-}
-
// OnGen is the activator's OnGen: a planned wake's outcome (spec §6.4, §8.2,
// §8.5). A healthy wake records the serving build and activates the config
// it started with; a first wake becomes the good build only once it has
@@ -5335,15 +5520,6 @@ func (m *Manager) OnGen(ev activator.GenEvent) {
case activator.GenFailed:
m.buildFailed(rt, ev)
}
- m.mu.Lock()
- w := m.workers[rt.Host]
- m.mu.Unlock()
- if w != nil {
- select {
- case w.ch <- ev:
- default:
- }
- }
}
func (m *Manager) wakeConfigActive(host string) {
@@ -5394,22 +5570,399 @@ git commit -m "zdd-5 task 10: the wake planner and planned-wake outcomes"
### Task 11: Socketless attempts and the recycle hook
**Files:**
+- Create: `internal/activator/workerswap.go`, `internal/activator/workerswap_test.go`
+- Modify: `internal/activator/activator.go` (`actEntry`: `attemptBuilds`, `retireTook`; plan 4's `respawnWorkerLocked`: time the retire of the old child)
- Create: `internal/rollout/worker.go`, `internal/rollout/worker_test.go`
- Modify: `internal/rollout/scheduler.go` (`execute`)
**Interfaces:**
-- Consumes: Task 8's `Activator.BounceFor`, `Activator.WakeWorker`; Task 10's `workerWait`, `Manager.workers`, `OnGen`; Task 6's `gate`, `start`.
+- Consumes: plan 4 Task 9's `respawnWorkerLocked(e *actEntry, stopOld Proc, why string, onto *release.StagedIdentity)`, `e.workerServing`, `stagedActivator`, `stagedFake`, `fakeIdentity`; `e.respawnDone`, `e.workerRespawning` (on `main`); plan 3's `swapChecksLocked`, `abandon`, `errSwapKind`, `errSwapBusy`, `errSwapStopped`, `observeSettle`, `settleOK`/`settleExited`/`settleCancelled` (Tasks 3–5); Task 9's `e.firstWatch`, `e.unexpectedExit`, `firstWakeSettle`, `shrinkFirstWake`, `genOf`, `bounceText`, `procCrashed` (on `main`); Task 8's `Activator`, `resolve`, `done`; Task 6's `gate`, `start`.
- Produces:
```go
- const workerAttemptTimeout = 2 * time.Minute
- func (s *scheduler) executeWorker(ctx context.Context, d dispatch, rt registry.Route) activator.SwapResult
+ // activator
+ func (a *Service) ReplaceWorker(ctx context.Context, at Attempt, fallback *release.StagedIdentity, newest func(host string) uint64) SwapResult
+ // actEntry gains: attemptBuilds []string (the builds a socketless attempt in flight may exec, for the generations file, Task 12)
+ // retireTook time.Duration (how long the last respawn took to retire its old child)
+ // rollout
+ func (s *scheduler) executeWorker(ctx context.Context, rt registry.Route, a activator.Attempt) activator.SwapResult
func (q *appQueue) admitOne(host string, now time.Time) (dispatch, bool)
func (m *Manager) CoalesceRecycle(host string) bool // plan 4's Service.CoalesceRecycle
```
-Spec §6.5 says a failed socketless `release`, `install` or `retry` attempt "is treated as a failed first wake". So a worker attempt is a stop (`BounceFor`) and a wake (`WakeWorker`), and the wake runs Task 9's planned path: Task 10's planner boots the attempt's build, the settle window is the readiness check, and an overlap failure starts the good build in its place. The stop releases and re-takes the lease, unlike §7.9's same-entry respawn (see the report).
+Spec §7.9: workers and native sidecar members "keep stop-then-start, on the respawn-on-same-entry path (`respawnWorkerLocked`), with the readiness and failure rules in §6.5". So a socketless attempt never stops the entry and never wakes it: `ReplaceWorker` retires the old child and execs the attempt's staged build on the same entry, keeping the lease and the replicator, then watches it for 10 s. Spec §6.5's failure rules decide what runs afterwards: a failed `restart` or `config` attempt respawns the build it replaced; a failed `release`, `install` or `retry` attempt "is treated as a failed first wake", so in `overlap` mode the good build is respawned in its place, and in `hold` mode nothing falls back and the instance stays down (§6.4, §8.3).
-- [ ] **Step 1: Write the failing tests**
+- [ ] **Step 1: Write the failing activator tests**
+
+Create `internal/activator/workerswap_test.go`:
+
+```go
+package activator
+
+import (
+ "context"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+func workerAttempt(cause Cause, mode SwapMode, onto release.StagedIdentity) Attempt {
+ return Attempt{Host: "bot.x.internal", Cause: cause, Build: onto.Version, StagedPath: onto.StagedPath,
+ Digest: onto.Digest, Demand: onto.Demand, Mode: mode, SchedRev: 1}
+}
+
+func one(string) uint64 { return 1 }
+
+// Spec §7.9: the attempt respawns on the SAME entry. The lease is never
+// released or re-taken, and the new child execs the attempt's staged build.
+func TestReplaceWorkerKeepsTheEntryAndItsLease(t *testing.T) {
+ shrinkFirstWake(t, 55*time.Second, 25*time.Second, 100*time.Millisecond)
+ a, b, _, _, _ := stagedActivator(t)
+ host := "bot.x.internal"
+ if err := a.WakeWorker(alwaysOnWorkerRoute(host)); err != nil {
+ t.Fatalf("WakeWorker: %v", err)
+ }
+ acquires, _, _ := b.counts()
+ res := a.ReplaceWorker(context.Background(), workerAttempt(CauseRelease, ModeOverlap, fakeIdentity("v2")), nil, one)
+ if res.Outcome != SwapDone || res.Gen.Build != "v2" || res.Gen.Digest != fakeIdentity("v2").Digest {
+ t.Fatalf("ReplaceWorker = %+v", res)
+ }
+ if got, _, _ := b.counts(); got != acquires {
+ t.Fatalf("lease acquires %d -> %d: a socketless attempt must keep the lease (spec §7.9)", acquires, got)
+ }
+ if calls := b.calls(); len(calls) != 2 || calls[1] == nil || *calls[1] != fakeIdentity("v2") {
+ t.Fatalf("spawns %v, want the wake, then v2 by its staged path", calls)
+ }
+ if got := a.entryServing(host); got != fakeIdentity("v2") {
+ t.Fatalf("serving %+v", got)
+ }
+}
+
+// Spec §6.5: a failed release in overlap mode respawns the good build in
+// its place, on the same entry, and reports the failure.
+func TestReplaceWorkerFallsBackToTheGoodBuild(t *testing.T) {
+ shrinkFirstWake(t, 55*time.Second, 25*time.Second, 300*time.Millisecond)
+ a, b, _, _, _ := stagedActivator(t)
+ host := "bot.x.internal"
+ if err := a.WakeWorker(alwaysOnWorkerRoute(host)); err != nil {
+ t.Fatal(err)
+ }
+ good := fakeIdentity("v1")
+ go func() {
+ waitFor(t, "the v2 child", func() bool { return len(b.calls()) == 2 })
+ time.Sleep(50 * time.Millisecond)
+ b.lastWorker().crash()
+ }()
+ res := a.ReplaceWorker(context.Background(), workerAttempt(CauseRelease, ModeOverlap, fakeIdentity("v2")), &good, one)
+ if res.Outcome != SwapFailed || res.Gen.Build != "v1" {
+ t.Fatalf("ReplaceWorker = %+v, want failed with the good v1 serving", res)
+ }
+ if calls := b.calls(); len(calls) != 3 || *calls[2] != good {
+ t.Fatalf("spawns %v, want the good build last", calls)
+ }
+ if got := a.entryServing(host); got != good {
+ t.Fatalf("serving %+v, want the good build", got)
+ }
+}
+
+// Spec §6.4, §8.3: a hold build that ran never falls back. The instance
+// stays down.
+func TestReplaceWorkerInHoldModeStaysDown(t *testing.T) {
+ shrinkFirstWake(t, 55*time.Second, 25*time.Second, 300*time.Millisecond)
+ a, b, _, _, _ := stagedActivator(t)
+ host := "bot.x.internal"
+ if err := a.WakeWorker(alwaysOnWorkerRoute(host)); err != nil {
+ t.Fatal(err)
+ }
+ good := fakeIdentity("v1")
+ go func() {
+ waitFor(t, "the v2 child", func() bool { return len(b.calls()) == 2 })
+ time.Sleep(50 * time.Millisecond)
+ b.lastWorker().crash()
+ }()
+ res := a.ReplaceWorker(context.Background(), workerAttempt(CauseRelease, ModeHold, fakeIdentity("v2")), &good, one)
+ if res.Outcome != SwapDown {
+ t.Fatalf("ReplaceWorker = %+v, want down", res)
+ }
+ if len(b.calls()) != 2 {
+ t.Fatalf("spawns %v: a hold build fell back", b.calls())
+ }
+ waitFor(t, "the entry to stop", func() bool { return !a.Owns(host) })
+}
+
+// Spec §6.5: a failed restart respawns the build it replaced, and writes
+// nothing against it.
+func TestReplaceWorkerRestartFailureRespawnsTheServingBuild(t *testing.T) {
+ shrinkFirstWake(t, 55*time.Second, 25*time.Second, 300*time.Millisecond)
+ a, b, _, _, _ := stagedActivator(t)
+ host := "bot.x.internal"
+ if err := a.WakeWorker(alwaysOnWorkerRoute(host)); err != nil {
+ t.Fatal(err)
+ }
+ serving := a.entryServing(host)
+ go func() {
+ waitFor(t, "the restarted child", func() bool { return len(b.calls()) == 2 })
+ time.Sleep(50 * time.Millisecond)
+ b.lastWorker().crash()
+ }()
+ res := a.ReplaceWorker(context.Background(), workerAttempt(CauseRestart, ModeOverlap, serving), nil, one)
+ if res.Outcome != SwapFailed {
+ t.Fatalf("ReplaceWorker = %+v", res)
+ }
+ waitFor(t, "the serving build back", func() bool { return len(b.calls()) == 3 && !a.entryRespawning(host) })
+ if calls := b.calls(); *calls[2] != serving {
+ t.Fatalf("respawned %+v, want the serving build", calls[2])
+ }
+}
+
+// Spec §6.5: a clean exit inside the 10 s is a one-shot's success.
+func TestReplaceWorkerOneShotCleanExitIsReady(t *testing.T) {
+ shrinkFirstWake(t, 55*time.Second, 25*time.Second, 300*time.Millisecond)
+ a, b, _, _, _ := stagedActivator(t)
+ if err := a.WakeWorker(alwaysOnWorkerRoute("bot.x.internal")); err != nil {
+ t.Fatal(err)
+ }
+ go func() {
+ waitFor(t, "the v2 child", func() bool { return len(b.calls()) == 2 })
+ b.lastWorker().exit()
+ }()
+ if res := a.ReplaceWorker(context.Background(), workerAttempt(CauseRelease, ModeOverlap, fakeIdentity("v2")), nil, one); res.Outcome != SwapDone {
+ t.Fatalf("ReplaceWorker = %+v, want done", res)
+ }
+}
+
+// A superseded attempt touches nothing.
+func TestReplaceWorkerSupersededSpawnsNothing(t *testing.T) {
+ a, b, _, _, _ := stagedActivator(t)
+ if err := a.WakeWorker(alwaysOnWorkerRoute("bot.x.internal")); err != nil {
+ t.Fatal(err)
+ }
+ res := a.ReplaceWorker(context.Background(), workerAttempt(CauseRelease, ModeOverlap, fakeIdentity("v2")), nil, func(string) uint64 { return 2 })
+ if res.Outcome != SwapAbandoned || len(b.calls()) != 1 {
+ t.Fatalf("ReplaceWorker = %+v, spawns %v", res, b.calls())
+ }
+}
+```
+
+`stagedActivator`, `entryServing` and `stagedFake` are plan 4 Task 9's and Task 8's (`worker_identity_test.go`, `worker_staged_test.go`); `shrinkFirstWake` is Task 9's; `alwaysOnWorkerRoute`, `waitFor` and `entryRespawning` are on `main`.
+
+- [ ] **Step 2: Run them to see them fail**
+
+Run: `TMPDIR=/tmp/k go test ./internal/activator/ -run 'TestReplaceWorker' -count=1`
+Expected: FAIL, `a.ReplaceWorker undefined`.
+
+- [ ] **Step 3: Time the retire, and add the entry fields**
+
+In `activator.go`, add to `actEntry` (beside Task 9's fields):
+
+```go
+ // attemptBuilds names the builds a socketless attempt in flight may exec
+ // (its own and its fallback), so the generations file pins them before
+ // anything execs them (Task 12). retireTook is how long the last respawn
+ // took to retire its old child: the "swapped" event's drain time for a
+ // worker (spec §9.4). Both written under a.mu.
+ attemptBuilds []string
+ retireTook time.Duration
+```
+
+In plan 4's `respawnWorkerLocked` goroutine, replace the whole retire block
+
+```go
+ if stopOld != nil {
+ stopCtx, cancel := context.WithTimeout(context.Background(), actStopTimeout)
+ if serr := stopOld.Stop(stopCtx); serr != nil {
+ log.Printf("activator: worker %s respawn: retiring the old child: %v", host, serr)
+ }
+ cancel()
+ }
+```
+
+with this, which times the old child's stop on the wall clock:
+
+```go
+ var retireTook time.Duration
+ if stopOld != nil {
+ retireStart := time.Now()
+ stopCtx, cancel := context.WithTimeout(context.Background(), actStopTimeout)
+ if serr := stopOld.Stop(stopCtx); serr != nil {
+ log.Printf("activator: worker %s respawn: retiring the old child: %v", host, serr)
+ }
+ cancel()
+ retireTook = time.Since(retireStart)
+ }
+```
+
+In the `default:` arm, beside plan 4's `e.workerServing = ident`, add `e.retireTook = retireTook`.
+
+- [ ] **Step 4: Write `ReplaceWorker`**
+
+Create `internal/activator/workerswap.go`:
+
+```go
+package activator
+
+import (
+ "context"
+ "fmt"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// ReplaceWorker runs one socketless attempt (zero-downtime spec §6.5, §7.9)
+// on an awake worker or native sidecar member: stop-then-start on the SAME
+// entry, through respawnWorkerLocked, so the lease and the replicator stay
+// put. The new child must still be running firstWakeSettle (10 s) after it
+// started; a clean exit inside that window is a one-shot's success. On a
+// failure:
+// - restart or config: the build it replaced is respawned (spec §6.5);
+// - release, install or retry in hold mode: nothing falls back, and the
+// entry stops, so the instance stays down (spec §6.4, §8.3);
+// - release, install or retry in overlap mode: fallback, the host's good
+// build, is respawned in its place; with none, the entry stops, as a
+// failed wake does today (spec §8.2).
+//
+// The scheduler runs the cohort-failure transition from the result. newest
+// is called with a.mu held, as Swap's is.
+func (a *Service) ReplaceWorker(ctx context.Context, at Attempt, fallback *release.StagedIdentity, newest func(host string) uint64) SwapResult {
+ onto := release.StagedIdentity{Version: at.Build, StagedPath: at.StagedPath, Digest: at.Digest, Demand: at.Demand}
+ a.mu.Lock()
+ e := a.entries[at.Host]
+ if err := a.swapChecksLocked(e, at, newest); err != nil {
+ a.mu.Unlock()
+ return abandon(err)
+ }
+ if !e.worker {
+ a.mu.Unlock()
+ return abandon(errSwapKind)
+ }
+ if e.workerRespawning {
+ a.mu.Unlock()
+ return abandon(errSwapBusy)
+ }
+ prev := e.workerServing
+ e.attemptBuilds = []string{onto.Version}
+ if fallback != nil {
+ e.attemptBuilds = append(e.attemptBuilds, fallback.Version)
+ }
+ // Spec §9.4: the bounce event names the attempt's cause.
+ emitEvent(a.Logs, e.route, "info", bounceText(true, at.Cause))
+ a.respawnWorkerLocked(e, e.proc, "attempt ("+string(at.Cause)+")", &onto)
+ done := e.respawnDone
+ a.mu.Unlock()
+ defer func() {
+ a.mu.Lock()
+ e.attemptBuilds = nil
+ a.mu.Unlock()
+ }()
+
+ if !a.awaitRespawn(ctx, done) {
+ return abandon(errSwapStopped)
+ }
+ a.mu.Lock()
+ if a.entries[at.Host] != e || e.state != actAwake {
+ a.mu.Unlock()
+ return abandon(errSwapStopped)
+ }
+ if !sameArtifact(e.workerServing, onto) {
+ // Nothing of the attempt's build ran: the backend refused the staged
+ // copy (release.ErrStagedCopyChanged), the config gate stood the
+ // worker down, or the exec failed. Put the build it replaced back,
+ // as a failed restart does (spec §6.5). This is a staging refusal,
+ // never a failure of the build.
+ if prev.StagedPath != "" && !e.workerRespawning {
+ a.respawnWorkerLocked(e, nil, "restoring "+prev.Version, &prev)
+ }
+ a.mu.Unlock()
+ return abandon(fmt.Errorf("staging: worker not respawned onto %s", onto.Version))
+ }
+ obsCtx, cancel := context.WithCancel(ctx)
+ e.firstWatch, e.unexpectedExit = cancel, false
+ proc, n, drain := e.proc, e.gen.N, e.retireTook
+ a.mu.Unlock()
+
+ r := observeSettle(obsCtx, proc, firstWakeSettle)
+ cancel()
+ served := func(id release.StagedIdentity) Gen {
+ g := genOf(id)
+ g.N = n
+ return g
+ }
+
+ a.mu.Lock()
+ e.firstWatch = nil
+ unexpected := e.unexpectedExit
+ mine := a.entries[at.Host] == e && e.state == actAwake
+ switch {
+ case r == settleOK, r == settleExited && !procCrashed(proc):
+ a.mu.Unlock()
+ return SwapResult{Outcome: SwapDone, Gen: served(onto), Drain: drain}
+ case r == settleCancelled && !unexpected:
+ // An intentional stop cancels the observation and records nothing
+ // (spec §6.4).
+ a.mu.Unlock()
+ return abandon(errSwapStopped)
+ }
+ reason := "exited within " + firstWakeSettle.String() + " of starting"
+ adoption := at.Cause == CauseRelease || at.Cause == CauseInstall || at.Cause == CauseRetry
+ switch {
+ case !adoption:
+ if mine && prev.StagedPath != "" && !e.workerRespawning {
+ a.respawnWorkerLocked(e, nil, "restoring "+prev.Version, &prev)
+ }
+ a.mu.Unlock()
+ return SwapResult{Outcome: SwapFailed, Reason: reason}
+ case at.Mode == ModeHold:
+ if mine {
+ a.beginStopLocked(e)
+ }
+ a.mu.Unlock()
+ return SwapResult{Outcome: SwapDown, Reason: reason}
+ case fallback == nil || !mine:
+ if mine {
+ a.beginStopLocked(e)
+ }
+ a.mu.Unlock()
+ return SwapResult{Outcome: SwapFailed, Reason: reason + "; no good build to fall back to"}
+ }
+ a.respawnWorkerLocked(e, nil, "falling back to "+fallback.Version, fallback)
+ done = e.respawnDone
+ a.mu.Unlock()
+ a.awaitRespawn(ctx, done)
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ if a.entries[at.Host] == e && sameArtifact(e.workerServing, *fallback) {
+ return SwapResult{Outcome: SwapFailed, Gen: served(*fallback), Reason: reason}
+ }
+ return SwapResult{Outcome: SwapFailed, Reason: reason + "; the good build did not start either"}
+}
+
+// sameArtifact reports whether the backend exec'd exactly the chosen bytes:
+// the staged path and its digest (the release demand is not the backend's
+// to report).
+func sameArtifact(got, want release.StagedIdentity) bool {
+ return got.StagedPath == want.StagedPath && got.Digest == want.Digest
+}
+
+// awaitRespawn waits for a respawn's goroutine to finish with every child it
+// made (respawnDone closes last). False when ctx ended first.
+func (a *Service) awaitRespawn(ctx context.Context, done chan struct{}) bool {
+ select {
+ case <-done:
+ return true
+ case <-ctx.Done():
+ return false
+ }
+}
+```
+
+A worker child that crashes during the observation is not recycled by the sweep meanwhile: the scheduler's `CoalesceRecycle` (Step 7) answers true for a host whose attempt is running, so the attempt makes the next spawn.
+
+- [ ] **Step 5: Run the activator tests**
+
+Run: `TMPDIR=/tmp/k go test ./internal/activator/ -count=1`
+Expected: `ok github.com/carlosframework/platform/internal/activator`
+
+- [ ] **Step 6: Write the failing rollout tests**
Create `internal/rollout/worker_test.go`:
@@ -5420,7 +5973,6 @@ import (
"testing"
"github.com/carlosframework/platform/internal/activator"
- "github.com/carlosframework/platform/internal/registry"
"github.com/carlosframework/platform/internal/release"
)
@@ -5435,41 +5987,29 @@ func workerHarness(t *testing.T, hosts ...string) *harness {
return h
}
-func TestAWorkerReleaseIsAStopAndAFirstWake(t *testing.T) {
+// Spec §7.9: a socketless attempt is ReplaceWorker on the same entry, never
+// Swap and never a stop and a wake. It carries the host's good build as its
+// fallback.
+func TestAWorkerReleaseReplacesOnTheSameEntry(t *testing.T) {
h := workerHarness(t, "bot")
h.stage("v2")
- h.act.wake = func(rt registry.Route) error {
- h.act.serve(rt.Host, "v2")
- g, _ := h.act.Current(rt.Host)
- go func() {
- h.m.OnGen(activator.GenEvent{Kind: activator.GenHealthy, Route: rt, Gen: g, FirstWake: true})
- h.m.OnGen(activator.GenEvent{Kind: activator.GenSettled, Route: rt, Gen: g, FirstWake: true})
- }()
- return nil
- }
h.install("v2", "bot")
h.pass()
waitFor(t, "the good build to move to v2", func() bool { return h.rows.get("bot").GoodVersion == "v2" })
- calls, _, _ := h.act.snapshot()
- if len(calls) != 0 {
- t.Fatal("a socketless attempt went through Swap; it is stop-then-start (spec §7.9)")
+ if calls, _, _ := h.act.snapshot(); len(calls) != 0 {
+ t.Fatal("a socketless attempt went through Swap")
}
- h.act.mu.Lock()
- defer h.act.mu.Unlock()
- if len(h.act.bounced) != 1 || h.act.bounced[0] != "bot" {
- t.Fatalf("bounced %v", h.act.bounced)
+ r := h.act.replacements()
+ if len(r) != 1 || r[0].a.Build != "v2" || r[0].fallback == nil || r[0].fallback.Version != "v1" {
+ t.Fatalf("replacements %+v, want one onto v2 with the good v1 behind it", r)
}
}
func TestAFailedWorkerReleaseFallsBackAndHalts(t *testing.T) {
h := workerHarness(t, "bot")
h.stage("v2")
- h.act.wake = func(rt registry.Route) error {
- h.act.serve(rt.Host, "v1")
- g, _ := h.act.Current(rt.Host)
- go h.m.OnGen(activator.GenEvent{Kind: activator.GenFellBack, Route: rt, Gen: g, FirstWake: true,
- Mode: activator.ModeOverlap, Failed: release.StagedIdentity{Version: "v2"}, Reason: "exited within its settle window"})
- return nil
+ h.act.worker = func(a activator.Attempt, fb *release.StagedIdentity) activator.SwapResult {
+ return activator.SwapResult{Outcome: activator.SwapFailed, Reason: "exited within 10s of starting"}
}
h.install("v2", "bot")
h.pass()
@@ -5507,12 +6047,7 @@ func TestARecycleCoalescesOnlyWithAnAdmissibleRelease(t *testing.T) {
}
```
-- [ ] **Step 2: Run them to see them fail**
-
-Run: `go test ./internal/rollout/ -run 'TestAWorkerRelease|TestAFailedWorkerRelease|TestARecycleCoalesces' -count=1`
-Expected: FAIL, `q.admitOne undefined`.
-
-- [ ] **Step 3: Write the worker path**
+- [ ] **Step 7: Write the worker path**
Create `internal/rollout/worker.go`:
@@ -5525,70 +6060,30 @@ import (
"github.com/carlosframework/platform/internal/activator"
"github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
)
-// workerAttemptTimeout bounds how long a socketless attempt waits for its
-// wake's outcome: the settle window, a fallback's settle and the wake's own
-// work fit inside it many times over.
-const workerAttemptTimeout = 2 * time.Minute
-
-// executeWorker runs one attempt on a socketless host (spec §6.5, §7.9): stop
-// it, then wake it. While the attempt is registered in m.workers, the
-// planner boots the attempt's build, and the wake's outcome comes back on the
-// channel. No outcome at all (the wake failed before anything executed: a
-// lease, a staging refusal) is not a build failure.
-func (s *scheduler) executeWorker(ctx context.Context, d dispatch, rt registry.Route) activator.SwapResult {
+// executeWorker runs one resolved attempt on a socketless host (spec §6.5,
+// §7.9) through the activator's same-entry replacement. The host's good
+// build, by its retained identity, goes with it as the fallback for a failed
+// adoption in overlap mode; the activator decides whether it applies.
+func (s *scheduler) executeWorker(ctx context.Context, rt registry.Route, a activator.Attempt) activator.SwapResult {
m := s.m
- w := &workerWait{id: d.ID, ch: make(chan activator.GenEvent, 4)}
- m.mu.Lock()
- m.workers[rt.Host] = w
- m.mu.Unlock()
- defer func() {
- m.mu.Lock()
- if m.workers[rt.Host] == w {
- delete(m.workers, rt.Host)
- }
- m.mu.Unlock()
- }()
- m.emit(rt, EvSwapping, "info", textSwapping(d.ID.Build, d.ID.Cause))
- m.d.Act.BounceFor(rt.Host, d.ID.Cause)
- werr := m.d.Act.WakeWorker(rt)
- wait := workerAttemptTimeout
- if werr != nil {
- // A wake that failed reports its outcome, if any, right after it
- // returns (Task 9's afterWake runs once e.ready has closed).
- wait = time.Second
- }
- timer := time.NewTimer(wait)
- defer timer.Stop()
- for {
- select {
- case ev := <-w.ch:
- switch ev.Kind {
- case activator.GenHealthy:
- if !ev.FirstWake {
- return activator.SwapResult{Outcome: activator.SwapDone, Gen: ev.Gen}
- }
- case activator.GenSettled:
- return activator.SwapResult{Outcome: activator.SwapDone, Gen: ev.Gen}
- case activator.GenFellBack:
- return activator.SwapResult{Outcome: activator.SwapFailed, Gen: ev.Gen, Reason: ev.Reason}
- case activator.GenFailed:
- return activator.SwapResult{Outcome: activator.SwapDown, Reason: ev.Reason}
- }
- case <-timer.C:
- reason := "the worker's wake reported no outcome"
- if werr != nil {
- reason = werr.Error()
- }
- s.mu.Lock()
- s.q.deferHost(rt.Host, m.d.Now().Add(stagingDeferral))
- s.mu.Unlock()
- return activator.SwapResult{Outcome: activator.SwapAbandoned, Reason: reason}
- case <-ctx.Done():
- return activator.SwapResult{Outcome: activator.SwapAbandoned, Reason: ctx.Err().Error()}
- }
+ var fallback *release.StagedIdentity
+ if g := rt.Good(); g.Version != "" && g.Digest != "" && g.Version != a.Build {
+ fallback = &release.StagedIdentity{Version: g.Version, Digest: g.Digest, Demand: g.Demand,
+ StagedPath: release.VersionPath(m.d.OptDir, release.RouteAccount(rt), g.Version, rt.App)}
+ }
+ m.emit(rt, EvSwapping, "info", textSwapping(a.Build, a.Cause))
+ res := m.d.Act.ReplaceWorker(ctx, a, fallback, s.newest)
+ if res.Outcome == activator.SwapAbandoned {
+ // A staging refusal or a busy entry: try again after one
+ // stage-relay window, as a refused swap does.
+ s.mu.Lock()
+ s.q.deferHost(rt.Host, m.d.Now().Add(stagingDeferral))
+ s.mu.Unlock()
}
+ return res
}
// admitOne is admit for one host, for the recycle hook: the host's pending
@@ -5608,14 +6103,19 @@ func (q *appQueue) admitOne(host string, now time.Time) (dispatch, bool) {
// CoalesceRecycle is plan 4's Service.CoalesceRecycle (spec §7.9). It is
// called under the activator's lock, so it calls nothing in the activator:
-// the attempt it admits runs on its own goroutine. True means the attempt
-// makes the spawn and the recycle is skipped.
+// the attempt it admits runs on its own goroutine. True means an attempt
+// makes the spawn and the recycle is skipped: one admitted now, or one
+// already running for the host (ReplaceWorker respawns that child itself).
func (m *Manager) CoalesceRecycle(host string) bool {
s := m.schedulerOf(host)
if s == nil {
return false
}
s.mu.Lock()
+ if h := s.q.hosts[host]; h != nil && h.running != nil {
+ s.mu.Unlock()
+ return true
+ }
d, ok := s.q.admitOne(host, m.d.Now())
s.mu.Unlock()
if !ok {
@@ -5626,26 +6126,39 @@ func (m *Manager) CoalesceRecycle(host string) bool {
}
```
-In `scheduler.go`'s `execute`, directly after the `rt, err := m.d.Routes.Get(...)` block, add:
+In `scheduler.go`'s `execute`, replace
+
+```go
+ m.emit(*rt, EvSwapping, "info", textSwapping(a.Build, a.Cause))
+ s.done(d, m.d.Act.Swap(ctx, a, s.newest), a.Mode)
+```
+
+with
```go
if rt.IsSidecar() {
- s.done(d, s.executeWorker(ctx, d, *rt), "")
+ // Workers and native sidecar members keep stop-then-start on the
+ // same entry (spec §7.9).
+ s.done(d, s.executeWorker(ctx, *rt, a), a.Mode)
return
}
+ m.emit(*rt, EvSwapping, "info", textSwapping(a.Build, a.Cause))
+ s.done(d, m.d.Act.Swap(ctx, a, s.newest), a.Mode)
```
-- [ ] **Step 4: Run them to see them pass**
+`resolve` already reads a worker's serving identity through `Current` (Task 9: `genOf(e.workerServing)`), so a `restart` or `config` attempt names the captured build, and an adoption is checked against the canonical demand like any other.
+
+- [ ] **Step 8: Run them to see them pass**
Run: `go test ./internal/rollout/ -count=1 -race`
Expected: `ok github.com/carlosframework/platform/internal/rollout`
-- [ ] **Step 5: Commit**
+- [ ] **Step 9: Commit**
```bash
-go vet ./internal/rollout/ && test -z "$(gofmt -l internal/rollout)"
-git add internal/rollout/worker.go internal/rollout/worker_test.go internal/rollout/scheduler.go
-git commit -m "zdd-5 task 11: socketless attempts and the recycle hook"
+go build ./... && go vet ./internal/rollout/ ./internal/activator/ && test -z "$(gofmt -l internal/rollout internal/activator)"
+git add internal/activator/workerswap.go internal/activator/workerswap_test.go internal/activator/activator.go internal/rollout/worker.go internal/rollout/worker_test.go internal/rollout/scheduler.go
+git commit -m "zdd-5 task 11: socketless attempts on the same entry, and the recycle hook"
```
---
@@ -5659,7 +6172,7 @@ git commit -m "zdd-5 task 11: socketless attempts and the recycle hook"
- Modify: `internal/host/versionsweep.go`, `internal/host/versionsweep_test.go`
**Interfaces:**
-- Consumes: plan 4's `release.Generations`, `GenerationsPath`, `WriteGenerations`, `ReadGenerations`, `GenerationPins`, `BoxPins`; plan 3's `e.ops.members`, `e.pin`; Task 9's `e.plan`.
+- Consumes: plan 4's `release.Generations`, `GenerationsPath`, `WriteGenerations`, `ReadGenerations`, `GenerationPins`, `BoxPins`; plan 3's `e.ops.members`, `e.pin`, `swapOverlap`'s `a.spawnSuccessor(` call (Task 10); Task 9's `e.plan`; Task 11's `e.attemptBuilds`.
- Produces:
```go
// Service gains: GenerationsFile string (empty: never written, the flag off)
@@ -5850,6 +6363,9 @@ func (a *Service) generationsLocked() release.Generations {
for _, m := range e.ops.members {
set[m.gen.Build] = true
}
+ for _, v := range e.attemptBuilds { // a socketless attempt's build and fallback (Task 11)
+ set[v] = true
+ }
delete(set, "")
if len(set) == 0 {
continue
@@ -5896,11 +6412,12 @@ Add to `Service` in `activator.go`, beside `Planner`:
Call `a.persistGenerations()` (never while holding `a.mu`) at each of these points:
- `Start`: as its first statement, so the file exists before the first spawn.
- `doWake`: directly after the planning block Task 9 added (after `a.mu.Unlock()` that stores `e.plan`), and before step 1. Add it after the `e.pin` hold path too: put the call outside the `if a.Planner != nil && e.pin == nil {` block so it runs for both.
+- `doClaimWake`: directly after the planning block Task 9 added there, before `policy := startPolicy{…}` (claimed wakes never reach `doWake`'s planning block).
- `wake`: after the `a.afterWake(e, w, err)` line Task 9 added.
- `finishStop`: after `close(e.stopped)`.
- `sweepOnce`: as its last statement, after `a.mu.Unlock()`: one rewrite per sweep catches any change the points above miss.
-- plan 3's `swapOverlap`: directly before its `gb.StartGen(` call (the successor is already in the operation set), and plan 3's `leaveSet`: before `close(m.done)`.
-- plan 4's `respawnWorkerLocked` goroutine: after the `default:` arm installs the new child and unlocks (`a.mu.Unlock()` before its `log.Printf("activator: worker %s respawned …`).
+- plan 3's `swapOverlap`: directly before its `g, proc, spawnErr := a.spawnSuccessor(mctx, gb, e, rt, work, m.gen)` line (plan 3 Task 10; the successor is already in the operation set), and plan 3's `leaveSet`: before `close(m.done)`.
+- plan 4's `respawnWorkerLocked` goroutine, twice: directly after the "Still wanted?" block's `a.mu.Unlock()` (before the spawn, so a socketless attempt's `e.attemptBuilds` are on disk before its build execs), and after the `default:` arm installs the new child and unlocks (`a.mu.Unlock()` before its `log.Printf("activator: worker %s respawned …`).
- [ ] **Step 4: Put the good build in the pin sets**
@@ -5963,7 +6480,7 @@ git commit -m "zdd-5 task 12: the generations file; GoodVersion and generations
- Modify: `internal/status/snapshot.go` (on `17362eaf`: `Probes` ends at line 168 after `ReleaseRefusals`; `BoxMetrics` near 419; `CollectBox`'s "something known" condition at 826 and its `BoxMetrics` literal near 860)
- Test: `internal/status/snapshot_test.go` (append)
- Create: `internal/rollout/status.go`, `internal/rollout/status_test.go`
-- Modify: `internal/rollout/manager.go` (`Deps.OnChange`, `Manager.observing`), `scheduler.go` (`done`), `wakeplan.go` (`OnGen`)
+- Modify: `internal/rollout/manager.go` (`Deps.OnChange`, `Manager.observing`, `observe`, `retryBuild`), `scheduler.go` (`done`), `wakeplan.go` (`OnGen`)
**Interfaces:**
- Produces (plan 6 reads these names; they match plan 6's assumed rows):
@@ -6049,6 +6566,24 @@ func TestAFailedFirstWakeShowsAsHeld(t *testing.T) {
}
}
+// Plan 6's deploy watch counts a sleeper as "installed, awaiting first
+// wake" once its box reports B installed. No attempt runs for a sleeper, so
+// the pass that first sees B must republish.
+func TestANewlyInstalledBuildIsPublishedAtOnce(t *testing.T) {
+ h := newHarness(t, "a")
+ var changes atomic.Int32
+ h.m.d.OnChange = func() { changes.Add(1) }
+ h.act.mu.Lock()
+ delete(h.act.gens, "a") // asleep
+ h.act.mu.Unlock()
+ h.stage("v2")
+ h.install("v2", "a")
+ h.pass()
+ if changes.Load() == 0 {
+ t.Fatal("a newly installed build waited for the five-minute status tick")
+ }
+}
+
func TestReasonWords(t *testing.T) {
for text, want := range map[string]string{
"health check failed": status.ReasonHealth,
@@ -6065,11 +6600,11 @@ func TestReasonWords(t *testing.T) {
}
```
-Add `"github.com/carlosframework/platform/internal/release"` to that file's imports.
+Add `"sync/atomic"` and `"github.com/carlosframework/platform/internal/release"` to that file's imports.
- [ ] **Step 2: Run them to see them fail**
-Run: `go test ./internal/status/ ./internal/rollout/ -run 'TestTheRolloutReport|TestStatusReports|TestAFailedFirstWakeShows|TestReasonWords' -count=1`
+Run: `go test ./internal/status/ ./internal/rollout/ -run 'TestTheRolloutReport|TestStatusReports|TestAFailedFirstWakeShows|TestANewlyInstalledBuild|TestReasonWords' -count=1`
Expected: FAIL, `undefined: RolloutReport`.
- [ ] **Step 3: Add the report to status**
@@ -6165,7 +6700,19 @@ func (m *Manager) changed() {
}
```
-Call `m.changed()` as the last statement of `scheduler.done` (before the deferred `s.poke()` runs, so add it as a `defer m.changed()` directly after `defer s.poke()`) and of `Manager.OnGen`. In `OnGen`, add first:
+Call `m.changed()` as the last statement of `scheduler.done` (before the deferred `s.poke()` runs, so add it as a `defer m.changed()` directly after `defer s.poke()`) and of `Manager.OnGen`.
+
+A box whose instances are all asleep has no attempt to finish, so the installed build must be republished when the pass first sees it, or plan 6's deploy watch waits for the five-minute tick. In `Manager.observe`, declare `moved := false` before the `for k, list := range byApp {` loop, change `s.q.wantBuild(x.rt.Host, cause, x.rt.Version, now)` to `if s.q.wantBuild(x.rt.Host, cause, x.rt.Version, now) { moved = true }`, and directly before `return keys` add:
+
+```go
+ if moved {
+ m.changed() // a newly installed build is news for the deploy watch
+ }
+```
+
+In `retryBuild`, add `m.changed()` after its `s.poke()`: both `RetryHost` and an app-wide `carlos promote --retry` clear a hold the watch shows, and both go through it.
+
+In `OnGen`, add first:
```go
m.mu.Lock()
@@ -6293,7 +6840,7 @@ git commit -m "zdd-5 task 13: the rollout report in box status"
- Modify: `internal/host/restart.go` (`restartRequestLeg` at 860)
**Interfaces:**
-- Consumes: everything above; plan 1's `activator.SetGenerations`; plan 3's `Config.Swap`; `release.Channel`, `release.CanonicalDemand`, `appconfig.Get`, `appconfig.Callbacks.Delivered`.
+- Consumes: everything above; plan 1's `activator.SetGenerations`; plan 2's `ExecBackend.GenContainment`; `BoxLabel()` (`metrics_loop.go`); `release.Channel`, `release.CanonicalDemand`, `appconfig.Get`, `appconfig.Callbacks.Delivered`.
- Produces:
```go
func armedRollout() *rollout.Manager // nil unless CARLOS_SWAP=overlap; plan 6 calls it in fastConvergeLoop
@@ -6463,7 +7010,7 @@ func armRollout(ctx context.Context, getenv func(string) string, svc *activator.
}
platform := runtime.GOOS + "-" + runtime.GOARCH
m := rollout.NewManager(ctx, rollout.Deps{
- Act: svc, Routes: reg, Book: book, OptDir: release.OptDir(), DataDir: dd,
+ Act: svc, Routes: reg, Book: book, OptDir: release.OptDir(), DataDir: dd, Box: BoxLabel(),
Demand: func(ctx context.Context, acct, app, ver, path string) (string, string, error) {
return release.CanonicalDemand(ctx, store, acct, app, ver, platform, path)
},
@@ -6560,11 +7107,13 @@ func rolloutStatusProbe() *status.RolloutReport {
In `runAgent`, directly before `svc = activator.New(*cfg, &activator.SplitBackend{` (line 378), add:
```go
- // CARLOS_SWAP=overlap turns on generations (plan 1) and swaps
- // (plan 3) before the Service exists, which reads both once.
+ // CARLOS_SWAP=overlap turns on generations (plan 1), and with
+ // them plan 3's Swap, before the Service exists, which reads the
+ // switch once in New. Plan 2's per-generation containment goes on
+ // with them: an overlap needs each generation in its own child.
swapOn := rollout.Enabled(os.Getenv)
activator.SetGenerations(swapOn)
- cfg.Swap = swapOn
+ eb.GenContainment = swapOn
```
After `svc.Logs = logSink` (line 393, still before `startActivator(...)`), add:
@@ -6679,14 +7228,15 @@ git commit -m "zdd-5 task 14: wire CARLOS_SWAP=overlap"
**Interfaces:**
- Produces:
```go
- type stageRelay struct { /* pending map[string]string, get, touch */ }
+ type stageRelay struct { /* pending map[string]string, get, refused, touch */ }
func (r *stageRelay) note(host, ver string)
func (r *stageRelay) tick() bool
func armStageRelay(reg *registry.Registry, dd string)
func touchStageMarker(dd string)
```
+- Consumes: `release.ReadRefusal(bakeDir, host string) (*release.Refusal, error)` and `Refusal.Version` (on `main`, `9e628024`); root writes these records into the agent's data dir, its bake dir.
-Spec §7.7: "The agent touches it again every `stageRequestEvery` while any exec route's channel head isn't installed, so a touch missed during a running pass is repeated." The relay learns a channel head from the pass that could not stage it (`OnStageBlocked`), which runs only past bake holds; so it keeps asking until root's stamp shows that version installed, the route goes, or a newer head replaces it.
+Spec §7.7: "The agent touches it again every `stageRequestEvery` while any exec route's channel head isn't installed, so a touch missed during a running pass is repeated." The relay learns a channel head from the pass that could not stage it (`OnStageBlocked`), which runs only past bake holds. It keeps asking until root has recorded an outcome for that (host, version), and no longer: root's stamp shows the version installed, or root wrote a refusal record naming it (a wrong program, unproven bytes, or a unit that did not come up: the records from `9e628024` that every later pass reads before staging). It also stops when the route goes or a newer head replaces it. Without the refusal check a refused head would be touched every 15 s forever, and each touch runs a root pass.
- [ ] **Step 1: Write the failing test**
@@ -6700,14 +7250,18 @@ import (
"testing"
"github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
)
+func noRefusal(string) (*release.Refusal, error) { return nil, nil }
+
func TestTheRelayAsksAgainUntilTheHeadIsInstalled(t *testing.T) {
rows := map[string]*registry.Route{"a": {Host: "a", Version: "v1"}, "b": {Host: "b", Version: "v1"}}
touches := 0
r := &stageRelay{pending: map[string]string{},
- get: func(h string) (*registry.Route, error) { return rows[h], nil },
- touch: func() { touches++ }}
+ get: func(h string) (*registry.Route, error) { return rows[h], nil },
+ refused: noRefusal,
+ touch: func() { touches++ }}
if r.tick() || touches != 0 {
t.Fatal("touched with nothing outstanding")
}
@@ -6727,11 +7281,36 @@ func TestTheRelayAsksAgainUntilTheHeadIsInstalled(t *testing.T) {
t.Fatal("an unreadable row is not proof the head is installed")
}
}
+
+// Root's refusal record is an outcome too: once root has refused a
+// (host, version), asking again only reruns the refusal.
+func TestTheRelayStopsAtRootsRefusal(t *testing.T) {
+ rows := map[string]*registry.Route{"a": {Host: "a", Version: "v1"}, "b": {Host: "b", Version: "v1"}}
+ refusals := map[string]*release.Refusal{"a": {Host: "a", Version: "v2", Kind: release.RefusalWrongProgram}}
+ touches := 0
+ r := &stageRelay{pending: map[string]string{},
+ get: func(h string) (*registry.Route, error) { return rows[h], nil },
+ refused: func(h string) (*release.Refusal, error) { return refusals[h], nil },
+ touch: func() { touches++ }}
+ r.note("a", "v2")
+ if r.tick() || touches != 0 {
+ t.Fatal("touched for a head root has refused")
+ }
+ r.note("b", "v3")
+ refusals["b"] = &release.Refusal{Host: "b", Version: "v2"} // an older refusal names another version
+ if !r.tick() || touches != 1 {
+ t.Fatal("a refusal of another version is no outcome for this head")
+ }
+ r.refused = func(string) (*release.Refusal, error) { return nil, errors.New("permission denied") }
+ if !r.tick() {
+ t.Fatal("an unreadable refusal record is not an outcome")
+ }
+}
```
- [ ] **Step 2: Run it to see it fail**
-Run: `TMPDIR=/tmp/k go test ./internal/host/ -run TestTheRelayAsksAgain -count=1`
+Run: `TMPDIR=/tmp/k go test ./internal/host/ -run 'TestTheRelayAsksAgain|TestTheRelayStopsAtRootsRefusal' -count=1`
Expected: FAIL, `undefined: stageRelay`.
- [ ] **Step 3: Write the relay**
@@ -6750,16 +7329,21 @@ import (
"time"
"github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
)
// stageRelay retransmits the stage request (spec §7.7). carlos-stage.path
// does not act on a change made while carlos-stage.service is running, so a
// touch that lands mid-pass is lost; this asks again every stageRequestEvery
-// until root's stamp shows the head installed. Armed only with the flag on.
+// until root has recorded an outcome for the head: its stamp shows the head
+// installed, or its refusal record names it. Armed only with the flag on.
type stageRelay struct {
mu sync.Mutex
pending map[string]string // host -> the channel head a pass could not stage
get func(host string) (*registry.Route, error)
+ // refused reads root's refusal record for a host (release.ReadRefusal
+ // over the data dir): nil when there is none.
+ refused func(host string) (*release.Refusal, error)
touch func()
}
@@ -6770,7 +7354,8 @@ func (r *stageRelay) note(host, ver string) {
}
// tick touches the marker once if any head is still outstanding, and drops
-// the ones root has installed or whose route is gone.
+// the ones root has installed or refused, or whose route is gone. A row or a
+// record that cannot be read is no outcome, so it keeps the head pending.
func (r *stageRelay) tick() bool {
r.mu.Lock()
snap := make(map[string]string, len(r.pending))
@@ -6785,7 +7370,16 @@ func (r *stageRelay) tick() bool {
outstanding = true
continue
}
- if rt == nil || rt.Version == ver {
+ done := rt == nil || rt.Version == ver
+ if !done {
+ rec, rerr := r.refused(host)
+ if rerr != nil {
+ outstanding = true
+ continue
+ }
+ done = rec != nil && rec.Version == ver
+ }
+ if done {
r.mu.Lock()
if r.pending[host] == ver {
delete(r.pending, host)
@@ -6813,7 +7407,9 @@ func touchStageMarker(dd string) {
// armStageRelay starts the retransmission loop for the life of the process.
func armStageRelay(reg *registry.Registry, dd string) {
- r := &stageRelay{pending: map[string]string{}, get: reg.Get, touch: func() { touchStageMarker(dd) }}
+ r := &stageRelay{pending: map[string]string{}, get: reg.Get,
+ refused: func(host string) (*release.Refusal, error) { return release.ReadRefusal(dd, host) },
+ touch: func() { touchStageMarker(dd) }}
armedStageRelay.Store(r)
go func() {
t := time.NewTicker(stageRequestEvery)
@@ -7211,7 +7807,7 @@ Expected: no output from `gofmt`, every package `ok`, and three silent cross-bui
```bash
grep -rn 'CARLOS_SWAP\|rollout.Enabled' --include=*.go internal | grep -v _test.go
```
-Expected: `rollout.Enabled` is read only in `internal/host/agent.go` (the `SetGenerations`/`Config.Swap` line) and `internal/host/rollout.go`, and `SwapEnv` only in `internal/rollout`. With the flag unset, `TestTheFlagOffArmsNothing` (Task 14) and `TestWithoutAPlannerTheWakeIsTodays` (Task 9) are the proof.
+Expected: `rollout.Enabled` is read only in `internal/host/agent.go` (the `SetGenerations`/`GenContainment` lines) and `internal/host/rollout.go`, and `SwapEnv` only in `internal/rollout`. With the flag unset, `TestTheFlagOffArmsNothing` (Task 14) and `TestWithoutAPlannerTheWakeIsTodays` (Task 9) are the proof.
- [ ] **Step 3: Push and describe**
diff --git a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-surfaces.md b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-surfaces.md
index a2a0cf770..0aa7cabff 100644
--- a/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-surfaces.md
+++ b/docs/superpowers/plans/2026-10-07-zero-downtime-deploys-6-surfaces.md
@@ -14,7 +14,7 @@
## Global Constraints
-- `swap` takes `overlap` or `hold`; the default is `overlap`. `swap_concurrency` is an integer, 1 or more; the default is 1 (spec §4.1). Stored as `"swap"` and `"swap_concurrency"` in the default bundle (plan 5).
+- `swap` takes `overlap` or `hold`; the default is `overlap`. `swap_concurrency` is an integer, 1 or more; the default is 1 (spec §4.1). Stored as `"swap"` and `"swap_concurrency"` in the default bundle (plan 5 Task 3), defaults as absent keys: plan 5 reads `swap_concurrency` 0 as 1, and its `ValidSwapConcurrency` accepts 0, so this plan's doors refuse 0 themselves.
- The swap policy is an app setting, changed by app members, and reaches boxes with the app's config (spec §4.1, §11). The CLI door is person-only (`svcrole.Set{}`).
- `Manifest.Swap` is `"hold"` (`release.DemandHold`) or empty, additive, absent from older manifests (spec §4.2). A re-ship of the same bytes with a different demand is a different release (409).
- **Retry** (spec rev 14 §8.4): the app-wide retry is `carlos promote --retry <version> <channel>`, already on `main`. There is no signed retry object and no `carlos deploy --retry`. Every surface that reports a failed build names that command. `carlos instances retry <host>` clears one host's hold. `carlos restart` never clears a failure record.
@@ -38,21 +38,24 @@
## Names this plan takes from plans 4 and 5
-Plans 4 and 5 are written (plan 5 was still being revised when this plan was). Rows marked **assumed** name something plan 5 had not yet written down; **Task 0 checks every row against the merged code.** If a name differs, fix it in this plan (a find-and-replace on the design branch) before Task 1.
+Every row below is checked against the written plans 4 and 5 (their Interfaces blocks and code); **Task 0 checks every row again against the merged code.** If a name differs there, fix it in this plan (a find-and-replace on the design branch) before Task 1.
| Name | Where | From | Used in tasks |
|---|---|---|---|
-| `Manifest.Swap string` `json:"swap,omitempty"`; `const DemandHold = "hold"`; `release.Ship` refuses any other value | `internal/release` | plan 4 Task 1 | 7 |
-| `appconfig.Manifest.Swap string` `json:"swap,omitempty"`, `SwapConcurrency int` `json:"swap_concurrency,omitempty"`; `SwapOverlap = "overlap"`, `SwapHold = "hold"`; `ValidSwap(v string) bool` (`""`, overlap, hold); `ValidSwapConcurrency(n int) bool` (n ≥ 0) | `internal/appconfig/appconfig.go` | plan 5 Task 3 | 3, 4, 6 |
-| `registry.Route.ServingVersion string` | `internal/registry` | plan 5 Task 2 | 13 |
-| `func (m *rollout.Manager) RetryHost(host string) bool`: "the box half of `carlos instances retry`; plan 6 adds the door" | `internal/rollout` | plan 5 Task 8 | 11 |
-| event texts `textSwapping(build string, cause activator.Cause)`, `textSwapped(build)`, `textSwapFailed(build, serving, reason)`, `textSwapDown(build)`, `textCohortHalted(build, reason)`, `textFallback(failed, good)`, `textConfigPending(reason string, wait time.Duration)`, `textStagingRefused(build)`, all returning `string`, drafts that plan 6 rewrites | `internal/rollout/events.go` | plan 5 Task 8 | 14 |
-| **assumed:** `func armedRollout() *rollout.Manager` (nil unless `CARLOS_SWAP=overlap`), callable in `fastConvergeLoop` | `internal/host/rollout.go` | plan 5 (file listed, task not yet written) | 11 |
-| **assumed:** `func bounceText(worker bool, cause Cause) string`, the one place `BounceFor` words its event | `internal/activator/wakeplan.go` | plan 5 Task 10 | 14 |
-| **assumed:** `status.BoxMetrics.Rollout *RolloutReport` `json:"rollout,omitempty"`, published only with the flag on; `RolloutReport{Hosts map[string]HostRollout; Apps map[string]AppRollout}`, `Apps` keyed `"<acct>/<app>"`; `HostRollout{Account, App, Installed, Serving, Good string; Awake bool; Attempt, AttemptBuild, HeldBy, Reason string; ConfigPending bool}`; `AppRollout{Queued, Running, Settling int}` (`Settling` counts canary settles and first-wake observations) | `internal/status/snapshot.go` | plan 5 (`RolloutReport` named in its file list) | 8, 10, 11, 12 |
-| **assumed:** attempt words `status.AttemptQueued "queued"`, `AttemptRunning "running"`, `AttemptSettling "settling"`, `AttemptFailed "failed"`, `AttemptDown "down"`; reason words `status.ReasonHealth "health"`, `ReasonSettle "settle"`, `ReasonExit "exit"`, `ReasonStaging "staging"`, `ReasonReused "reused"` | `internal/status` | plan 5 | 2, 10 |
-| **assumed:** the box republishes `status/boxes/<label>.json` within 2 s of every rollout transition, not only on the 5-minute tick | `internal/host` | plan 5 | 10 |
-| **assumed:** a failed first wake marks its host `Attempt: AttemptFailed`, `AttemptBuild: B`, `HeldBy: B` | `internal/rollout` | plan 5 | 10 |
+| `release.Manifest.Swap string` `json:"swap,omitempty"`; `release.Ship` refuses any value but `""` and `DemandHold` | `internal/release/release.go` | plan 4 Task 1 | 7 |
+| `const DemandHold = "hold"`; `func NormalizeDemand(swap string) string` (boxes only; plan 6 never calls it) | `internal/release/demand.go` | plan 4 Task 1 | 7 |
+| `appconfig.Manifest.Swap string` `json:"swap,omitempty"`, `SwapConcurrency int` `json:"swap_concurrency,omitempty"`; `SwapOverlap = "overlap"`, `SwapHold = "hold"`; `ValidSwap(v string) bool` (`""`, overlap, hold); `ValidSwapConcurrency(n int) bool` (n ≥ 0: 0 is unset, meaning 1, so the doors refuse 0 themselves) | `internal/appconfig/appconfig.go` | plan 5 Task 3 | 3, 4, 6 |
+| `rollout.PolicyFromManifest(m *appconfig.Manifest) AppPolicy` (the box's reading, which `appconfig.PolicyOf` matches) | `internal/rollout/policy.go` | plan 5 Task 3 | 3 |
+| `registry.Route.ServingVersion string` | `internal/registry/registry.go` | plan 5 Task 2 | 13 |
+| `func (m *Manager) RetryHost(host string) bool`: "the box half of `carlos instances retry`; plan 6 adds the door" | `internal/rollout/manager.go` | plan 5 Task 8 | 11 |
+| event texts `textSwapping(build string, cause activator.Cause)`, `textSwapped(build string, drain time.Duration)` (drain: how long the retiring generation took to drain and exit), `textSwapFailed(build, serving, reason string)`, `textSwapDown(build string)`, `textCohortHalted(build, box, reason string)` (box: the box label; the event sink already carries account, app and host), `textFallback(failed, good string)`, `textConfigPending(reason string, wait time.Duration)`, `textStagingRefused(build string)`, all returning `string`, drafts that plan 6 rewrites; `reason` is the attempt's free diagnostic text | `internal/rollout/events.go` | plan 5 Task 8 | 14 |
+| `func reasonWord(text string) string`: an attempt's diagnostic text to one of the closed `status.Reason*` words; anything it does not recognise is `ReasonHealth` | `internal/rollout/status.go` | plan 5 Task 13 | 14 |
+| `func armedRollout() *rollout.Manager` (nil unless `CARLOS_SWAP=overlap`); armed in `runAgent` before `startActivator`, so it is set before `fastConvergeLoop` runs | `internal/host/rollout.go` | plan 5 Task 14 | 11 |
+| `func bounceText(worker bool, cause Cause) string`, the one place `BounceFor` and plan 5 Task 11's `ReplaceWorker` (a socketless attempt's same-entry stop-then-start) word the bounce event | `internal/activator/wakeplan.go` | plan 5 Task 9 | 14 |
+| `status.BoxMetrics.Rollout *RolloutReport` `json:"rollout,omitempty"` (and `Probes.Rollout`), filled by `rolloutStatusProbe()`, nil with the flag off; `RolloutReport{Hosts map[string]HostRollout; Apps map[string]AppRollout; Failures []RolloutFailure}`, `Apps` keyed `"<acct>/<app>"`; `HostRollout{Account, App, Installed, Serving, Good string; Awake bool; Attempt, AttemptBuild, HeldBy, Reason string; ConfigPending bool; ConfigError string}` (JSON `account`, `app`, `installed`, `serving`, `good`, `awake`, `attempt`, `attempt_build`, `held_by`, `reason`, `config_pending`, `config_error`); `AppRollout{Queued, Running, Settling int}` (`Settling` counts canary settles and first-wake observations); `RolloutFailure{Account, App, Build, Host, Mode, Reason string; PromotedAt, At time.Time; HeldDown []string}` (`Host` is the instance the build failed on) | `internal/status/snapshot.go` | plan 5 Task 13 | 8, 10, 11, 12 |
+| attempt words `status.AttemptQueued "queued"`, `AttemptRunning "running"`, `AttemptSettling "settling"`, `AttemptFailed "failed"`, `AttemptDown "down"`; reason words `status.ReasonHealth "health"`, `ReasonSettle "settle"`, `ReasonExit "exit"`, `ReasonStaging "staging"`, `ReasonReused "reused"` | `internal/status/snapshot.go` | plan 5 Task 13 | 2, 8, 10 |
+| What plan 5's report says, which the join and the watch rely on: `Serving` is `""` and `Awake` false for an asleep instance (`Awake` is `Serving != ""`), so an asleep instance's previous build is `Good`; **every** host a failure record holds (the one that failed and its siblings) reads `Attempt: AttemptFailed` (or `AttemptDown` for a hold-mode host left down), `AttemptBuild: B`, `HeldBy: B`, `Reason` a closed word, and only `Failures[].Host` tells the failed one apart (Task 8 does); a failed first wake reads the same (`TestAFailedFirstWakeShowsAsHeld`); `ConfigError` is free text and is never shown | `internal/rollout/status.go` | plan 5 Task 13 | 8, 9, 10, 12 |
+| The box republishes `status/boxes/<label>.json` about 2 s after a rollout transition: `Deps.OnChange` (called at the end of `scheduler.done` and `Manager.OnGen`, by a pass whose `observe` sees a newly installed build, and by `retryBuild`, so a box of sleepers reports B installed at once) sends on `statusNudge` (cap 1), and `statusLoop` sleeps 2 s, so bursts coalesce, then ticks | `internal/rollout/manager.go`, `internal/host/rollout.go`, `agent.go` | plan 5 Tasks 13, 14 | 10 |
## Choices this plan makes where the spec is silent
@@ -66,11 +69,11 @@ Plans 4 and 5 are written (plan 5 was still being revised when this plan was). R
## Spec and code questions (not resolved here; raised with Paul)
-- **Q1.** Box snapshots are written every `status.SnapshotPeriod` (5 minutes, `internal/status/fleet.go:119`). A watch that ends on scheduler state needs it within seconds. This plan assumes plan 5 republishes on every rollout transition.
-- **Q2.** Plan 5 has not yet written down its box-status report, the attempt and reason words, `armedRollout` or the bounce text's home (the **assumed** rows above).
+- **Q1.** Box snapshots are written every `status.SnapshotPeriod` (5 minutes, `internal/status/fleet.go:119`). A watch that ends on scheduler state needs them within seconds. Plan 5 republishes about 2 s after `scheduler.done` and `Manager.OnGen` (Tasks 13, 14), but not when a pass first observes B installed and starts no attempt. A box whose targeted instances are all asleep then shows B installed only at its next 5-minute tick, after the deploy watch's deadline. Plan 5's `Manager.Pass` (and `Nudge`) should call `m.changed()` when `observe` moves any host's installed build, and `RetryHost` after it clears a hold.
+- **Q2.** Plan 5's report marks every host a failure record holds as `failed`, the failed instance and its siblings alike, where spec §9.1 calls the siblings **held**. Task 8 tells them apart by `RolloutReport.Failures[].Host`. If plan 5 changes `Status()` to set `Attempt` only on the record's own host, that step in Task 8 becomes a no-op and can go.
- **Q3.** The first non-default swap policy on an app with no `config.json` creates one with no vars. `SyncExec` removes a route's config file for an absent bundle and writes one for a present-but-empty bundle, so every box sees a config delivery: today a bounce, under plan 5 a `config` attempt for every awake instance. §4.1 does not say whether that one-time restart is acceptable.
- **Q4.** Rev 14 dropped the signed retry object. A host retry now rides the narrowed-restart log, which is unsigned, so a box that can write the bucket could clear another host's hold. Spec §11 argued for signing only for the app-wide retry, which `promote --retry` keeps.
-- **Q5.** Plan 5's `textSwapped(build)` carries no drain time and `textCohortHalted(build, reason)` no app or box, though §9.4 lists both. Plan 5's failure reasons look like free text ("health check failed"), not closed words; `rolloutwords.Reason` then renders "it failed (health check failed)", which is unreviewed text inside reviewed copy.
+- **Q5 (resolved).** §9.4's "swapped (with the retiring generation's drain time)" and "cohort halted (app, box, build, reason)" are carried by plan 5's `textSwapped(build string, drain time.Duration)` and `textCohortHalted(build, box, reason string)`; app and host ride the event sink. Plan 5's reasons are free text ("health check failed"), so Task 14 passes every reason through plan 5's `reasonWord` before `rolloutwords.Reason`: no free text reaches reviewed copy, and `zdd.reason.unknown` is only reached by a console or CLI reading a word from a newer box.
## File Structure
@@ -96,7 +99,7 @@ Plans 4 and 5 are written (plan 5 was still being revised when this plan was). R
| `instances_e2e_test.go` | `apps set` and `instances retry` through a real console | 11 |
| `internal/console/instancepage.go`, `templates/app_instance.html`, `browserretry.go` (new) | installed, serving, failure hold, Retry | 12 |
| `internal/status/snapshot.go`, `internal/console/apisidecars.go`, `sidecars_cli.go` | `carlos sidecars ls` reads `ServingVersion` | 13 |
-| `internal/rollout/events.go` (plan 5's), `internal/activator/wakeplan.go` (plan 5's) | events read their words from `rolloutwords` | 14 |
+| `internal/rollout/events.go` (plan 5's), `internal/activator/wakeplan.go` (plan 5's); `internal/rollout/manager_test.go`, `wakeplan_test.go`, `internal/activator/wakeplan_test.go` (plan 5's) | events read their words from `rolloutwords`; plan 5's tests stop pinning draft wording | 14 |
| `docs/site/cli.md` | `apps set`, the watch, `--swap`, `promote --retry`, the migrations rule, `instances retry`, sidecars | 15 |
| `approved_copy_test.go`, `approved_copy_zdd_test.go` (new) | the approved-copy pin | 16 |
@@ -118,34 +121,39 @@ git merge-base --is-ancestor 9e628024 HEAD && echo "promote --retry present"
Expected: a count of at least 2 (plans 4 and 5 merged), then `promote --retry present`.
-- [ ] **Step 2: Check every assumed name.**
+- [ ] **Step 2: Check every name this plan takes from plans 4 and 5.**
```bash
cd /home/paulca/amadan.net/carlos/platform-zdd-6
grep -n 'Swap .*json:"swap,omitempty"' internal/release/release.go
-grep -rn 'DemandHold *=' internal/release/*.go | grep -v _test
-grep -n 'SwapConcurrency int' internal/appconfig/appconfig.go
-grep -rn 'func ValidSwap(\|func ValidSwapConcurrency(\|SwapOverlap *=' internal/appconfig/*.go | grep -v _test
-grep -n 'ServingVersion' internal/registry/registry.go | head -3
-grep -n 'func (m \*Manager) RetryHost(host string) bool' internal/rollout/*.go
-grep -n 'func textSwapping\|func textSwapped\|func textSwapFailed\|func textSwapDown\|func textCohortHalted\|func textFallback\|func textConfigPending\|func textStagingRefused' internal/rollout/events.go
-grep -rn 'func armedRollout' internal/host/*.go
-grep -rn 'func bounceText' internal/activator/*.go
+grep -n 'DemandHold *=\|func NormalizeDemand(swap string) string' internal/release/demand.go
+grep -n 'SwapConcurrency int *`json:"swap_concurrency,omitempty"`' internal/appconfig/appconfig.go
+grep -n 'func ValidSwap(v string) bool\|func ValidSwapConcurrency(n int) bool { return n >= 0 }\|SwapOverlap *=' internal/appconfig/appconfig.go
+grep -n 'func PolicyFromManifest(m \*appconfig.Manifest) AppPolicy' internal/rollout/policy.go
+grep -n 'ServingVersion string' internal/registry/registry.go
+grep -n 'func (m \*Manager) RetryHost(host string) bool' internal/rollout/manager.go
+grep -n 'func textSwapping(build string, cause activator.Cause) string\|func textSwapped(build string, drain time.Duration) string\|func textSwapFailed(build, serving, reason string) string\|func textSwapDown(build string) string\|func textCohortHalted(build, box, reason string) string\|func textFallback(failed, good string) string\|func textConfigPending(reason string, wait time.Duration) string\|func textStagingRefused(build string) string' internal/rollout/events.go
+grep -n 'func reasonWord(text string) string' internal/rollout/status.go
+grep -n 'func armedRollout() \*rollout.Manager' internal/host/rollout.go
+grep -n 'func bounceText(worker bool, cause Cause) string' internal/activator/wakeplan.go
grep -n 'Rollout \*RolloutReport' internal/status/snapshot.go
-grep -rn 'type RolloutReport struct\|type HostRollout struct\|type AppRollout struct' internal/status/*.go
-grep -rn 'AttemptQueued\|AttemptDown\|ReasonHealth\|ReasonReused' internal/status/*.go | grep -v _test
+grep -n 'type RolloutReport struct\|type HostRollout struct\|type AppRollout struct\|type RolloutFailure struct\|Failures \[\]RolloutFailure\|ConfigError *string' internal/status/snapshot.go
+grep -n 'AttemptQueued *=\|AttemptRunning *=\|AttemptSettling *=\|AttemptFailed *=\|AttemptDown *=\|ReasonHealth *=\|ReasonSettle *=\|ReasonExit *=\|ReasonStaging *=\|ReasonReused *=' internal/status/snapshot.go
+grep -n 'OnChange func()' internal/rollout/manager.go
+grep -n 'statusNudge' internal/host/rollout.go internal/host/agent.go
grep -rn 'func SetSwapPolicy\|func PolicyOf\|func GetSwapPolicy' internal/appconfig/ || echo "none: Task 3 adds them"
```
-Expected: every `grep` before the last prints at least one line, and `textSwapping` through `textStagingRefused` print eight; the last prints `none: Task 3 adds them`. If anything is missing or named differently, stop: fix the table and every use of the name in this plan, commit that on `design-zero-downtime-deploys`, and start again.
+Expected: every `grep` before the last prints at least one line; the `text*` grep prints eight, the status-types grep six, the attempt and reason grep ten, and `statusNudge` at least three (its declaration, the `OnChange` send, the `statusLoop` receive); the last prints `none: Task 3 adds them`. If anything is missing or named differently, stop: fix the table and every use of the name in this plan, commit that on `design-zero-downtime-deploys`, and start again.
- [ ] **Step 3: Confirm the flag gates the report and the manager.**
```bash
-grep -rn 'Rollout\b\|armedRollout' internal/host/*.go | grep -v _test | head
+grep -n 'p.Rollout = rolloutStatusProbe()' internal/host/agent.go
+grep -n 'func rolloutStatusProbe\|rollout.Enabled(getenv)\|rolloutMgr.Store(m)' internal/host/rollout.go
```
-Expected: the code filling `BoxMetrics.Rollout` and `armedRollout` both depend on `CARLOS_SWAP=overlap`. If a flag-off box publishes a report or arms a manager, stop and raise it: Task 10's "byte-identical with the flag off" relies on it.
+Expected: the probe line in `statusTick`, and in `rollout.go` the probe returning nil when `armedRollout()` is nil, `armRollout` returning early unless `rollout.Enabled`, and the one `rolloutMgr.Store(m)`: the code filling `BoxMetrics.Rollout` and `armedRollout` both depend on `CARLOS_SWAP=overlap`. If a flag-off box publishes a report or arms a manager, stop and raise it: Task 10's "byte-identical with the flag off" relies on it.
---
@@ -257,12 +265,12 @@ Expected, one file each: `deploy.go`, `docs/site/cli.md`, `docs/site/cli.md`, `i
{"id": "zdd.reason.exit", "section": "Rollout words (shared)", "label": "Reason: exit", "text": "it exited before it was ready", "context": "Same, for a process that quit before its health check."},
{"id": "zdd.reason.staging", "section": "Rollout words (shared)", "label": "Reason: staging", "text": "the copy on the box did not match the release", "context": "Same, for a staged binary whose checksum did not match."},
{"id": "zdd.reason.reused", "section": "Rollout words (shared)", "label": "Reason: reused name", "text": "this version name was shipped twice with different content, so ship it under a new name", "context": "Same, when a version name was reused for different bytes."},
- {"id": "zdd.reason.unknown", "section": "Rollout words (shared)", "label": "Reason: unknown", "text": "it failed (%s)", "context": "Fallback for a reason this binary does not know. %s is the box's own word or text for it."},
+ {"id": "zdd.reason.unknown", "section": "Rollout words (shared)", "label": "Reason: unknown", "text": "it failed (%s)", "context": "Fallback for a reason word this binary does not know, from a box newer than the CLI or console reading it. %s is the box's own one-word name for it, such as 'health'."},
{"id": "zdd.event.swapping", "section": "Events (platform log)", "label": "Swapping", "text": "starting build %s beside the running one (%s)", "context": "Platform log line when an overlap swap begins. Build, cause."},
- {"id": "zdd.event.swapped", "section": "Events (platform log)", "label": "Swapped", "text": "now serving build %s", "context": "Log line after the switch. Build."},
+ {"id": "zdd.event.swapped", "section": "Events (platform log)", "label": "Swapped", "text": "now serving build %s. The previous build finished its requests and stopped in %s", "context": "Log line after the switch. Build, then how long the old build took to finish its open requests and exit, like 1.2s or 25s."},
{"id": "zdd.event.swap-failed", "section": "Events (platform log)", "label": "Swap failed", "text": "build %s did not take over, because %s. Still serving build %s", "context": "Log line when the new build failed and the old one kept serving. Build, reason, serving build."},
{"id": "zdd.event.hold-down", "section": "Events (platform log)", "label": "Hold-mode failure", "text": "build %s failed and nothing is serving. This app swaps in hold mode, so nothing falls back by itself. Roll back with carlos rollback", "context": "Log line when a hold-mode build failed after the old one stopped. Build."},
- {"id": "zdd.event.cohort-halted", "section": "Events (platform log)", "label": "Rollout stopped", "text": "rollout of build %s stopped on this box, because %s. Instances that have not served it stay on their last good build", "context": "Log line when a box halts a rollout. Build, reason."},
+ {"id": "zdd.event.cohort-halted", "section": "Events (platform log)", "label": "Rollout stopped", "text": "rollout of build %s stopped on box %s, because %s. Instances on that box that have not served it stay on their last good build", "context": "Log line when a box halts a rollout. Build, box label, reason. The log already shows the app and the instance it came from."},
{"id": "zdd.event.fallback-served", "section": "Events (platform log)", "label": "Fallback served", "text": "build %s failed to start, so the last good build, %s, is serving instead", "context": "Log line when an asleep instance's first wake on a new build failed and the previous build answered."},
{"id": "zdd.event.config-pending", "section": "Events (platform log)", "label": "Config waiting", "text": "new config reached this box but is not in use yet, because %s. Trying again in %s", "context": "Log line when a config change could not start a process with it. Reason, wait like 2m0s."},
{"id": "zdd.event.staging-refused", "section": "Events (platform log)", "label": "Staging refused", "text": "build %s is not staged correctly on this box, so the box asked for it to be staged again", "context": "Log line when the staged binary did not match the release. Build."},
@@ -431,7 +439,7 @@ Expected: `git status --short` prints nothing.
- Create: `internal/rolloutwords/words.go`, `internal/rolloutwords/words_test.go`
**Interfaces:**
-- Consumes: `status.ReasonHealth`, `ReasonSettle`, `ReasonExit`, `ReasonStaging`, `ReasonReused` (plan 5, assumed), in the test only.
+- Consumes: `status.ReasonHealth`, `ReasonSettle`, `ReasonExit`, `ReasonStaging`, `ReasonReused` (plan 5 Task 13, `internal/status/snapshot.go`), in the test only.
- Produces: `func Cause(c string) string`, `func Reason(r string) string`; consts `EventSwapping`, `EventSwapped`, `EventSwapFailed`, `EventHoldDown`, `EventCohortHalted`, `EventFallbackServed`, `EventConfigPending`, `EventStagingRefused`, `EventBounceInstance`, `EventBounceSidecar`, `StateHeld`, `StateRollingOut`, `StateAsleep`, `StateConfigPending`. Imports only `fmt`, so `internal/rollout`, `internal/activator`, `internal/console` and the CLI can all use it.
- [ ] **Step 1: Write the failing test.** Create `internal/rolloutwords/words_test.go`:
@@ -478,8 +486,8 @@ func TestNoEmDashInAnyWord(t *testing.T) {
func TestVerbsAreWhereTheCallersPutThem(t *testing.T) {
words := allWords()
for name, want := range map[string]int{
- "EventSwapping": 2, "EventSwapped": 1, "EventSwapFailed": 3, "EventHoldDown": 1,
- "EventCohortHalted": 2, "EventFallbackServed": 2, "EventConfigPending": 2,
+ "EventSwapping": 2, "EventSwapped": 2, "EventSwapFailed": 3, "EventHoldDown": 1,
+ "EventCohortHalted": 3, "EventFallbackServed": 2, "EventConfigPending": 2,
"EventStagingRefused": 1, "EventBounceInstance": 1, "EventBounceSidecar": 1,
"StateHeld": 1, "StateRollingOut": 0, "StateAsleep": 1, "StateConfigPending": 0, "reasonUnknown": 1,
} {
@@ -579,10 +587,10 @@ func Reason(r string) string {
// plus the bounce. TestVerbsAreWhereTheCallersPutThem pins each verb count.
const (
EventSwapping = "starting build %s beside the running one (%s)" // build, cause
- EventSwapped = "now serving build %s" // build
+ EventSwapped = "now serving build %s. The previous build finished its requests and stopped in %s" // build, drain
EventSwapFailed = "build %s did not take over, because %s. Still serving build %s" // build, reason, serving
EventHoldDown = "build %s failed and nothing is serving. This app swaps in hold mode, so nothing falls back by itself. Roll back with carlos rollback" // build
- EventCohortHalted = "rollout of build %s stopped on this box, because %s. Instances that have not served it stay on their last good build" // build, reason
+ EventCohortHalted = "rollout of build %s stopped on box %s, because %s. Instances on that box that have not served it stay on their last good build" // build, box, reason
EventFallbackServed = "build %s failed to start, so the last good build, %s, is serving instead" // failed, good
EventConfigPending = "new config reached this box but is not in use yet, because %s. Trying again in %s" // reason, wait
EventStagingRefused = "build %s is not staged correctly on this box, so the box asked for it to be staged again" // build
@@ -2099,10 +2107,10 @@ Expected: `ok`, nothing else.
- Create: `internal/console/rolloutjoin_test.go`
**Interfaces:**
-- Consumes: `status.BoxMetrics.Rollout`, `HostRollout`, `AppRollout` (plan 5, assumed).
+- Consumes: `status.BoxMetrics.Rollout`, `RolloutReport` (`Hosts`, `Apps`, `Failures`), `HostRollout`, `AppRollout`, `RolloutFailure`, `status.AttemptFailed` (plan 5 Task 13).
- Produces:
- - `type rolloutView struct{ Box, Installed, Serving, Good string; Awake bool; Attempt, AttemptBuild, HeldBy, Reason string; ConfigPending, SchedulerIdle bool; account, app string }`, JSON keys `box`, `installed`, `serving`, `good`, `awake`, `attempt`, `attempt_build`, `held_by`, `reason`, `config_pending`, `scheduler_idle`.
- - `func joinBoxes(boxes []status.Box, now time.Time) boxJoin`; `func (j boxJoin) rollout(host, acct, app string) *rolloutView` (nil unless the report names this account and app). Task 13 adds the `serving` map to the same fold.
+ - `type rolloutView struct{ Box, Installed, Serving, Good string; Awake bool; Attempt, AttemptBuild, HeldBy, Reason string; ConfigPending, SchedulerIdle bool; account, app string }`, JSON keys `box`, `installed`, `serving`, `good`, `awake`, `attempt`, `attempt_build`, `held_by`, `reason`, `config_pending`, `scheduler_idle`. `HostRollout.ConfigError` (free text) is deliberately not carried.
+ - `func joinBoxes(boxes []status.Box, now time.Time) boxJoin`; `func (j boxJoin) rollout(host, acct, app string) *rolloutView` (nil unless the report names this account and app). A host plan 5 reports as `AttemptFailed` that is not the `Host` of a matching `Failures` record is joined with `Attempt` and `AttemptBuild` empty: held by a sibling's failure (spec §9.1), `HeldBy` kept. Task 13 adds the `serving` map to the same fold.
- Instance-list records carry `"rollout"` (omitted when nil); promote answers carry `"held_until"` (RFC 3339) when `held`.
- [ ] **Step 1: Write the failing tests.** Create `internal/console/rolloutjoin_test.go`:
@@ -2188,6 +2196,25 @@ func TestJoinBoxesIdleNeedsNothingQueuedRunningOrSettling(t *testing.T) {
}
}
+// Plan 5 reports every host a failure record holds as failed. Spec §9.1:
+// the one the build failed on is failed, its siblings are held.
+func TestJoinBoxesTellsFailedFromHeld(t *testing.T) {
+ failed := status.HostRollout{Account: "bab", App: "shop", Installed: "v2", Serving: "v1", Good: "v1", Awake: true,
+ Attempt: status.AttemptFailed, AttemptBuild: "v2", HeldBy: "v2", Reason: status.ReasonHealth}
+ b := status.Box{GeneratedAt: time.Now(), Label: "x", Metrics: &status.BoxMetrics{Rollout: &status.RolloutReport{
+ Hosts: map[string]status.HostRollout{"a": failed, "b": failed},
+ Apps: map[string]status.AppRollout{"bab/shop": {}},
+ Failures: []status.RolloutFailure{{Account: "bab", App: "shop", Build: "v2", Host: "a", Reason: status.ReasonHealth}},
+ }}}
+ j := joinBoxes([]status.Box{b}, time.Now())
+ if a := j.rollout("a", "bab", "shop"); a.Attempt != status.AttemptFailed || a.AttemptBuild != "v2" || a.HeldBy != "v2" {
+ t.Errorf("a, where v2 failed = %+v", a)
+ }
+ if sib := j.rollout("b", "bab", "shop"); sib.Attempt != "" || sib.AttemptBuild != "" || sib.HeldBy != "v2" || sib.Reason != status.ReasonHealth {
+ t.Errorf("b, held by a's failure = %+v", sib)
+ }
+}
+
func TestAPIPromoteHeldSaysWhenItLifts(t *testing.T) {
fx := newConsoleFixture(t)
if _, err := fx.st.CreateAppIn(OpsAccountID, "titogo", "paul@keymail.dev", time.Now()); err != nil {
@@ -2232,7 +2259,7 @@ func TestAPIPromoteHeldSaysWhenItLifts(t *testing.T) {
- [ ] **Step 2: Run them and see them fail.**
-Run: `GOFLAGS=-mod=mod GOPROXY=off go test ./internal/console/ -run 'TestAPIInstanceListCarriesTheRollout|TestJoinBoxes|TestAPIPromoteHeldSaysWhenItLifts' -count=1`
+Run: `GOFLAGS=-mod=mod GOPROXY=off go test ./internal/console/ -run 'TestAPIInstanceListCarriesTheRollout|TestJoinBoxes|TestAPIPromoteHeldSaysWhenItLifts' -count=1` (`TestJoinBoxes` covers `TestJoinBoxesIdleNeedsNothingQueuedRunningOrSettling` and `TestJoinBoxesTellsFailedFromHeld`)
Expected: FAIL, `undefined: rolloutView`.
- [ ] **Step 3: Write the join.** In `internal/console/apiinstances.go`, add to `instanceListRecord` after `DBBytes *int64 ...` (line 196):
@@ -2302,7 +2329,18 @@ func joinBoxes(boxes []status.Box, now time.Time) boxJoin {
j.dbBytes[host] = n
}
if ro := b.Metrics.Rollout; ro != nil {
+ // Plan 5 marks every host a failure record holds as failed. The
+ // record's own Host is the one the build failed on; the rest are
+ // held by it (spec §9.1), so their attempt is cleared and HeldBy
+ // kept. A hold-mode host left down (AttemptDown) stays as it is.
+ failedOn := map[[4]string]bool{}
+ for _, f := range ro.Failures {
+ failedOn[[4]string{f.Account, f.App, f.Build, f.Host}] = true
+ }
for host, hr := range ro.Hosts {
+ if hr.Attempt == status.AttemptFailed && !failedOn[[4]string{hr.Account, hr.App, hr.AttemptBuild, host}] {
+ hr.Attempt, hr.AttemptBuild = "", ""
+ }
a := ro.Apps[hr.Account+"/"+hr.App]
j.rollouts[host] = rolloutView{
Box: b.Label, Installed: hr.Installed, Serving: hr.Serving, Good: hr.Good, Awake: hr.Awake,
@@ -2397,10 +2435,14 @@ func TestInstalledAndServingCells(t *testing.T) {
if installedCell(old) != "v1" || servingCell(old) != "—" {
t.Errorf("no rollout: %q, %q", installedCell(old), servingCell(old))
}
- r := apiInstanceRecord{StagedVersion: "v1", Rollout: &apiRollout{Installed: "v2", Serving: "v1"}}
+ r := apiInstanceRecord{StagedVersion: "v1", Rollout: &apiRollout{Installed: "v2", Serving: "v1", Awake: true}}
if installedCell(r) != "v2" || servingCell(r) != "v1" {
t.Errorf("rollout cells = %q, %q", installedCell(r), servingCell(r))
}
+ asleep := apiInstanceRecord{StagedVersion: "v1", Rollout: &apiRollout{Installed: "v2", Good: "v1"}}
+ if servingCell(asleep) != "—" {
+ t.Errorf("an asleep instance serves nothing: %q", servingCell(asleep))
+ }
}
func TestRolloutReason(t *testing.T) {
@@ -2411,8 +2453,11 @@ func TestRolloutReason(t *testing.T) {
{nil, ""},
{&apiRollout{Installed: "v2", Serving: "v2", Awake: true}, ""},
{&apiRollout{Installed: "v2", Serving: "v1", Awake: true, HeldBy: "v2"}, fmt.Sprintf(rolloutwords.StateHeld, "v2")},
- {&apiRollout{Installed: "v2", Serving: "v1", HeldBy: "v2"}, fmt.Sprintf(rolloutwords.StateHeld, "v2")},
- {&apiRollout{Installed: "v2", Serving: "v1"}, fmt.Sprintf(rolloutwords.StateAsleep, "v2")},
+ // Plan 5 reports no serving build for an asleep instance: its last
+ // good build is what it ran.
+ {&apiRollout{Installed: "v2", Good: "v1", HeldBy: "v2"}, fmt.Sprintf(rolloutwords.StateHeld, "v2")},
+ {&apiRollout{Installed: "v2", Good: "v1"}, fmt.Sprintf(rolloutwords.StateAsleep, "v2")},
+ {&apiRollout{Installed: "v2", Good: "v2"}, ""},
{&apiRollout{Installed: "v2", Serving: "v1", Awake: true}, rolloutwords.StateRollingOut},
{&apiRollout{Installed: "v2", Serving: "v2", Awake: true, ConfigPending: true}, rolloutwords.StateConfigPending},
} {
@@ -2497,16 +2542,18 @@ func servingCell(r apiInstanceRecord) string {
}
// rolloutReason is the REASON cell (spec §9.1): why the serving build is not
-// the installed one, or that new config waits for a process (§8.1).
+// the installed one, or that new config waits for a process (§8.1). An asleep
+// instance serves nothing (plan 5 reports Serving "" for it), so its installed
+// build is compared with Good, the build it last ran.
func rolloutReason(r *apiRollout) string {
switch {
case r == nil:
return ""
case r.HeldBy != "":
return fmt.Sprintf(rolloutwords.StateHeld, r.HeldBy)
- case r.Installed != r.Serving && !r.Awake:
+ case !r.Awake && r.Installed != r.Good:
return fmt.Sprintf(rolloutwords.StateAsleep, r.Installed)
- case r.Installed != r.Serving:
+ case r.Awake && r.Installed != r.Serving:
return rolloutwords.StateRollingOut
case r.ConfigPending:
return rolloutwords.StateConfigPending
@@ -2567,7 +2614,7 @@ Expected: `ok`, nothing else.
- Modify: `deploy.go:359-362` (the held line), `:400-455` (the watch), `:1491-1580` (lift `checkTarget` out of `watchVersion`)
**Interfaces:**
-- Consumes: `apiRollout`, `apiInstanceRecord.Rollout`, `apiPromoteResult.HeldUntil` (Task 9); `rolloutwords.Reason` (Task 2); `status.Attempt*` (plan 5, assumed); `watchVersion`, `watchTarget`, `watchDiagnostics`, `watchOrigin` (existing).
+- Consumes: `apiRollout`, `apiInstanceRecord.Rollout`, `apiPromoteResult.HeldUntil` (Task 9); `rolloutwords.Reason` (Task 2); `status.AttemptQueued`, `AttemptRunning`, `AttemptSettling`, `AttemptFailed`, `AttemptDown`, `status.ReasonHealth`, `ReasonExit` (plan 5 Task 13; failed and held already told apart by Task 8's join; an asleep instance reports `Serving` `""`); `watchVersion`, `watchTarget`, `watchDiagnostics`, `watchOrigin` (existing).
- Produces:
- `func checkTarget(client *http.Client, t watchTarget, ver string, n int64) (live bool, diag string)`
- `type rolloutState string` (`rolloutPending`, `rolloutSwapped`, `rolloutAsleep`, `rolloutFailed`, `rolloutHeld`), `func (s rolloutState) final() bool`, `func classifyRollout(r *apiRollout, ver string) rolloutState`
@@ -2604,15 +2651,15 @@ func TestClassifyRollout(t *testing.T) {
want rolloutState
}{
{"box has not installed B", apiRollout{Installed: "b1", Serving: "b1", Awake: true}, rolloutPending},
- {"idle scheduler, B not installed", apiRollout{Installed: "b1", Serving: "b1", SchedulerIdle: true}, rolloutPending},
+ {"idle scheduler, B not installed", apiRollout{Installed: "b1", Good: "b1", SchedulerIdle: true}, rolloutPending},
{"swapped", apiRollout{Installed: B, Serving: B, Awake: true}, rolloutSwapped},
- {"asleep with B installed", apiRollout{Installed: B, Serving: "b1"}, rolloutAsleep},
+ {"asleep with B installed", apiRollout{Installed: B, Good: "b1"}, rolloutAsleep},
{"queued", apiRollout{Installed: B, Serving: "b1", Awake: true, Attempt: status.AttemptQueued, AttemptBuild: B}, rolloutPending},
{"canary settling", apiRollout{Installed: B, Serving: "b1", Awake: true, Attempt: status.AttemptSettling, AttemptBuild: B}, rolloutPending},
{"failed here", apiRollout{Installed: B, Serving: "b1", Awake: true, Attempt: status.AttemptFailed, AttemptBuild: B, HeldBy: B, Reason: status.ReasonHealth}, rolloutFailed},
{"hold mode, down", apiRollout{Installed: B, Attempt: status.AttemptDown, AttemptBuild: B, HeldBy: B}, rolloutFailed},
{"held by a sibling's failure", apiRollout{Installed: B, Serving: "b1", Awake: true, HeldBy: B}, rolloutHeld},
- {"asleep and held", apiRollout{Installed: B, Serving: "b1", HeldBy: B}, rolloutHeld},
+ {"asleep and held", apiRollout{Installed: B, Good: "b1", HeldBy: B}, rolloutHeld},
{"an older build's failure is not B's", apiRollout{Installed: B, Serving: B, Awake: true, Attempt: status.AttemptFailed, AttemptBuild: "b0"}, rolloutSwapped},
} {
r := tc.r
@@ -2659,11 +2706,11 @@ func TestWatchRolloutFollowsTheBoxToDone(t *testing.T) {
var calls atomic.Int64
list := sweeps(&calls,
[]apiInstanceRecord{rec("a.test", &apiRollout{Box: "x1", Installed: "b1", Serving: "b1", Awake: true, SchedulerIdle: true}),
- rec("b.test", &apiRollout{Box: "x1", Installed: "b1", Serving: "b1", SchedulerIdle: true})},
+ rec("b.test", &apiRollout{Box: "x1", Installed: "b1", Good: "b1", SchedulerIdle: true})},
[]apiInstanceRecord{rec("a.test", &apiRollout{Box: "x1", Installed: B, Serving: "b1", Awake: true, Attempt: status.AttemptSettling, AttemptBuild: B}),
- rec("b.test", &apiRollout{Box: "x1", Installed: B, Serving: "b1"})},
+ rec("b.test", &apiRollout{Box: "x1", Installed: B, Good: "b1"})},
[]apiInstanceRecord{rec("a.test", &apiRollout{Box: "x1", Installed: B, Serving: B, Awake: true, SchedulerIdle: true}),
- rec("b.test", &apiRollout{Box: "x1", Installed: B, Serving: "b1", SchedulerIdle: true})},
+ rec("b.test", &apiRollout{Box: "x1", Installed: B, Good: "b1", SchedulerIdle: true})},
)
if err := testWatch(&out, list, noURL(t)).run(targetsFor("a.test", "b.test")); err != nil {
t.Fatalf("watch: %v\n%s", err, out.String())
@@ -2688,7 +2735,7 @@ func TestWatchRolloutNoOpDeployEndsAtOnce(t *testing.T) {
var calls atomic.Int64
list := sweeps(&calls, []apiInstanceRecord{
rec("a.test", &apiRollout{Box: "x1", Installed: B, Serving: B, Awake: true, SchedulerIdle: true}),
- rec("b.test", &apiRollout{Box: "x1", Installed: B, Serving: B, SchedulerIdle: true}),
+ rec("b.test", &apiRollout{Box: "x1", Installed: B, Good: B, SchedulerIdle: true}),
})
if err := testWatch(io.Discard, list, noURL(t)).run(targetsFor("a.test", "b.test")); err != nil {
t.Fatal(err)
@@ -2721,7 +2768,7 @@ func TestWatchRolloutOpenSettleHoldsTheWatch(t *testing.T) {
var calls atomic.Int64
list := sweeps(&calls, []apiInstanceRecord{
rec("a.test", &apiRollout{Box: "x1", Installed: B, Serving: B, Awake: true}),
- rec("b.test", &apiRollout{Box: "x1", Installed: B, Serving: "b1"}),
+ rec("b.test", &apiRollout{Box: "x1", Installed: B, Good: "b1"}),
})
err := testWatch(io.Discard, list, noURL(t)).run(targetsFor("a.test", "b.test"))
if err == nil || !strings.Contains(err.Error(), fmt.Sprintf(watchBoxBusy, "x1")) {
@@ -3408,7 +3455,7 @@ The request rides the narrowed-restart log (`apps/<acct>/<app>/restart-scope.jso
- Test: `internal/release/restartscope_test.go`, `internal/host/restartscope_test.go`, `instances_retry_test.go` (new)
**Interfaces:**
-- Consumes: `rollout.Manager.RetryHost` (plan 5); `armedRollout` (plan 5, assumed); `boxJoin.rollout`, `loadBoxJoin` (Task 8); `leadingArg` (Task 5); `release.AppendRestartScope`, `release.LoadRestartScopes` (existing).
+- Consumes: `func (m *Manager) RetryHost(host string) bool` (plan 5 Task 8, `internal/rollout/manager.go`); `func armedRollout() *rollout.Manager` (plan 5 Task 14, `internal/host/rollout.go`); plan 5 Task 14's `restartRequestLeg` branch (`feedRestarts(routes, moved, scoped, mem.gen, m.Restart)` with the flag on); `boxJoin.rollout`, `loadBoxJoin` (Task 8); `leadingArg` (Task 5); `release.AppendRestartScope`, `release.LoadRestartScopes` (existing).
- Produces:
- `release.RestartScopeRetry = "retry"`, accepted by `ValidRestartScope`.
- `restartMemory.retryHost func(host string) bool` (nil: retries are logged and dropped).
@@ -3582,7 +3629,7 @@ In `internal/host/agent.go`, after `restartMem := newRestartMemory()` (line 2181
}
```
-If plan 5 arms the manager after this line, move these lines to just after plan 5's arming, inside the same function.
+`fastConvergeLoop` is where `restartMem` lives (plan 5 Task 14 edits its `convergeLegFor` call a few lines below). Plan 5 arms the manager in `runAgent`, directly after `svc.Logs = logSink` and before `startActivator`, so `armedRollout()` already answers when this line runs. The retry block above sits before plan 5's `feedRestarts` branch and strips the retries from `scoped`, so `feedRestarts` and `bounceScopedRoutes` never see one.
- [ ] **Step 4: Run the release and host packages.**
@@ -4150,15 +4197,19 @@ func (d *instanceDetail) PromoteRetryText() string {
// RolloutNote explains a serving build that differs from the installed one
// when no failure holds it: rolling out, or asleep until the next request.
+// Plan 5 reports Serving "" for an asleep instance, so asleep compares the
+// installed build with Good, the build it last ran.
func (d *instanceDetail) RolloutNote() string {
r := d.Rollout
- if r == nil || r.HeldBy != "" || r.Installed == r.Serving {
+ switch {
+ case r == nil || r.HeldBy != "":
return ""
- }
- if !r.Awake {
+ case !r.Awake && r.Installed != r.Good:
return fmt.Sprintf(swapCopy.Asleep, r.Installed)
+ case r.Awake && r.Installed != r.Serving:
+ return fmt.Sprintf(swapCopy.Rolling, r.Installed, r.Serving)
}
- return fmt.Sprintf(swapCopy.Rolling, r.Installed, r.Serving)
+ return ""
}
```
@@ -4308,7 +4359,7 @@ After plan 4, a sidecar's registry `Version` is root's install stamp, and the wo
- Test: `internal/status/snapshot_test.go`, `internal/console/apisidecars_test.go:702-756`
**Interfaces:**
-- Consumes: `registry.Route.ServingVersion` (plan 5).
+- Consumes: `registry.Route.ServingVersion` (plan 5 Task 2).
- Produces: `status.BoxMetrics.RouteServing map[string]string` `json:"route_serving,omitempty"`; `boxJoin.serving map[string]string`.
- [ ] **Step 1: Write the failing tests.** Append to `internal/status/snapshot_test.go`:
@@ -4417,16 +4468,17 @@ Expected: `ok` for each, nothing else. The status dry-run leak scan (`internal/s
### Task 14: Event wording
-Plan 5 words every rollout event through the text functions in `internal/rollout/events.go`, and the stop-then-start bounce through `bounceText` in `internal/activator/wakeplan.go` (assumed name). This task rewrites their bodies with the approved phrases; signatures stay. The flag-off literal in `activator.go` (`bounceLocked`) is not touched.
+Plan 5 words every rollout event through the eight text functions in `internal/rollout/events.go` (plan 5 Task 8), and the stop-then-start bounce through `bounceText` in `internal/activator/wakeplan.go` (plan 5 Task 9). This task rewrites their bodies with the approved phrases; signatures stay. Plan 5's emit sites pass the attempt's free diagnostic text as `reason` (`res.Reason`, `rec.Reason`), so every body that words a reason maps it through plan 5's `reasonWord` (Task 13) first: only the closed words reach `rolloutwords.Reason`, and no free text reaches reviewed copy. The flag-off literal in `activator.go` (`bounceLocked`) is not touched.
**Files:**
- Modify: `internal/rollout/events.go` (plan 5's)
- Modify: `internal/activator/wakeplan.go` (plan 5's `bounceText`)
- Create: `internal/rollout/events_words_test.go`
+- Modify (plan 5's tests that pin draft wording): `internal/rollout/manager_test.go`, `internal/rollout/wakeplan_test.go`, `internal/activator/wakeplan_test.go`
**Interfaces:**
-- Consumes: `rolloutwords.Event*`, `Cause`, `Reason` (Task 2); `activator.Cause` (index).
-- Produces: nothing new; plan 5's emit sites now print approved text.
+- Consumes: `rolloutwords.Event*`, `Cause`, `Reason` (Task 2); `activator.Cause`, `CauseRelease`, `CauseConfig` (index); plan 5 Task 8's `textSwapping(build string, cause activator.Cause)`, `textSwapped(build string, drain time.Duration)`, `textSwapFailed(build, serving, reason string)`, `textSwapDown(build string)`, `textCohortHalted(build, box, reason string)`, `textFallback(failed, good string)`, `textConfigPending(reason string, wait time.Duration)`, `textStagingRefused(build string)`; plan 5 Task 13's `reasonWord(text string) string` and `status.Reason*`; plan 5 Task 9's `bounceText(worker bool, cause Cause) string`.
+- Produces: nothing new; plan 5's emit sites now print approved text. A drain time prints rounded to a tenth of a second.
- [ ] **Step 1: Write the failing test.** Create `internal/rollout/events_words_test.go`:
@@ -4435,22 +4487,26 @@ package rollout
import (
"fmt"
+ "strings"
"testing"
"time"
"github.com/carlosframework/platform/internal/activator"
"github.com/carlosframework/platform/internal/rolloutwords"
+ "github.com/carlosframework/platform/internal/status"
)
+// The reasons below are plan 5's own diagnostic texts (its TestReasonWords),
+// as its emit sites pass them.
func TestEventTextsAreTheApprovedOnes(t *testing.T) {
cases := map[string][2]string{
"swapping": {textSwapping("v2", activator.CauseRelease), fmt.Sprintf(rolloutwords.EventSwapping, "v2", rolloutwords.Cause("release"))},
- "swapped": {textSwapped("v2"), fmt.Sprintf(rolloutwords.EventSwapped, "v2")},
- "swap failed": {textSwapFailed("v2", "v1", "health"), fmt.Sprintf(rolloutwords.EventSwapFailed, "v2", rolloutwords.Reason("health"), "v1")},
+ "swapped": {textSwapped("v2", 1234*time.Millisecond), fmt.Sprintf(rolloutwords.EventSwapped, "v2", "1.2s")},
+ "swap failed": {textSwapFailed("v2", "v1", "health check failed"), fmt.Sprintf(rolloutwords.EventSwapFailed, "v2", rolloutwords.Reason(status.ReasonHealth), "v1")},
"down": {textSwapDown("v2"), fmt.Sprintf(rolloutwords.EventHoldDown, "v2")},
- "halted": {textCohortHalted("v2", "settle"), fmt.Sprintf(rolloutwords.EventCohortHalted, "v2", rolloutwords.Reason("settle"))},
+ "halted": {textCohortHalted("v2", "box-7", "swap: successor exited during its settle"), fmt.Sprintf(rolloutwords.EventCohortHalted, "v2", "box-7", rolloutwords.Reason(status.ReasonSettle))},
"fallback": {textFallback("v2", "v1"), fmt.Sprintf(rolloutwords.EventFallbackServed, "v2", "v1")},
- "config pending": {textConfigPending("exit", 2*time.Minute), fmt.Sprintf(rolloutwords.EventConfigPending, rolloutwords.Reason("exit"), (2 * time.Minute).String())},
+ "config pending": {textConfigPending("exited within 10s of becoming healthy", 2*time.Minute), fmt.Sprintf(rolloutwords.EventConfigPending, rolloutwords.Reason(status.ReasonExit), (2 * time.Minute).String())},
"staging refused": {textStagingRefused("v2"), fmt.Sprintf(rolloutwords.EventStagingRefused, "v2")},
}
for name, c := range cases {
@@ -4459,6 +4515,21 @@ func TestEventTextsAreTheApprovedOnes(t *testing.T) {
}
}
}
+
+// No free text reaches reviewed copy: an unrecognised diagnostic is worded as
+// a closed reason, never quoted.
+func TestEventTextsNeverCarryTheDiagnostic(t *testing.T) {
+ const diag = "GET / answered 500"
+ for name, s := range map[string]string{
+ "swap failed": textSwapFailed("v2", "v1", diag),
+ "halted": textCohortHalted("v2", "box-7", diag),
+ "config pending": textConfigPending(diag, time.Minute),
+ } {
+ if strings.Contains(s, diag) || strings.Contains(s, "500") {
+ t.Errorf("%s carries the diagnostic: %q", name, s)
+ }
+ }
+}
```
- [ ] **Step 2: Run it and see it fail.**
@@ -4473,16 +4544,25 @@ func textSwapping(build string, cause activator.Cause) string {
return fmt.Sprintf(rolloutwords.EventSwapping, build, rolloutwords.Cause(string(cause)))
}
-func textSwapped(build string) string { return fmt.Sprintf(rolloutwords.EventSwapped, build) }
+// textSwapped's drain is how long the retiring generation took to drain and
+// exit, printed to a tenth of a second.
+func textSwapped(build string, drain time.Duration) string {
+ return fmt.Sprintf(rolloutwords.EventSwapped, build, drain.Round(100*time.Millisecond).String())
+}
+// reason, here and below, is the attempt's free diagnostic text: reasonWord
+// closes it to one of status's reason words before it is worded, so no free
+// text reaches reviewed copy.
func textSwapFailed(build, serving, reason string) string {
- return fmt.Sprintf(rolloutwords.EventSwapFailed, build, rolloutwords.Reason(reason), serving)
+ return fmt.Sprintf(rolloutwords.EventSwapFailed, build, rolloutwords.Reason(reasonWord(reason)), serving)
}
func textSwapDown(build string) string { return fmt.Sprintf(rolloutwords.EventHoldDown, build) }
-func textCohortHalted(build, reason string) string {
- return fmt.Sprintf(rolloutwords.EventCohortHalted, build, rolloutwords.Reason(reason))
+// box is the box's label; the event sink already carries account, app and
+// host (spec §9.4: "cohort halted (app, box, build, reason)").
+func textCohortHalted(build, box, reason string) string {
+ return fmt.Sprintf(rolloutwords.EventCohortHalted, build, box, rolloutwords.Reason(reasonWord(reason)))
}
func textFallback(failed, good string) string {
@@ -4490,7 +4570,7 @@ func textFallback(failed, good string) string {
}
func textConfigPending(reason string, wait time.Duration) string {
- return fmt.Sprintf(rolloutwords.EventConfigPending, rolloutwords.Reason(reason), wait.String())
+ return fmt.Sprintf(rolloutwords.EventConfigPending, rolloutwords.Reason(reasonWord(reason)), wait.String())
}
func textStagingRefused(build string) string {
@@ -4498,7 +4578,7 @@ func textStagingRefused(build string) string {
}
```
-In `internal/activator/wakeplan.go`, add the import and replace `bounceText`'s body with:
+In `internal/activator/wakeplan.go`, add the import `"github.com/carlosframework/platform/internal/rolloutwords"` (it already imports `fmt`) and replace `bounceText`'s body with:
```go
format := rolloutwords.EventBounceInstance
@@ -4508,6 +4588,63 @@ In `internal/activator/wakeplan.go`, add the import and replace `bounceText`'s b
return fmt.Sprintf(format, rolloutwords.Cause(string(cause)))
```
+- [ ] **Step 3b: Re-key plan 5's tests that pin its draft wording.** Plan 5's tests assert six draft phrases. Find each by its assertion in the named test (the draft literal is whatever plan 5's merged test carries; the lines below are plan 5 Task 8, 9 and 10's) and replace it so it holds for any approved wording. Where an argument is not known to the test (the drain time, the box label, the attempt's diagnostic), the assertion is the text up to that argument.
+
+In `internal/rollout/manager_test.go` (add `"strings"` and `"time"` to its imports if absent):
+
+```go
+ // TestAReleaseRollsOutCanaryFirstThenPaced, replacing h.events.has("swapped to build v2"):
+ swapped, _, _ := strings.Cut(textSwapped("v2", time.Hour), time.Hour.String())
+ if !h.events.has(swapped) {
+ t.Fatal("no swapped event")
+ }
+```
+
+```go
+ // TestAFailedCanaryHaltsTheCohort, replacing h.events.has("rollout of build v2 halted"):
+ halted, _, _ := strings.Cut(textCohortHalted("v2", "\x00", ""), "\x00")
+ if !h.events.has(halted) {
+ t.Fatal("no cohort-halted event")
+ }
+```
+
+```go
+ // TestStagingRefusalWritesNoFailureRecord, replacing h.events.has("not staged correctly"):
+ if !h.events.has(textStagingRefused("v2")) {
+ t.Fatal("no staging event")
+ }
+```
+
+```go
+ // TestAFailedConfigAttemptRetriesWithBackoff, replacing h.events.has("config delivered, not active"):
+ pending, _, _ := strings.Cut(rolloutwords.EventConfigPending, "%s")
+ waitFor(t, "the pending event", func() bool { return h.events.has(pending) })
+```
+
+(`manager_test.go` also gains the import `"github.com/carlosframework/platform/internal/rolloutwords"`.)
+
+In `internal/rollout/wakeplan_test.go`:
+
+```go
+ // TestAFellBackWakeRecordsTheFailureAndKeepsTheGoodBuild, replacing h.events.has("served build v1 instead"):
+ if !h.events.has(textFallback("v2", "v1")) {
+ t.Fatal("no fallback event")
+ }
+```
+
+In `internal/activator/wakeplan_test.go` (add `"fmt"` and `"github.com/carlosframework/platform/internal/rolloutwords"` to its imports):
+
+```go
+ // TestBounceForNamesTheCause, replacing sink.msgs[0] != "instance restarting (config)":
+ want := fmt.Sprintf(rolloutwords.EventBounceInstance, rolloutwords.Cause(string(CauseConfig)))
+ if len(sink.msgs) != 1 || sink.msgs[0] != want {
+ t.Fatalf("events = %v, want %q", sink.msgs, want)
+ }
+```
+
+Run: `cd /home/paulca/amadan.net/carlos/platform-zdd-6 && grep -rn 'events.has("\|msgs\[0\] != "' internal/rollout/ internal/activator/`
+Expected: nothing: no test pins a phrase by its literal any more.
+
- [ ] **Step 4: Run both packages and the flag-off proof.**
Run: `GOFLAGS=-mod=mod GOPROXY=off go test ./internal/rollout/ -count=1 && TMPDIR=/tmp/k GOFLAGS=-mod=mod GOPROXY=off go test ./internal/activator/ -count=1 && GOFLAGS=-mod=mod GOPROXY=off go vet ./internal/rollout/ ./internal/activator/ && gofmt -l internal/rollout internal/activator`
diff --git a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md
index ba5c65be9..771653717 100644
--- a/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md
+++ b/docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md
@@ -1,6 +1,6 @@
# Zero-downtime deploys: overlap swaps and paced rollout
-Status: rev 15, 2026-10-07. Rev 15 keeps the canonical socket path as a symlink to the current generation, for outside consumers (#103), found while planning. Rev 14 adopts `carlos promote --retry` from `main` (9e628024) as the retry signal, in place of a separate signed retry object. That is a simplification found while planning. **Accepted by Astra in round 13** (`CODEX-FINDINGS-ZERO-DOWNTIME-R13.md`). Awaiting Paul's review. Discussions #231 (deploy pickup) and #230
+Status: rev 16, 2026-10-07. Rev 16 says a sleeper's config is acknowledged at delivery with today's hibernating marker (§8.1). Rev 15 keeps the canonical socket path as a symlink to the current generation, for outside consumers (#103), found while planning. Rev 14 adopts `carlos promote --retry` from `main` (9e628024) as the retry signal, in place of a separate signed retry object. That is a simplification found while planning. **Accepted by Astra in round 13** (`CODEX-FINDINGS-ZERO-DOWNTIME-R13.md`). Awaiting Paul's review. Discussions #231 (deploy pickup) and #230
(per-app restart pace, first-route health step) feed into this. Astra
rejected rev 1 with 17 findings, rev 2 with 12 new ones, rev 3 with 7, rev 4
with 3 and rev 5 with 4. The findings files are
@@ -619,7 +619,10 @@ the cause:
*config delivered, not active*, with the last failure. A newer delivered
revision replaces the pending attempt. A sleeping instance needs no
attempt: its next wake starts with the delivered file, and that makes it
- active.
+ active. **A sleeper is acknowledged at delivery, with today's hibernating
+ marker** (`internal/appconfig/syncexec.go`). That marker already means
+ "active from the next wake", which is true for a sleeper, so the
+ "acknowledged only when active" rule above applies to awake instances.
### 8.2 A failed first wake (`overlap` mode)