amadan
pipelines slice 1: the ladder, enforced and visible (#11)
* release: the channel ladder as ordered vocabulary canary -> edge -> beta -> stable, frozen. Rung() maps a channel name onto its rung (only canary nests), RungIndex() gives its height, and PromotionAllowed() encodes the one rule the ladder exists for: beta is entered from edge and stable from beta, so "it was signed off on beta" means something. A fresh build enters at edge, because merging is what puts a build there; canary has no gate in either direction, because production runs on canary/* channels right now and a canary is proven before it is a candidate at all. Every refusal names -hotfix: an invisible bypass is indistinguishable from a broken gate. Vocabulary only — no store, no pointer, no route touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * promote: enforce the ladder, with -hotfix as the one bypass carlos promote now reads the app's existing channel pointers to work out which rung the version already occupies, and refuses a skip: beta is entered from edge, stable from beta. The refusal names -hotfix, which bypasses ordering and is reported as a bypass. Three deliberate non-refusals, because the fleet depends on them: canary/* is never gated (go.tito.io and keymail.dev run on canary channels today), a repeat is a no-op success so running the same command twice cannot fail, and a legacy-key app is never short-circuited into staying on its legacy key. Enforcement lives in the new PromoteOnLadder, not in Promote: the migration verb and the console's canary flow keep the unchecked primitive, so nothing that legitimately points a channel outside the ladder changes behaviour. Promotion still moves a signed pointer and nothing else — no route, no environment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * release: record every promotion on the pointer it moves The channel pointer grows an audit trail: version, from-rung, to-rung, actor, hotfix, timestamp — newest first, capped at 20. It rides inside the SIGNED payload, so the trail is exactly as trustworthy as the pointer itself; a log that can be edited without breaking the signature proves nothing. Additive, and tested as such: pointers written before this change carry no "audit" key, parse with none, verify, promote and roll back exactly as before. There are live ones in production buckets, and the fleet is still running binaries that predate the field — which is fine in both directions, because the signature covers the payload STRING and unknown JSON keys are ignored. Promote takes the record as a variadic argument, so every mechanical caller (migration, tests) is untouched and unrecorded, while the two paths a person actually promotes through — carlos promote and the console — name who did it. Rollback appends nothing (it is not a promotion) but carries the trail forward rather than erasing it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * console: the ladder, on the page and on the promote path One section on the app page: canary, edge, beta, stable, each with what it holds, when it got there and who put it there, and the hosts following it listed under the channel they subscribe to. The audit trail is a day old, so most pointers have none — those rungs say "by whom is not recorded" rather than crashing or guessing. The promote controls between adjacent rungs post to the app's EXISTING promote endpoint rather than a new one, and that endpoint moved from release.Promote to release.PromoteOnLadder. That is the substance of this change: the console drew a ladder it did not enforce, so a hand-written POST could put anything on stable. Now the same door serves the button and the POST, it refuses skips, and it reports a repeat as the no-op it is. Stable carries its own gate (spec §3): an account owner, then requireFreshAuth — called LAST, after the ownership check and after CSRF, so a member probing it still meets the byte-identical 404 rather than a redirect into Keymail that would confirm what the 404 hides. A member promoting to beta is unchanged, which is the gate table's own rule. A canary route is badged; a canary on the app's primary host is badged unmissably, because an unreviewed build in front of every user is exactly what spec §5 refuses to leave discoverable only by reading the registry. "Production" is inferred from the host being the app's apex, and the inference lives alone in prodWiredCanaryHosts with a note saying to delete it whole when named environments land. Three existing tests walked paths the ladder now refuses (v2 lives on a canary channel, and canary never climbs). They were rewritten to walk a build up the ladder instead, or to seed the state they were really about through the release primitive. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * adopt: arm the timer on install, and show when it isn't carlos-adopt.timer was enabled on the flagship and never once active: `systemctl enable` writes the .wants symlink and stops there, so the timer does not arm until the box next reboots — and nothing had. No NEXT, no LAST, no release ever adopted itself, and every deploy silently needed someone to run the adoption pass by hand. A scheduled channel flip would have been a no-op nobody noticed. Two halves, because fixing the install path without making the failure visible would just mean the next variant of it is silent too. Install: bootstrap's systemctl sequence is now one list (systemctlPlan) that a live run executes and a staging run prints, so the two can't drift. Timers get an explicit start after the enable; services deliberately don't, since carlos-edge still needs its binary and host.env staged first. A staging root (-root <dir>) executes nothing at all — that path is how a LIVE box refreshes its unit files, and arming from inside a temp root would reach out of it. Visibility: the box snapshot carries adoption_running — true, false, or a loud null when the probe couldn't ask systemd. It aggregates worst-first (one known-dead timer beats any number of healthy boxes; an unknown blocks a cheerful true) and appears per-box as well as fleet-wide, because "something isn't adopting" without a label just means ssh'ing into every box in turn. The allowlist gains the field explicitly: it says whether the platform's own timer is armed on an opaquely-labelled machine, and nothing about a tenant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * release: pin the channel names production is actually running on The ladder is only safe to ship because promotion into a canary channel is never refused, and the two names that matter are not hypothetical: go.tito.io follows canary/platform-adoption and keymail.dev follows canary/keymail today. A ladder change that refused either would mean production could no longer be deployed, and no other test names them. Also pins that edge -> stable is refused and that the refusal tells the operator about -hotfix, since a gate whose bypass is undiscoverable gets worked around in worse ways. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> ---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
27 files changed,
+1889
−83
cli.go+33 −3internal/console/icons.go+23 −3internal/console/ladder.go+226 −0internal/console/ladder_test.go+241 −0internal/console/server.go+59 −5internal/console/server_test.go+41 −19internal/console/templates/app.html+2 −0internal/console/templates/fleet.html+6 −0internal/console/templates/ladder.html+64 −0internal/console/templates/layout.html+33 −0internal/console/view.go+23 −0internal/host/agent.go+5 −0internal/host/bootstrap.go+53 −10internal/host/host_test.go+100 −0internal/host/probes.go+40 −0internal/release/ladder.go+110 −0internal/release/ladder_test.go+100 −0internal/release/laddercheck_test.go+46 −0internal/release/release.go+202 −7internal/release/release_test.go+260 −0internal/status/assets/index.html+11 −0internal/status/contract_test.go+24 −18internal/status/dryrun.go+7 −5internal/status/fleet.go+58 −6internal/status/snapshot.go+22 −7internal/status/status_test.go+98 −0main.go+2 −0
diff --git a/cli.go b/cli.go| index 339f7f896..59bc6d033 100644 |
| --- a/cli.go |
| +++ b/cli.go |
| @@ -268,18 +268,48 @@ func runShip(args []string) { |
| fmt.Printf("shipped %s %s (%d artifacts)\n", *app, *ver, len(files)) |
| } |
| +// runPromote points a channel at a shipped version, with the ladder enforced: |
| +// beta is entered from edge and stable from beta, so "it was signed off on |
| +// beta" means something. Canary channels are never refused (the fleet runs on |
| +// them), a repeated command is a no-op success, and -hotfix is the one |
| +// bypass — explicit, so it can be seen. |
| func runPromote(args []string) { |
| fs := flag.NewFlagSet("promote", flag.ExitOnError) |
| app := fs.String("app", "", "app name") |
| + hotfix := fs.Bool("hotfix", false, "bypass the channel ladder (recorded as a hotfix)") |
| acctOf := accountFlags(fs) |
| fs.Parse(args) |
| if *app == "" || fs.NArg() != 2 { |
| - log.Fatal("usage: carlos promote -app <a> <version> <channel>") |
| + log.Fatal("usage: carlos promote -app <a> [-hotfix] <version> <channel>") |
| } |
| - if err := release.Promote(context.Background(), openStore(), signingKey(), acctOf(), *app, fs.Arg(1), fs.Arg(0)); err != nil { |
| + version, channel := fs.Arg(0), fs.Arg(1) |
| + res, err := release.PromoteOnLadder(context.Background(), openStore(), signingKey(), |
| + acctOf(), *app, channel, version, |
| + release.PromoteOptions{Hotfix: *hotfix, Actor: promoteActor()}) |
| + if err != nil { |
| log.Fatal(err) |
| } |
| - fmt.Printf("promoted %s %s -> %s\n", *app, fs.Arg(0), fs.Arg(1)) |
| + switch { |
| + case res.NoOp: |
| + fmt.Printf("%s %s is already on %s — nothing to do\n", *app, version, channel) |
| + case res.Hotfix: |
| + fmt.Printf("promoted %s %s -> %s (HOTFIX: ladder bypassed, recorded)\n", *app, version, channel) |
| + default: |
| + fmt.Printf("promoted %s %s -> %s\n", *app, version, channel) |
| + } |
| +} |
| + |
| +// promoteActor names who is promoting, for the pointer's audit trail: the |
| +// logged-in console identity when this terminal has one, and otherwise the |
| +// operator's user@host. Never fatal and never blocking — a release verb runs |
| +// from CI as often as from a laptop, and "we don't know who" is a worse |
| +// answer than "someone on this machine", but neither is worth refusing a |
| +// promotion over. |
| +func promoteActor() string { |
| + if c, err := loggedInCredential(""); err == nil && c.Address != "" { |
| + return c.Address |
| + } |
| + return defaultLabel() |
| } |
| func runChannels(args []string) { |
diff --git a/internal/console/icons.go b/internal/console/icons.go| index 2597ae982..0956bb164 100644 |
| --- a/internal/console/icons.go |
| +++ b/internal/console/icons.go |
| @@ -60,14 +60,34 @@ const ( |
| // "New account", in the same column the check sits in (Lucide: plus). |
| iconPlus template.HTML = iconOpen + `<path d="M5 12h14"/><path d="M12 5v14"/></svg>` |
| + |
| + // The step between two rungs of the ladder. It points DOWN because the |
| + // ladder is drawn the way the spec draws it — canary at the top, stable |
| + // at the foot — so the arrow follows the page, not the metaphor |
| + // (Lucide: arrow-down). |
| + iconArrowDown template.HTML = iconOpen + `<path d="M12 5v14"/><path d="m19 12-7 7-7-7"/></svg>` |
| + |
| + // A route following a canary channel (Lucide: flask-conical). |
| + iconFlask template.HTML = iconOpen + |
| + `<path d="M14 2v6a2 2 0 0 0 .245.96l5.51 10.08A2 2 0 0 1 18 22H6a2 2 0 0 1-1.755-2.96l5.51-10.08A2 2 0 0 0 10 8V2"/>` + |
| + `<path d="M6.453 15h11.094"/><path d="M8.5 2h7"/></svg>` |
| + |
| + // A canary that looks production-wired — the one badge on the app page |
| + // that has to stop a reader (Lucide: triangle-alert). |
| + iconTriangleAlert template.HTML = iconOpen + |
| + `<path d="m21.73 18-8-14a2 2 0 0 0-3.48 0l-8 14A2 2 0 0 0 4 21h16a2 2 0 0 0 1.73-3"/>` + |
| + `<path d="M12 9v4"/><path d="M12 17h.01"/></svg>` |
| ) |
| // icons maps each Lucide slug to its vendored markup. Slugs are the Lucide |
| // names so the mapping back to lucide.dev stays obvious. |
| var icons = map[string]template.HTML{ |
| - "chevron-down": iconChevronDown, |
| - "check": iconCheck, |
| - "plus": iconPlus, |
| + "chevron-down": iconChevronDown, |
| + "check": iconCheck, |
| + "plus": iconPlus, |
| + "arrow-down": iconArrowDown, |
| + "flask-conical": iconFlask, |
| + "triangle-alert": iconTriangleAlert, |
| } |
| // icon renders one chrome icon by its Lucide slug: {{icon "chevron-down"}}. |
diff --git a/internal/console/ladder.go b/internal/console/ladder.go| new file mode 100644 |
| index 000000000..cb05575a0 |
| --- /dev/null |
| +++ b/internal/console/ladder.go |
| @@ -0,0 +1,226 @@ |
| +package console |
| + |
| +// The app page's ladder section — the console half of the pipelines spec |
| +// (§3 the ladder, §5 guard rails, §7 console surface). |
| +// |
| +// One screen that answers "what is on each rung, who put it there, and which |
| +// hosts are serving it", plus the only promotions the ladder permits: the |
| +// step from one rung to the next. It is assembled from what the page already |
| +// reads — the app's channel pointers and its registry routes — so it can |
| +// disagree with neither. |
| +// |
| +// Two things it deliberately is not. It is not a second promote path: the |
| +// controls post to the app's one promote endpoint, which enforces the ladder |
| +// and the passkey gate for everybody, so a hand-written POST meets the same |
| +// doors as a button. And it is not an environment view — environments do not |
| +// exist yet (spec §4, deferred), which is why the one place this file guesses |
| +// at "production" is quarantined in prodWiredCanaryHosts below. |
| + |
| +import ( |
| + "sort" |
| + "strings" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// LadderRung is one rung of the app's ladder, with everything sitting on it. |
| +type LadderRung struct { |
| + // Rung is the rung's name: canary, edge, beta or stable. |
| + Rung string |
| + // Gate is what it takes to get here, in one line (spec §3's gate table). |
| + Gate string |
| + // Channels are the channels belonging to this rung — one for edge, beta |
| + // and stable; any number for canary, which is the rung with names. |
| + Channels []LadderChannel |
| + // Up describes the step to the next rung: the promote control, or the |
| + // reason there is not one. nil on stable, which is the top. |
| + Up *LadderStep |
| +} |
| + |
| +// LadderChannel is one channel on a rung: what it holds and who is serving it. |
| +type LadderChannel struct { |
| + Channel, Version string |
| + PromotedAt time.Time |
| + // Actor is who promoted the version this channel holds, read from the |
| + // pointer's signed audit trail. "" is the ordinary case for now, not an |
| + // error: the trail is new and every pointer written before it has none. |
| + Actor string |
| + // Hotfix records that this promotion bypassed the ladder. |
| + Hotfix bool |
| + // Err carries a pointer that would not read or verify, and a channel that |
| + // hosts follow but that nothing has ever been published to. Both are one |
| + // line of bad news on one rung, never a failed page. |
| + Err string |
| + // Routes are the hosts subscribed to this channel. |
| + Routes []LadderRoute |
| +} |
| + |
| +// LadderRoute is one host under the channel it follows. |
| +type LadderRoute struct { |
| + Host string |
| + // Canary is whether this host follows a canary channel at all. |
| + Canary bool |
| + // ProdWired is the conspicuous case: a canary in front of everybody. See |
| + // prodWiredCanaryHosts for exactly how much this claim is worth. |
| + ProdWired bool |
| +} |
| + |
| +// LadderStep is the move from one rung to the one above it. |
| +type LadderStep struct { |
| + // To is the rung above. |
| + To string |
| + // Version is what the rung below currently holds — the version the |
| + // control would promote. "" when the rung below holds nothing, or holds |
| + // it on a pointer that will not read, and so there is nothing to offer. |
| + Version string |
| + // Note is shown instead of a control: why this step is not a promotion, |
| + // or why it cannot be offered right now. |
| + Note string |
| + // StepUp warns that taking this step will ask for a passkey. Display |
| + // only — the gate itself is server-side, in the promote handler. |
| + StepUp bool |
| +} |
| + |
| +// gates are spec §3's gate table, in the words a person reading the page |
| +// needs rather than the words the table uses. |
| +var gates = map[string]string{ |
| + release.RungCanary: "No gate — a canary is proven before it is a candidate, and may be wired to anything, including production.", |
| + release.RungEdge: "Merged to main. Nothing is promoted onto edge.", |
| + release.RungBeta: "Sign-off by an account member, recorded.", |
| + release.RungStable: "Sign-off by an account owner with a passkey, recorded, after the bake window.", |
| +} |
| + |
| +// assembleLadder folds an app's channels and routes into the four rungs. |
| +// |
| +// A channel outside the ladder's vocabulary (and any host following one) is |
| +// not on the ladder and does not appear here — the Channels table below still |
| +// lists it. A host following a rung channel that has no pointer DOES appear, |
| +// as an error line: a host serving nothing is exactly the kind of thing this |
| +// section exists to stop being invisible. |
| +func assembleLadder(channels []ChannelView, routes []RouteView) []LadderRung { |
| + prod := prodWiredCanaryHosts(routes) |
| + byChannel := map[string][]LadderRoute{} |
| + for _, rt := range routes { |
| + rung, ok := release.Rung(rt.Channel) |
| + byChannel[rt.Channel] = append(byChannel[rt.Channel], LadderRoute{ |
| + Host: rt.Host, |
| + Canary: ok && rung == release.RungCanary, |
| + ProdWired: prod[rt.Host], |
| + }) |
| + } |
| + entries := map[string][]LadderChannel{} |
| + for _, cv := range channels { |
| + rung, ok := release.Rung(cv.Name) |
| + if !ok { |
| + continue |
| + } |
| + entries[rung] = append(entries[rung], LadderChannel{ |
| + Channel: cv.Name, Version: cv.Version, PromotedAt: cv.PromotedAt, |
| + Actor: cv.Actor, Hotfix: cv.Hotfix, Err: cv.Err, |
| + Routes: byChannel[cv.Name], |
| + }) |
| + delete(byChannel, cv.Name) |
| + } |
| + // Whatever is left is a host following a channel with no pointer behind |
| + // it. Only rung channels are shown, for the reason in the doc comment. |
| + for ch, rts := range byChannel { |
| + rung, ok := release.Rung(ch) |
| + if !ok { |
| + continue |
| + } |
| + entries[rung] = append(entries[rung], LadderChannel{ |
| + Channel: ch, Routes: rts, |
| + Err: "nothing is published on this channel — these hosts are following a pointer that does not exist", |
| + }) |
| + } |
| + out := make([]LadderRung, 0, 4) |
| + ladder := []string{release.RungCanary, release.RungEdge, release.RungBeta, release.RungStable} |
| + for i, rung := range ladder { |
| + chans := entries[rung] |
| + // Map iteration ordered nothing above; canary is the rung that can |
| + // hold several, and a list that reshuffles per request is unreadable. |
| + sort.Slice(chans, func(a, b int) bool { return chans[a].Channel < chans[b].Channel }) |
| + lr := LadderRung{Rung: rung, Gate: gates[rung], Channels: chans} |
| + if i+1 < len(ladder) { |
| + lr.Up = stepUp(rung, ladder[i+1], chans) |
| + } |
| + out = append(out, lr) |
| + } |
| + return out |
| +} |
| + |
| +// stepUp describes the move from one rung to the next: a control when the |
| +// ladder permits one and there is something to promote, otherwise the reason. |
| +// The rule it encodes is release.PromotionAllowed's, in the direction the page |
| +// draws — nothing is promoted onto edge, and the two real steps are edge→beta |
| +// and beta→stable. |
| +func stepUp(from, to string, chans []LadderChannel) *LadderStep { |
| + s := &LadderStep{To: to, StepUp: to == release.RungStable} |
| + if to == release.RungEdge { |
| + s.Note = "Nothing is promoted onto edge — merging to main is what puts a build there." |
| + return s |
| + } |
| + // edge and beta hold exactly one channel, named for the rung. |
| + for _, c := range chans { |
| + if c.Err == "" && c.Version != "" { |
| + s.Version = c.Version |
| + break |
| + } |
| + } |
| + if s.Version == "" { |
| + s.Note = "Nothing on " + from + " to promote." |
| + } |
| + return s |
| +} |
| + |
| +// prodWiredCanaryHosts is THE INFERENCE, and it is alone in this file on |
| +// purpose. |
| +// |
| +// Spec §5 wants a canary wired to a production environment flagged |
| +// conspicuously, because an unreviewed build serving everybody must not be |
| +// discoverable only by reading the registry. But environments (spec §4) do |
| +// not exist yet — they are a subsystem with their own plan — so there is |
| +// nothing on a route that says "production" today. |
| +// |
| +// Until there is, production is inferred from the route being the app's |
| +// PRIMARY host: the apex it is reached at, picked as fewest labels, then |
| +// shortest, then alphabetical. A canary on `keymail.dev` is in front of |
| +// everybody; a canary on `rc.keymail.dev` is a side door somebody opened on |
| +// purpose. It is a heuristic and it is allowed to be wrong in both directions |
| +// — it flags, it never restricts. |
| +// |
| +// WHEN ENVIRONMENTS LAND: delete the heuristic and answer from the route's |
| +// environment instead. This function is the whole of what has to change; its |
| +// callers only ever ask "is this host a prod-wired canary". |
| +func prodWiredCanaryHosts(routes []RouteView) map[string]bool { |
| + primary := "" |
| + better := func(a, b string) bool { |
| + la, lb := strings.Count(a, "."), strings.Count(b, ".") |
| + if la != lb { |
| + return la < lb |
| + } |
| + if len(a) != len(b) { |
| + return len(a) < len(b) |
| + } |
| + return a < b |
| + } |
| + for _, rt := range routes { |
| + if rt.Host == "" { |
| + continue |
| + } |
| + if primary == "" || better(rt.Host, primary) { |
| + primary = rt.Host |
| + } |
| + } |
| + out := map[string]bool{} |
| + for _, rt := range routes { |
| + if rt.Host != primary { |
| + continue |
| + } |
| + if rung, ok := release.Rung(rt.Channel); ok && rung == release.RungCanary { |
| + out[rt.Host] = true |
| + } |
| + } |
| + return out |
| +} |
diff --git a/internal/console/ladder_test.go b/internal/console/ladder_test.go| new file mode 100644 |
| index 000000000..b9bac411a |
| --- /dev/null |
| +++ b/internal/console/ladder_test.go |
| @@ -0,0 +1,241 @@ |
| +package console |
| + |
| +// The app page's ladder section (pipelines spec §3, §5, §7). Four properties, |
| +// in the order they would hurt if lost: the four rungs say what is on them, |
| +// the promote control is owner-only, `stable` cannot be reached without a |
| +// passkey, and a skip up the ladder is refused however it is posted. |
| + |
| +import ( |
| + "context" |
| + "net/http" |
| + "net/url" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + "github.com/carlosframework/platform/internal/registry" |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// shipVersion adds one more shipped release to the fixture app — fixture |
| +// state, built through the real primitive so the console reads it exactly as |
| +// it reads a real one. |
| +func shipVersion(t *testing.T, fx *consoleFixture, app, version string) { |
| + t.Helper() |
| + m := release.Manifest{ |
| + Schema: 1, App: app, Version: version, Kind: "binary", |
| + BuiltAt: "2026-07-03T10:00:00Z", |
| + Artifacts: map[string]release.Artifact{"linux-amd64": {File: app}}, |
| + } |
| + if err := release.Ship(context.Background(), fx.rel, release.OpsAccount, m, |
| + map[string][]byte{"linux-amd64": []byte("binary " + version)}); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| + |
| +// seedChannel points a channel at a version through the UNCHECKED primitive. |
| +// That is deliberate: these are arrangements of the world the console then has |
| +// to read and gate, not promotions under test. |
| +func seedChannel(t *testing.T, fx *consoleFixture, app, channel, version string, rec ...release.Promotion) { |
| + t.Helper() |
| + if err := release.Promote(context.Background(), fx.rel, fx.priv, |
| + release.OpsAccount, app, channel, version, rec...); err != nil { |
| + t.Fatal(err) |
| + } |
| +} |
| + |
| +// rungBlock cuts one rung's markup out of the rendered page, so "the beta rung |
| +// shows v4" is an assertion about the beta rung and not about the page |
| +// containing "v4" somewhere. |
| +func rungBlock(t *testing.T, body, rung string) string { |
| + t.Helper() |
| + _, after, ok := strings.Cut(body, `id="rung-`+rung+`"`) |
| + if !ok { |
| + t.Fatalf("no %s rung in the ladder:\n%s", rung, body) |
| + } |
| + if before, _, ok := strings.Cut(after, `id="rung-`); ok { |
| + return before |
| + } |
| + before, _, _ := strings.Cut(after, "</ol>") |
| + return before |
| +} |
| + |
| +// routeLine narrows further, to the one line a host is listed on. |
| +func routeLine(t *testing.T, block, host string) string { |
| + t.Helper() |
| + for _, li := range strings.Split(block, "<li") { |
| + if strings.Contains(li, ">"+host+"</a>") { |
| + return li |
| + } |
| + } |
| + t.Fatalf("host %s is not listed in:\n%s", host, block) |
| + return "" |
| +} |
| + |
| +// The four rungs, in order, each saying what it holds, when it got there and |
| +// who put it there. The audit trail is new, so most live pointers have none: |
| +// the rung that has one names the actor, and the rung that does not still |
| +// renders — saying so — rather than crashing or inventing a name. |
| +func TestLadderRendersFourRungs(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + shipVersion(t, fx, "kass", "v3") |
| + shipVersion(t, fx, "kass", "v4") |
| + seedChannel(t, fx, "kass", "edge", "v3", |
| + release.Promotion{Version: "v3", To: "edge", Actor: "ci@keymail.dev"}) |
| + seedChannel(t, fx, "kass", "beta", "v4", |
| + release.Promotion{Version: "v4", From: "edge", To: "beta", Actor: "kim@keymail.dev"}) |
| + |
| + c := signInPaul(t, fx) |
| + status, _, body := get(t, c, fx.u("/apps/kass")) |
| + if status != http.StatusOK { |
| + t.Fatalf("app page = %d", status) |
| + } |
| + // Fixture: stable v1 and canary/x v2, both written before the audit trail |
| + // existed; edge and beta seeded above. |
| + for rung, version := range map[string]string{ |
| + "canary": "v2", "edge": "v3", "beta": "v4", "stable": "v1", |
| + } { |
| + block := rungBlock(t, body, rung) |
| + if !strings.Contains(block, ">"+version+"<") { |
| + t.Errorf("the %s rung does not show %s:\n%s", rung, version, block) |
| + } |
| + } |
| + if b := rungBlock(t, body, "beta"); !strings.Contains(b, "kim@keymail.dev") { |
| + t.Errorf("the beta rung does not say who promoted it:\n%s", b) |
| + } |
| + if b := rungBlock(t, body, "edge"); !strings.Contains(b, "ci@keymail.dev") { |
| + t.Errorf("the edge rung does not say who promoted it:\n%s", b) |
| + } |
| + // The pre-audit pointer: rendered, and honest about what it cannot say. |
| + if b := rungBlock(t, body, "stable"); !strings.Contains(b, "not recorded") { |
| + t.Errorf("a pointer with no audit trail did not say so:\n%s", b) |
| + } |
| + // Routes sit under the rung they subscribe to. |
| + if b := rungBlock(t, body, "stable"); !strings.Contains(b, "kass.carlosapp.dev") { |
| + t.Errorf("the stable rung does not list the hosts following it:\n%s", b) |
| + } |
| +} |
| + |
| +// The promote control is the owner's. A member still sees the whole ladder — |
| +// visibility is not the thing being restricted — and is offered no button. |
| +func TestLadderPromoteControlIsOwnerOnly(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + shipVersion(t, fx, "kass", "v3") |
| + seedChannel(t, fx, "kass", "edge", "v3") |
| + |
| + c := signInPaul(t, fx) |
| + _, _, body := get(t, c, fx.u("/apps/kass")) |
| + if !strings.Contains(body, "Promote v3 to beta") { |
| + t.Fatalf("owner is offered no edge -> beta control:\n%s", body) |
| + } |
| + // The control posts to the app's one promote endpoint — the same door, |
| + // with the same ladder and step-up gates behind it. |
| + if !strings.Contains(body, `action="/a/bab/apps/kass/promote"`) { |
| + t.Fatal("the ladder control does not post to the promote endpoint") |
| + } |
| + |
| + if err := fx.st.AddMember(OpsAccountID, "sam@keymail.dev", RoleMember, time.Now()); err != nil { |
| + t.Fatal(err) |
| + } |
| + fx.setUser("sam@keymail.dev") |
| + cm := newBrowser(t) |
| + signUp(t, fx, cm, "code-sam-ladder") |
| + status, _, mbody := get(t, cm, fx.u("/apps/kass")) |
| + if status != http.StatusOK { |
| + t.Fatalf("member app page = %d, want 200", status) |
| + } |
| + if !strings.Contains(mbody, `id="rung-stable"`) { |
| + t.Fatal("a member cannot see the ladder at all") |
| + } |
| + if strings.Contains(mbody, "Promote v3 to beta") { |
| + t.Error("a member is offered the ladder's promote control") |
| + } |
| +} |
| + |
| +// Stable is the rung that puts a build in front of every user, so a live |
| +// session is not enough on its own. Without the passkey the pointer does not |
| +// move; with it, the held promotion resumes and is recorded. |
| +func TestLadderPromoteToStableNeedsStepUp(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + shipVersion(t, fx, "kass", "v3") |
| + seedChannel(t, fx, "kass", "beta", "v3") |
| + c := signInPaul(t, fx) |
| + csrf := sessionToken(t, fx, c) |
| + |
| + status, loc, _ := postForm(t, c, fx.u("/apps/kass/promote"), |
| + url.Values{"csrf": {csrf}, "channel": {"stable"}, "version": {"v3"}}) |
| + if status != http.StatusSeeOther || !strings.HasPrefix(loc, fx.fk.srv.URL) { |
| + t.Fatalf("un-stepped-up stable promotion = %d -> %q, want a redirect into Keymail", status, loc) |
| + } |
| + if p := fx.channelVersion(t, "kass", "stable"); p.Version != "v1" { |
| + t.Fatalf("stable moved to %s with no passkey confirmation", p.Version) |
| + } |
| + |
| + if status, _, body := completeStepUp(t, fx, c, loc); status != http.StatusSeeOther { |
| + t.Fatalf("resumed promotion = %d:\n%s", status, body) |
| + } |
| + p := fx.channelVersion(t, "kass", "stable") |
| + if p.Version != "v3" { |
| + t.Fatalf("stable = %s after a confirmed promotion, want v3", p.Version) |
| + } |
| + if len(p.Audit) == 0 { |
| + t.Fatal("the promotion recorded nothing on the pointer it moved") |
| + } |
| + if a := p.Audit[0]; a.Version != "v3" || a.From != "beta" || a.To != "stable" || |
| + a.Actor != "paul@keymail.dev" || a.Hotfix { |
| + t.Fatalf("audit[0] = %+v, want v3 beta->stable by paul, no hotfix", a) |
| + } |
| +} |
| + |
| +// A skip is refused whoever asks and however confirmed: edge -> stable never |
| +// happens through the console, because "it was signed off on beta" is only |
| +// worth anything if everything on stable came through beta. |
| +func TestLadderRefusesASkip(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + shipVersion(t, fx, "kass", "v3") |
| + seedChannel(t, fx, "kass", "edge", "v3") |
| + c := signInPaul(t, fx) |
| + csrf := sessionToken(t, fx, c) |
| + |
| + // postGated walks the passkey round trip, so this is the refusal of a |
| + // FULLY confirmed owner — the ladder, not the gate, is what stops it. |
| + postGated(t, fx, c, fx.u("/apps/kass/promote"), |
| + url.Values{"csrf": {csrf}, "channel": {"stable"}, "version": {"v3"}}) |
| + if p := fx.channelVersion(t, "kass", "stable"); p.Version != "v1" { |
| + t.Fatalf("a skip from edge moved stable to %s", p.Version) |
| + } |
| + if _, _, body := get(t, c, fx.u("/apps/kass")); !strings.Contains(body, "stable is entered from beta only") { |
| + t.Fatalf("no ladder refusal flash:\n%s", body) |
| + } |
| +} |
| + |
| +// A canary is badged; a canary on the app's primary host is badged |
| +// unmissably, because that is an unreviewed build serving everybody |
| +// (spec §5). Both badges are chrome, so both are icons, not glyphs. |
| +func TestProdWiredCanaryIsBadgedDistinctly(t *testing.T) { |
| + fx := newConsoleFixture(t) |
| + // The app's apex host follows a canary channel — and a side host follows |
| + // the same one, which is the ordinary, unremarkable case. |
| + fx.addRoute(t, registry.Route{Host: "kass.carlosapp.dev", Kind: "instance", Status: "ready", |
| + Version: "v2", App: "kass", Channel: "canary/x", Addr: "127.0.0.1:9001"}) |
| + fx.addRoute(t, registry.Route{Host: "rc.kass.carlosapp.dev", Kind: "instance", Status: "ready", |
| + Version: "v2", App: "kass", Channel: "canary/x", Addr: "127.0.0.1:9002"}) |
| + c := signInPaul(t, fx) |
| + _, _, body := get(t, c, fx.u("/apps/kass")) |
| + block := rungBlock(t, body, "canary") |
| + |
| + apex := routeLine(t, block, "kass.carlosapp.dev") |
| + if !strings.Contains(apex, "chip-prod") { |
| + t.Errorf("the app's primary host on a canary channel is not flagged:\n%s", apex) |
| + } |
| + side := routeLine(t, block, "rc.kass.carlosapp.dev") |
| + if !strings.Contains(side, "chip-canary") { |
| + t.Errorf("a canary route carries no canary badge:\n%s", side) |
| + } |
| + if strings.Contains(side, "chip-prod") { |
| + t.Errorf("a side canary is flagged as production-wired:\n%s", side) |
| + } |
| + if !strings.Contains(apex, "<svg") || !strings.Contains(side, "<svg") { |
| + t.Error("the badges are text glyphs, not icons") |
| + } |
| +} |
diff --git a/internal/console/server.go b/internal/console/server.go| index 7c26c3422..af9dde602 100644 |
| --- a/internal/console/server.go |
| +++ b/internal/console/server.go |
| @@ -1161,9 +1161,12 @@ func (s *Server) handleNewApp(w http.ResponseWriter, r *http.Request) { |
| http.Redirect(w, r, home+"apps/"+url.PathEscape(name), http.StatusSeeOther) |
| } |
| -// handlePromote points a channel at a shipped version, from the app page. |
| +// handlePromote points a channel at a shipped version. It is the app's ONE |
| +// promote door — the releases table posts to it, and so does every step of the |
| +// ladder section — which is what makes the ladder and the passkey gate below |
| +// worth anything: a hand-written POST meets exactly the doors a button does. |
| func (s *Server) handlePromote(w http.ResponseWriter, r *http.Request) { |
| - addr, raw, acct, _, app, ok := s.accountApp(w, r) |
| + addr, raw, acct, role, app, ok := s.accountApp(w, r) |
| if !ok { |
| return |
| } |
| @@ -1199,9 +1202,28 @@ func (s *Server) handlePromote(w http.ResponseWriter, r *http.Request) { |
| back("err", "Version "+version+" is not a shipped release of "+app+".") |
| return |
| } |
| - if err := release.Promote(ctx, s.cfg.Releases, s.cfg.Key, acct.Sqid, app, channel, version); err != nil { |
| + // The rung's own gate, LAST — after membership, after the app check, after |
| + // CSRF, and after every refusal that costs nothing. Order is load-bearing: |
| + // a non-member must meet the byte-identical 404 before step-up is ever |
| + // mentioned, or the redirect into Keymail would confirm what the 404 is |
| + // there to hide. |
| + if !s.promoteGate(w, r, addr, raw, role, channel) { |
| + return |
| + } |
| + // PromoteOnLadder, not Promote: the ladder is enforced here, not merely |
| + // drawn. It also records the promotion on the pointer it moves — what |
| + // moved, from which rung to which, and which member said so. |
| + res, err := release.PromoteOnLadder(ctx, s.cfg.Releases, s.cfg.Key, acct.Sqid, app, channel, version, |
| + release.PromoteOptions{Actor: addr}) |
| + if err != nil { |
| log.Printf("console: promote %s %s -> %s: %v", app, version, channel, err) |
| - back("err", "Promote failed: "+err.Error()) |
| + // The ladder's own refusals are the useful text here — they name the |
| + // rung this build is on and the one it would have to come through. |
| + back("err", "Promote refused: "+err.Error()) |
| + return |
| + } |
| + if res.NoOp { |
| + back("ok", channel+" already serves "+version+" — nothing to do.") |
| return |
| } |
| s.audit(acct, app, addr, "promote", "promoted "+version+" to "+channel) |
| @@ -1212,6 +1234,28 @@ func (s *Server) handlePromote(w http.ResponseWriter, r *http.Request) { |
| back("ok", msg) |
| } |
| +// promoteGate applies the gate the TARGET RUNG carries (pipelines spec §3's |
| +// gate table). Below stable, membership — which the caller has already |
| +// proved — is the gate, and this waves the promotion through. |
| +// |
| +// Stable is different in kind: it puts a build in front of every user of the |
| +// app, so it takes an account OWNER, and a live session is not sufficient |
| +// proof that the owner is the one asking. A member probing it gets the same |
| +// 404 a stranger gets, and only then, last of all, is the passkey demanded. |
| +// |
| +// It returns false when it has already written the response — either the 404 |
| +// or the step-up redirect — and the caller must simply return. |
| +func (s *Server) promoteGate(w http.ResponseWriter, r *http.Request, addr, raw, role, channel string) bool { |
| + if rung, _ := release.Rung(channel); rung != release.RungStable { |
| + return true |
| + } |
| + if role != RoleOwner { |
| + http.NotFound(w, r) |
| + return false |
| + } |
| + return s.requireFreshAuth(w, r, addr, raw) |
| +} |
| + |
| // handleRollback swaps a channel back to its previous version. |
| func (s *Server) handleRollback(w http.ResponseWriter, r *http.Request) { |
| addr, raw, acct, _, app, ok := s.accountApp(w, r) |
| @@ -1343,7 +1387,17 @@ func (s *Server) handleNewCanary(w http.ResponseWriter, r *http.Request) { |
| back("err", "Could not check the release store. Please try again.") |
| return |
| } |
| - if err := release.Promote(ctx, s.cfg.Releases, s.cfg.Key, acct.Sqid, app, channel, version); err != nil { |
| + // Plain Promote, deliberately, where handlePromote uses PromoteOnLadder: |
| + // the target is always canary/<name>, the one rung the ladder never |
| + // refuses in either direction, so enforcement here could only turn the |
| + // "new means new" refusal above into a less useful message. Nothing |
| + // reachable from this handler can name another rung. |
| + // |
| + // Creating a canary is still a promotion onto the canary rung — recorded |
| + // like any other, so "who put this build in front of users" has an answer |
| + // for the rung most likely to be wired to production. |
| + rec := release.Promotion{To: release.RungCanary, Actor: addr} |
| + if err := release.Promote(ctx, s.cfg.Releases, s.cfg.Key, acct.Sqid, app, channel, version, rec); err != nil { |
| log.Printf("console: canary %s %s -> %s: %v", app, version, channel, err) |
| back("err", "Creating the canary failed: "+err.Error()) |
| return |
diff --git a/internal/console/server_test.go b/internal/console/server_test.go| index d0a597ad1..f7fac57e2 100644 |
| --- a/internal/console/server_test.go |
| +++ b/internal/console/server_test.go |
| @@ -482,38 +482,55 @@ func TestPromoteFlow(t *testing.T) { |
| t.Fatal("keyed fixture rendered the disabled note") |
| } |
| - // Promote v2 to stable (select field). |
| + // A build on no rung yet enters the ladder at edge, and the free-text |
| + // field is what creates that channel the first time. (The fixture's own |
| + // builds cannot move: v1 is already on stable and v2 is a canary, and the |
| + // ladder promotes neither — see TestLadderRefusesASkip.) |
| + shipVersion(t, fx, "kass", "v3") |
| status, loc, _ := postForm(t, c, fx.u("/apps/kass/promote"), |
| - url.Values{"csrf": {csrf}, "version": {"v2"}, "channel": {"stable"}}) |
| + url.Values{"csrf": {csrf}, "version": {"v3"}, "channel": {"stable"}, "new_channel": {"edge"}}) |
| if status != http.StatusSeeOther || loc != fx.path("/apps/kass") { |
| t.Fatalf("promote = %d -> %q", status, loc) |
| } |
| - if p := fx.channelVersion(t, "kass", "stable"); p.Version != "v2" || len(p.History) == 0 || p.History[0] != "v1" { |
| - t.Fatalf("stable pointer = %+v, want v2 with v1 in history", p) |
| + if p := fx.channelVersion(t, "kass", "edge"); p.Version != "v3" { |
| + t.Fatalf("edge pointer = %+v, want v3", p) |
| } |
| _, _, body = get(t, c, fx.u("/apps/kass")) |
| - if !strings.Contains(body, "Promoted v2 to stable.") || |
| + if !strings.Contains(body, "Promoted v3 to edge.") || |
| !strings.Contains(body, "bake window") || !strings.Contains(body, "rollback never bakes") { |
| t.Fatalf("promote flash missing bake note:\n%s", body) |
| } |
| - // A brand-new channel via the free-text field; beta also bakes. |
| + // One rung at a time, up to stable — which is the rung that asks for a |
| + // passkey, so postGated walks the confirmation the console demands. |
| postForm(t, c, fx.u("/apps/kass/promote"), |
| - url.Values{"csrf": {csrf}, "version": {"v1"}, "channel": {"stable"}, "new_channel": {"beta"}}) |
| - if p := fx.channelVersion(t, "kass", "beta"); p.Version != "v1" { |
| - t.Fatalf("beta pointer = %+v, want v1", p) |
| + url.Values{"csrf": {csrf}, "version": {"v3"}, "new_channel": {"beta"}}) |
| + if p := fx.channelVersion(t, "kass", "beta"); p.Version != "v3" { |
| + t.Fatalf("beta pointer = %+v, want v3", p) |
| } |
| - if _, _, body := get(t, c, fx.u("/apps/kass")); !strings.Contains(body, "Promoted v1 to beta.") { |
| + if _, _, body := get(t, c, fx.u("/apps/kass")); !strings.Contains(body, "Promoted v3 to beta.") { |
| t.Fatal("no beta promote flash") |
| } |
| + postGated(t, fx, c, fx.u("/apps/kass/promote"), |
| + url.Values{"csrf": {csrf}, "version": {"v3"}, "channel": {"stable"}}) |
| + if p := fx.channelVersion(t, "kass", "stable"); p.Version != "v3" || len(p.History) == 0 || p.History[0] != "v1" { |
| + t.Fatalf("stable pointer = %+v, want v3 with v1 in history", p) |
| + } |
| + |
| + // Running the same promotion again is a no-op success, not a failure. |
| + postGated(t, fx, c, fx.u("/apps/kass/promote"), |
| + url.Values{"csrf": {csrf}, "version": {"v3"}, "channel": {"stable"}}) |
| + if _, _, body := get(t, c, fx.u("/apps/kass")); !strings.Contains(body, "already serves v3") { |
| + t.Fatalf("a repeated promotion was not a no-op:\n%s", body) |
| + } |
| - // Unshipped version: refused, pointer unmoved. |
| + // Unshipped version: refused before any gate, pointer unmoved. |
| postForm(t, c, fx.u("/apps/kass/promote"), |
| url.Values{"csrf": {csrf}, "version": {"v9"}, "channel": {"stable"}}) |
| if _, _, body := get(t, c, fx.u("/apps/kass")); !strings.Contains(body, "not a shipped release") { |
| t.Fatal("no bad-version flash") |
| } |
| - if p := fx.channelVersion(t, "kass", "stable"); p.Version != "v2" { |
| + if p := fx.channelVersion(t, "kass", "stable"); p.Version != "v3" { |
| t.Fatalf("stable moved to %s on a refused promote", p.Version) |
| } |
| @@ -554,10 +571,11 @@ func TestRollbackFlow(t *testing.T) { |
| t.Fatal("no empty-history rollback flash") |
| } |
| - // Promote v2 over v1, then roll back: pointer returns to v1 and v2 |
| - // becomes the (re-)rollback target. |
| - postForm(t, c, fx.u("/apps/kass/promote"), |
| - url.Values{"csrf": {csrf}, "version": {"v2"}, "channel": {"stable"}}) |
| + // Put v2 over v1 on stable, then roll back: the pointer returns to v1 and |
| + // v2 becomes the (re-)rollback target. Seeded through the release |
| + // primitive rather than promoted through the console, because this test is |
| + // about rollback and the ladder has its own (TestLadderRefusesASkip). |
| + seedChannel(t, fx, "kass", "stable", "v2") |
| if _, _, body := get(t, c, fx.u("/apps/kass")); !strings.Contains(body, `name="channel" value="stable"`) { |
| t.Fatal("no rollback button after stable gained history") |
| } |
| @@ -811,8 +829,12 @@ func TestRecentActivity(t *testing.T) { |
| t.Fatal("activity section rendered before any action") |
| } |
| + // A promotion the ladder allows (a fresh build entering at edge), and a |
| + // rollback of a channel seeded with something to roll back to. |
| + shipVersion(t, fx, "kass", "v3") |
| + seedChannel(t, fx, "kass", "stable", "v2") |
| postForm(t, c, fx.u("/apps/kass/promote"), |
| - url.Values{"csrf": {csrf}, "version": {"v2"}, "channel": {"stable"}}) |
| + url.Values{"csrf": {csrf}, "version": {"v3"}, "new_channel": {"edge"}}) |
| postForm(t, c, fx.u("/apps/kass/rollback"), |
| url.Values{"csrf": {csrf}, "channel": {"stable"}}) |
| postForm(t, c, fx.u("/apps/kass/canary"), |
| @@ -823,7 +845,7 @@ func TestRecentActivity(t *testing.T) { |
| _, _, body := get(t, c, fx.u("/apps/kass")) |
| for _, want := range []string{ |
| "Recent activity", "paul@keymail.dev", |
| - "promoted v2 to stable", "rolled back stable to v1", |
| + "promoted v3 to edge", "rolled back stable to v1", |
| "created canary rc at v1", "retired canary rc", |
| "CLI promotions don", |
| } { |
| @@ -832,7 +854,7 @@ func TestRecentActivity(t *testing.T) { |
| } |
| } |
| // Newest first: the retire outranks the promote. |
| - if strings.Index(body, "retired canary rc") > strings.Index(body, "promoted v2 to stable") { |
| + if strings.Index(body, "retired canary rc") > strings.Index(body, "promoted v3 to edge") { |
| t.Fatal("activity is not newest-first") |
| } |
| } |
diff --git a/internal/console/templates/app.html b/internal/console/templates/app.html| index 68c50820a..fbb3fc1a1 100644 |
| --- a/internal/console/templates/app.html |
| +++ b/internal/console/templates/app.html |
| @@ -16,6 +16,8 @@ |
| </p> |
| {{end}} |
| +{{template "ladder" .}} |
| + |
| <h2>Domains</h2> |
| {{if .Detail.Domains}} |
| <div class="tablewrap"> |
diff --git a/internal/console/templates/fleet.html b/internal/console/templates/fleet.html| index 1348914ed..f82bcf944 100644 |
| --- a/internal/console/templates/fleet.html |
| +++ b/internal/console/templates/fleet.html |
| @@ -23,6 +23,10 @@ |
| <h2>Replication</h2> |
| <p class="tile-value">{{boolWord .Fleet.Replication.Healthy "healthy" "unhealthy"}}</p> |
| </div> |
| + <div class="tile"> |
| + <h2>Adoption</h2> |
| + <p class="tile-value">{{boolWord .Fleet.AdoptionRunning "running" "not running"}}</p> |
| + </div> |
| </section> |
| <h2>Boxes</h2> |
| @@ -32,6 +36,7 @@ |
| <tr> |
| <th>Label</th><th>Region</th><th>Load</th> |
| <th>Static</th><th>Resident</th><th>Hibernating</th><th>Awake</th><th>Asleep</th> |
| + <th>Adoption</th> |
| <th></th> |
| </tr> |
| {{range .Fleet.Boxes}} |
| @@ -44,6 +49,7 @@ |
| <td>{{.Routes.Hibernating}}</td> |
| <td>{{intOrDash .Routes.Awake}}</td> |
| <td>{{intOrDash .Routes.Asleep}}</td> |
| + <td>{{boolWord .AdoptionRunning "running" "not running"}}</td> |
| <td>{{if .Stale}}<span class="chip chip-stale">stale</span>{{end}}</td> |
| </tr> |
| {{end}} |
diff --git a/internal/console/templates/ladder.html b/internal/console/templates/ladder.html| new file mode 100644 |
| index 000000000..b7ade3e01 |
| --- /dev/null |
| +++ b/internal/console/templates/ladder.html |
| @@ -0,0 +1,64 @@ |
| +{{define "ladder"}} |
| +<h2>Ladder</h2> |
| +<p class="small">A channel is maturity, not a place: the same build climbs |
| +canary → edge → beta → stable, and a promotion moves a signed pointer without |
| +touching what any host is wired to.</p> |
| +<ol class="ladder"> |
| + {{range .Detail.Ladder}} |
| + <li class="rung" id="rung-{{.Rung}}"> |
| + <p class="rung-name">{{.Rung}}</p> |
| + {{if .Channels}} |
| + {{range .Channels}} |
| + <div class="rung-channel"> |
| + {{if .Err}} |
| + <p class="error"><code>{{.Channel}}</code> — {{.Err}}</p> |
| + {{else}} |
| + <p><code>{{.Channel}}</code> holds <code>{{.Version}}</code> |
| + <span class="small">promoted {{fmtTime .PromotedAt}} |
| + {{if .Actor}}by {{.Actor}}{{else}}— by whom is not recorded, on a pointer written before the trail existed{{end}} |
| + {{if .Hotfix}}· ladder bypassed (hotfix){{end}}</span></p> |
| + {{end}} |
| + {{if .Routes}} |
| + <ul class="routes"> |
| + {{range .Routes}} |
| + <li><a href="https://{{.Host}}">{{.Host}}</a> |
| + {{if .ProdWired}}<span class="chip chip-prod">{{icon "triangle-alert"}} canary in front of everybody</span> |
| + {{else if .Canary}}<span class="chip chip-canary">{{icon "flask-conical"}} canary</span>{{end}} |
| + </li> |
| + {{end}} |
| + </ul> |
| + {{else}} |
| + <p class="small">No host follows this channel.</p> |
| + {{end}} |
| + </div> |
| + {{end}} |
| + {{else}} |
| + <p class="small">Nothing on {{.Rung}}.</p> |
| + {{end}} |
| + <p class="small gate">{{.Gate}}</p> |
| + {{if .Up}} |
| + <div class="step"> |
| + {{icon "arrow-down"}} |
| + {{if and .Up.Version $.CanPromote $.IsOwner}} |
| + <form method="post" action="{{$.Base}}/apps/{{$.Detail.App}}/promote"> |
| + <input type="hidden" name="csrf" value="{{$.CSRF}}"> |
| + <input type="hidden" name="channel" value="{{.Up.To}}"> |
| + <input type="hidden" name="version" value="{{.Up.Version}}"> |
| + <button type="submit" class="linkish">Promote {{.Up.Version}} to {{.Up.To}}</button> |
| + </form> |
| + {{if .Up.StepUp}}<span class="small">— confirms with your passkey</span>{{end}} |
| + {{else if .Up.Note}} |
| + <span class="small">{{.Up.Note}}</span> |
| + {{else}} |
| + <span class="small">Promoting to {{.Up.To}} is the account owner's to do.</span> |
| + {{end}} |
| + </div> |
| + {{end}} |
| + </li> |
| + {{end}} |
| +</ol> |
| +<p class="small">A canary is never refused by the ladder and never gates |
| +anything: it is where a change is proven before it is a candidate. Everything |
| +above edge is entered one rung at a time — the only bypass is |
| +<code>carlos promote -hotfix</code>, which says so on the pointer it moves.</p> |
| +{{end}} |
diff --git a/internal/console/templates/layout.html b/internal/console/templates/layout.html| index eb87bd008..e79152697 100644 |
| --- a/internal/console/templates/layout.html |
| +++ b/internal/console/templates/layout.html |
| @@ -141,9 +141,42 @@ |
| padding: 0.15rem 0.55rem; border-radius: 999px; margin: 0.15rem 0.3rem 0.15rem 0; |
| } |
| .chip-stale { border: 1px solid var(--warn, #a6790f); color: var(--warn, #a6790f); } |
| + /* A canary is bordered — a fact about a route, not a warning. A canary on |
| + the app's primary host is FILLED, because an unreviewed build in front of |
| + every user is the one thing on this page a reader must not skim past. */ |
| + .chip-canary { border: 1px solid var(--accent); color: var(--accent); } |
| + .chip-prod { background: var(--bad); color: var(--bg); } |
| + .chip .icon { vertical-align: -0.1em; margin-right: 0.2em; } |
| .chip-up { background: var(--accent-soft); color: #2f7a4f; } |
| .chip-down { background: var(--accent-soft); color: var(--bad); } |
| .chips { display: flex; flex-wrap: wrap; margin: 0 0 1rem; } |
| + /* The release ladder: four rungs drawn top-down in the order the spec |
| + draws them (canary first, stable at the foot), with the promotion |
| + between two of them rendered as the step it is. list-style is off |
| + because the rung's own name is its marker. */ |
| + .ladder { list-style: none; margin: 0 0 1rem; padding: 0; max-width: 40rem; } |
| + .rung { |
| + border: 1px solid var(--rule); border-radius: 0.6rem; padding: 0.8rem 1rem; |
| + } |
| + .rung + .rung { margin-top: 0.4rem; } |
| + .rung-name { |
| + font-family: ui-sans-serif, system-ui, -apple-system, sans-serif; |
| + font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em; |
| + font-weight: 600; color: var(--muted); margin: 0 0 0.3rem; |
| + } |
| + .rung p { margin: 0 0 0.2rem; } |
| + .rung .gate { margin: 0.5rem 0 0; } |
| + .rung .routes { list-style: none; margin: 0.3rem 0 0.6rem; padding: 0; font-size: 0.95rem; } |
| + .rung .routes li { padding: 0.1rem 0; } |
| + /* The step to the next rung is the last band of the rung it leaves: ruled |
| + off, full width, so the arrow reads as "and then down to here". */ |
| + .step { |
| + display: flex; align-items: baseline; gap: 0.5rem; flex-wrap: wrap; |
| + margin: 0.7rem -1rem -0.8rem; padding: 0.5rem 1rem; |
| + border-top: 1px solid var(--rule); |
| + } |
| + .step form { display: flex; align-items: baseline; gap: 0.5rem; margin: 0; } |
| + .step > .icon { color: var(--muted); align-self: center; } |
| th { |
| font-family: ui-sans-serif, system-ui, -apple-system, sans-serif; |
| font-size: 0.78rem; text-transform: uppercase; letter-spacing: 0.05em; |
diff --git a/internal/console/view.go b/internal/console/view.go| index 25856c73e..76fb54068 100644 |
| --- a/internal/console/view.go |
| +++ b/internal/console/view.go |
| @@ -27,6 +27,10 @@ type AppView struct { |
| Domains []DomainView |
| Channels []ChannelView |
| Canaries []CanaryView |
| + // Ladder is the four rungs with what sits on each (ladder.go) — the same |
| + // channels and routes as the fields above, arranged as the pipeline they |
| + // actually are rather than as two alphabetical lists. |
| + Ladder []LadderRung |
| // RouteAccount is the account the app's ROUTES say it belongs to, when |
| // that disagrees with the account whose page this is; "" when they agree |
| // or there are no routes. It is the half-migrated state — see |
| @@ -118,6 +122,13 @@ type ChannelView struct { |
| // HasHistory is whether the pointer records previous versions — i.e. |
| // whether a rollback has somewhere to go. |
| HasHistory bool |
| + // Actor is who promoted the version this channel now holds, and Hotfix |
| + // whether they bypassed the ladder to do it — both read from the |
| + // pointer's own signed audit trail. Actor is "" for every pointer written |
| + // before the trail existed, which is most of the live fleet: that is |
| + // missing information, not an error, and the page says so. |
| + Actor string |
| + Hotfix bool |
| // Err degrades a missing or invalid channel pointer to one row's error |
| // text instead of a failed page. |
| Err string |
| @@ -212,6 +223,7 @@ func Assemble(regPath string, s release.Store, pub ed25519.PublicKey, acct strin |
| Domains: domainsByApp[app], |
| Channels: channels, |
| Canaries: assembleCanaries(channels, byApp[app]), |
| + Ladder: assembleLadder(channels, byApp[app]), |
| RouteAccount: mismatch, |
| }) |
| } |
| @@ -272,6 +284,17 @@ func assembleChannels(ctx context.Context, s release.Store, pub ed25519.PublicKe |
| Name: name, Version: p.Version, PromotedAt: p.PromotedAt, |
| HasHistory: len(p.History) > 0, |
| } |
| + // The trail is newest first, but a rollback carries it forward |
| + // without appending: the entry that explains the CURRENT version is |
| + // the newest one naming it, not simply the newest one. No match — an |
| + // old pointer, or a version that arrived by rollback — leaves Actor |
| + // empty, which the page renders as "not recorded". |
| + for _, a := range p.Audit { |
| + if a.Version == p.Version { |
| + cv.Actor, cv.Hotfix = a.Actor, a.Hotfix |
| + break |
| + } |
| + } |
| if hold := release.HoldFor(release.DefaultHolds, name); hold > 0 { |
| if until := p.PromotedAt.Add(hold); now.Before(until) { |
| cv.BakedUntil = until |
diff --git a/internal/host/agent.go b/internal/host/agent.go| index abdc6a214..07780366e 100644 |
| --- a/internal/host/agent.go |
| +++ b/internal/host/agent.go |
| @@ -489,6 +489,11 @@ func statusLoop(reg *registry.Registry) { |
| MemFrac: readMemFrac(), |
| DiskFrac: readDiskFrac(), |
| Listening: wrapListeningProbe(unitBackedSockets(routes), listeningProbe), |
| + // One systemctl call per 5-minute tick, asking the only question |
| + // that catches an enabled-but-never-armed adoption timer — the |
| + // flagship's silent no-adopt state, which nothing on any page |
| + // could see before this. |
| + AdoptRunning: adoptTimerActive(), |
| // LagSecs stays nil: litestream's per-DB replication lag isn't |
| // surfaced up to the agent process anywhere yet — the |
| // replication machinery (internal/lease's Replicator) lives |
diff --git a/internal/host/bootstrap.go b/internal/host/bootstrap.go| index 060dd29f0..024587fd8 100644 |
| --- a/internal/host/bootstrap.go |
| +++ b/internal/host/bootstrap.go |
| @@ -49,6 +49,42 @@ const tmpfilesConf = "d /run/carlos 0750 " + serviceUser + " " + serviceUser + " |
| // replication bucket is unaffected. |
| var enabledUnits = []string{"carlos-edge.service", "carlos-adopt.timer"} |
| +// systemctlPlan is the exact systemctl sequence a live bootstrap runs — and |
| +// the same sequence a staging run (-root <dir>) PRINTS instead of running, so |
| +// the two can never drift into describing different behaviour. |
| +// |
| +// Timers get an explicit `start` after the `enable`, and that line is the |
| +// point of this function: `systemctl enable` writes a .wants symlink and |
| +// stops there, which arms the timer at the NEXT boot and not a moment sooner. |
| +// carlos-adopt.timer sat enabled-but-inactive on the flagship for weeks for |
| +// exactly this reason — symlink present, nothing rebooted, no release ever |
| +// adopted itself, every deploy silently needing a hand-run adoption pass. A |
| +// re-run is harmless: starting an already-running timer is a no-op, which |
| +// keeps bootstrap idempotent. |
| +// |
| +// Services are deliberately NOT started. carlos-edge needs its binary at |
| +// /opt/carlos/carlos and /etc/carlos/host.env staged first — neither of which |
| +// bootstrap can do for the operator, which is why it ends by telling them to. |
| +// Starting timers is safe in a way starting services is not: a timer that |
| +// fires before the binary lands just fails one adoption pass and tries again |
| +// on the next tick. |
| +func systemctlPlan(units []string) [][]string { |
| + plan := [][]string{ |
| + {"daemon-reload"}, |
| + append([]string{"enable"}, units...), |
| + } |
| + var timers []string |
| + for _, u := range units { |
| + if strings.HasSuffix(u, ".timer") { |
| + timers = append(timers, u) |
| + } |
| + } |
| + if len(timers) > 0 { |
| + plan = append(plan, append([]string{"start"}, timers...)) |
| + } |
| + return plan |
| +} |
| + |
| // RunBootstrap is the `carlos bootstrap` verb. Idempotent — safe to re-run. |
| func RunBootstrap(args []string) { |
| fs := flag.NewFlagSet("bootstrap", flag.ExitOnError) |
| @@ -61,9 +97,10 @@ func RunBootstrap(args []string) { |
| // bootstrap provisions (or re-converges) a box under root: units into |
| // <root>/etc/systemd/system, the standard dirs, the tmpfiles.d entry, and — |
| -// only when root is "/" — the service user, ownership, and systemctl |
| -// daemon-reload + enable. A non-"/" root is the staging/test path: files |
| -// only, with the privileged steps printed instead of run. |
| +// only when root is "/" — the service user, ownership, and the systemctl |
| +// sequence (daemon-reload, enable, and arming the timers; see systemctlPlan). |
| +// A non-"/" root is the staging/test path: files only, with the privileged |
| +// steps printed instead of run. |
| func bootstrap(root string) error { |
| live := filepath.Clean(root) == "/" |
| unitDir := filepath.Join(root, "etc", "systemd", "system") |
| @@ -131,7 +168,14 @@ func bootstrap(root string) error { |
| fmt.Printf("bootstrap: staged under %s — on a real box (-root /) it would also:\n", root) |
| fmt.Printf(" useradd --system --user-group --no-create-home --shell /usr/sbin/nologin %s\n", serviceUser) |
| fmt.Printf(" chown -R %s:%s /data /run/carlos\n", serviceUser, serviceUser) |
| - fmt.Printf(" systemctl daemon-reload && systemctl enable %s\n", strings.Join(enabledUnits, " ")) |
| + // Printed, never run: a staging root is how a LIVE box refreshes its |
| + // unit files (pinfra's update-platform-binary.sh stages into |
| + // /tmp/carlos-unitstage), so arming anything from here would reach out |
| + // of the temp root and touch the running system from a command whose |
| + // entire contract is "write files, change nothing". |
| + for _, args := range systemctlPlan(enabledUnits) { |
| + fmt.Printf(" systemctl %s\n", strings.Join(args, " ")) |
| + } |
| return nil |
| } |
| @@ -145,13 +189,12 @@ func bootstrap(root string) error { |
| return err |
| } |
| } |
| - if err := systemctl("daemon-reload"); err != nil { |
| - return err |
| - } |
| - if err := systemctl(append([]string{"enable"}, enabledUnits...)...); err != nil { |
| - return err |
| + for _, args := range systemctlPlan(enabledUnits) { |
| + if err := systemctl(args...); err != nil { |
| + return err |
| + } |
| } |
| - fmt.Printf("bootstrap: units installed, %s enabled\n", strings.Join(enabledUnits, " + ")) |
| + fmt.Printf("bootstrap: units installed, %s enabled (timers armed, not just symlinked)\n", strings.Join(enabledUnits, " + ")) |
| fmt.Println(" next: copy the carlos binary to /opt/carlos/carlos, stage /etc/carlos/host.env, then: systemctl start carlos-edge") |
| return nil |
| } |
diff --git a/internal/host/host_test.go b/internal/host/host_test.go| index ea0a3b211..4312df011 100644 |
| --- a/internal/host/host_test.go |
| +++ b/internal/host/host_test.go |
| @@ -385,3 +385,103 @@ func TestIsCanaryRoute(t *testing.T) { |
| } |
| } |
| } |
| + |
| +// ---- Pipelines Task 5: the install path arms timers ------------------------- |
| + |
| +// TestSystemctlPlanArmsTimers is the regression test for the flagship's |
| +// silent-adoption bug: carlos-adopt.timer was enabled — the .wants symlink was |
| +// right there — but never active, because `systemctl enable` only writes the |
| +// symlink and the timer does not arm until something starts it or the box |
| +// reboots. Nothing had rebooted, so no release ever adopted itself and every |
| +// deploy quietly needed a hand-run `systemctl start carlos-adopt.service`. |
| +// Every timer bootstrap enables must therefore also be started, and no |
| +// SERVICE may be (carlos-edge needs its binary and host.env staged first — |
| +// bootstrap's own closing line tells the operator to start it). |
| +func TestSystemctlPlanArmsTimers(t *testing.T) { |
| + plan := systemctlPlan(enabledUnits) |
| + if len(plan) == 0 || plan[0][0] != "daemon-reload" { |
| + t.Fatalf("plan must reload units first, got %v", plan) |
| + } |
| + |
| + started := map[string]bool{} |
| + enabled := map[string]bool{} |
| + for _, args := range plan { |
| + for _, unit := range args[1:] { |
| + switch args[0] { |
| + case "start": |
| + started[unit] = true |
| + case "enable": |
| + enabled[unit] = true |
| + } |
| + } |
| + } |
| + for _, u := range enabledUnits { |
| + if !enabled[u] { |
| + t.Errorf("%s: enabled by the plan? no — bootstrap must still enable everything it did before", u) |
| + } |
| + if strings.HasSuffix(u, ".timer") && !started[u] { |
| + t.Errorf("%s: enabled but never started — that is exactly the bug (enable only symlinks; the timer arms at the next boot)", u) |
| + } |
| + if strings.HasSuffix(u, ".service") && started[u] { |
| + t.Errorf("%s: bootstrap must not start a service — the binary and /etc/carlos/host.env may not be staged yet", u) |
| + } |
| + } |
| +} |
| + |
| +// TestBootstrapStagingRunsNoSystemctl guards the other half of the same |
| +// change: `carlos bootstrap -root <dir>` is how the unit-refresh path stages |
| +// units into a temp root on a LIVE box (see pinfra's |
| +// update-platform-binary.sh, which stages into /tmp/carlos-unitstage). Arming |
| +// timers from a staging run would touch the live system from a command whose |
| +// entire contract is "write files, change nothing" — so a staging run must |
| +// print what a live one would do and execute none of it. The fake systemctl on |
| +// PATH records any invocation; the file must never appear. |
| +func TestBootstrapStagingRunsNoSystemctl(t *testing.T) { |
| + bin := t.TempDir() |
| + log := filepath.Join(t.TempDir(), "systemctl-calls") |
| + script := "#!/bin/sh\necho \"$@\" >> " + log + "\n" |
| + if err := os.WriteFile(filepath.Join(bin, "systemctl"), []byte(script), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) |
| + |
| + if err := bootstrap(t.TempDir()); err != nil { |
| + t.Fatalf("bootstrap: %v", err) |
| + } |
| + if b, err := os.ReadFile(log); err == nil { |
| + t.Errorf("staging bootstrap ran systemctl: %s", b) |
| + } |
| +} |
| + |
| +// TestAdoptTimerStateReading pins the probe's honesty rules at the seam that |
| +// needs no systemd to test: systemd's own word for the timer's state maps to |
| +// running/not-running, and anything this probe cannot interpret — no systemd |
| +// at all, a systemctl that failed to run, a word from a future version — |
| +// reports nil (unknown), never a reassuring true. |
| +func TestAdoptTimerStateReading(t *testing.T) { |
| + cases := []struct { |
| + out string |
| + want *bool |
| + }{ |
| + {"active\n", boolPtr(true)}, |
| + {"inactive\n", boolPtr(false)}, |
| + {"failed\n", boolPtr(false)}, |
| + {"deactivating\n", boolPtr(false)}, |
| + {"unknown\n", boolPtr(false)}, // systemd answered: it has no such unit loaded |
| + {"", nil}, // systemctl never ran (not a systemd box) |
| + {"something-new\n", nil}, // a word this probe does not understand |
| + } |
| + for _, c := range cases { |
| + got := parseIsActive(c.out) |
| + switch { |
| + case c.want == nil && got != nil: |
| + t.Errorf("parseIsActive(%q) = %v, want nil", c.out, *got) |
| + case c.want != nil && got == nil: |
| + t.Errorf("parseIsActive(%q) = nil, want %v", c.out, *c.want) |
| + case c.want != nil && *got != *c.want: |
| + t.Errorf("parseIsActive(%q) = %v, want %v", c.out, *got, *c.want) |
| + } |
| + } |
| +} |
| + |
| +func boolPtr(v bool) *bool { return &v } |
diff --git a/internal/host/probes.go b/internal/host/probes.go| index ea5d93ff2..4fe69a30c 100644 |
| --- a/internal/host/probes.go |
| +++ b/internal/host/probes.go |
| @@ -13,6 +13,7 @@ package host |
| import ( |
| "bufio" |
| "os" |
| + "os/exec" |
| "strconv" |
| "strings" |
| "syscall" |
| @@ -20,6 +21,45 @@ import ( |
| "github.com/carlosframework/platform/internal/registry" |
| ) |
| +// adoptTimerUnit is the unit whose liveness IS "this box adopts releases on |
| +// its own" — the same name bootstrap installs and arms (see bootstrap.go's |
| +// enabledUnits). |
| +const adoptTimerUnit = "carlos-adopt.timer" |
| + |
| +// adoptTimerActive asks systemd whether the adoption timer is ACTIVE, not |
| +// merely enabled. The distinction is the whole point: `systemctl enable` |
| +// writes a .wants symlink and nothing more, so a timer can sit enabled and |
| +// never armed until the box next reboots — which is exactly how the flagship |
| +// ran for weeks without a single automatic adoption. `is-active` is the only |
| +// question that distinguishes the two. |
| +// |
| +// The exit status is deliberately ignored in favour of the printed word: |
| +// is-active exits non-zero for every not-active state while still printing |
| +// which state it is, so the error would tell us less than the output already |
| +// does. A systemctl that couldn't run at all prints nothing, and nothing maps |
| +// to nil. |
| +func adoptTimerActive() *bool { |
| + out, _ := exec.Command("systemctl", "is-active", adoptTimerUnit).Output() |
| + return parseIsActive(string(out)) |
| +} |
| + |
| +// parseIsActive maps systemd's is-active vocabulary onto the tri-state the |
| +// status page publishes. "unknown" — systemd's answer when it has no such unit |
| +// loaded — is a genuine false: a box with no adoption timer adopts nothing. |
| +// Anything this function does not recognise (empty output because systemctl |
| +// isn't there, a word a future systemd invents) is nil, because the one answer |
| +// this probe must never invent is a reassuring one. |
| +func parseIsActive(out string) *bool { |
| + yes, no := true, false |
| + switch strings.TrimSpace(out) { |
| + case "active": |
| + return &yes |
| + case "inactive", "failed", "deactivating", "unknown": |
| + return &no |
| + } |
| + return nil |
| +} |
| + |
| // readMachineID reads /etc/machine-id, trimmed. Unreadable (missing file, |
| // permission denied — plausible inside some containers) -> "", NOT a |
| // hash-of-hostname fallback. A hostname-derived id would look exactly like a |
diff --git a/internal/release/ladder.go b/internal/release/ladder.go| new file mode 100644 |
| index 000000000..c7c2d089f |
| --- /dev/null |
| +++ b/internal/release/ladder.go |
| @@ -0,0 +1,110 @@ |
| +// The channel ladder — the platform's frozen vocabulary for build maturity |
| +// (pipelines spec §3). |
| +// |
| +// canary/<name> a specific change, shipped from a worktree or a branch. |
| +// ↓ No gate in and no gate out: canary is where work happens. |
| +// edge merged to main. Nothing is promoted ONTO edge — merging is |
| +// ↓ what puts a build there. |
| +// beta promoted from edge on sign-off. |
| +// ↓ |
| +// stable promoted from beta on sign-off, after the bake window. |
| +// |
| +// Ordering is enforced from edge upward only, and `carlos promote -hotfix` is |
| +// the one bypass — an explicit, recorded one, because a bypass that is |
| +// invisible is indistinguishable from a broken gate. |
| +// |
| +// This file is vocabulary and rules, nothing else: it touches no store, no |
| +// pointer and no route. In particular, promotion moves a signed channel |
| +// pointer and NEVER changes a route's environment — the ladder proves nothing |
| +// if beta and stable can be wired to different glue. |
| +package release |
| + |
| +import ( |
| + "fmt" |
| + "strings" |
| +) |
| + |
| +// The rungs. Frozen: a channel name outside this vocabulary (plus canary's |
| +// nested names) is not a rung, and promotion onto it is not a promotion. |
| +const ( |
| + RungCanary = "canary" |
| + RungEdge = "edge" |
| + RungBeta = "beta" |
| + RungStable = "stable" |
| +) |
| + |
| +// rungs is the ladder, bottom first — the order RungIndex reports. |
| +var rungs = []string{RungCanary, RungEdge, RungBeta, RungStable} |
| + |
| +// RungIndex is a rung's height: 0 canary, 1 edge, 2 beta, 3 stable, -1 for |
| +// anything that is not a rung name (including a channel name like |
| +// "canary/x" — that is Rung's job). |
| +func RungIndex(rung string) int { |
| + for i, r := range rungs { |
| + if r == rung { |
| + return i |
| + } |
| + } |
| + return -1 |
| +} |
| + |
| +// Rung reports which rung a CHANNEL name belongs to: "stable" -> "stable", |
| +// "canary/keymail" -> "canary". Only canary nests (a canary is a named |
| +// change, so it needs a name); anything else is a single segment, and a name |
| +// outside the vocabulary belongs to no rung at all. |
| +func Rung(channel string) (string, bool) { |
| + if !ValidChannel(channel) { |
| + return "", false |
| + } |
| + name, _, nested := strings.Cut(channel, "/") |
| + if nested && name != RungCanary { |
| + return "", false |
| + } |
| + if RungIndex(name) < 0 { |
| + return "", false |
| + } |
| + return name, true |
| +} |
| + |
| +// PromotionAllowed reports whether a build may move onto rung `to`, given the |
| +// rung it currently occupies (`from` — the highest, if several; "" when it is |
| +// on none). nil means go ahead; every refusal names -hotfix, because the |
| +// bypass has to be as discoverable as the gate. |
| +// |
| +// Legal: edge -> beta, beta -> stable, and a build that is on no rung yet |
| +// entering at edge. Canary is unconditional in both directions of the |
| +// argument list — production runs on canary/* channels, and nothing about |
| +// maturity may stop a canary shipping. |
| +func PromotionAllowed(from, to string) error { |
| + if RungIndex(to) < 0 { |
| + return fmt.Errorf("%q is not a rung (%s) — use -hotfix to point a channel outside the ladder", |
| + to, strings.Join(rungs, ", ")) |
| + } |
| + switch { |
| + case to == RungCanary: |
| + // No gate in: a canary is proven BEFORE it is a candidate. |
| + return nil |
| + case to == RungEdge && from == "": |
| + // A fresh build entering the ladder — the merge that produced it is |
| + // the gate, and this is what an automatic edge promotion does. |
| + return nil |
| + case to == RungBeta && from == RungEdge: |
| + return nil |
| + case to == RungStable && from == RungBeta: |
| + return nil |
| + } |
| + if to == RungEdge { |
| + return fmt.Errorf("nothing is promoted onto edge — merging to main is what puts a build there, and this one is on %s; use -hotfix to override", |
| + describeRung(from)) |
| + } |
| + return fmt.Errorf("%s is entered from %s only, and this build is on %s — use -hotfix to bypass the ladder", |
| + to, rungs[RungIndex(to)-1], describeRung(from)) |
| +} |
| + |
| +// describeRung names a rung for a human, including the "on nothing yet" case. |
| +func describeRung(rung string) string { |
| + if rung == "" { |
| + return "no rung yet" |
| + } |
| + return rung |
| +} |
diff --git a/internal/release/ladder_test.go b/internal/release/ladder_test.go| new file mode 100644 |
| index 000000000..496b38dfa |
| --- /dev/null |
| +++ b/internal/release/ladder_test.go |
| @@ -0,0 +1,100 @@ |
| +package release |
| + |
| +import ( |
| + "strings" |
| + "testing" |
| +) |
| + |
| +func TestRungOfChannel(t *testing.T) { |
| + cases := []struct { |
| + channel string |
| + rung string |
| + ok bool |
| + }{ |
| + {"stable", RungStable, true}, |
| + {"beta", RungBeta, true}, |
| + {"edge", RungEdge, true}, |
| + {"canary", RungCanary, true}, |
| + {"canary/keymail", RungCanary, true}, |
| + {"canary/platform-adoption", RungCanary, true}, |
| + {"canary/rehearsal", RungCanary, true}, |
| + {"canary/x", RungCanary, true}, |
| + // Only canary nests, and only names in the frozen vocabulary are rungs. |
| + {"beta/x", "", false}, |
| + {"stable/x", "", false}, |
| + {"dev", "", false}, |
| + {"wip", "", false}, |
| + {"", "", false}, |
| + {"canary/", "", false}, |
| + {"canary/a/b", "", false}, |
| + } |
| + for _, c := range cases { |
| + got, ok := Rung(c.channel) |
| + if got != c.rung || ok != c.ok { |
| + t.Errorf("Rung(%q) = (%q, %v), want (%q, %v)", c.channel, got, ok, c.rung, c.ok) |
| + } |
| + } |
| +} |
| + |
| +func TestRungIndexOrdersTheLadder(t *testing.T) { |
| + if RungIndex(RungCanary) != 0 || RungIndex(RungEdge) != 1 || |
| + RungIndex(RungBeta) != 2 || RungIndex(RungStable) != 3 { |
| + t.Fatalf("ladder order wrong: canary=%d edge=%d beta=%d stable=%d", |
| + RungIndex(RungCanary), RungIndex(RungEdge), RungIndex(RungBeta), RungIndex(RungStable)) |
| + } |
| + for _, unknown := range []string{"", "dev", "prod", "canary/x"} { |
| + if got := RungIndex(unknown); got != -1 { |
| + t.Errorf("RungIndex(%q) = %d, want -1", unknown, got) |
| + } |
| + } |
| +} |
| + |
| +func TestPromotionAllowedLegalTransitions(t *testing.T) { |
| + legal := []struct{ from, to string }{ |
| + // The two ordered steps, which is the whole ladder above edge. |
| + {RungEdge, RungBeta}, |
| + {RungBeta, RungStable}, |
| + // A fresh build enters the ladder at edge (merging put it there). |
| + {"", RungEdge}, |
| + // Canary has no gate in: production runs on canary/* channels today |
| + // and shipping to one is never refused, whatever the build is on. |
| + {"", RungCanary}, |
| + {RungCanary, RungCanary}, |
| + {RungEdge, RungCanary}, |
| + {RungBeta, RungCanary}, |
| + {RungStable, RungCanary}, |
| + } |
| + for _, c := range legal { |
| + if err := PromotionAllowed(c.from, c.to); err != nil { |
| + t.Errorf("PromotionAllowed(%q, %q) = %v, want nil", c.from, c.to, err) |
| + } |
| + } |
| +} |
| + |
| +func TestPromotionAllowedRefusesSkipsAndNamesHotfix(t *testing.T) { |
| + refused := []struct{ from, to string }{ |
| + {RungEdge, RungStable}, // the skip the ladder exists to stop |
| + {"", RungBeta}, // straight onto beta, having been nowhere |
| + {"", RungStable}, // straight onto stable, having been nowhere |
| + {RungCanary, RungEdge}, // nothing is promoted from canary to edge |
| + {RungCanary, RungBeta}, // ... nor past it |
| + {RungCanary, RungStable}, // |
| + {RungStable, RungBeta}, // no going back down the ladder |
| + {RungBeta, RungEdge}, // |
| + {RungEdge, RungEdge}, // same-rung re-promotion of another build |
| + {RungBeta, RungBeta}, // |
| + {RungStable, RungStable}, // |
| + {RungEdge, "dev"}, // not a rung at all |
| + {RungEdge, ""}, // |
| + } |
| + for _, c := range refused { |
| + err := PromotionAllowed(c.from, c.to) |
| + if err == nil { |
| + t.Errorf("PromotionAllowed(%q, %q) = nil, want a refusal", c.from, c.to) |
| + continue |
| + } |
| + if !strings.Contains(err.Error(), "-hotfix") { |
| + t.Errorf("PromotionAllowed(%q, %q) = %q, want the refusal to name -hotfix", c.from, c.to, err) |
| + } |
| + } |
| +} |
diff --git a/internal/release/laddercheck_test.go b/internal/release/laddercheck_test.go| new file mode 100644 |
| index 000000000..6b778e15c |
| --- /dev/null |
| +++ b/internal/release/laddercheck_test.go |
| @@ -0,0 +1,46 @@ |
| +package release_test |
| + |
| +import ( |
| + "testing" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// The two channel names production is running on RIGHT NOW must never be |
| +// refused by the ladder — go.tito.io follows canary/platform-adoption and |
| +// keymail.dev follows canary/keymail. If these fail, production cannot deploy. |
| +func TestLiveProductionChannelsAreNeverRefused(t *testing.T) { |
| + for _, ch := range []string{"canary/platform-adoption", "canary/keymail", "canary/console", "canary/rehearsal"} { |
| + rung, ok := release.Rung(ch) |
| + if !ok || rung != release.RungCanary { |
| + t.Fatalf("Rung(%q) = %q,%v — want canary,true", ch, rung, ok) |
| + } |
| + // Promotion INTO a canary channel from anywhere must be allowed. |
| + for _, from := range []string{"", "canary", "edge", "beta", "stable"} { |
| + if err := release.PromotionAllowed(from, release.RungCanary); err != nil { |
| + t.Errorf("PromotionAllowed(%q -> canary) refused: %v", from, err) |
| + } |
| + } |
| + } |
| +} |
| + |
| +// The skip that matters: nothing reaches stable without beta, and the refusal |
| +// tells the operator the way out. |
| +func TestEdgeToStableIsRefusedAndNamesTheBypass(t *testing.T) { |
| + err := release.PromotionAllowed(release.RungEdge, release.RungStable) |
| + if err == nil { |
| + t.Fatal("edge -> stable was allowed; the ladder means nothing") |
| + } |
| + if got := err.Error(); !contains(got, "-hotfix") { |
| + t.Errorf("refusal %q does not name -hotfix", got) |
| + } |
| +} |
| + |
| +func contains(s, sub string) bool { |
| + for i := 0; i+len(sub) <= len(s); i++ { |
| + if s[i:i+len(sub)] == sub { |
| + return true |
| + } |
| + } |
| + return false |
| +} |
diff --git a/internal/release/release.go b/internal/release/release.go| index 1ffada312..a621bdb8b 100644 |
| --- a/internal/release/release.go |
| +++ b/internal/release/release.go |
| @@ -76,10 +76,42 @@ type ChannelPointer struct { |
| // rollback returns to where you were (Eleven versions.go's |
| // current<->previous swap, generalised). |
| History []string `json:"history,omitempty"` |
| + // Audit is the promotion trail, newest first, max auditMax: how this |
| + // channel came to point where it does, and who said so. ADDITIVE — a |
| + // pointer written before the trail existed has no "audit" key, parses |
| + // with none, and is not an error; that is most of the live fleet. |
| + Audit []Promotion `json:"audit,omitempty"` |
| } |
| -// historyMax bounds a pointer's rollback history. |
| -const historyMax = 10 |
| +// Promotion is one entry in a channel pointer's audit trail. It rides inside |
| +// the SIGNED payload, so the trail is as trustworthy as the pointer itself — |
| +// an audit log that can be edited without breaking the signature is a log |
| +// that proves nothing. |
| +type Promotion struct { |
| + Version string `json:"version"` |
| + // From is the rung the version occupied before this promotion (highest, |
| + // if several); empty when it was on none. To is the rung it moved onto, |
| + // empty for a channel outside the ladder's vocabulary. |
| + From string `json:"from,omitempty"` |
| + To string `json:"to,omitempty"` |
| + // Actor is who promoted — a console member's address, or the operator's |
| + // user@host at the CLI. Empty when the caller did not say. |
| + Actor string `json:"actor,omitempty"` |
| + // Hotfix records that the ladder was bypassed. This is the whole reason |
| + // the field exists: a bypass nobody can see is a broken gate. |
| + Hotfix bool `json:"hotfix,omitempty"` |
| + At time.Time `json:"at"` |
| +} |
| + |
| +// historyMax bounds a pointer's rollback history; auditMax bounds its |
| +// promotion trail. The trail is longer because it answers questions about the |
| +// past ("when did this reach stable, and who signed it off") that rollback |
| +// targets do not — but it is still bounded: a signed pointer that grows |
| +// without limit eventually stops being fetchable. |
| +const ( |
| + historyMax = 10 |
| + auditMax = 20 |
| +) |
| // Bucket keys, account-scoped: app names are unique WITHIN an account, so the |
| // account sqid is part of every key and the key path itself carries the |
| @@ -282,6 +314,18 @@ func validatePointer(p *ChannelPointer, wantApp, wantChannel string) error { |
| return fmt.Errorf("pointer history has invalid version %q", v) |
| } |
| } |
| + // The audit trail is checked loosely and only for shape: an old pointer |
| + // carries none, and a trail is a record of what happened, not an |
| + // instruction — refusing a pointer the fleet needs over a malformed |
| + // history entry would turn a bookkeeping bug into an outage. |
| + if len(p.Audit) > auditMax { |
| + return fmt.Errorf("pointer audit trail too long (%d > %d)", len(p.Audit), auditMax) |
| + } |
| + for _, a := range p.Audit { |
| + if a.Version != "" && !ValidVersion(a.Version) { |
| + return fmt.Errorf("pointer audit has invalid version %q", a.Version) |
| + } |
| + } |
| return nil |
| } |
| @@ -423,7 +467,12 @@ func GetManifest(ctx context.Context, s Store, acct, app, version string) (*Mani |
| // consumes. An existing pointer must verify against this key's public half |
| // before it is replaced: history from a document we can't trust doesn't get |
| // carried forward. |
| -func Promote(ctx context.Context, s Store, priv ed25519.PrivateKey, acct, app, channel, version string) error { |
| +// |
| +// An optional rec appends an entry to the pointer's audit trail (who, from |
| +// which rung, hotfix or not). It is variadic because most callers — the |
| +// migration verb, the tests, anything moving a pointer mechanically — have no |
| +// promotion to record; a promotion a person made does. |
| +func Promote(ctx context.Context, s Store, priv ed25519.PrivateKey, acct, app, channel, version string, rec ...Promotion) error { |
| if !ValidAccount(acct) { |
| return fmt.Errorf("promote: invalid account %q", acct) |
| } |
| @@ -463,10 +512,19 @@ func Promote(ctx context.Context, s Store, priv ed25519.PrivateKey, acct, app, c |
| default: |
| return err |
| } |
| - next := &ChannelPointer{App: app, Channel: channel, Version: version, History: old.History} |
| + next := &ChannelPointer{App: app, Channel: channel, Version: version, |
| + History: old.History, Audit: old.Audit} |
| if old.Version != "" && old.Version != version { |
| next.History = prependCapped(old.Version, old.History, historyMax) |
| } |
| + if len(rec) > 0 { |
| + e := rec[0] |
| + e.Version = version // the entry describes THIS promotion, whatever it said |
| + if e.At.IsZero() { |
| + e.At = time.Now().UTC().Truncate(time.Second) |
| + } |
| + next.Audit = prependCapped(e, old.Audit, auditMax) |
| + } |
| wrapper, err := signPointer(priv, next) |
| if err != nil { |
| return err |
| @@ -475,6 +533,138 @@ func Promote(ctx context.Context, s Store, priv ed25519.PrivateKey, acct, app, c |
| return err |
| } |
| +// PromoteOptions carries what the ladder needs to know about a promotion |
| +// beyond "which version, which channel". |
| +type PromoteOptions struct { |
| + // Hotfix bypasses the ordering check entirely — the one sanctioned way |
| + // past a gate, and recorded as such (spec §3: a bypass that is invisible |
| + // is indistinguishable from a broken gate). |
| + Hotfix bool |
| + // Actor is who is promoting, recorded in the pointer's audit trail: a |
| + // console member's address, or the operator's user@host at the CLI. |
| + Actor string |
| +} |
| + |
| +// PromoteResult says what a ladder promotion actually did, so the caller can |
| +// tell the operator. |
| +type PromoteResult struct { |
| + // From is the rung the version occupied before this promotion (the |
| + // highest, if several); "" when it was on no rung. |
| + From string |
| + // To is the rung of the target channel; "" when the channel is not a rung |
| + // at all, which only a hotfix can reach. |
| + To string |
| + // NoOp is a repeat: the channel already pointed at this version, so |
| + // nothing was written and nothing failed. |
| + NoOp bool |
| + // Hotfix is whether the ordering check was bypassed. |
| + Hotfix bool |
| +} |
| + |
| +// PromoteOnLadder is Promote with the channel ladder enforced: it works out |
| +// which rung this version already occupies (from the app's existing channel |
| +// pointers), checks the transition, and only then moves the pointer. |
| +// |
| +// Three things it deliberately does NOT do. It never refuses a canary — the |
| +// fleet runs on canary/* channels and maturity has no say over them. It never |
| +// fails a repeat: a channel that already points at this version is a no-op |
| +// success, so running the same command twice is not an error. And it never |
| +// touches a route — promotion moves a signed pointer and nothing else, so |
| +// beta and stable are always wired to whatever their routes say they are. |
| +// |
| +// Plain Promote stays as it was: this is the verb's entry point, not a |
| +// replacement, so nothing that legitimately points a channel outside the |
| +// ladder (a migration, the console's canary flow) changes behaviour. |
| +func PromoteOnLadder(ctx context.Context, s Store, priv ed25519.PrivateKey, acct, app, channel, version string, opt PromoteOptions) (PromoteResult, error) { |
| + res := PromoteResult{Hotfix: opt.Hotfix} |
| + if !ValidAccount(acct) { |
| + return res, fmt.Errorf("promote: invalid account %q", acct) |
| + } |
| + if !ValidApp(app) { |
| + return res, fmt.Errorf("promote: invalid app %q", app) |
| + } |
| + if !ValidChannel(channel) { |
| + return res, fmt.Errorf("promote: invalid channel %q", channel) |
| + } |
| + if !ValidVersion(version) || version == "dev" { |
| + return res, fmt.Errorf("promote: invalid version %q", version) |
| + } |
| + res.To, _ = Rung(channel) |
| + pub, _ := priv.Public().(ed25519.PublicKey) |
| + |
| + // A repeat is a no-op, checked before anything else: re-running a command |
| + // must not fail, whichever rung the build has since climbed to. Only the |
| + // ACCOUNT-QUALIFIED pointer counts — an app still on a legacy key has not |
| + // been written to its qualified key yet, and short-circuiting here would |
| + // leave it there. |
| + if cur, _, err := s.Get(ctx, ChannelKey(acct, app, channel)); err == nil { |
| + if p, verr := verifyPointer(pub, cur, app, channel); verr == nil && p.Version == version { |
| + res.NoOp = true |
| + res.From, _ = Rung(channel) |
| + return res, nil |
| + } |
| + } else if !errors.Is(err, os.ErrNotExist) { |
| + return res, fmt.Errorf("promote: read %s: %w", ChannelKey(acct, app, channel), err) |
| + } |
| + |
| + from, err := VersionRung(ctx, s, pub, acct, app, version) |
| + if err != nil { |
| + return res, fmt.Errorf("promote: %w", err) |
| + } |
| + res.From = from |
| + if !opt.Hotfix { |
| + // A channel that is not a rung is named as itself in the refusal — |
| + // "wip is not a rung" reads better than an empty target. |
| + to := res.To |
| + if to == "" { |
| + to = channel |
| + } |
| + if err := PromotionAllowed(from, to); err != nil { |
| + return res, fmt.Errorf("promote %s %s -> %s: %w", app, version, channel, err) |
| + } |
| + } |
| + rec := Promotion{From: res.From, To: res.To, Actor: opt.Actor, Hotfix: opt.Hotfix} |
| + if err := Promote(ctx, s, priv, acct, app, channel, version, rec); err != nil { |
| + return res, err |
| + } |
| + return res, nil |
| +} |
| + |
| +// VersionRung reports the highest rung any of this app's channels currently |
| +// points at version — "how mature is this build", answered from the pointers |
| +// themselves rather than a side table that could disagree with them. "" means |
| +// the version is on no rung (it may still be shipped, or on a channel outside |
| +// the ladder's vocabulary). |
| +// |
| +// A channel whose pointer cannot be read or verified is skipped rather than |
| +// fatal: an unreadable pointer can only LOWER the answer, which makes the |
| +// ordering check stricter, never looser. |
| +func VersionRung(ctx context.Context, s Store, pub ed25519.PublicKey, acct, app, version string) (string, error) { |
| + // Verification needs a real key: ed25519.Verify PANICS on a wrong-sized |
| + // one, and a read-only console runs with none. |
| + if len(pub) != ed25519.PublicKeySize { |
| + return "", fmt.Errorf("version rung: no usable release public key") |
| + } |
| + keys, prefix, _, err := ListWithFallback(ctx, s, ChannelsPrefix(acct, app), LegacyChannelsPrefix(app)) |
| + if err != nil { |
| + return "", fmt.Errorf("list channels of %s: %w", app, err) |
| + } |
| + best := "" |
| + for _, k := range keys { |
| + name := strings.TrimSuffix(strings.TrimPrefix(k, prefix), ".json") |
| + rung, ok := Rung(name) |
| + if !ok || RungIndex(rung) <= RungIndex(best) { |
| + continue // not a rung, or no higher than what we already found |
| + } |
| + p, err := Channel(ctx, s, pub, acct, app, name) |
| + if err != nil || p.Version != version { |
| + continue |
| + } |
| + best = rung |
| + } |
| + return best, nil |
| +} |
| + |
| // Channel resolves a channel's current pointer, verifying its signature — |
| // the one read path the fleet uses, so nothing unsigned ever reaches a box. |
| // Not-found is os.ErrNotExist-wrapped. |
| @@ -521,6 +711,9 @@ func Rollback(ctx context.Context, s Store, priv ed25519.PrivateKey, acct, app, |
| Channel: channel, |
| Version: old.History[0], |
| History: prependCapped(old.Version, old.History[1:], historyMax), |
| + // A rollback is not a promotion and appends nothing, but it must not |
| + // erase how the channel got here either. |
| + Audit: old.Audit, |
| } |
| wrapper, err := signPointer(priv, next) |
| if err != nil { |
| @@ -530,9 +723,11 @@ func Rollback(ctx context.Context, s Store, priv ed25519.PrivateKey, acct, app, |
| return err |
| } |
| -// prependCapped puts v at the head of hist, dropping the tail beyond max. |
| -func prependCapped(v string, hist []string, max int) []string { |
| - out := append([]string{v}, hist...) |
| +// prependCapped puts v at the head of list, dropping the tail beyond max — |
| +// the newest-first, bounded shape both the rollback history and the audit |
| +// trail have. |
| +func prependCapped[T any](v T, list []T, max int) []T { |
| + out := append([]T{v}, list...) |
| if len(out) > max { |
| out = out[:max] |
| } |
diff --git a/internal/release/release_test.go b/internal/release/release_test.go| index af5c038ab..235666e50 100644 |
| --- a/internal/release/release_test.go |
| +++ b/internal/release/release_test.go |
| @@ -3,10 +3,12 @@ package release |
| import ( |
| "context" |
| "crypto/ed25519" |
| + "encoding/base64" |
| "encoding/json" |
| "errors" |
| "fmt" |
| "os" |
| + "strings" |
| "testing" |
| "time" |
| ) |
| @@ -277,3 +279,261 @@ func TestSignStampsPromotedAt(t *testing.T) { |
| t.Fatalf("PromotedAt = %v, want stamped at signing time", got.PromotedAt) |
| } |
| } |
| + |
| +// The ladder, end to end through the store: a build walks edge -> beta -> |
| +// stable, a skip is refused with the bypass named, -hotfix places it anyway, |
| +// canary is never refused, and repeating a promotion is a no-op success. |
| +func TestPromoteOnLadderEnforcesOrdering(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", "linux-arm64", []byte("bin v1")) |
| + shipRelease(t, s, "bab", "myapp", "v2", "linux-arm64", []byte("bin v2")) |
| + |
| + promote := func(version, channel string, opt PromoteOptions) (PromoteResult, error) { |
| + return PromoteOnLadder(ctx, s, priv, "bab", "myapp", channel, version, opt) |
| + } |
| + |
| + // A fresh build enters at edge, then climbs one rung at a time. |
| + if _, err := promote("v1", "edge", PromoteOptions{}); err != nil { |
| + t.Fatalf("v1 -> edge: %v", err) |
| + } |
| + res, err := promote("v1", "beta", PromoteOptions{}) |
| + if err != nil { |
| + t.Fatalf("v1 edge -> beta: %v", err) |
| + } |
| + if res.From != RungEdge || res.To != RungBeta || res.NoOp { |
| + t.Fatalf("v1 edge -> beta: result = %+v", res) |
| + } |
| + if _, err := promote("v1", "stable", PromoteOptions{}); err != nil { |
| + t.Fatalf("v1 beta -> stable: %v", err) |
| + } |
| + if p, err := Channel(ctx, s, pub, "bab", "myapp", "stable"); err != nil || p.Version != "v1" { |
| + t.Fatalf("stable = %+v, %v; want v1", p, err) |
| + } |
| + |
| + // A build that has been nowhere cannot land on stable, and the refusal |
| + // says how to override it. |
| + _, err = promote("v2", "stable", PromoteOptions{}) |
| + if err == nil { |
| + t.Fatalf("v2 (on no rung) -> stable was allowed") |
| + } |
| + if !strings.Contains(err.Error(), "-hotfix") { |
| + t.Fatalf("refusal = %q, want it to name -hotfix", err) |
| + } |
| + if p, _ := Channel(ctx, s, pub, "bab", "myapp", "stable"); p.Version != "v1" { |
| + t.Fatalf("stable moved to %s on a refused promotion", p.Version) |
| + } |
| + // The skip is refused too — that is what the ladder is for. |
| + if _, err := promote("v2", "edge", PromoteOptions{}); err != nil { |
| + t.Fatalf("v2 -> edge: %v", err) |
| + } |
| + if _, err := promote("v2", "stable", PromoteOptions{}); err == nil { |
| + t.Fatalf("v2 edge -> stable (a skip) was allowed") |
| + } |
| + |
| + // -hotfix places it anyway, and says so. |
| + res, err = promote("v2", "stable", PromoteOptions{Hotfix: true}) |
| + if err != nil { |
| + t.Fatalf("v2 -> stable -hotfix: %v", err) |
| + } |
| + if !res.Hotfix || res.To != RungStable { |
| + t.Fatalf("hotfix result = %+v", res) |
| + } |
| + if p, _ := Channel(ctx, s, pub, "bab", "myapp", "stable"); p.Version != "v2" { |
| + t.Fatalf("stable = %s after hotfix, want v2", p.Version) |
| + } |
| + |
| + // Repeating a promotion is a no-op success, never an error — even though |
| + // v1 now sits on beta while stable has moved past it. |
| + res, err = promote("v1", "beta", PromoteOptions{}) |
| + if err != nil { |
| + t.Fatalf("re-promoting v1 to the beta it already holds: %v", err) |
| + } |
| + if !res.NoOp { |
| + t.Fatalf("re-promotion result = %+v, want NoOp", res) |
| + } |
| +} |
| + |
| +// Production runs on canary/* channels right now: ordering must never refuse |
| +// one, whatever the build is already on. |
| +func TestPromoteOnLadderNeverRefusesCanary(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", "linux-arm64", []byte("bin v1")) |
| + |
| + for _, channel := range []string{"canary/keymail", "canary/platform-adoption", "canary/rehearsal"} { |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "myapp", channel, "v1", PromoteOptions{}); err != nil { |
| + t.Fatalf("promote v1 -> %s: %v", channel, err) |
| + } |
| + if p, err := Channel(ctx, s, pub, "bab", "myapp", channel); err != nil || p.Version != "v1" { |
| + t.Fatalf("%s = %+v, %v", channel, p, err) |
| + } |
| + } |
| + // ...including a build that has already reached stable. |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "myapp", "edge", "v1", PromoteOptions{Hotfix: true}); err != nil { |
| + t.Fatalf("hotfix v1 -> edge: %v", err) |
| + } |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "myapp", "canary/late", "v1", PromoteOptions{}); err != nil { |
| + t.Fatalf("promote a build that is on edge back onto a canary: %v", err) |
| + } |
| +} |
| + |
| +// VersionRung answers "how mature is this build" from the channel pointers |
| +// that exist, and reports the HIGHEST rung when several point at it. |
| +func TestVersionRungReportsHighestRung(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", "linux-arm64", []byte("bin v1")) |
| + |
| + if rung, err := VersionRung(ctx, s, pub, "bab", "myapp", "v1"); err != nil || rung != "" { |
| + t.Fatalf("VersionRung with no pointers = %q, %v; want \"\"", rung, err) |
| + } |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "canary/x", "v1"); err != nil { |
| + t.Fatalf("Promote: %v", err) |
| + } |
| + if rung, err := VersionRung(ctx, s, pub, "bab", "myapp", "v1"); err != nil || rung != RungCanary { |
| + t.Fatalf("VersionRung on canary = %q, %v", rung, err) |
| + } |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "beta", "v1"); err != nil { |
| + t.Fatalf("Promote: %v", err) |
| + } |
| + if rung, err := VersionRung(ctx, s, pub, "bab", "myapp", "v1"); err != nil || rung != RungBeta { |
| + t.Fatalf("VersionRung on canary+beta = %q, %v; want the highest", rung, err) |
| + } |
| + // A channel that is not a rung says nothing about maturity. |
| + shipRelease(t, s, "bab", "myapp", "v2", "linux-arm64", []byte("bin v2")) |
| + if err := Promote(ctx, s, priv, "bab", "myapp", "wip", "v2"); err != nil { |
| + t.Fatalf("Promote: %v", err) |
| + } |
| + if rung, err := VersionRung(ctx, s, pub, "bab", "myapp", "v2"); err != nil || rung != "" { |
| + t.Fatalf("VersionRung of a build on a non-rung channel = %q, %v; want \"\"", rung, err) |
| + } |
| +} |
| + |
| +// Every promotion is recorded on the pointer it moves: what moved, from which |
| +// rung to which, who did it, whether it bypassed the ladder, and when. |
| +func TestPromotionRecordsAnAuditEntry(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", "linux-arm64", []byte("bin v1")) |
| + shipRelease(t, s, "bab", "myapp", "v2", "linux-arm64", []byte("bin v2")) |
| + |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "myapp", "edge", "v1", |
| + PromoteOptions{Actor: "paul@keymail.dev"}); err != nil { |
| + t.Fatalf("v1 -> edge: %v", err) |
| + } |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "myapp", "beta", "v1", |
| + PromoteOptions{Actor: "paul@keymail.dev"}); err != nil { |
| + t.Fatalf("v1 -> beta: %v", err) |
| + } |
| + p, err := Channel(ctx, s, pub, "bab", "myapp", "beta") |
| + if err != nil { |
| + t.Fatalf("Channel beta: %v", err) |
| + } |
| + if len(p.Audit) != 1 { |
| + t.Fatalf("beta audit = %+v, want one entry", p.Audit) |
| + } |
| + got := p.Audit[0] |
| + if got.Version != "v1" || got.From != RungEdge || got.To != RungBeta || |
| + got.Actor != "paul@keymail.dev" || got.Hotfix || got.At.IsZero() { |
| + t.Fatalf("audit entry = %+v", got) |
| + } |
| + |
| + // A hotfix says so, and the trail keeps what came before it, newest first. |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "myapp", "beta", "v2", |
| + PromoteOptions{Actor: "james@keymail.dev", Hotfix: true}); err != nil { |
| + t.Fatalf("v2 -> beta -hotfix: %v", err) |
| + } |
| + p, err = Channel(ctx, s, pub, "bab", "myapp", "beta") |
| + if err != nil { |
| + t.Fatalf("Channel beta: %v", err) |
| + } |
| + if len(p.Audit) != 2 { |
| + t.Fatalf("beta audit = %+v, want two entries", p.Audit) |
| + } |
| + if !p.Audit[0].Hotfix || p.Audit[0].Version != "v2" || p.Audit[0].Actor != "james@keymail.dev" { |
| + t.Fatalf("newest audit entry = %+v, want the hotfix", p.Audit[0]) |
| + } |
| + if p.Audit[1].Version != "v1" || p.Audit[1].Hotfix { |
| + t.Fatalf("older audit entry = %+v, want v1's ordinary promotion", p.Audit[1]) |
| + } |
| + // A no-op repeat records nothing: nothing happened. |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "myapp", "beta", "v2", |
| + PromoteOptions{Actor: "james@keymail.dev"}); err != nil { |
| + t.Fatalf("repeat promotion: %v", err) |
| + } |
| + if p, _ := Channel(ctx, s, pub, "bab", "myapp", "beta"); len(p.Audit) != 2 { |
| + t.Fatalf("a no-op repeat appended an audit entry: %+v", p.Audit) |
| + } |
| + // Rollback is not a promotion, but it must not lose the trail either. |
| + if err := Rollback(ctx, s, priv, "bab", "myapp", "beta"); err != nil { |
| + t.Fatalf("Rollback: %v", err) |
| + } |
| + if p, _ := Channel(ctx, s, pub, "bab", "myapp", "beta"); len(p.Audit) != 2 { |
| + t.Fatalf("rollback dropped the audit trail: %+v", p.Audit) |
| + } |
| +} |
| + |
| +// The audit trail is additive: pointers written before it existed — and there |
| +// are live ones in production buckets — must still verify and parse, with no |
| +// audit data and no error. This is the exact payload the pre-audit signPointer |
| +// produced, signed the same way. |
| +func TestPointerWrittenBeforeAuditStillParses(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + priv, pub := testKeys(t) |
| + shipRelease(t, s, "bab", "myapp", "v1", "linux-arm64", []byte("bin v1")) |
| + |
| + old := `{ |
| + "schema": 1, |
| + "app": "myapp", |
| + "channel": "beta", |
| + "version": "v1", |
| + "promoted_at": "2026-07-01T10:00:00Z", |
| + "history": [ |
| + "v0" |
| + ] |
| +}` |
| + b64 := base64.StdEncoding.EncodeToString([]byte(old)) |
| + wrapper, err := json.MarshalIndent(signedWrapper{ |
| + Payload: b64, |
| + Sig: base64.StdEncoding.EncodeToString(ed25519.Sign(priv, []byte(b64))), |
| + }, "", " ") |
| + if err != nil { |
| + t.Fatalf("marshal wrapper: %v", err) |
| + } |
| + if _, err := s.Put(ctx, ChannelKey("bab", "myapp", "beta"), wrapper, ""); err != nil { |
| + t.Fatalf("plant pre-audit pointer: %v", err) |
| + } |
| + |
| + p, err := Channel(ctx, s, pub, "bab", "myapp", "beta") |
| + if err != nil { |
| + t.Fatalf("Channel on a pre-audit pointer: %v", err) |
| + } |
| + if p.Version != "v1" || len(p.History) != 1 || p.History[0] != "v0" { |
| + t.Fatalf("pre-audit pointer = %+v", p) |
| + } |
| + if p.Audit != nil { |
| + t.Fatalf("pre-audit pointer has audit data: %+v", p.Audit) |
| + } |
| + // It still answers "which rung is this build on", and still promotes — |
| + // the trail simply starts at the next promotion. |
| + if rung, err := VersionRung(ctx, s, pub, "bab", "myapp", "v1"); err != nil || rung != RungBeta { |
| + t.Fatalf("VersionRung on a pre-audit pointer = %q, %v", rung, err) |
| + } |
| + if _, err := PromoteOnLadder(ctx, s, priv, "bab", "myapp", "stable", "v1", |
| + PromoteOptions{Actor: "paul@keymail.dev"}); err != nil { |
| + t.Fatalf("promote a pre-audit build: %v", err) |
| + } |
| + sp, err := Channel(ctx, s, pub, "bab", "myapp", "stable") |
| + if err != nil { |
| + t.Fatalf("Channel stable: %v", err) |
| + } |
| + if len(sp.Audit) != 1 || sp.Audit[0].From != RungBeta || sp.Audit[0].To != RungStable { |
| + t.Fatalf("stable audit = %+v", sp.Audit) |
| + } |
| +} |
diff --git a/internal/status/assets/index.html b/internal/status/assets/index.html| index 04bb6842b..4c3580849 100644 |
| --- a/internal/status/assets/index.html |
| +++ b/internal/status/assets/index.html |
| @@ -26,6 +26,10 @@ |
| <h2>Replication</h2> |
| <p class="tile-value" id="tile-replication"></p> |
| </div> |
| + <div class="tile"> |
| + <h2>Adoption</h2> |
| + <p class="tile-value" id="tile-adoption"></p> |
| + </div> |
| </section> |
| <section> |
| @@ -36,6 +40,7 @@ |
| <tr> |
| <th>Label</th><th>Region</th><th>Load</th> |
| <th>Static</th><th>Resident</th><th>Hibernating</th><th>Awake</th><th>Asleep</th> |
| + <th>Adoption</th> |
| <th></th> |
| </tr> |
| </thead> |
| @@ -135,6 +140,11 @@ |
| var repl = f.replication || {}; |
| document.getElementById("tile-replication").replaceChildren(asBool(repl.healthy, "healthy", "unhealthy")); |
| + |
| + // "not running" is the loud reading this tile exists for: a fleet whose |
| + // adoption timers aren't armed is one where nothing picks up a release on |
| + // its own, which used to be invisible from anywhere. |
| + document.getElementById("tile-adoption").replaceChildren(asBool(f.adoption_running, "running", "not running")); |
| } |
| function renderBoxes(f) { |
| @@ -148,6 +158,7 @@ |
| routeCols.forEach(function (k) { |
| tr.appendChild(td(asText(routes[k]))); |
| }); |
| + tr.appendChild(td(asBool(b.adoption_running, "running", "not running"))); |
| var staleTd = document.createElement("td"); |
| if (b.stale) { |
| // Bordered, not filled — a "trust this less" flag, not an outage |
diff --git a/internal/status/contract_test.go b/internal/status/contract_test.go| index 154202123..9243aa624 100644 |
| --- a/internal/status/contract_test.go |
| +++ b/internal/status/contract_test.go |
| @@ -24,7 +24,11 @@ import ( |
| // strings ("" for root, "fieldname" for nested, "fieldname[]" for array |
| // elements); values are the allowed field names at that path. |
| var allowedBoxKeys = map[string][]string{ |
| - "": {"generated_at", "label", "region", "routes", "load", "capacity", "replication"}, |
| + // "adoption_running" is a deliberate publication (pipelines spec §5): it |
| + // says whether the platform's own release-adoption timer is armed on this |
| + // box. It carries no tenant, hostname or machine identity — only whether |
| + // an opaquely-labelled box is picking up its own releases. |
| + "": {"generated_at", "label", "region", "routes", "load", "capacity", "replication", "adoption_running"}, |
| "routes": {"static", "resident", "hibernating", "awake", "asleep"}, |
| "replication": {"healthy", "lag_seconds"}, |
| } |
| @@ -32,14 +36,14 @@ var allowedBoxKeys = map[string][]string{ |
| // allowedFleetKeys are THE schema for Fleet's JSON output. Same friction as |
| // allowedBoxKeys: every new field is a deliberate publication decision. |
| var allowedFleetKeys = map[string][]string{ |
| - "": {"generated_at", "boxes", "instances", "capacity", "apps", "apps_unlisted", "services", "replication"}, |
| - "boxes[]": {"label", "region", "load", "stale", "routes"}, |
| - "boxes[].routes": {"static", "resident", "hibernating", "awake", "asleep"}, |
| - "instances": {"total", "static", "resident", "hibernating", "awake", "asleep"}, |
| - "capacity": {"level"}, |
| - "apps[]": {"name", "instances", "healthy", "channel", "version"}, |
| - "services[]": {"name", "up"}, |
| - "replication": {"healthy", "lag_seconds"}, |
| + "": {"generated_at", "boxes", "instances", "capacity", "apps", "apps_unlisted", "services", "replication", "adoption_running"}, |
| + "boxes[]": {"label", "region", "load", "stale", "routes", "adoption_running"}, |
| + "boxes[].routes": {"static", "resident", "hibernating", "awake", "asleep"}, |
| + "instances": {"total", "static", "resident", "hibernating", "awake", "asleep"}, |
| + "capacity": {"level"}, |
| + "apps[]": {"name", "instances", "healthy", "channel", "version"}, |
| + "services[]": {"name", "up"}, |
| + "replication": {"healthy", "lag_seconds"}, |
| } |
| // assertOnlyKeys walks the decoded JSON map recursively, ensuring every key |
| @@ -173,9 +177,10 @@ func fixtureRoutes() []registry.Route { |
| // produces non-nil results. It uses values in the middle bands so that |
| // load and capacity both resolve to band words (not nil). |
| func fixtureProbes() status.Probes { |
| - memFrac := 0.75 // Between 0.70 and 0.90 -> "busy" |
| - diskFrac := 0.80 // Between 0.70 and 0.90 -> "getting-full" |
| - lagSecs := int64(300) // 5 minutes, healthy (< 15 min) |
| + memFrac := 0.75 // Between 0.70 and 0.90 -> "busy" |
| + diskFrac := 0.80 // Between 0.70 and 0.90 -> "getting-full" |
| + lagSecs := int64(300) // 5 minutes, healthy (< 15 min) |
| + adopting := true // a real reading, so the field is scanned with a value in it |
| // Listening function returns true for hibernating routes, false otherwise. |
| listening := func(sock string) *bool { |
| @@ -188,12 +193,13 @@ func fixtureProbes() status.Probes { |
| } |
| return status.Probes{ |
| - MachineID: "test-machine-id", |
| - Region: "us-west-1", |
| - MemFrac: &memFrac, |
| - DiskFrac: &diskFrac, |
| - Listening: listening, |
| - LagSecs: &lagSecs, |
| + MachineID: "test-machine-id", |
| + Region: "us-west-1", |
| + MemFrac: &memFrac, |
| + DiskFrac: &diskFrac, |
| + Listening: listening, |
| + LagSecs: &lagSecs, |
| + AdoptRunning: &adopting, |
| } |
| } |
diff --git a/internal/status/dryrun.go b/internal/status/dryrun.go| index 6bbf50605..5b75cdaac 100644 |
| --- a/internal/status/dryrun.go |
| +++ b/internal/status/dryrun.go |
| @@ -87,12 +87,14 @@ func dryRunFixtures() ([]registry.Route, Probes) { |
| memFrac, diskFrac := 0.73, 0.95 |
| lag := int64(120) |
| + adopting := true // a known reading, so adoption_running publishes non-null and gets scanned |
| probes := Probes{ |
| - MachineID: "ip-10-0-0-7", |
| - Region: "us-east-1", |
| - MemFrac: &memFrac, |
| - DiskFrac: &diskFrac, |
| - LagSecs: &lag, |
| + MachineID: "ip-10-0-0-7", |
| + Region: "us-east-1", |
| + MemFrac: &memFrac, |
| + DiskFrac: &diskFrac, |
| + LagSecs: &lag, |
| + AdoptRunning: &adopting, |
| // Every hibernating route reports "listening" so Awake/Asleep publish |
| // non-null — an unexercised null field is a field DryRun can't scan. |
| Listening: func(string) *bool { v := true; return &v }, |
diff --git a/internal/status/fleet.go b/internal/status/fleet.go| index e75ad6fca..f0b9af913 100644 |
| --- a/internal/status/fleet.go |
| +++ b/internal/status/fleet.go |
| @@ -34,6 +34,14 @@ type FleetBox struct { |
| Load *string `json:"load"` |
| Stale bool `json:"stale"` |
| Routes RouteCounts `json:"routes"` |
| + // AdoptionRunning DOES appear per-box, unlike Replication and Capacity |
| + // above, because it is the one reading whose whole value is knowing WHICH |
| + // box has it wrong: "something in the fleet isn't adopting" sends an |
| + // operator to ssh into every box in turn, which is barely better than the |
| + // silence this field exists to end. It carries nothing about a tenant — |
| + // only whether the platform's own timer is armed on an opaquely-labelled |
| + // machine. |
| + AdoptionRunning *bool `json:"adoption_running"` |
| } |
| // Fleet is the merged feed: status/fleet.json. Every field here is either a |
| @@ -52,7 +60,8 @@ type Fleet struct { |
| Name string `json:"name"` |
| Up bool `json:"up"` |
| } `json:"services"` |
| - Replication Replication `json:"replication"` |
| + Replication Replication `json:"replication"` |
| + AdoptionRunning *bool `json:"adoption_running"` |
| } |
| // capacityRank orders the capacity band words worst-last, so "worst of a |
| @@ -90,17 +99,19 @@ func BuildFleet(ctx context.Context, s release.Store, routes []registry.Route, n |
| for i, b := range boxes { |
| stale[i] = now.Sub(b.GeneratedAt) > 3*period |
| f.Boxes = append(f.Boxes, FleetBox{ |
| - Label: b.Label, |
| - Region: b.Region, |
| - Load: b.Load, |
| - Stale: stale[i], |
| - Routes: b.Routes, |
| + Label: b.Label, |
| + Region: b.Region, |
| + Load: b.Load, |
| + Stale: stale[i], |
| + Routes: b.Routes, |
| + AdoptionRunning: b.AdoptionRunning, |
| }) |
| } |
| f.Instances = sumInstances(boxes) |
| f.Capacity.Level = worstCapacity(boxes) |
| f.Replication = worstReplication(boxes, stale) |
| + f.AdoptionRunning = worstAdoption(boxes, stale) |
| apps, unlisted := buildApps(routes) |
| f.Apps = apps |
| @@ -241,6 +252,47 @@ func worstReplication(boxes []Box, stale []bool) Replication { |
| return worst |
| } |
| +// worstAdoption folds the boxes' adoption readings into one fleet-wide |
| +// answer, worst-first: a single box that is KNOWN not to be adopting makes the |
| +// fleet false, because a real defect must never be averaged away by the |
| +// healthy boxes around it. Failing that, an unknown box nulls the aggregate — |
| +// "all the boxes we could ask said yes" is not the same claim as "the fleet is |
| +// adopting", and this package does not make the stronger claim from the weaker |
| +// evidence. Only when every voting box is known-true is the fleet true. |
| +// |
| +// Stale boxes don't vote at all (same rule as worstReplication): a snapshot |
| +// that stopped arriving means "we no longer know what that box is doing", and |
| +// a reading from twenty minutes ago is not evidence about now. That is a real |
| +// blind spot rather than a solved problem — but staleness is itself rendered |
| +// loudly per-box, so it is a visible blind spot, which is the whole point of |
| +// this field. |
| +// The order of the checks matters and is not the order the boxes happen to |
| +// arrive in: a known false is decided BEFORE any unknown is allowed to null |
| +// the result, so one unreachable box can never mask a box that reported a dead |
| +// timer. |
| +func worstAdoption(boxes []Box, stale []bool) *bool { |
| + anyUnknown, anyTrue := false, false |
| + for i, b := range boxes { |
| + if stale[i] { |
| + continue |
| + } |
| + switch { |
| + case b.AdoptionRunning == nil: |
| + anyUnknown = true |
| + case !*b.AdoptionRunning: |
| + no := false |
| + return &no |
| + default: |
| + anyTrue = true |
| + } |
| + } |
| + if anyUnknown || !anyTrue { |
| + return nil |
| + } |
| + yes := true |
| + return &yes |
| +} |
| + |
| // routeGroup collects one App name's routes in the order they appeared in |
| // the input slice — used for both the apps and services groupings below, |
| // which differ only in which Kind they filter on and how they reduce a |
diff --git a/internal/status/snapshot.go b/internal/status/snapshot.go| index 8a05fd2ec..534d2aac8 100644 |
| --- a/internal/status/snapshot.go |
| +++ b/internal/status/snapshot.go |
| @@ -28,6 +28,11 @@ type Probes struct { |
| DiskFrac *float64 // 0..1 proximity to the disk floor |
| Listening func(sock string) *bool // kernel socket table read; nil fn = unknown |
| LagSecs *int64 // seconds since litestream last progressed |
| + // AdoptRunning is whether release adoption is actually armed on this box |
| + // — the carlos-adopt.timer being ACTIVE, not merely enabled. nil is |
| + // "never asked / couldn't tell", and like every other nil here it must |
| + // publish as null: a box that can't answer is not a box that is fine. |
| + AdoptRunning *bool |
| } |
| // RouteCounts is aggregate-only: no host, no socket path, just how many. |
| @@ -62,6 +67,15 @@ type Box struct { |
| Load *string `json:"load"` // comfortable|busy|critical |
| Capacity *string `json:"capacity"` // plenty|getting-full|at-capacity |
| Replication Replication `json:"replication"` |
| + // AdoptionRunning is whether this box is actually picking up releases: |
| + // true = the adoption timer is armed and will fire, false = it is not and |
| + // nothing on this box will ever adopt on its own, null = unknown. It is a |
| + // bare bool rather than a band word because there is no spectrum here and |
| + // nothing private in the answer — a box either runs the platform's own |
| + // adoption pass or it doesn't, which says nothing about any tenant. The |
| + // flagship spent weeks in the false state with no way to see it; this |
| + // field is the way to see it. |
| + AdoptionRunning *bool `json:"adoption_running"` |
| } |
| // replHealthyWithin is the lag under which replication counts as healthy. |
| @@ -218,13 +232,14 @@ func CollectBox(routes []registry.Route, p Probes, now time.Time) Box { |
| } |
| return Box{ |
| - GeneratedAt: now, |
| - Label: Label(p.MachineID), |
| - Region: p.Region, |
| - Routes: counts, |
| - Load: bandWord(p.MemFrac, loadWords), |
| - Capacity: bandWord(worst, capacityWords), |
| - Replication: Replication{Healthy: healthy, LagSecs: p.LagSecs}, |
| + GeneratedAt: now, |
| + Label: Label(p.MachineID), |
| + Region: p.Region, |
| + Routes: counts, |
| + Load: bandWord(p.MemFrac, loadWords), |
| + Capacity: bandWord(worst, capacityWords), |
| + Replication: Replication{Healthy: healthy, LagSecs: p.LagSecs}, |
| + AdoptionRunning: p.AdoptRunning, |
| } |
| } |
diff --git a/internal/status/status_test.go b/internal/status/status_test.go| index 0c6b11eda..da4830b4c 100644 |
| --- a/internal/status/status_test.go |
| +++ b/internal/status/status_test.go |
| @@ -557,3 +557,101 @@ func TestDryRunSelfChecks(t *testing.T) { |
| t.Errorf("dry run must attest the self-check ran: %s", buf.String()) |
| } |
| } |
| + |
| +// ---- Pipelines Task 5: adoption health ------------------------------------- |
| + |
| +// TestAdoptionUnknownIsANull pins the honest-unknown half of the pipelines |
| +// spec's §5 ("adoption must actually run"): a box whose probe never managed to |
| +// ask systemd about the adoption timer publishes null, exactly like every |
| +// other unmeasured quantity in this package. The failure this guards against |
| +// is not a wrong reading, it is a REASSURING one — a default true would make |
| +// every box that can't answer look like a box that is adopting fine, which is |
| +// precisely the silence that let carlos-adopt.timer sit enabled-but-never- |
| +// started on the flagship for weeks with nothing on any page to show for it. |
| +func TestAdoptionUnknownIsANull(t *testing.T) { |
| + b := status.CollectBox(nil, status.Probes{MachineID: "m"}, time.Now()) // nil AdoptRunning |
| + if b.AdoptionRunning != nil { |
| + t.Errorf("adoption_running = %v, want null — an unasked question is not a healthy answer", *b.AdoptionRunning) |
| + } |
| + for _, want := range []bool{true, false} { |
| + got := status.CollectBox(nil, status.Probes{MachineID: "m", AdoptRunning: &want}, time.Now()).AdoptionRunning |
| + if got == nil || *got != want { |
| + t.Errorf("AdoptRunning=%v: adoption_running = %v, want %v", want, got, want) |
| + } |
| + } |
| +} |
| + |
| +// TestFleetAdoptionWorstOf covers the aggregate: a box that is KNOWN not to be |
| +// adopting is a defect the fleet number must never hide, so one false outranks |
| +// any number of trues; an unknown box can't be summed into a confident true, |
| +// so it nulls the aggregate instead; and a stale box doesn't vote at all |
| +// (staleness means "we no longer know", the same rule worstReplication uses). |
| +func TestFleetAdoptionWorstOf(t *testing.T) { |
| + yes, no := true, false |
| + cases := []struct { |
| + name string |
| + boxes map[string]*bool // machine id -> AdoptRunning |
| + stale map[string]bool // machine id -> written 16 minutes ago |
| + want *bool |
| + }{ |
| + {"all running", map[string]*bool{"a": &yes, "b": &yes}, nil, &yes}, |
| + {"one dead timer wins", map[string]*bool{"a": &yes, "b": &no}, nil, &no}, |
| + {"one unknown blocks a cheerful true", map[string]*bool{"a": &yes, "b": nil}, nil, nil}, |
| + {"a known defect beats an unknown", map[string]*bool{"a": nil, "b": &no}, nil, &no}, |
| + {"stale boxes do not vote", map[string]*bool{"a": &yes, "b": &no}, map[string]bool{"b": true}, &yes}, |
| + {"no boxes at all", nil, nil, nil}, |
| + } |
| + for _, c := range cases { |
| + t.Run(c.name, func(t *testing.T) { |
| + s := teststore(t) |
| + now := time.Now() |
| + for id, adopt := range c.boxes { |
| + at := now |
| + if c.stale[id] { |
| + at = now.Add(-16 * time.Minute) // > 3 x the 5-minute period |
| + } |
| + b := status.CollectBox(nil, status.Probes{MachineID: id, AdoptRunning: adopt}, at) |
| + if err := status.WriteBox(context.Background(), s, b); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + f, err := status.BuildFleet(context.Background(), s, nil, now, 5*time.Minute) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + switch { |
| + case c.want == nil && f.AdoptionRunning != nil: |
| + t.Errorf("fleet adoption_running = %v, want null", *f.AdoptionRunning) |
| + case c.want != nil && f.AdoptionRunning == nil: |
| + t.Errorf("fleet adoption_running = null, want %v", *c.want) |
| + case c.want != nil && *f.AdoptionRunning != *c.want: |
| + t.Errorf("fleet adoption_running = %v, want %v", *f.AdoptionRunning, *c.want) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// TestFleetBoxCarriesAdoption keeps the per-box row honest too: the fleet-wide |
| +// number says SOMETHING is not adopting, the boxes table says which label — |
| +// without it, an operator reading "not running" on a two-box fleet has to go |
| +// SSH into both to find out where the dead timer is. |
| +func TestFleetBoxCarriesAdoption(t *testing.T) { |
| + s := teststore(t) |
| + now := time.Now() |
| + no := false |
| + if err := status.WriteBox(context.Background(), s, |
| + status.CollectBox(nil, status.Probes{MachineID: "dead", AdoptRunning: &no}, now)); err != nil { |
| + t.Fatal(err) |
| + } |
| + f, err := status.BuildFleet(context.Background(), s, nil, now, 5*time.Minute) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(f.Boxes) != 1 { |
| + t.Fatalf("boxes = %d, want 1", len(f.Boxes)) |
| + } |
| + if f.Boxes[0].AdoptionRunning == nil || *f.Boxes[0].AdoptionRunning { |
| + t.Errorf("boxes[0].adoption_running = %v, want false — the row must name the box with the dead timer", |
| + f.Boxes[0].AdoptionRunning) |
| + } |
| +} |
diff --git a/main.go b/main.go| index fd1d403c6..36f67a4bb 100644 |
| --- a/main.go |
| +++ b/main.go |
| @@ -77,6 +77,8 @@ commands: |
| (headless? don't log in: pass -owner instead) |
| accounts create | list | migrate — the tenancy primitive |
| ship | promote | channels publish and promote releases (-account <sqid|name>) |
| + promote climbs the ladder canary -> edge -> beta |
| + -> stable; -hotfix is the recorded bypass |
| adopt run one release-adoption pass (the hourly timer's verb) |
| canary ship this worktree to an ephemeral canary channel |
| bootstrap prepare a host (units, dirs, user) |