amadan
regions: each box answers for its own hosting, and three doors stop sharing one bar
Follow-up to 78d5664, which two reviews found had the right instinct, a
false story and three real defects. This fixes all four.
**Hosting is answered per box now, never inferred across boxes.** The
previous version keyed it on data_bucket alone. An absent data_bucket is
exactly what an OLD WRITER and a bucket-less new box both produce, so it
had to decide when silence meant "no" — and it decided by asking whether
any OTHER box had spoken. The consequence: the first box in a fleet to
adopt a bucket-publishing build withdrew every region that had not yet
rolled, all of them healthy. Minutes for an uninterrupted fleet roll,
hours at one box an hour, unbounded if a roll stalled. One writer's
upgrade cannot establish another writer's capabilities.
So each box states its own answer. `hibernation_configured` is published
on every edges file, always, never omitempty — that is the point of it,
because only an explicit `false` can carry the third state:
absent → a writer predating the field: UNKNOWN
true → the activator is configured
false → it is not
Unknown is not no: a box that predates the field keeps the eligibility it
had before the field existed. That is deliberately generous, and it is
what makes a box roll safe in either direction. A create into a genuinely
unconfigured old box can still strand — but that is the bug we already
had, where refusing creates into regions that host perfectly well would be
a new one, caused by the fix.
It reports CONFIGURATION, not verified hosting: a box naming a bucket its
IAM role cannot reach publishes true and still fails every wake. Naming it
for what it actually knows is the point.
**The three doors stop sharing one bar.** They were asking three different
questions through one predicate, which is how a static site came to be
refused for want of an activator it never needed:
- Reachable (an EIP) is required by every kind. A box nothing can dial
homes nothing.
- Hosting is required only by a SOCKET-BACKED INSTANCE, which is the only
kind an activator spawns. A static route is served from bucket bytes by
every armed edge; an addr-backed service proxies to a process the
platform never starts. The create gate ran before the record's kind was
even read, so it applied to all three alike; it now asks
provision.Record's own DerivesPaths, so it cannot drift from what
reconcile will mint.
- Staging (published bucket coordinates) is the move door's bar, and it is
UNCONDITIONAL — no dormant path, because internal/host's moveTargets
drops any target with no data_bucket and runMoveRound then answers
"publishes no data bucket yet — waiting" forever. The door accepted such
a move on reachability alone and wedged a live instance's residence in
preparing with no way out but a manual cancel: the door and the worker
disagreeing about one fact, which is the failure this family of gates
exists to prevent. Unknown is acceptable for a create, which stakes one
new instance, and not for a move, which stakes a running one.
**The prose is corrected.** 78d5664 asserted, in six places, that tito's
five region boxes had no CARLOS_HIBERNATE_BUCKET and that this caused the
502 in discussion #110. That is false: every tito region box publishes a
data_bucket, so their activators are armed, and the reported 502 remains
unexplained — per-instance DNS resolving a region-placed host to the home
edge, so the request must cross the backhaul, is the current lead. Every
sentence resting on that story is gone. The log line no longer claims
every request 502s (it cannot see whether something outside the platform
is bound to that socket) and no longer gives env advice that only helps
one of the two roles that reach it.
The documented gap it should have cited all along is real and is now
closed at the source: region-host/README.md's minimum host.env listed four
variables and said "nothing more", so a box built strictly by the docs
would be an edge that cannot host. It now carries the hibernation block
and says what leaving it out costs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ggesa34CVBQ97guKaEi9jw
12 files changed,
+517
−162
infra/modules/region-host/README.md+27 −1infra/modules/region-host/variables.tf+13 −8internal/console/apiinstancemove.go+37 −4internal/console/apiinstancemove_test.go+6 −3internal/console/apiinstances.go+23 −2internal/console/instancesplacement.go+119 −60internal/console/instancesplacement_test.go+73 −45internal/console/placementdoors_test.go+137 −0internal/edge/edge.go+28 −22internal/edge/noactivator_test.go+11 −9internal/host/edgesfile.go+39 −4internal/host/edgesfile_test.go+4 −4
CI passed — run details
diff --git a/infra/modules/region-host/README.md b/infra/modules/region-host/README.md| index ace82e086..b10c3a8c0 100644 |
| --- a/infra/modules/region-host/README.md |
| +++ b/infra/modules/region-host/README.md |
| @@ -29,6 +29,30 @@ meant to be. |
| AWS_REGION=eu-west-1 |
| ``` |
| + That is enough to make the box an **edge**: it will terminate TLS, adopt |
| + releases, converge routes and serve static sites. It is not enough to run |
| + an app. If this region is to **host instances**, it also needs its own |
| + data bucket — the one `tofu apply` already made and granted the host role: |
| + |
| + ```sh |
| + CARLOS_HIBERNATE_BUCKET=carlos-myco-ie-123456789012 |
| + CARLOS_HIBERNATE_REGION=eu-west-1 |
| + CARLOS_HIBERNATE_PREFIX=instances |
| + ``` |
| + |
| + Without those, the hibernation activator never arms, and nothing on the |
| + box ever starts an instance's process. The failure is quiet from the |
| + box's side — the record is accepted, the route is minted and the binary |
| + is staged, and only the request finds out — so the box now publishes |
| + what it was told as `hibernation_configured` in its edges file, and the |
| + console refuses instance creates into a region whose boxes report false |
| + rather than accepting one that would strand. Add the variables and |
| + restart `carlos-edge` and the region arms itself within a minute or two. |
| + |
| + `CARLOS_HIBERNATE_REGION` is the AWS region of the **data bucket**, and |
| + like `AWS_REGION` above it is a real AWS region name, not a placement |
| + code. |
| + |
| `AWS_REGION` is the one that bites. It must name the region the |
| **deployment bucket** lives in, not the region this box is in. The AWS Go |
| SDK does not follow S3's cross-region redirects, so a box in Sydney |
| @@ -64,7 +88,9 @@ half never leaves your secret store and never touches a box. There is |
| nothing region-specific to create, copy or rotate when you add a region. |
| What a region does need is in the four steps above and nothing more: the |
| -binary, `host.env`, `carlos bootstrap`, start the edge. |
| +binary, `host.env`, `carlos bootstrap`, start the edge. The only thing that |
| +is easy to leave out is the hibernation block in step 2 — and leaving it out |
| +is what makes a region an edge that cannot host. |
| ## Arming the root passes |
diff --git a/infra/modules/region-host/variables.tf b/infra/modules/region-host/variables.tf| index 1c9e3164e..dddc4cd84 100644 |
| --- a/infra/modules/region-host/variables.tf |
| +++ b/infra/modules/region-host/variables.tf |
| @@ -40,14 +40,19 @@ variable "data_bucket_arn" { |
| Granting it is half the job. The box must also be TOLD the bucket, as |
| CARLOS_HIBERNATE_BUCKET (plus CARLOS_HIBERNATE_REGION) in |
| - /etc/carlos/host.env, or its activator never arms and the box is an edge |
| - that cannot host: it will accept an instance, mint the route, stage the |
| - binary — and then serve 502 forever, because only an armed activator ever |
| - creates the socket the edge is dialling. Five of tito's six regions sat in |
| - exactly that state (discussion #110). The console now reads the box's |
| - published data_bucket as its hosting signal, so a region that has not been |
| - told is reported unarmed and refused at create rather than accepted and |
| - stranded. |
| + /etc/carlos/host.env, or its hibernation activator never arms and the box |
| + is an edge that cannot host: it will accept an instance, mint the route |
| + and stage the binary, and then nothing on it ever starts the process. |
| + |
| + This module does not write host.env — user_data is yours — so it is a |
| + step that has to be remembered, and the bring-up README's minimum env |
| + does not list it. |
| + |
| + The box publishes what it was told, as hibernation_configured in its |
| + control/edges/<box>.json, and the console reads that: a region whose |
| + boxes report false is refused at instance create rather than accepted and |
| + stranded. The field reports CONFIGURATION only — a box naming a bucket |
| + its IAM role cannot reach still reports true and still fails every wake. |
| EOT |
| type = string |
| } |
diff --git a/internal/console/apiinstancemove.go b/internal/console/apiinstancemove.go| index 23e674b7c..0c7466c91 100644 |
| --- a/internal/console/apiinstancemove.go |
| +++ b/internal/console/apiinstancemove.go |
| @@ -110,6 +110,25 @@ func armedCodes(set map[string]bool) string { |
| return strings.Join(codes, ", ") |
| } |
| +// stagingCodes is armedCodes for the move door's stricter bar: the regions |
| +// that publish a staging bucket, which is the set a move can actually name. |
| +// A separate renderer rather than a flag on armedCodes, because a refusal |
| +// that listed "armed" regions a move still could not use would send the |
| +// caller round the same loop again. |
| +func stagingCodes(caps map[string]placementCapability) string { |
| + codes := make([]string, 0, len(caps)) |
| + for c, cap := range caps { |
| + if cap.Reachable && cap.Staging { |
| + codes = append(codes, c) |
| + } |
| + } |
| + slices.Sort(codes) |
| + if len(codes) == 0 { |
| + return "(none)" |
| + } |
| + return strings.Join(codes, ", ") |
| +} |
| + |
| // homeRegionIn/Out translate between the home region's NAME (Config's |
| // CONSOLE_HOME_REGION, what pickers and callers speak — issue #346) and the |
| // record's "" (the unconstrained value the fleet's reconcile has always |
| @@ -184,13 +203,27 @@ func (s *Server) handleAPIInstanceMoveRequest(w http.ResponseWriter, r *http.Req |
| // The armed gate is MANDATORY for moves — unlike create there is no |
| // dormant path, because a move needs a live target box to restore on; |
| // a deployment with no armed placements cannot be a move target at all. |
| - set, armed := armedPlacements(r.Context(), s.cfg.Releases) |
| - if !armed { |
| + // |
| + // And the bar here is STAGING, not merely hosting: the requirement is |
| + // unconditional and is the target's published data-bucket coordinates, |
| + // because that is precisely what the worker on the other side demands. |
| + // internal/host's moveTargets skips any target whose edges file carries |
| + // no data_bucket, and runMoveRound then answers "target region %s |
| + // publishes no data bucket yet — waiting" on every pass, forever. Before |
| + // this the door accepted such a target on reachability alone and the |
| + // move wedged in preparing with no way out but a manual cancel — the |
| + // door and the worker disagreeing about the same fact, which is the |
| + // failure this whole family of gates exists to prevent. Unknown is not |
| + // an acceptable answer for a move the way it is for a create: a create |
| + // into an unconfigured box strands one instance, while a move stakes a |
| + // LIVE instance's residence on a lane that does not exist. |
| + caps, published := placements(r.Context(), s.cfg.Releases) |
| + if !published { |
| jsonError(w, http.StatusBadRequest, "this deployment publishes no armed regions — nothing to move to") |
| return |
| } |
| - if !set[to] { |
| - jsonError(w, http.StatusBadRequest, "region "+to+" is not armed in this deployment — armed: "+armedCodes(set)) |
| + if !caps[to].Reachable || !caps[to].Staging { |
| + jsonError(w, http.StatusBadRequest, "region "+to+" cannot take a move in this deployment — it publishes no staging bucket. Regions that can: "+stagingCodes(caps)) |
| return |
| } |
| s.moveTransition(w, r, acct, app, host, provision.MoveRequest(to, time.Now()), http.StatusAccepted) |
diff --git a/internal/console/apiinstancemove_test.go b/internal/console/apiinstancemove_test.go| index 6b4a8a136..be75ef761 100644 |
| --- a/internal/console/apiinstancemove_test.go |
| +++ b/internal/console/apiinstancemove_test.go |
| @@ -38,9 +38,12 @@ func TestAPIInstanceMoveVerbs(t *testing.T) { |
| put("control/edges/uk-box.json", `{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","data_bucket":"data-uk","data_bucket_region":"eu-west-2","hosts":{}}`) |
| moveURL := u + "/acme.go.tito.io/move" |
| - // Unarmed target → 400 naming the set. |
| - if code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"se"}`); code != http.StatusBadRequest || !strings.Contains(out, "armed: uk") { |
| - t.Fatalf("move to unarmed = %d %s, want 400 naming the set", code, out) |
| + // A target with no staging bucket → 400 naming the set that can take one. |
| + // The move door's bar is the target's published data_bucket, because |
| + // that is exactly what internal/host's moveTargets demands before a move |
| + // can even be prepared. |
| + if code, _, out := rawReq(t, http.MethodPost, moveURL, token, `{"region":"se"}`); code != http.StatusBadRequest || !strings.Contains(out, "Regions that can: uk") { |
| + t.Fatalf("move to a region with no staging bucket = %d %s, want 400 naming the set", code, out) |
| } |
| // Same region → 409 (the transition's own conflict). |
| if code, _, out := rawReq(t, http.MethodPost, u+"/acme.go.tito.io/move", token, `{"region":"ie"}`); code != http.StatusBadRequest && code != http.StatusConflict { |
diff --git a/internal/console/apiinstances.go b/internal/console/apiinstances.go| index 6ce77a0c8..f72a32f95 100644 |
| --- a/internal/console/apiinstances.go |
| +++ b/internal/console/apiinstances.go |
| @@ -642,10 +642,31 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request) |
| // deployment's home", which always exists); and the whole gate is |
| // dormant until some box publishes a placement code, so a console rolled |
| // ahead of its fleet changes nothing. |
| + // |
| + // The gate has TWO bars, and which one applies depends on what is being |
| + // created. Every kind needs the region to be REACHABLE — a box nothing |
| + // can dial homes nothing. Only a socket-backed instance additionally |
| + // needs the region to HOST, because only that kind is spawned by an |
| + // activator: a static route is served from bucket bytes by every armed |
| + // edge with no process at all, and an addr-backed service proxies to a |
| + // process the platform never starts. Gating those two on hosting would |
| + // refuse a perfectly serveable static site in a region for want of a |
| + // hibernation bucket it would never have used. The predicate is |
| + // provision.Record's own (DerivesPaths), asked of a probe record rather |
| + // than restated here, so it cannot drift from what reconcile will mint. |
| if placed == "" && region != "" { |
| - if set, published := armedPlacements(ctx, s.cfg.Releases); published && !set[region] { |
| + probe := provision.Record{Kind: strings.TrimSpace(req.Kind), Addrs: req.Addrs} |
| + caps, published := placements(ctx, s.cfg.Releases) |
| + switch { |
| + case !published: |
| + // Dormant: the fleet predates the placement field. |
| + case !caps[region].Reachable: |
| + jsonError(w, http.StatusBadRequest, |
| + "region "+region+" is not armed in this deployment — armed: "+armedCodes(reachableSet(caps))) |
| + return |
| + case probe.DerivesPaths() && !caps[region].Hosting: |
| jsonError(w, http.StatusBadRequest, |
| - "region "+region+" is not armed in this deployment — armed: "+armedCodes(set)) |
| + "region "+region+" is reachable but cannot host an instance — no box there has its hibernation activator configured. Regions that can: "+armedCodes(hostingSet(caps))) |
| return |
| } |
| } |
diff --git a/internal/console/instancesplacement.go b/internal/console/instancesplacement.go| index 18d9d9824..ba26f71d9 100644 |
| --- a/internal/console/instancesplacement.go |
| +++ b/internal/console/instancesplacement.go |
| @@ -40,10 +40,19 @@ type edgesFilePlacementView struct { |
| Placement string `json:"placement"` |
| // DataBucket is the box's CARLOS_HIBERNATE_BUCKET, verbatim |
| // (internal/host/edgesfile.go writes it from that env and omits the key |
| - // entirely when it is unset). It is read here as the box's own answer to |
| - // "can I run an instance" — see armedPlacements. |
| - DataBucket string `json:"data_bucket"` |
| - Hosts map[string]json.RawMessage `json:"hosts"` |
| + // entirely when it is unset). It carries the move lane's actual |
| + // coordinates, and doubles as positive hosting evidence from a writer |
| + // too old to publish HibernationConfigured. |
| + DataBucket string `json:"data_bucket"` |
| + // HibernationConfigured is the box's own answer to "is my hibernation |
| + // activator wired" (internal/host/edgesfile.go). A POINTER because |
| + // absent and false are different facts and the difference is the whole |
| + // point: absent is a writer that predates the field, whose capability is |
| + // UNKNOWN, while false is a box that published the field and said no. |
| + // Reading absence as no is what made an earlier version withdraw healthy |
| + // regions mid-roll — see placements. |
| + HibernationConfigured *bool `json:"hibernation_configured"` |
| + Hosts map[string]json.RawMessage `json:"hosts"` |
| } |
| // loadEdgesJoin folds every readable edges file into host → the boxes that |
| @@ -101,70 +110,120 @@ func readEdgesFiles(ctx context.Context, s release.Store) []edgesFilePlacementVi |
| return out |
| } |
| -// armedPlacements folds the fleet's edges files into every published |
| -// placement code and whether it can take an instance today (issue #329): |
| -// armed means some box publishing the code can be reached AND can host, so |
| -// set[code] false is a region that EXISTS but cannot land one, which the |
| -// regions endpoint (#346) shows and the create door refuses alike. The |
| -// second return reports whether ANY file carried a placement code at all: |
| -// false means the fleet predates the field (or the read failed) and the |
| -// create gate stays DORMANT — the door then accepts what it always |
| -// accepted, so rolling boxes and console in either order is safe. |
| +// placementCapability is what one placement code's boxes can do, folded |
| +// across every box publishing that code. Three separate facts, because the |
| +// three doors that consult them are asking three different questions, and |
| +// conflating those is how a static site came to be refused for want of an |
| +// activator it never needed. |
| +type placementCapability struct { |
| + // Reachable: some box publishing this code has a dial address. A box |
| + // nothing can reach is not a landing target for anything. |
| + Reachable bool |
| + // Hosting: some reachable box publishing this code has its hibernation |
| + // activator configured. Only a SOCKET-BACKED INSTANCE needs this — a |
| + // static route is served from bucket bytes by every edge, and an |
| + // addr-backed service proxies to a process the platform never spawns. |
| + Hosting bool |
| + // Staging: some reachable box publishing this code publishes actual |
| + // data-bucket COORDINATES, which is the move door's requirement and is |
| + // strictly stronger than Hosting. internal/host's moveTargets drops any |
| + // target whose file carries no data_bucket, and runMoveRound then |
| + // reports "target region %s publishes no data bucket yet — waiting" for |
| + // as long as the record lives. A door admitting such a move promises |
| + // something the worker refuses forever. |
| + Staging bool |
| +} |
| + |
| +// placements folds the fleet's edges files into each published placement |
| +// code's capability (issue #329, discussion #110). The second return reports |
| +// whether ANY file carried a placement code at all: false means the fleet |
| +// predates the field (or the read failed) and the create gate stays DORMANT, |
| +// accepting what it always accepted, so rolling boxes and console in either |
| +// order is safe. |
| // |
| -// Reachable is an EIP: a box nothing can dial is not a landing target. |
| +// The hosting question is answered PER BOX, from that box's own |
| +// hibernation_configured field, and is never inferred across boxes. That |
| +// distinction is what discussion #110's review paid for. An earlier version |
| +// keyed hosting on data_bucket alone — and because an absent data_bucket is |
| +// exactly what an OLD WRITER and a bucket-less new box both produce, it had |
| +// to decide when silence meant "no". It decided by asking whether any OTHER |
| +// box had spoken, so the first box in a fleet to adopt a bucket-publishing |
| +// build withdrew every region that had not yet rolled, all of them healthy: |
| +// minutes for an uninterrupted fleet-wide roll, hours at one box an hour, |
| +// unbounded if a roll stalled. One writer's upgrade cannot establish another |
| +// writer's capabilities. |
| // |
| -// Can host is a published data bucket, and it is the half discussion #110 |
| -// paid for. Before it, armed was reachability alone — an EDGE-MEMBERSHIP |
| -// fact — while every consumer of this set reads it as "a region that can |
| -// take an instance right now". On tito those two diverged for five of six |
| -// regions: the boxes served TLS, converged routes and staged binaries |
| -// perfectly, so they were reachable and published their placement codes, |
| -// but none had been told a CARLOS_HIBERNATE_BUCKET. Their activators never |
| -// armed. An instance created into one of them was accepted, placed, staged |
| -// and routed, and then answered 502 forever, because the edge proxies a |
| -// hibernate-flagged route straight to a socket that only an armed |
| -// activator would ever create. |
| +// So each file answers only for itself, and UNKNOWN is not NO: |
| // |
| -// data_bucket is the box's own answer and not an inference: internal/host's |
| -// edges-file pass writes CARLOS_HIBERNATE_BUCKET into it verbatim and omits |
| -// the key when that env is unset, and it is the SAME env, read in the same |
| -// process, that decides whether activator.ConfigFromEnv arms the activator |
| -// at all (internal/host/agent.go). So a published data bucket means that |
| -// box's activator is armed, which is exactly the question a region picker |
| -// is asking. It heals with no console change: the moment a region box's |
| -// host.env grows the variable, its next edges-file pass publishes the key |
| -// and the region arms itself. |
| +// hibernation_configured present → that box's own answer, believed |
| +// absent → a writer predating the field: unknown, |
| +// and it keeps the eligibility it had |
| +// before the field existed |
| // |
| -// DORMANT until some box publishes one, the same roll-order rule the |
| -// placement field itself got and for the same reason. A fleet whose boxes |
| -// all predate the data_bucket field would otherwise go from "every region |
| -// armed" to "no region armed" the moment this console rolled, which is a |
| -// worse failure than the one it fixes — it would refuse creates into |
| -// regions that host perfectly well. With the gate dormant such a fleet |
| -// keeps the reachability-only answer it has always had. |
| -func armedPlacements(ctx context.Context, s release.Store) (set map[string]bool, published bool) { |
| - files := readEdgesFiles(ctx, s) |
| - // The dormancy read, taken over every file before any verdict: one box |
| - // publishing a data bucket is what makes the fleet's silence on the |
| - // field meaningful for every OTHER box. |
| - hostingKnown := false |
| - for _, f := range files { |
| - if f.DataBucket != "" { |
| - hostingKnown = true |
| - break |
| - } |
| - } |
| - set = map[string]bool{} |
| - for _, f := range files { |
| +// That last line is what makes a box roll safe in either direction, and it |
| +// is deliberately generous. A genuinely unconfigured box on an old build |
| +// stays eligible until it rolls and says so, so a create into it can still |
| +// strand — but that is the bug we already had, whereas refusing creates into |
| +// regions that host perfectly well would be a new one, caused BY the fix. |
| +// |
| +// None of these three is a promise that a wake will SUCCEED. A box whose IAM |
| +// role cannot reach the bucket it names, or whose CARLOS_HIBERNATE_REGION is |
| +// wrong, publishes hibernation_configured true and fails every lease. This |
| +// is configuration, which is all an edges file can honestly carry. |
| +func placements(ctx context.Context, s release.Store) (caps map[string]placementCapability, published bool) { |
| + caps = map[string]placementCapability{} |
| + for _, f := range readEdgesFiles(ctx, s) { |
| if f.Placement == "" { |
| continue |
| } |
| published = true |
| - armed := f.EIP != "" |
| - if hostingKnown { |
| - armed = armed && f.DataBucket != "" |
| + c := caps[f.Placement] |
| + // Every capability below is gated on reachability: a box with no |
| + // dial address contributes nothing, however well configured it is. |
| + // The code is still RECORDED, so an unreachable region reads as |
| + // known-and-unarmed rather than as absent. |
| + if f.EIP == "" { |
| + caps[f.Placement] = c |
| + continue |
| + } |
| + // Unknown is not no. An absent hibernation_configured with no |
| + // data_bucket is a writer that predates the field, and it keeps the |
| + // eligibility it had before the field existed. |
| + hosting := true |
| + if f.HibernationConfigured != nil { |
| + hosting = *f.HibernationConfigured |
| } |
| - set[f.Placement] = set[f.Placement] || armed |
| + c.Reachable = true |
| + c.Hosting = c.Hosting || hosting |
| + c.Staging = c.Staging || f.DataBucket != "" |
| + caps[f.Placement] = c |
| + } |
| + return caps, published |
| +} |
| + |
| +// armedPlacements is the picker answer in the shape its callers have always |
| +// taken: code -> "can take an instance right now", which is Reachable AND |
| +// Hosting. |
| +func armedPlacements(ctx context.Context, s release.Store) (set map[string]bool, published bool) { |
| + caps, published := placements(ctx, s) |
| + return hostingSet(caps), published |
| +} |
| + |
| +// hostingSet and reachableSet project a capability map onto the one axis a |
| +// particular refusal is about, so the list a caller is handed names regions |
| +// that would actually satisfy the bar they just failed. |
| +func hostingSet(caps map[string]placementCapability) map[string]bool { |
| + set := make(map[string]bool, len(caps)) |
| + for code, c := range caps { |
| + set[code] = c.Reachable && c.Hosting |
| + } |
| + return set |
| +} |
| + |
| +func reachableSet(caps map[string]placementCapability) map[string]bool { |
| + set := make(map[string]bool, len(caps)) |
| + for code, c := range caps { |
| + set[code] = c.Reachable |
| } |
| - return set, published |
| + return set |
| } |
diff --git a/internal/console/instancesplacement_test.go b/internal/console/instancesplacement_test.go| index 656914dd1..748c91100 100644 |
| --- a/internal/console/instancesplacement_test.go |
| +++ b/internal/console/instancesplacement_test.go |
| @@ -5,16 +5,16 @@ import ( |
| "testing" |
| ) |
| -// The hosting half of armed (discussion #110). Before it, armed was |
| -// reachability alone — an edge-membership fact — while every consumer reads |
| -// the set as "regions that can take an instance right now". On tito those two |
| -// diverged for five of six regions: reachable boxes that had never been told |
| -// a CARLOS_HIBERNATE_BUCKET, so their activators never armed, so an instance |
| -// placed there was accepted, staged and routed and then served 502 forever. A |
| -// box's published data_bucket is its own answer to "can I host", and it is |
| -// the same env that arms the activator. |
| -func TestArmedPlacementsRequiresHosting(t *testing.T) { |
| - fx, _ := instanceFixture(t, "code-inst-armedhosting") |
| +// Hosting is answered PER BOX, from that box's own hibernation_configured |
| +// field, and is never inferred from what another box publishes. An earlier |
| +// version keyed hosting on data_bucket alone, and because an absent |
| +// data_bucket is what an OLD WRITER and a bucket-less new box both produce, |
| +// it decided when silence meant "no" by asking whether any OTHER box had |
| +// spoken. One box adopting a bucket-publishing build then withdrew every |
| +// region that had not yet rolled, all of them healthy. These tests pin the |
| +// three states apart. |
| +func TestPlacementsAnswersHostingPerBox(t *testing.T) { |
| + fx, _ := instanceFixture(t, "code-inst-placementcaps") |
| ctx := context.Background() |
| put := func(key, body string) { |
| t.Helper() |
| @@ -23,53 +23,81 @@ func TestArmedPlacementsRequiresHosting(t *testing.T) { |
| } |
| } |
| - // Dormant: no box in the fleet publishes a data bucket at all, so the |
| - // hosting gate must not fire and armed stays exactly the reachability |
| - // answer these boxes have always had. A fleet whose boxes predate the |
| - // field must not lose every region the day this console rolls. |
| + // uk: an old writer — no hibernation_configured, no data_bucket. UNKNOWN, |
| + // and unknown keeps the eligibility it had before the field existed. |
| put("control/edges/uk-box.json", `{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","hosts":{}}`) |
| - put("control/edges/se-box.json", `{"v":1,"box":"se-box","eip":"","placement":"se","hosts":{}}`) |
| - set, published := armedPlacements(ctx, fx.rel) |
| + // ie: a new writer that can host. |
| + put("control/edges/ie-box.json", `{"v":1,"box":"ie-box","eip":"5.6.7.8","placement":"ie","data_bucket":"carlos-ie","hibernation_configured":true,"hosts":{}}`) |
| + // de: a new writer that CANNOT host, and says so explicitly. This is the |
| + // state no earlier version could express. |
| + put("control/edges/de-box.json", `{"v":1,"box":"de-box","eip":"9.9.9.9","placement":"de","hibernation_configured":false,"hosts":{}}`) |
| + // se: reachable by nothing. |
| + put("control/edges/se-box.json", `{"v":1,"box":"se-box","eip":"","placement":"se","data_bucket":"carlos-se","hibernation_configured":true,"hosts":{}}`) |
| + |
| + caps, published := placements(ctx, fx.rel) |
| if !published { |
| - t.Fatal("published = false, want true: both files carry a placement code") |
| + t.Fatal("published = false, want true") |
| } |
| - if !set["uk"] { |
| - t.Error("uk unarmed while the fleet publishes no data bucket at all — the gate must stay dormant") |
| + |
| + // The whole point: ie publishing a bucket must not make uk's silence |
| + // mean "no". uk stays eligible until uk itself says otherwise. |
| + if !caps["uk"].Hosting { |
| + t.Error("uk not hosting: it is an OLD WRITER with no opinion, and one box's upgrade must not withdraw another box's region") |
| } |
| - if set["se"] { |
| - t.Error("se armed with no EIP — the reachability half must still apply") |
| + if !caps["ie"].Hosting || !caps["ie"].Staging { |
| + t.Errorf("ie = %+v, want hosting and staging", caps["ie"]) |
| } |
| - |
| - // One box publishes a data bucket. That is what makes every other box's |
| - // silence on the field meaningful, so uk — reachable, but never told a |
| - // hibernate bucket — is now correctly unarmed. |
| - put("control/edges/ie-box.json", `{"v":1,"box":"ie-box","eip":"5.6.7.8","placement":"ie","data_bucket":"carlos-tito-ie-1","hosts":{}}`) |
| - set, published = armedPlacements(ctx, fx.rel) |
| - if !published { |
| - t.Fatal("published = false, want true") |
| + if caps["de"].Hosting { |
| + t.Error("de hosting: it published hibernation_configured false, which is an explicit no") |
| } |
| - if !set["ie"] { |
| - t.Error("ie unarmed: it is reachable and publishes a data bucket") |
| + if !caps["de"].Reachable { |
| + t.Error("de unreachable: it has an EIP — a box that cannot host is still an edge") |
| } |
| - if set["uk"] { |
| - t.Error("uk armed: reachable, but it publishes no data bucket, so its activator is not armed and an instance placed there would serve 502 forever") |
| + if caps["se"].Reachable || caps["se"].Hosting || caps["se"].Staging { |
| + t.Errorf("se = %+v, want nothing: no EIP gates every capability", caps["se"]) |
| } |
| - if set["se"] { |
| - t.Error("se armed: neither reachable nor hosting") |
| + |
| + // Staging is strictly stronger than Hosting, and uk is where they part: |
| + // eligible to host (unknown), but publishing no bucket coordinates, so |
| + // no move can name it — internal/host's moveTargets would drop it and |
| + // the worker would wait forever. |
| + if caps["uk"].Staging { |
| + t.Error("uk staging: it publishes no data_bucket, so a move there could never be prepared") |
| } |
| +} |
| - // A region arms itself the moment its box's host.env grows the variable — |
| - // no console change and no console restart, just the box's next |
| - // edges-file pass republishing the file it always publishes. |
| - _, etag, err := fx.rel.Get(ctx, "control/edges/uk-box.json") |
| - if err != nil { |
| - t.Fatal(err) |
| +// An explicit false is believed even when a SIBLING box in the same region |
| +// can host: the fold is an OR across boxes, so one capable box arms the |
| +// region, and that is correct — the record only needs one box to land on. |
| +func TestPlacementsFoldsAcrossBoxesInARegion(t *testing.T) { |
| + fx, _ := instanceFixture(t, "code-inst-placementfold") |
| + ctx := context.Background() |
| + put := func(key, body string) { |
| + t.Helper() |
| + if _, err := fx.rel.Put(ctx, key, []byte(body), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| } |
| - if _, err := fx.rel.Put(ctx, "control/edges/uk-box.json", |
| - []byte(`{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","data_bucket":"carlos-tito-uk-1","hosts":{}}`), etag); err != nil { |
| + put("control/edges/uk-a.json", `{"v":1,"box":"uk-a","eip":"1.1.1.1","placement":"uk","hibernation_configured":false,"hosts":{}}`) |
| + put("control/edges/uk-b.json", `{"v":1,"box":"uk-b","eip":"2.2.2.2","placement":"uk","data_bucket":"carlos-uk","hibernation_configured":true,"hosts":{}}`) |
| + |
| + caps, _ := placements(ctx, fx.rel) |
| + if !caps["uk"].Hosting || !caps["uk"].Staging { |
| + t.Errorf("uk = %+v, want hosting and staging: one capable box in the region is enough", caps["uk"]) |
| + } |
| +} |
| + |
| +// The dormancy that remains is the one the placement field itself shipped |
| +// with, and it is about placement codes, not about hosting: a fleet where no |
| +// box publishes a code at all leaves the create gate entirely dormant. |
| +func TestPlacementsDormantWithoutAnyPlacementCode(t *testing.T) { |
| + fx, _ := instanceFixture(t, "code-inst-placementdormant") |
| + ctx := context.Background() |
| + if _, err := fx.rel.Put(ctx, "control/edges/solo.json", |
| + []byte(`{"v":1,"box":"solo","eip":"1.2.3.4","hosts":{}}`), ""); err != nil { |
| t.Fatal(err) |
| } |
| - if set, _ = armedPlacements(ctx, fx.rel); !set["uk"] { |
| - t.Error("uk still unarmed after publishing a data bucket") |
| + if _, published := placements(ctx, fx.rel); published { |
| + t.Error("published = true with no placement code anywhere — the gate must stay dormant") |
| } |
| } |
diff --git a/internal/console/placementdoors_test.go b/internal/console/placementdoors_test.go| new file mode 100644 |
| index 000000000..da30abc65 |
| --- /dev/null |
| +++ b/internal/console/placementdoors_test.go |
| @@ -0,0 +1,137 @@ |
| +package console |
| + |
| +import ( |
| + "context" |
| + "net/http" |
| + "strings" |
| + "testing" |
| +) |
| + |
| +// edgesFor writes a fleet's edges files and returns the fixture's put helper |
| +// shape, so each door test states its fleet in one place. |
| +func putEdges(t *testing.T, fx *consoleFixture, files map[string]string) { |
| + t.Helper() |
| + for name, body := range files { |
| + if _, err := fx.rel.Put(context.Background(), "control/edges/"+name+".json", []byte(body), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| +} |
| + |
| +// Only a SOCKET-BACKED INSTANCE needs an armed activator on the target box. |
| +// A static route is served from bucket bytes by every armed edge with no |
| +// process at all, so refusing one for want of a hibernation bucket refuses a |
| +// site that would have served perfectly. The gate ran before the record's |
| +// kind was even read, so it applied to every kind alike. |
| +func TestCreateDoorLetsStaticIntoANonHostingRegion(t *testing.T) { |
| + fx, token := instanceFixture(t, "code-inst-statictonohost") |
| + m := titoManifest() |
| + m.Regions = nil |
| + m.Pool = "" |
| + enableInstances(t, fx, "titogo", m) |
| + // uk is reachable and explicitly cannot host; ie can. |
| + putEdges(t, fx, map[string]string{ |
| + "uk-box": `{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","hibernation_configured":false,"hosts":{}}`, |
| + "ie-box": `{"v":1,"box":"ie-box","eip":"5.6.7.8","placement":"ie","data_bucket":"carlos-ie","hibernation_configured":true,"hosts":{}}`, |
| + }) |
| + u := instancesURL(fx, "bab", "titogo") |
| + |
| + // A static site into uk: allowed. It needs an edge, not an activator. |
| + code, _, out := rawReq(t, http.MethodPost, u, token, |
| + `{"host":"brochure.go.tito.io","region":"uk","kind":"static"}`) |
| + if code != http.StatusCreated { |
| + t.Fatalf("create static into a reachable non-hosting region = %d %s, want 201 — a static route runs no process", code, out) |
| + } |
| + |
| + // An instance into the same region: refused, and the refusal says why |
| + // and lists somewhere that would work. |
| + code, _, out = rawReq(t, http.MethodPost, u, token, |
| + `{"host":"app.go.tito.io","region":"uk"}`) |
| + if code != http.StatusBadRequest { |
| + t.Fatalf("create instance into a non-hosting region = %d %s, want 400", code, out) |
| + } |
| + if !strings.Contains(out, "cannot host") || !strings.Contains(out, "ie") { |
| + t.Errorf("refusal %q should say the region cannot host and name one that can", out) |
| + } |
| +} |
| + |
| +// An addr-backed service proxies to a process the platform never spawns, so |
| +// it needs the region to exist, not to host. |
| +func TestCreateDoorLetsAddrBackedServiceIntoANonHostingRegion(t *testing.T) { |
| + fx, token := instanceFixture(t, "code-inst-svctonohost") |
| + m := titoManifest() |
| + m.Regions = nil |
| + m.Pool = "" |
| + enableInstances(t, fx, "titogo", m) |
| + putEdges(t, fx, map[string]string{ |
| + "uk-box": `{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","hibernation_configured":false,"hosts":{}}`, |
| + }) |
| + code, _, out := rawReq(t, http.MethodPost, instancesURL(fx, "bab", "titogo"), token, |
| + `{"host":"svc.go.tito.io","region":"uk","kind":"service","addrs":["10.0.0.4:8080"]}`) |
| + if code != http.StatusCreated { |
| + t.Fatalf("create addr-backed service into a non-hosting region = %d %s, want 201", code, out) |
| + } |
| +} |
| + |
| +// An UNREACHABLE region is refused for every kind — a box nothing can dial |
| +// homes nothing, static included. |
| +func TestCreateDoorRefusesEveryKindIntoAnUnreachableRegion(t *testing.T) { |
| + fx, token := instanceFixture(t, "code-inst-unreachable") |
| + m := titoManifest() |
| + m.Regions = nil |
| + m.Pool = "" |
| + enableInstances(t, fx, "titogo", m) |
| + putEdges(t, fx, map[string]string{ |
| + "se-box": `{"v":1,"box":"se-box","eip":"","placement":"se","hosts":{}}`, |
| + "ie-box": `{"v":1,"box":"ie-box","eip":"5.6.7.8","placement":"ie","data_bucket":"carlos-ie","hibernation_configured":true,"hosts":{}}`, |
| + }) |
| + code, _, out := rawReq(t, http.MethodPost, instancesURL(fx, "bab", "titogo"), token, |
| + `{"host":"brochure.go.tito.io","region":"se","kind":"static"}`) |
| + if code != http.StatusBadRequest || !strings.Contains(out, "not armed") { |
| + t.Fatalf("create static into an unreachable region = %d %s, want 400 not-armed", code, out) |
| + } |
| +} |
| + |
| +// The move door's bar is the target's published STAGING bucket, and it is |
| +// unconditional. internal/host's moveTargets drops any target whose edges |
| +// file carries no data_bucket, and runMoveRound then answers "publishes no |
| +// data bucket yet — waiting" forever, so a door that admitted the move would |
| +// wedge a LIVE instance's residence on a lane that does not exist. |
| +func TestMoveDoorRequiresAStagingBucket(t *testing.T) { |
| + fx, token := instanceFixture(t, "code-inst-movestaging") |
| + m := titoManifest() |
| + m.Regions = nil |
| + m.Pool = "" |
| + enableInstances(t, fx, "titogo", m) |
| + // uk is an old writer: reachable, eligible to HOST (unknown is not no), |
| + // but it publishes no bucket coordinates, so it can take no move. |
| + putEdges(t, fx, map[string]string{ |
| + "uk-box": `{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","hosts":{}}`, |
| + "ie-box": `{"v":1,"box":"ie-box","eip":"5.6.7.8","placement":"ie","data_bucket":"carlos-ie","hibernation_configured":true,"hosts":{}}`, |
| + }) |
| + u := instancesURL(fx, "bab", "titogo") |
| + |
| + // uk IS a legal create target — unknown hosting stays eligible. This |
| + // instance is created in ie, though, so the move below is a genuine |
| + // cross-region request and cannot be refused by the same-region conflict |
| + // instead of by the bar under test. |
| + if code, _, out := rawReq(t, http.MethodPost, u, token, |
| + `{"host":"mover.go.tito.io","region":"ie"}`); code != http.StatusCreated { |
| + t.Fatalf("create into ie = %d %s, want 201", code, out) |
| + } |
| + if code, _, out := rawReq(t, http.MethodPost, u, token, |
| + `{"host":"stayer.go.tito.io","region":"uk"}`); code != http.StatusCreated { |
| + t.Fatalf("create into an unknown-hosting region = %d %s, want 201 — unknown is not no", code, out) |
| + } |
| + // uk is NOT a legal move target: it publishes no staging bucket. |
| + code, _, out := rawReq(t, http.MethodPost, u+"/mover.go.tito.io/move", token, `{"region":"uk"}`) |
| + if code != http.StatusBadRequest { |
| + t.Fatalf("move to a region with no staging bucket = %d %s, want 400", code, out) |
| + } |
| + if !strings.Contains(out, "staging bucket") { |
| + t.Errorf("refusal %q should name the missing staging bucket — the worker's own reason", out) |
| + } |
| + if !strings.Contains(out, "ie") { |
| + t.Errorf("refusal %q should list the regions that can take a move", out) |
| + } |
| +} |
diff --git a/internal/edge/edge.go b/internal/edge/edge.go| index a58f17548..90fbc697b 100644 |
| --- a/internal/edge/edge.go |
| +++ b/internal/edge/edge.go |
| @@ -592,32 +592,38 @@ func (e *Edge) Reload() error { |
| if hib { |
| managed[t.Host] = true |
| } |
| - // A hibernate-flagged instance route on a box with NO activator is |
| - // unservable, and silently so — which is what discussion #110 cost. |
| - // The row is real, the binary is staged, the socket path is derived, |
| - // and nothing will ever create that socket, because only an armed |
| - // activator spawns the child that binds it. Every request falls into |
| - // unixProxy's dial and comes back as the stdlib ReverseProxy's bare |
| - // 502 — a status that reads as "your backend is broken" for a |
| - // backend that was never going to exist. On tito this ran for |
| - // fifteen minutes across five regions before anyone opened an SSM |
| - // session, and the box knew the answer the whole time. |
| + // A hibernate-flagged instance route on a box with no activator has |
| + // nothing on this box that would ever spawn its process: the row is |
| + // real and the socket path is derived, but only an armed activator |
| + // starts the child that binds it, so a request that reaches this |
| + // route here falls into unixProxy's dial and comes back as the |
| + // stdlib ReverseProxy's bare, bodiless 502 — a status that reads as |
| + // "your backend is broken" about a backend nothing here will start. |
| // |
| - // The fix for the CONSUMER half is the console's (armedPlacements |
| - // now requires a published data bucket, so such a region is not |
| - // offered and create refuses it). This is the box's half: say it |
| - // out loud, name the variable, and let anyone reading the journal |
| - // stop looking at the app. |
| + // Diagnostic only, and deliberately careful about what it claims. It |
| + // says what this box knows — an eligible route, no activator — and |
| + // not that every request 502s, because it cannot see whether |
| + // something else is answering: a process started outside the |
| + // platform could be bound to that socket already. The one deployment |
| + // shape it was written for is a box whose /etc/carlos/host.env was |
| + // hand-assembled and never given CARLOS_HIBERNATE_BUCKET, which the |
| + // region-host bring-up docs make easy to do. |
| // |
| - // Throttled, because it is a standing condition — true on every |
| - // reload for as long as the row and the missing env both exist — and |
| - // keyed per host, a value the box itself controls (logThrottled's |
| - // rule). Not an error and not a refusal to serve: `carlos edge` run |
| - // without the agent has no activator by design, and taking the route |
| - // out of the table would only trade a 502 for a 404. |
| + // The advice names both roles rather than guessing which one is |
| + // running, because the fix differs and only one of them has one: |
| + // `carlos edge` standalone passes a nil activator regardless of the |
| + // environment (RunWithActivator), so an unconditional "set this |
| + // variable" would send that operator after something that cannot |
| + // help them. Saying both costs one clause and is true either way. |
| + // |
| + // Throttled: a standing condition, true on every reload for as long |
| + // as the row and the missing activator both exist. Keyed per host — |
| + // a value the box itself controls, per logThrottled's rule. Not a |
| + // refusal to serve: dropping the route would only trade a 502 for a |
| + // 404, and the row may yet be served once an activator arms. |
| if e.act == nil && t.Kind == "instance" && t.Hibernate && !viaTunnel { |
| e.logThrottled("no-activator "+t.Host, fmt.Sprintf( |
| - "route %s is a hibernating instance but this box has no activator — nothing will ever create %s, so every request 502s. Set CARLOS_HIBERNATE_BUCKET in /etc/carlos/host.env and restart carlos-edge.", |
| + "route %s is a hibernating instance but this box has no activator, so nothing here will create %s or start its process. If this box runs `carlos agent`, set CARLOS_HIBERNATE_BUCKET in /etc/carlos/host.env and restart carlos-edge; `carlos edge` on its own never runs instances.", |
| t.Host, t.Socket)) |
| } |
| m[t.Host] = upstream{proxy: proxy, route: t, hibernating: hib} |
diff --git a/internal/edge/noactivator_test.go b/internal/edge/noactivator_test.go| index aba0af722..9300cd573 100644 |
| --- a/internal/edge/noactivator_test.go |
| +++ b/internal/edge/noactivator_test.go |
| @@ -7,14 +7,16 @@ import ( |
| "github.com/carlosframework/platform/internal/registry" |
| ) |
| -// A hibernate-flagged instance route on a box with no activator is |
| -// unservable, and before discussion #110 it was unservable in silence: the |
| -// row is real, the socket path is derived, and nothing will ever create that |
| -// socket, so every request comes back as unixProxy's bare stdlib 502. On tito |
| -// that ran for fifteen minutes across five regions before anyone opened an |
| -// SSM session, and the box knew the answer the whole time. Reload must name |
| -// the condition, the host, the socket nobody will bind, and the variable that |
| -// fixes it. |
| +// A hibernate-flagged instance route on a box with no activator has nothing |
| +// on that box that would ever spawn its process, and before this it said so |
| +// nowhere: the row is real and the socket path is derived, so a request |
| +// reaching it here dials a socket nothing will bind and comes back as a |
| +// bodiless 502 that reads as a broken backend. Reload must name the host, |
| +// the socket, and the fix. |
| +// |
| +// The message is deliberately careful about what it claims — it reports what |
| +// this box knows, not that every request 502s, since it cannot see whether |
| +// something outside the platform is already bound to that socket. |
| func TestReloadNamesAHibernatingRouteWithNoActivator(t *testing.T) { |
| buf := captureLog(t) |
| @@ -39,7 +41,7 @@ func TestReloadNamesAHibernatingRouteWithNoActivator(t *testing.T) { |
| "region-probe.example.com", |
| "/run/carlos/region-probe.example.com.sock", |
| "CARLOS_HIBERNATE_BUCKET", |
| - "502", |
| + "carlos edge", |
| } { |
| if !strings.Contains(got, want) { |
| t.Errorf("reload log does not mention %q — an operator reading the journal still cannot tell this from a broken app.\ngot: %s", want, got) |
diff --git a/internal/host/edgesfile.go b/internal/host/edgesfile.go| index e6b89b944..dd5e55791 100644 |
| --- a/internal/host/edgesfile.go |
| +++ b/internal/host/edgesfile.go |
| @@ -94,10 +94,40 @@ type edgesFile struct { |
| // from the same file set everything else reads (instance-move spec). |
| // Additive + omitempty like Placement; a box with no hibernate bucket |
| // (activator disabled) publishes neither key. |
| - DataBucket string `json:"data_bucket,omitempty"` |
| - DataBucketRegion string `json:"data_bucket_region,omitempty"` |
| - Generation string `json:"generation"` |
| - Hosts map[string]edgeHost `json:"hosts"` |
| + DataBucket string `json:"data_bucket,omitempty"` |
| + DataBucketRegion string `json:"data_bucket_region,omitempty"` |
| + // HibernationConfigured is this box's own answer to "is my hibernation |
| + // activator wired" — true exactly when CARLOS_HIBERNATE_BUCKET is set, |
| + // which is the same env, read in the same process, that decides whether |
| + // activator.ConfigFromEnv builds a Config at all (internal/host/agent.go |
| + // arms the activator iff that Config is non-nil, and log.Fatalfs rather |
| + // than continuing if the env is set but malformed — so a live agent |
| + // process cannot publish this field disagreeing with its own activator). |
| + // |
| + // Deliberately NOT omitempty, and this is the whole point of the field. |
| + // A reader needs to tell three states apart, and only an explicitly |
| + // published `false` can carry the third: |
| + // |
| + // absent → the writer predates this field; its capability is UNKNOWN |
| + // true → the activator is configured |
| + // false → the activator is not configured |
| + // |
| + // Before it, the only signal was data_bucket, and its absence was |
| + // ambiguous in exactly the way that matters: an old writer and a |
| + // bucket-less new box produce byte-identical files. A console reading |
| + // that had to guess, and the guess it made was to infer one box's |
| + // silence from ANOTHER box's speech — so the first box in a fleet to |
| + // publish a bucket withdrew every region that had not yet rolled, all |
| + // of them healthy. One writer's upgrade cannot establish another |
| + // writer's capabilities; this field is how each box answers for itself. |
| + // |
| + // It reports CONFIGURATION, not verified hosting readiness: a box whose |
| + // IAM role cannot reach the bucket it names, or whose |
| + // CARLOS_HIBERNATE_REGION is wrong, publishes true here and still fails |
| + // every wake. Naming it for what it actually knows is the point. |
| + HibernationConfigured bool `json:"hibernation_configured"` |
| + Generation string `json:"generation"` |
| + Hosts map[string]edgeHost `json:"hosts"` |
| } |
| // edgesFilePass builds this box's edges file from the routes currently in |
| @@ -407,6 +437,11 @@ func edgesFileLoop(reg *registry.Registry) { |
| Placement: strings.TrimSpace(os.Getenv("CARLOS_REGION")), |
| DataBucket: strings.TrimSpace(os.Getenv("CARLOS_HIBERNATE_BUCKET")), |
| DataBucketRegion: strings.TrimSpace(os.Getenv("CARLOS_HIBERNATE_REGION")), |
| + // Same env, same trim, one line apart, deliberately: this boolean |
| + // must never disagree with the bucket beside it, and the way to |
| + // guarantee that is to derive both from the one read rather than to |
| + // ask two questions that could drift. |
| + HibernationConfigured: strings.TrimSpace(os.Getenv("CARLOS_HIBERNATE_BUCKET")) != "", |
| } |
| log.Printf("agent: edges file loop armed (5min + jitter, %s)", edgesFileKey(box)) |
| last := seedEdgesFileGate(ctx, store, box) |
diff --git a/internal/host/edgesfile_test.go b/internal/host/edgesfile_test.go| index d43ea1c99..58bd8d686 100644 |
| --- a/internal/host/edgesfile_test.go |
| +++ b/internal/host/edgesfile_test.go |
| @@ -268,7 +268,7 @@ func TestEdgesFileRawWireBodyMatchesSchema(t *testing.T) { |
| t.Fatalf("generation %q is not RFC3339: %v", probe.Generation, err) |
| } |
| - want := `{"v":1,"box":"quiet-harbour","eip":"203.0.113.10","generation":"` + |
| + want := `{"v":1,"box":"quiet-harbour","eip":"203.0.113.10","hibernation_configured":false,"generation":"` + |
| probe.Generation + `","hosts":{"app.example.com":{"role":"home","platform":false}}}` |
| if string(got) != want { |
| t.Fatalf("raw wire body =\n%s\nwant\n%s", got, want) |
| @@ -310,7 +310,7 @@ func TestEdgesFileRawWireBodyRegionField(t *testing.T) { |
| t.Fatal(err) |
| } |
| gen := genOf(t, got) |
| - want := `{"v":1,"box":"quiet-harbour","eip":"203.0.113.10","generation":"` + |
| + want := `{"v":1,"box":"quiet-harbour","eip":"203.0.113.10","hibernation_configured":false,"generation":"` + |
| gen + `","hosts":{"app.example.com":{"role":"home","platform":false}}}` |
| if string(got) != want { |
| t.Fatalf("raw wire body (region unset) =\n%s\nwant\n%s", got, want) |
| @@ -329,7 +329,7 @@ func TestEdgesFileRawWireBodyRegionField(t *testing.T) { |
| t.Fatal(err) |
| } |
| gen := genOf(t, got) |
| - want := `{"v":1,"box":"quiet-harbour","eip":"203.0.113.10","region":"eu-west-1","generation":"` + |
| + want := `{"v":1,"box":"quiet-harbour","eip":"203.0.113.10","region":"eu-west-1","hibernation_configured":false,"generation":"` + |
| gen + `","hosts":{"app.example.com":{"role":"home","platform":false}}}` |
| if string(got) != want { |
| t.Fatalf("raw wire body (region set) =\n%s\nwant\n%s", got, want) |
| @@ -882,7 +882,7 @@ func TestEdgesFileRawWireBodyPlacementField(t *testing.T) { |
| if err := json.Unmarshal(got, &probe); err != nil { |
| t.Fatal(err) |
| } |
| - want := `{"v":1,"box":"quiet-harbour","eip":"203.0.113.10","region":"eu-west-2","placement":"uk","generation":"` + |
| + want := `{"v":1,"box":"quiet-harbour","eip":"203.0.113.10","region":"eu-west-2","placement":"uk","hibernation_configured":false,"generation":"` + |
| probe.Generation + `","hosts":{"app.example.com":{"role":"home","platform":false}}}` |
| if string(got) != want { |
| t.Fatalf("raw wire body (placement set) =\n%s\nwant\n%s", got, want) |