amadan

carlos / platform Public

Merge pull request #348 from carlosframework/region-at-the-door

region at the door: armed-placement validation on instance create + CARLOS_REGION to exec children (#329)
Paul Campbell pushed by paul@keymail.dev d56ac0035ac864554f6b5205230a1ceb3ebe3510 parent 9a056e5parent 67cae8a
9 files changed, +457 −46
  • docs/superpowers/specs/2026-08-28-instance-move-design.md +187 −0
  • internal/activator/backend_exec.go +13 −0
  • internal/activator/backend_exec_test.go +47 −0
  • internal/console/apiinstances.go +20 −0
  • internal/console/apiinstances_test.go +49 −0
  • internal/console/instancesplacement.go +47 −12
  • internal/host/agent.go +6 −3
  • internal/host/edgesfile.go +20 −7
  • internal/host/edgesfile_test.go +68 −24
diff --git a/docs/superpowers/specs/2026-08-28-instance-move-design.md b/docs/superpowers/specs/2026-08-28-instance-move-design.md
new file mode 100644
index 000000000..f283f8a97
--- /dev/null
+++ b/docs/superpowers/specs/2026-08-28-instance-move-design.md
@@ -0,0 +1,187 @@
+# Instance moves: two-phase, data-following (issue #331)
+
+Status: DRAFT for review — 2026-08-28
+Companion issues: #329 (region on create — shipped separately), #330
+(replica_regions — shape only, untouched here), titogo residence design
+§5/§8.3 (the consumer contract this serves).
+
+## Decisions taken in conversation (Paul, 2026-08-28)
+
+- Cross-bucket IAM is **push-only into a staging prefix** (`moves/`),
+ never into `instances/` — a box can stage state toward a sibling
+ region but can never touch another region's live replica lineage.
+- The platform moves **the instance DB and record only**. The uploads
+ bulk-copy stays with the tenant (titogo's router), which owns the
+ bucket, the credentials, and the knowledge of which keys belong to
+ which instance. The move state machine exposes the phases so the
+ tenant can sequence its own copy inside the frozen window. If a
+ per-instance uploads prefix ever becomes a declared platform concept
+ (a record field), the move widens to carry it without unwinding this
+ design.
+
+## The invariant
+
+*Residence is a property of a route; the record's `region` field is the
+residence carrier* (ingress spec §Anticipated #2). This spec never adds
+a second arbiter: at every instant exactly one region's box is in scope
+for the record, and the flip of `region` is the LAST act of the source.
+Never two writers falls out of that plus the freeze the tenant holds
+across the cut.
+
+## Wire shape
+
+The record gains one additive, omitempty object — present only while a
+move is in flight:
+
+```json
+"move": {
+ "to": "uk",
+ "state": "preparing" | "ready" | "cutting",
+ "requested_at": "<RFC3339>",
+ "ready_at": "<RFC3339, when state >= ready>"
+}
+```
+
+Records written before this field parse unchanged (wire-shape rule).
+`move` is written by the console (the two verbs below) and by the
+SOURCE box (the `preparing → ready` promotion and the terminal
+clear-and-flip); each write is the narrow CAS `SetRecordRegion` already
+uses. The record's `region` stays the SOURCE region until the cut
+completes — every reader that knows nothing about moves keeps a
+consistent answer throughout.
+
+## The two verbs
+
+Console API, on one instance, gated exactly like every other instance
+door (app bearer; the tenant's router holds the token — never the
+instance itself):
+
+- `POST .../instances/{host}/move` body `{"region":"uk"}` — validated
+ against the armed-placement set (#329's gate, mandatory here: a move
+ to an unarmed region is always a 400, dormant never applies because a
+ move NEEDS a live target box). Refused 409 if a move is already in
+ flight, 400 if `to` equals the current region. Writes
+ `move:{to,state:"preparing"}`.
+- `POST .../instances/{host}/move/cut` — refused 409 unless
+ `move.state == "ready"`. Writes `move.state:"cutting"`. The tenant
+ calls this only after freezing writes; the platform does not verify
+ the freeze (it cannot — the freeze is app-level), it documents it.
+- `DELETE .../instances/{host}/move` — abort, legal in `preparing` or
+ `ready` (clears `move`; the source drops its second replica; staged
+ bytes in the target bucket are garbage that the next prepare
+ overwrites and the target's post-restore cleanup deletes). Refused
+ 409 in `cutting`: past that point the move completes forward.
+- State is read, not pushed: the existing single-record GET already
+ returns the record verbatim, so `move.state` is pollable today with
+ zero new read surface.
+
+## What the source box does
+
+The reconcile/activator pass on the box whose `CARLOS_REGION` matches
+`region` (the source — the record still points at it):
+
+1. **preparing**: alongside its normal own-bucket replication, the
+ activator opens a SECOND litestream replica of the instance DB to
+ `s3://<target-data-bucket>/moves/<host>/` — the lease package's
+ `ReplicateWithLease` pointed at the sibling bucket's staging prefix.
+ Target bucket name and AWS region are derived, not configured
+ per-move: each box publishes its data bucket coordinates as two new
+ additive fields in its edges file (`data_bucket`, `data_bucket_region`
+ — same additive rule as `placement`), so the source reads the
+ target's coordinates from the same file set everything else reads.
+ A wake is forced if the instance is asleep? NO — a parked instance's
+ state is already fully in the source bucket; for a parked instance
+ the "second replica" is a bucket-to-staging copy performed by the
+ source box (its IAM can read its own bucket and push to the sibling's
+ staging), and prepare completes without ever waking the child.
+2. **ready promotion**: once the staging replica reports caught up
+ (litestream generation synced; for the parked case, copy complete),
+ the source CASes `move.state: "preparing" → "ready"`, stamping
+ `ready_at`. The second replica keeps streaming — ready is a level,
+ not an edge; writes after ready keep flowing to staging.
+3. **cutting**: on seeing `state:"cutting"`, the source (in order):
+ final `SyncAndWait` on the staging replica → stop the child (normal
+ stopping path, which also checkpoints to the source's OWN bucket —
+ the abort-recovery copy) → close both replicators → **delete the
+ local DB, WAL/SHM, and litestream meta dirs** (the warm-cache purge;
+ without it a later move back would resurrect a stale local file — the
+ activator's warm-cache trust is exactly right for crashes and exactly
+ wrong here) → one CAS that sets `region: <to>` and clears `move` →
+ remove its own route row on its next pass (the record has left its
+ scope, the existing retirement path does this unprompted).
+ If the box dies mid-cut, the record still says `cutting` with the old
+ region: the source resumes the sequence on restart (every step is
+ idempotent — sync again, stop a stopped child, delete absent files,
+ CAS already-flipped is a no-op conflict it re-reads).
+
+## What the target box does
+
+Nothing move-specific until the flip — then its ordinary pass sees a
+record newly in scope and mints the route. One addition in the
+activator's `Restore`: when the local DB is absent AND
+`instances/<host>/` in its own bucket has no replica, it checks
+`moves/<host>/` (own bucket, own IAM); if present, server-side copy
+`moves/<host>/ → instances/<host>/`, restore from there, `CheckDB` as
+always, and delete `moves/<host>/` on success. A corrupt staged replica
+quarantines exactly like a corrupt own one — and the abort story still
+stands, because the source's last own-bucket checkpoint is intact and
+`region` can be pointed back by hand (`SetRecordRegion` — the operator
+recovery door that already exists).
+
+## IAM (the infra half — platform-infrastructure + tito repo applies)
+
+`region-host` (and `region-remote-host`) gain an optional
+`sibling_staging_arns` list; when set, the role gains `s3:PutObject` on
+`<each sibling data bucket>/moves/*` — and nothing else: no Get, no
+List, no Delete, never `instances/*`. Residency-at-rest holds
+(cross-region reads remain impossible at IAM); the write blast radius
+is confined to staging keys no box ever trusts as live state without
+its own restore-time `CheckDB`. Litestream against a push-only prefix:
+the lease/generation dance needs List on its own prefix — checked
+during build; if litestream cannot run push-only, prepare falls back to
+periodic own-checkpoint + copy-into-staging (same convergence, chunkier
+cadence), and the spec's contract (`ready` = staged copy caught up)
+does not change.
+
+## Instance env
+
+Already shipped with #329: every exec child gets `CARLOS_REGION` (the
+box's placement code), agent-owned in both directions. An app that
+lands on the target box after a move reads the new value at boot —
+titogo's boot-clears-freeze rule needs exactly this.
+
+## Out of scope
+
+- Uploads (tenant-side, by decision above).
+- `replica_regions` behavior (#330 stays shape-only).
+- Moving unit-backed, addr-backed, static, or everywhere-class routes:
+ the verbs refuse anything but an exec-backed `kind:"instance"` route
+ (400 naming the kind) — the only class whose state the platform owns.
+- Fleet placements (`fleet/<name>`): refused; fleets keep D6
+ state-does-not-follow semantics on purpose.
+- Automatic freeze verification; region-aware unit economics; the
+ member-facing residence door (titogo builds on these verbs).
+
+## Testing
+
+- Record CAS: move-field transitions (prepare/ready/cut/abort/conflict)
+ against the fake store, including the 409 matrix.
+- Source pass: fake-store + fake-litestream harness proving the cut
+ ordering (final sync BEFORE stop BEFORE local purge BEFORE flip), the
+ parked-instance prepare (no wake), resume-from-`cutting` idempotence,
+ and that abort leaves the source serving with its own replication
+ intact.
+- Target restore: staging fallback (copy, restore, CheckDB, cleanup),
+ corrupt-staging quarantine, and precedence (own `instances/` replica
+ beats staging when both exist — a re-landed instance must not regress
+ to an older staged copy).
+- Console: verb gates (bearer, kind, armed region, state machine),
+ poll-through-GET.
+- The never-two-writers argument pinned by an ordering test: a request
+ hitting the source during `cutting` after stop is a 503 from the edge
+ (route gone on next pass) while the target has not yet minted. The
+ length of that window is the target's reconcile latency, NOT assumed:
+ measure it during build (the adopt path/timer cadence is the floor)
+ and record it in the runbook — the tenant's freeze must be budgeted
+ to cover it, and if it measures in minutes rather than seconds, a
+ serial-bump nudge on the record write is the follow-up, not a redesign.
diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go
index ddd0893b5..a8fac9f93 100644
--- a/internal/activator/backend_exec.go
+++ b/internal/activator/backend_exec.go
@@ -60,6 +60,14 @@ type ExecBackend struct {
// but a carlos app is a separate binary from the release store — there is
// nothing sensible to invent here, and the adopt pass owns the symlinks.
Materialize func(ctx context.Context, host string) error
+ // Region is the box's CARLOS placement code (CARLOS_REGION), handed to
+ // every exec child as its own CARLOS_REGION (issue #329, titogo's
+ // residence design: the app reads where it lives — e.g. to clear a move
+ // freeze after landing). Agent-owned in BOTH directions, exactly like
+ // the control-channel vars below: set from what the agent knows, and
+ // STRIPPED when the agent knows nothing — a tenant's config env must not
+ // be able to tell an app it runs somewhere it does not.
+ Region string
// Cache, when non-nil, adds the purge verb to each spawned instance's
// control socket so it can invalidate its own cached responses right
// after a write. Nil disables that ONE verb, not the socket: the control
@@ -626,6 +634,11 @@ func (b *ExecBackend) Start(ctx context.Context, rt registry.Route) (Proc, strin
} else {
childVars = withoutVars(childVars, "CARLOS_PURGE")
}
+ if b.Region != "" {
+ childVars = withVar(childVars, "CARLOS_REGION", b.Region)
+ } else {
+ childVars = withoutVars(childVars, "CARLOS_REGION")
+ }
// Record the build we're about to exec — read the symlink immediately before
// the start so it names the version the kernel will resolve. Reading it here,
// rather than back in wake() after the health wait, closes a real window: a
diff --git a/internal/activator/backend_exec_test.go b/internal/activator/backend_exec_test.go
index 6bc46a846..7e3fb3a8a 100644
--- a/internal/activator/backend_exec_test.go
+++ b/internal/activator/backend_exec_test.go
@@ -1,10 +1,13 @@
package activator
import (
+ "context"
"os"
"os/exec"
"path/filepath"
+ "strings"
"testing"
+ "time"
"github.com/carlosframework/platform/internal/cgroup"
"github.com/carlosframework/platform/internal/registry"
@@ -56,3 +59,47 @@ func TestSpawnInCgroupAppliesTheTier(t *testing.T) {
t.Errorf("memory.max = %q, want the children tier", got)
}
}
+
+// The box's placement code reaches every exec child as CARLOS_REGION (issue
+// #329) — agent-owned in both directions: set from ExecBackend.Region, and
+// STRIPPED when the backend has none, even when a tenant's own env file
+// tries to supply it. Same rule and same reason as CARLOS_CONTROL_SOCKET /
+// CARLOS_PURGE: an app must not be told it runs somewhere the agent never
+// vouched for.
+func TestExecBackendStartChildRegion(t *testing.T) {
+ for _, tc := range []struct {
+ name, backendRegion, want string
+ wantSet bool
+ }{
+ {"set from the backend", "uk", "uk", true},
+ {"stripped when the backend has none", "", "", false},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ b := testExecBackend(t)
+ b.Region = tc.backendRegion
+ host := "region-env.example.com"
+ _, envPath := stageRecordingLiveFullEnv(t, b, host, "v1")
+ // A tenant-supplied value that must never survive as-is.
+ if err := os.WriteFile(filepath.Join(b.DataDir, host+".config.env"), []byte("CARLOS_REGION=forged\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ rt := registry.Route{Host: host, Kind: "instance", Socket: filepath.Join(t.TempDir(), "instance.sock")}
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+ defer cancel()
+ proc, _, _ := b.Start(ctx, rt) // fake binary never binds the socket; health-wait error expected
+ if proc != nil {
+ defer proc.Stop(context.Background())
+ }
+ child := map[string]string{}
+ for _, line := range strings.Split(waitForFile(t, envPath, time.Second), "\n") {
+ if k, v, ok := strings.Cut(line, "="); ok {
+ child[k] = v
+ }
+ }
+ got, ok := child["CARLOS_REGION"]
+ if ok != tc.wantSet || got != tc.want {
+ t.Fatalf("child CARLOS_REGION = %q (set=%v), want %q (set=%v)", got, ok, tc.want, tc.wantSet)
+ }
+ })
+ }
+}
diff --git a/internal/console/apiinstances.go b/internal/console/apiinstances.go
index 64650e82b..954ca0e79 100644
--- a/internal/console/apiinstances.go
+++ b/internal/console/apiinstances.go
@@ -604,6 +604,26 @@ func (s *Server) handleAPIInstanceCreate(w http.ResponseWriter, r *http.Request)
jsonError(w, http.StatusBadRequest, msg)
return
}
+ // The armed-placement gate (issue #329): a non-empty region that names
+ // no armed box is an instance NOTHING will ever mint — the record would
+ // sit dark forever, which is worse than a refusal that lists the real
+ // choices. Placed apps skipped (their value is a fleet, validated by its
+ // own rail above); the unconstrained "" region skipped (it means "the
+ // deployment's home", which always exists); and the whole gate is
+ // dormant until some box publishes a placement code, so a console rolled
+ // ahead of its fleet changes nothing.
+ if placed == "" && region != "" {
+ if set, armed := armedPlacements(ctx, s.cfg.Releases); armed && !set[region] {
+ codes := make([]string, 0, len(set))
+ for c := range set {
+ codes = append(codes, c)
+ }
+ slices.Sort(codes)
+ jsonError(w, http.StatusBadRequest,
+ "region "+region+" is not armed in this deployment — armed: "+strings.Join(codes, ", "))
+ return
+ }
+ }
channel := strings.TrimSpace(req.Channel)
if channel == "" {
// The manifest's channel, then the documented default (§3.1). An
diff --git a/internal/console/apiinstances_test.go b/internal/console/apiinstances_test.go
index 8870c16df..b9a5bd7ff 100644
--- a/internal/console/apiinstances_test.go
+++ b/internal/console/apiinstances_test.go
@@ -2135,3 +2135,52 @@ func TestAPIInstanceListServedBy(t *testing.T) {
t.Errorf("get beta = %d %s, want 200 with no served_by key", code, out)
}
}
+
+// TestAPIInstanceCreateArmedPlacementGate (issue #329): once any box
+// publishes a placement code in its edges file, a create naming a region no
+// armed box serves is refused with the real choices — a record nothing will
+// ever mint is a dark instance, worse than a 400. Dormant before that
+// (legacy fleet: behaviour unchanged); a placement without an EIP is not a
+// landing target; the unconstrained "" region always passes.
+func TestAPIInstanceCreateArmedPlacementGate(t *testing.T) {
+ fx, token := instanceFixture(t, "code-inst-armed")
+ m := titoManifest()
+ m.Regions = nil // the manifest allow-list is a separate rail; keep it out
+ enableInstances(t, fx, "titogo", m)
+ u := instancesURL(fx, "bab", "titogo")
+ put := func(key, body string) {
+ t.Helper()
+ if _, err := fx.rel.Put(context.Background(), key, []byte(body), ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+
+ // DORMANT: no edges file carries a placement — any shape-valid region
+ // is accepted, exactly as before the gate existed.
+ put("control/edges/legacy.json", `{"v":1,"box":"legacy","eip":"9.9.9.9","hosts":{}}`)
+ if code, _, out := rawReq(t, http.MethodPost, u, token, `{"host":"acme.go.tito.io","region":"zz-typo"}`); code != http.StatusCreated {
+ t.Fatalf("dormant create = %d %s, want 201", code, out)
+ }
+
+ // ARMED: uk has an EIP, se does not, legacy still publishes none.
+ put("control/edges/uk-box.json", `{"v":1,"box":"uk-box","eip":"1.2.3.4","region":"eu-west-2","placement":"uk","hosts":{}}`)
+ put("control/edges/se-box.json", `{"v":1,"box":"se-box","eip":"","region":"eu-north-1","placement":"se","hosts":{}}`)
+
+ if code, _, out := rawReq(t, http.MethodPost, u, token, `{"host":"beta.go.tito.io","region":"uk"}`); code != http.StatusCreated {
+ t.Fatalf("create region uk (armed) = %d %s, want 201", code, out)
+ }
+ if code, _, out := rawReq(t, http.MethodPost, u, token, `{"host":"gamma.go.tito.io"}`); code != http.StatusCreated {
+ t.Fatalf("create with unconstrained region = %d %s, want 201", code, out)
+ }
+ code, _, out := rawReq(t, http.MethodPost, u, token, `{"host":"delta.go.tito.io","region":"zz-typo"}`)
+ if code != http.StatusBadRequest {
+ t.Fatalf("create region zz-typo (armed) = %d %s, want 400", code, out)
+ }
+ if !strings.Contains(out, "armed: uk") {
+ t.Fatalf("refusal %q does not name the armed set", out)
+ }
+ // se published a placement but no EIP: not a landing target.
+ if code, _, out := rawReq(t, http.MethodPost, u, token, `{"host":"eps.go.tito.io","region":"se"}`); code != http.StatusBadRequest {
+ t.Fatalf("create region se (no EIP) = %d %s, want 400", code, out)
+ }
+}
diff --git a/internal/console/instancesplacement.go b/internal/console/instancesplacement.go
index 9ae896c8e..649325f8d 100644
--- a/internal/console/instancesplacement.go
+++ b/internal/console/instancesplacement.go
@@ -33,10 +33,12 @@ type servedBy struct {
// capacity, and "where is this instance" is the question, not "who is its
// write home" (that is the home map's question, internal/edge).
type edgesFilePlacementView struct {
- V int `json:"v"`
- Box string `json:"box"`
- Region string `json:"region"`
- Hosts map[string]json.RawMessage `json:"hosts"`
+ V int `json:"v"`
+ Box string `json:"box"`
+ EIP string `json:"eip"`
+ Region string `json:"region"`
+ Placement string `json:"placement"`
+ Hosts map[string]json.RawMessage `json:"hosts"`
}
// loadEdgesJoin folds every readable edges file into host → the boxes that
@@ -49,14 +51,31 @@ type edgesFilePlacementView struct {
// all, the same shape filter the other readers apply.
func loadEdgesJoin(ctx context.Context, s release.Store) map[string][]servedBy {
out := map[string][]servedBy{}
+ for _, f := range readEdgesFiles(ctx, s) {
+ for host := range f.Hosts {
+ out[host] = append(out[host], servedBy{Box: f.Box, Region: f.Region})
+ }
+ }
+ for host := range out {
+ sort.Slice(out[host], func(i, j int) bool { return out[host][i].Box < out[host][j].Box })
+ }
+ return out
+}
+
+// readEdgesFiles is the shared best-effort read under both consumers:
+// list, shape-filter, parse, skip anything unreadable or unversioned. A
+// failed list is an empty answer, logged — the callers each degrade to
+// their pre-#337/#329 behaviour rather than failing the request.
+func readEdgesFiles(ctx context.Context, s release.Store) []edgesFilePlacementView {
if s == nil {
- return out
+ return nil
}
keys, err := s.List(ctx, edgesFilePrefix)
if err != nil {
log.Printf("console: api instances: could not list %s for the placement join: %v", edgesFilePrefix, err)
- return out
+ return nil
}
+ var out []edgesFilePlacementView
for _, key := range keys {
name, ok := strings.CutPrefix(key, edgesFilePrefix)
if !ok || !strings.HasSuffix(name, ".json") || strings.Contains(name, "/") {
@@ -72,12 +91,28 @@ func loadEdgesJoin(ctx context.Context, s release.Store) map[string][]servedBy {
log.Printf("console: api instances: skipping %s in the placement join: not a versioned edges file", key)
continue
}
- for host := range f.Hosts {
- out[host] = append(out[host], servedBy{Box: f.Box, Region: f.Region})
- }
- }
- for host := range out {
- sort.Slice(out[host], func(i, j int) bool { return out[host][i].Box < out[host][j].Box })
+ out = append(out, f)
}
return out
}
+
+// armedPlacements folds the fleet's edges files into the set of placement
+// codes an instance may be created into (issue #329): every code published
+// by a box that also has a dial address (an EIP — a box nothing can reach
+// is not a landing target). armed reports whether ANY file carried a
+// placement code at all: false means the fleet predates the field (or the
+// read failed) and the gate stays DORMANT — the door then accepts what it
+// always accepted, so rolling boxes and console in either order is safe.
+func armedPlacements(ctx context.Context, s release.Store) (set map[string]bool, armed bool) {
+ set = map[string]bool{}
+ for _, f := range readEdgesFiles(ctx, s) {
+ if f.Placement == "" {
+ continue
+ }
+ armed = true
+ if f.EIP != "" {
+ set[f.Placement] = true
+ }
+ }
+ return set, armed
+}
diff --git a/internal/host/agent.go b/internal/host/agent.go
index bd8f84dbb..13bb1580c 100644
--- a/internal/host/agent.go
+++ b/internal/host/agent.go
@@ -270,9 +270,12 @@ func RunAgent(args []string) {
// gate is still the hibernate bucket env even though unit tenants
// never touch the bucket — one opt-in switch for the whole activator.
eb := &activator.ExecBackend{
- Cfg: *cfg,
- OptDir: release.OptDir(),
- DataDir: dd,
+ Cfg: *cfg,
+ OptDir: release.OptDir(),
+ DataDir: dd,
+ // The box's placement code, into every exec child's env
+ // (issue #329) — same env the reconcile pass scopes by.
+ Region: strings.TrimSpace(os.Getenv("CARLOS_REGION")),
Cache: cache,
Materialize: materializeHook(reg, dd),
Cgroups: cgm,
diff --git a/internal/host/edgesfile.go b/internal/host/edgesfile.go
index c2e724dcf..d57bbd1aa 100644
--- a/internal/host/edgesfile.go
+++ b/internal/host/edgesfile.go
@@ -73,10 +73,19 @@ type edgeHost struct {
// batch-wide if it ever reached here (plan review catch, 2026-08-23).
// edgesFileRegion resolves the real thing.
type edgesFile struct {
- V int `json:"v"`
- Box string `json:"box"`
- EIP string `json:"eip"`
- Region string `json:"region,omitempty"`
+ V int `json:"v"`
+ Box string `json:"box"`
+ EIP string `json:"eip"`
+ Region string `json:"region,omitempty"`
+ // Placement is this box's CARLOS placement code — CARLOS_REGION verbatim
+ // ("ie", "uk"), the OTHER region vocabulary (see Region above, issue
+ // #329). Additive and omitempty like Region: a pre-placement body parses
+ // unchanged, and a box with no CARLOS_REGION set (single-box
+ // deployments, the legal unset state) publishes no key at all. The
+ // console's create door folds every file's Placement into the armed set
+ // a requested instance region is validated against — dormant until at
+ // least one box publishes the field, so roll order does not matter.
+ Placement string `json:"placement,omitempty"`
Generation string `json:"generation"`
Hosts map[string]edgeHost `json:"hosts"`
}
@@ -114,7 +123,7 @@ type edgesFile struct {
// once per alias per process (cleared when the alias publishes).
var withheldAliasSaid sync.Map
-func edgesFilePass(ctx context.Context, s release.Store, reg *registry.Registry, box, eip, region string, last *[]byte) error {
+func edgesFilePass(ctx context.Context, s release.Store, reg *registry.Registry, box, eip, region, placement string, last *[]byte) error {
routes, err := reg.List()
if err != nil {
return fmt.Errorf("edges: list: %w", err)
@@ -248,7 +257,7 @@ func edgesFilePass(ctx context.Context, s release.Store, reg *registry.Registry,
}
}
- f := edgesFile{V: 1, Box: box, EIP: eip, Region: region, Hosts: hosts}
+ f := edgesFile{V: 1, Box: box, EIP: eip, Region: region, Placement: placement, Hosts: hosts}
gate, err := json.Marshal(f)
if err != nil {
return fmt.Errorf("edges: marshal: %w", err)
@@ -359,6 +368,10 @@ func edgesFileLoop(reg *registry.Registry) {
// per pass). readPlacementRegion is the one IMDS call; edgesFileRegion
// applies the CARLOS_AWS_REGION override on top, pure.
region := edgesFileRegion(os.Getenv, readPlacementRegion())
+ // The placement code alongside it: CARLOS_REGION, the same env the
+ // reconcile pass scopes by — the two vocabularies travel in one file so
+ // a reader can join them without a second source.
+ placement := strings.TrimSpace(os.Getenv("CARLOS_REGION"))
log.Printf("agent: edges file loop armed (5min + jitter, %s)", edgesFileKey(box))
last := seedEdgesFileGate(ctx, store, box)
delay := time.Minute
@@ -366,7 +379,7 @@ func edgesFileLoop(reg *registry.Registry) {
time.Sleep(delay)
delay = 5*time.Minute + rand.N(time.Minute)
eip := readPublicIP()
- if err := edgesFilePass(ctx, store, reg, box, eip, region, &last); err != nil {
+ if err := edgesFilePass(ctx, store, reg, box, eip, region, placement, &last); err != nil {
log.Printf("agent: edges file pass: %v", err)
}
}
diff --git a/internal/host/edgesfile_test.go b/internal/host/edgesfile_test.go
index 11c140ffa..2c219db67 100644
--- a/internal/host/edgesfile_test.go
+++ b/internal/host/edgesfile_test.go
@@ -55,7 +55,7 @@ func TestEdgesFilePublishesHomedHosts(t *testing.T) {
s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
seedRoute(t, reg, "app.example.com")
var lastBody []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &lastBody); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {
t.Fatal(err)
}
got, _, err := s.Get(context.Background(), "control/edges/annie.json")
@@ -86,7 +86,7 @@ func TestEdgesFilePublishesWithEmptyEIP(t *testing.T) {
s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
seedRoute(t, reg, "app.example.com")
var lastBody []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", &lastBody); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", "", &lastBody); err != nil {
t.Fatal(err)
}
got, _, err := s.Get(context.Background(), "control/edges/annie.json")
@@ -112,7 +112,7 @@ func TestEdgesFileSkipsWriteWhenUnchanged(t *testing.T) {
seedRoute(t, reg, "app.example.com")
var lastBody []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &lastBody); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {
t.Fatal(err)
}
if s.puts != 1 {
@@ -122,7 +122,7 @@ func TestEdgesFileSkipsWriteWhenUnchanged(t *testing.T) {
// Second pass, identical registry state: a fresh generation timestamp
// would differ from the first pass's, but the change-gate excludes
// Generation, so this must not write.
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &lastBody); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {
t.Fatal(err)
}
if s.puts != 1 {
@@ -131,7 +131,7 @@ func TestEdgesFileSkipsWriteWhenUnchanged(t *testing.T) {
// A registry change makes the next pass write again.
seedRoute(t, reg, "second.example.com")
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &lastBody); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {
t.Fatal(err)
}
if s.puts != 2 {
@@ -151,12 +151,12 @@ func TestEdgesFileHoldsLastKnownGoodEIPOnTransientBlank(t *testing.T) {
seedRoute(t, reg, "app.example.com")
var last []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &last); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &last); err != nil {
t.Fatal(err)
}
// Simulate a transient IMDS blip: this tick's probe returns "".
- if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", &last); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", "", &last); err != nil {
t.Fatal(err)
}
@@ -186,7 +186,7 @@ func TestEdgesFileExcludesNonReadyRows(t *testing.T) {
}
var last []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &last); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &last); err != nil {
t.Fatal(err)
}
got, _, err := s.Get(context.Background(), "control/edges/annie.json")
@@ -219,7 +219,7 @@ func TestEdgesFileWildcardRowPublishesVerbatim(t *testing.T) {
}
var last []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &last); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &last); err != nil {
t.Fatal(err)
}
got, _, err := s.Get(context.Background(), "control/edges/annie.json")
@@ -250,7 +250,7 @@ func TestEdgesFileRawWireBodyMatchesSchema(t *testing.T) {
seedRoute(t, reg, "app.example.com")
var last []byte
- if err := edgesFilePass(context.Background(), s, reg, "quiet-harbour", "203.0.113.10", "", &last); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "quiet-harbour", "203.0.113.10", "", "", &last); err != nil {
t.Fatal(err)
}
got, _, err := s.Get(context.Background(), "control/edges/quiet-harbour.json")
@@ -302,7 +302,7 @@ func TestEdgesFileRawWireBodyRegionField(t *testing.T) {
t.Run("unset region is omitted entirely", func(t *testing.T) {
var last []byte
- if err := edgesFilePass(context.Background(), s, reg, "quiet-harbour", "203.0.113.10", "", &last); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "quiet-harbour", "203.0.113.10", "", "", &last); err != nil {
t.Fatal(err)
}
got, _, err := s.Get(context.Background(), "control/edges/quiet-harbour.json")
@@ -321,7 +321,7 @@ func TestEdgesFileRawWireBodyRegionField(t *testing.T) {
s2, reg2 := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
seedRoute(t, reg2, "app.example.com")
var last []byte
- if err := edgesFilePass(context.Background(), s2, reg2, "quiet-harbour", "203.0.113.10", "eu-west-1", &last); err != nil {
+ if err := edgesFilePass(context.Background(), s2, reg2, "quiet-harbour", "203.0.113.10", "eu-west-1", "", &last); err != nil {
t.Fatal(err)
}
got, _, err := s2.Get(context.Background(), "control/edges/quiet-harbour.json")
@@ -423,7 +423,7 @@ func TestEdgesFileSeedGateSkipsWriteOnUnchangedBucketState(t *testing.T) {
// A prior process (or the prior pass of this one) already published.
var priming []byte
- if err := edgesFilePass(context.Background(), real, reg, "annie", "1.2.3.4", "", &priming); err != nil {
+ if err := edgesFilePass(context.Background(), real, reg, "annie", "1.2.3.4", "", "", &priming); err != nil {
t.Fatal(err)
}
@@ -435,7 +435,7 @@ func TestEdgesFileSeedGateSkipsWriteOnUnchangedBucketState(t *testing.T) {
}
counting := &putCountingStore{Store: real}
- if err := edgesFilePass(context.Background(), counting, reg, "annie", "1.2.3.4", "", &last); err != nil {
+ if err := edgesFilePass(context.Background(), counting, reg, "annie", "1.2.3.4", "", "", &last); err != nil {
t.Fatal(err)
}
if counting.puts != 0 {
@@ -454,7 +454,7 @@ func TestEdgesFileSeedHoldsEIPThroughRestartBlip(t *testing.T) {
seedRoute(t, reg, "app.example.com")
var priming []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &priming); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &priming); err != nil {
t.Fatal(err)
}
@@ -466,7 +466,7 @@ func TestEdgesFileSeedHoldsEIPThroughRestartBlip(t *testing.T) {
// This tick both hits a transient IMDS blip AND sees a real registry
// change, so the pass must write.
seedRoute(t, reg, "second.example.com")
- if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", &last); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", "", &last); err != nil {
t.Fatal(err)
}
@@ -495,13 +495,13 @@ func TestEdgesFileUnseededLastPublishesBlankEIPOnFirstTickBlip(t *testing.T) {
seedRoute(t, reg, "app.example.com")
var priming []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &priming); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &priming); err != nil {
t.Fatal(err)
}
// The pre-fix shape: a fresh process's *last starts nil (no seed read).
var unseeded []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", &unseeded); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "", "", "", &unseeded); err != nil {
t.Fatal(err)
}
@@ -545,7 +545,7 @@ func TestEdgesFileSeedPreRegionBucketWritesOnceWhenRegionArrives(t *testing.T) {
// This build now has CARLOS_REGION set: the first pass must write, once.
counting := &putCountingStore{Store: real}
- if err := edgesFilePass(context.Background(), counting, reg, "annie", "1.2.3.4", "eu-west-1", &last); err != nil {
+ if err := edgesFilePass(context.Background(), counting, reg, "annie", "1.2.3.4", "eu-west-1", "", &last); err != nil {
t.Fatal(err)
}
if counting.puts != 1 {
@@ -565,7 +565,7 @@ func TestEdgesFileSeedPreRegionBucketWritesOnceWhenRegionArrives(t *testing.T) {
// A second, identical-region pass must not write again — the gate now
// includes region on both sides.
- if err := edgesFilePass(context.Background(), counting, reg, "annie", "1.2.3.4", "eu-west-1", &last); err != nil {
+ if err := edgesFilePass(context.Background(), counting, reg, "annie", "1.2.3.4", "eu-west-1", "", &last); err != nil {
t.Fatal(err)
}
if counting.puts != 1 {
@@ -588,7 +588,7 @@ func TestEdgesFileMarksPlatformRows(t *testing.T) {
}
}
var lastBody []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &lastBody); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {
t.Fatal(err)
}
body, _, err := s.Get(context.Background(), "control/edges/annie.json")
@@ -654,7 +654,7 @@ func TestEdgesFilePublishesPointingAliasesOfServedPrimaries(t *testing.T) {
publish := func() map[string]hostView {
t.Helper()
var lastBody []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &lastBody); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {
t.Fatal(err)
}
body, _, err := s.Get(context.Background(), "control/edges/annie.json")
@@ -734,7 +734,7 @@ func TestEdgesFilePublishesDelegatedWildcardParents(t *testing.T) {
}
}
var lastBody []byte
- if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", &lastBody); err != nil {
+ if err := edgesFilePass(context.Background(), s, reg, "annie", "1.2.3.4", "", "", &lastBody); err != nil {
t.Fatal(err)
}
body, _, err := s.Get(context.Background(), "control/edges/annie.json")
@@ -797,7 +797,7 @@ func TestEdgesFileBoxNameJoinsTheStatusSnapshotLabel(t *testing.T) {
s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
seedRoute(t, reg, "app.example.com")
var last []byte
- if err := edgesFilePass(ctx, s, reg, box, "203.0.113.7", "eu-west-1", &last); err != nil {
+ if err := edgesFilePass(ctx, s, reg, box, "203.0.113.7", "eu-west-1", "", &last); err != nil {
t.Fatal(err)
}
if err := status.WriteBox(ctx, s, snapshot); err != nil {
@@ -808,3 +808,47 @@ func TestEdgesFileBoxNameJoinsTheStatusSnapshotLabel(t *testing.T) {
t.Fatalf("fold = %v, want this box's own EIP — the two writers disagree about its name", got)
}
}
+
+// The placement code (issue #329): CARLOS_REGION verbatim, additive between
+// region and generation, omitted entirely when the box has none — the
+// console's armed-placement gate stays dormant on a fleet that never
+// publishes it, which is what makes roll order irrelevant.
+func TestEdgesFileRawWireBodyPlacementField(t *testing.T) {
+ s, reg := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg, "app.example.com")
+ var last []byte
+ if err := edgesFilePass(context.Background(), s, reg, "quiet-harbour", "203.0.113.10", "eu-west-2", "uk", &last); err != nil {
+ t.Fatal(err)
+ }
+ got, _, err := s.Get(context.Background(), "control/edges/quiet-harbour.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var probe struct {
+ Placement string `json:"placement"`
+ Generation string `json:"generation"`
+ }
+ if err := json.Unmarshal(got, &probe); err != nil {
+ t.Fatal(err)
+ }
+ want := `{"v":1,"box":"quiet-harbour","eip":"203.0.113.10","region":"eu-west-2","placement":"uk","generation":"` +
+ probe.Generation + `","hosts":{"app.example.com":{"role":"home","platform":false}}}`
+ if string(got) != want {
+ t.Fatalf("raw wire body (placement set) =\n%s\nwant\n%s", got, want)
+ }
+
+ // Unset placement publishes no key at all.
+ s2, reg2 := release.NewDirStore(t.TempDir()), edgesTestRegistry(t)
+ seedRoute(t, reg2, "app.example.com")
+ var last2 []byte
+ if err := edgesFilePass(context.Background(), s2, reg2, "quiet-harbour", "203.0.113.10", "", "", &last2); err != nil {
+ t.Fatal(err)
+ }
+ got2, _, err := s2.Get(context.Background(), "control/edges/quiet-harbour.json")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if strings.Contains(string(got2), "placement") {
+ t.Fatalf("raw wire body with no CARLOS_REGION carries a placement key: %s", got2)
+ }
+}