amadan

carlos / platform Public

Box identity Option B rev 4 and plan rev 2: answer Astra's round 3

Spec: a fresh decision per attempt and owned only after the Put
succeeds, so a failed replace cannot renew authority; the complete
collected snapshot is what the decision writes; a 60 s mutation deadline
on every gated write and retireAfter of three periods plus that deadline
as the retirement barrier; a platform box with a split or prefixed
status store does not arm, which makes the arbitration namespace
enforceable; configured and fleet verdicts never expire; the relay's
whole store is gated; a seed without our box_id is discarded; readers
combine completeness monotonically and read evidence and observations
before the early return; cleared alert rules are kept as Firing:false
for a day and the watcher reads the fleet object first; both fleet files
are written conditionally after held.Err(); one identity path in
Publishable; carlos-edge.service; both object locations resolved and
printed; the operator commands are carlos status flags. Plan: the
concrete failure sequences as tests, squash landing stated, cli.md and
README tasks, the full gate. Findings committed unedited as
CODEX-FINDINGS-BOX-IDENTITY-B-R3.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude 91b5012aba1da39ec8f6a7940a654a30e14c916c parent fb59cb2
3 files changed, +1829 −866
  • docs/superpowers/plans/2026-10-06-box-identity-detect.md +1271 −588
  • docs/superpowers/specs/2026-10-06-box-identity-detect-design.md +378 −278
  • docs/superpowers/specs/CODEX-FINDINGS-BOX-IDENTITY-B-R3.md +180 −0

CI passed — run details

diff --git a/docs/superpowers/plans/2026-10-06-box-identity-detect.md b/docs/superpowers/plans/2026-10-06-box-identity-detect.md
index 68d496de0..16e7afd87 100644
--- a/docs/superpowers/plans/2026-10-06-box-identity-detect.md
+++ b/docs/superpowers/plans/2026-10-06-box-identity-detect.md
@@ -4,61 +4,67 @@
**Goal:** Two boxes whose machine ids hash to the same label stop silently overwriting each other's `status/boxes/<label>.json` and `control/edges/<label>.json`: the second writer refuses, alerts, and takes no claim work, and nothing moves.
-**Architecture:** `status.DecideLabel` reads the label's snapshot and returns a verdict (owned, foreign, twin, unproven, unknown) from a private `box_id` and the cloud instance id; the agent decides at arm and every status tick, writes the verdict to `<dataDir>/identity.json`, and every label-keyed publisher in every process writes through a `gatedStore` that refuses unless `Publishable(label, target)`. The console learns of collisions from a `box_id` mismatch between the two objects, from evidence objects the refusing box writes, and from a fleet observation file the status lease holder keeps; all three fail automatic region eligibility closed and feed one per-label alert object. Two CLI verbs let an operator check a label before a box joins and free a label after a collision.
+**Architecture:** `status.DecideLabel` reads the label's snapshot and returns a verdict (owned, foreign, twin, unproven, unknown) from a private `box_id` and the cloud instance id, writing the box's complete snapshot only on owned; the agent decides at arm and every status tick, writes the verdict to `<dataDir>/identity.json`, and every label-keyed publisher in every process writes through `host.Gated`, which refuses each `Put`/`Delete` unless `Publishable(label)` and bounds it with a 60 s deadline. The console learns of collisions from a `box_id` mismatch between the two objects, from evidence objects the refusing box writes, and from an observation file the status lease holder keeps; all three fail automatic region eligibility closed and feed one per-label alert object. Two `carlos status` flags let an operator check a label before a box joins and free a label after a collision.
-**Tech Stack:** Go 1.25 (`go.mod`), `release.Store` (S3 and DirStore), `internal/status`, `internal/host`, `internal/edgesfile`, `internal/console`, `internal/relay`, `internal/metrics`.
+**Tech Stack:** Go 1.25, `release.Store` (S3 and DirStore), `internal/status`, `internal/host`, `internal/edgesfile`, `internal/console`, `internal/relay`, `internal/metrics`.
-**Spec:** `docs/superpowers/specs/2026-10-06-box-identity-detect-design.md` (rev 3). Its §2.1 table, §2.2 entry-point table and §3 reader rules are normative; this plan cites them by section.
+**Spec:** `docs/superpowers/specs/2026-10-06-box-identity-detect-design.md` (rev 4). §2.1 (the verdict table), §2.2 (the gate, `mutationTTL`, `retireAfter`), §2.6 (the arbitration-namespace rule) and §3 (reader rules, monotonic completeness) are normative. Line numbers below are against the branch after its merge of main (`fb59cb23`).
+
+**Landing:** the task commits are not independently shippable (Task 2 arms snapshot refusal before Task 4 gates the other publishers and before Task 8's recovery tooling exists). The branch is reviewed whole and lands on main as **one squash** (`amadan branch merge -squash`, spec §8).
## Global Constraints
- No object key, prefix or Route53 `SetIdentifier` changes (spec §1).
- `box_id` and `instance_id` never appear in `status/fleet.json`, `status/index.html`, any account page, the instances API response, or an alert mail body (spec §1, §3, §7).
-- A label is never reassigned automatically; only `carlos fleet retire` deletes a label's objects, and only when both are stale (spec §2.1, §2.3).
-- `owned` is recorded only after a successful conditional write; no verdict is promoted by reading back the box's own bytes (spec §2.1).
-- Every `Put` and `Delete` under a label-keyed key goes through `gatedStore`; reads and lists are not gated (spec §2.2).
-- Copy voice: plain sentences, no em dashes, no new user-facing strings beyond the log lines and the two CLI verbs' output in this plan (house rule).
-- Cross-compile gate: CI builds darwin and windows; anything Linux-only in `internal/host` needs `//go:build linux` and a `!linux` stub.
-- Tests use `release.NewDirStore(t.TempDir())`; the DirStore arbitrates only writers sharing one handle (spec §2.5).
-- Run the package tests with `GOFLAGS=-buildvcs=false go test ./internal/<pkg>/...`; the full gate needs the `-exec` wrapper from the repo notes.
+- A label is never reassigned automatically; only `carlos status --retire-label` deletes a label's objects, and only when its snapshot is older than `retireAfter = 3*status.SnapshotPeriod + mutationTTL` (spec §2.2, §2.3).
+- `owned` is recorded only after a successful conditional write; each decision attempt starts fresh; a failed `Put` is `unknown` (spec §2.1).
+- Every `Put`, `Delete` and `DeleteIfMatch` under a label-keyed key goes through `host.Gated`, which applies `mutationTTL = 60 * time.Second` as a context deadline; reads and lists are not gated (spec §2.2).
+- A platform box whose status store is not the deployment bucket at the root does not arm (spec §2.6).
+- Copy voice: plain sentences, no em dashes (house rule).
+- Cross-compile: CI builds darwin, windows and linux-arm64 (`AGENTS.md`); anything Linux-only in `internal/host` needs `//go:build linux` and a `!linux` stub.
+- The gate before any commit (`AGENTS.md:184-187`): `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l .` clean. Inside the sandbox: `GOFLAGS=-buildvcs=false`; the full test run needs the `-exec` wrapper from the repo notes so socket tests get a short `TMPDIR`.
+- A new `carlos` flag is documented in `docs/site/cli.md` under the verb's heading (`AGENTS.md:242`).
+- Tests use `release.NewDirStore(t.TempDir())`; one handle arbitrates (spec §2.5).
## Review Focus
-Inputs the spec implies but no task's tests exercised at first draft; each line's test is added to the owning task below.
+Inputs the spec implies but no task's tests exercised at first draft; each line's test is in the owning task below.
-1. A snapshot body that parses but has `generated_at` zero (an old or hand-edited object): `BoxStale` says stale; the verdict must still be foreign or unproven, never owned. Task 2.
-2. A `box_id` in the object that is not 32 lower-case hex (a hand edit): treated as present and different, so foreign, not as absent. Task 2.
-3. `identity.json` written by a previous release of the agent with a different `v` or no `store`: `Publishable` must answer false, not panic or default to open. Task 3.
-4. The lease holder reads a `status/identity/fleet.json` whose `contested_until` is in the far future (clock skew on a previous publisher): it must never shorten it, and must still clear it when the time passes. Task 7.
-5. `carlos fleet retire` on a label whose edges file is absent but snapshot stale: deletes the snapshot only, exits 0, prints that the edges file was already gone. Task 8.
+1. A snapshot body with `generated_at` zero (hand-edited): `BoxStale` says stale; the verdict must still be foreign or unproven, never owned. Task 2.
+2. A `box_id` in the object that is not 32 lower-case hex: present and different, so foreign, never absent. Task 2.
+3. `identity.json` from a previous shape (`v` 0, no `store`): `Publishable` false, no panic. Task 3.
+4. A `contested_until` far in the future from a skewed previous publisher: never shortened, and still cleared when it passes. Task 7.
+5. `--retire-label` with the edges file present but the snapshot absent: deletes the edges file with a printed warning, exit 0. Task 8.
---
### Task 1: `box_id` on the wire
**Files:**
-- Modify: `internal/status/snapshot.go:387-445` (Box), `:523-531` (Label), `:535` (CollectBox), `:674`
+- Modify: `internal/status/snapshot.go:387-445` (Box), `:523-531` (Label), `:535-680` (CollectBox)
- Modify: `internal/edgesfile/edgesfile.go:68-79` (File)
-- Modify: `internal/host/edgesfile.go:157-160` (edgesFile)
+- Modify: `internal/host/edgesfile.go:157-162` (edgesFile)
- Test: `internal/status/status_test.go`, `internal/edgesfile/edgesfile_test.go`, `internal/host/edgesfile_test.go`
**Interfaces:**
-- Produces: `status.BoxID(machineID string) string` (32 lower-case hex, `""` for `""`); `status.Box.BoxID string json:"box_id,omitempty"`; `status.Probes` unchanged; `edgesfile.File.BoxID string json:"box_id,omitempty"`; `host.edgesFile.BoxID string json:"box_id,omitempty"`.
+- Produces: `status.BoxID(machineID string) string` (32 lower-case hex, `""` for `""`); `status.Box.BoxID string json:"box_id,omitempty"`; `edgesfile.File.BoxID string json:"box_id,omitempty"`; `host.edgesFile.BoxID string json:"box_id,omitempty"`.
- [ ] **Step 1: Write the failing tests**
```go
// internal/status/status_test.go
func TestBoxIDIsThirtyTwoHexAndCollisionFree(t *testing.T) {
- a, b := status.BoxID("00000000000000000000000000000018"), status.BoxID("0000000000000000000000000000003c")
- if status.Label("00000000000000000000000000000018") != status.Label("0000000000000000000000000000003c") {
+ const m1, m2 = "00000000000000000000000000000018", "0000000000000000000000000000003c"
+ if status.Label(m1) != status.Label(m2) {
t.Fatal("fixture: these two machine ids must share a label (edsger-knitting)")
}
+ a, b := status.BoxID(m1), status.BoxID(m2)
if a == b {
t.Fatal("BoxID must differ where Label collides")
}
+ hex32 := regexp.MustCompile(`^[0-9a-f]{32}$`)
for _, id := range []string{a, b} {
- if !regexp.MustCompile(`^[0-9a-f]{32}$`).MatchString(id) {
+ if !hex32.MatchString(id) {
t.Fatalf("BoxID %q is not 32 lower-case hex", id)
}
}
@@ -67,7 +73,7 @@ func TestBoxIDIsThirtyTwoHexAndCollisionFree(t *testing.T) {
}
sum := sha256.Sum256([]byte("m"))
if want := hex.EncodeToString(sum[:16]); status.BoxID("m") != want {
- t.Fatalf("BoxID = %q, want first 16 bytes of sha256: %q", status.BoxID("m"), want)
+ t.Fatalf("BoxID = %q, want %q", status.BoxID("m"), want)
}
}
@@ -76,9 +82,9 @@ func TestCollectBoxCarriesBoxIDPrivately(t *testing.T) {
if b.BoxID != status.BoxID("m") {
t.Fatalf("Box.BoxID = %q, want %q", b.BoxID, status.BoxID("m"))
}
- body, _ := json.Marshal(status.CollectBox(nil, status.Probes{MachineID: ""}, time.Now()))
+ body, _ := json.Marshal(status.CollectBox(nil, status.Probes{}, time.Now()))
if strings.Contains(string(body), "box_id") {
- t.Fatal("an empty machine id must omit box_id, not publish an empty one")
+ t.Fatal("an empty machine id must omit box_id")
}
}
```
@@ -87,11 +93,8 @@ func TestCollectBoxCarriesBoxIDPrivately(t *testing.T) {
// internal/edgesfile/edgesfile_test.go
func TestParseCarriesBoxID(t *testing.T) {
f, err := edgesfile.Parse([]byte(`{"v":1,"box":"annie-welding","box_id":"0123456789abcdef0123456789abcdef","eip":"1.2.3.4","hosts":{}}`))
- if err != nil {
- t.Fatal(err)
- }
- if f.BoxID != "0123456789abcdef0123456789abcdef" {
- t.Fatalf("BoxID = %q", f.BoxID)
+ if err != nil || f.BoxID != "0123456789abcdef0123456789abcdef" {
+ t.Fatalf("BoxID = %q, err %v", f.BoxID, err)
}
old, err := edgesfile.Parse([]byte(`{"v":1,"box":"annie-welding","eip":"1.2.3.4","hosts":{}}`))
if err != nil || old.BoxID != "" {
@@ -104,7 +107,7 @@ func TestParseCarriesBoxID(t *testing.T) {
// internal/host/edgesfile_test.go
func TestEdgesFilePassPublishesBoxID(t *testing.T) {
s := release.NewDirStore(t.TempDir())
- reg := newTestRegistry(t) // the helper the existing edgesFilePass tests use
+ reg := edgesTestRegistry(t)
var last []byte
ident := edgesFile{Box: "annie", BoxID: "0123456789abcdef0123456789abcdef"}
if err := edgesFilePass(context.Background(), s, reg, ident, "1.2.3.4", &last); err != nil {
@@ -127,20 +130,19 @@ func TestEdgesFilePassPublishesBoxID(t *testing.T) {
- [ ] **Step 2: Run the tests to verify they fail**
Run: `GOFLAGS=-buildvcs=false go test ./internal/status/ ./internal/edgesfile/ ./internal/host/ -run 'BoxID|CarriesBoxID|PublishesBoxID' -v`
-Expected: FAIL, `undefined: status.BoxID`, `f.BoxID undefined`, `unknown field BoxID`.
+Expected: FAIL with `undefined: status.BoxID`, `f.BoxID undefined`, `unknown field BoxID`.
- [ ] **Step 3: Implement**
-```go
-// internal/status/snapshot.go, after Label:
+`internal/status/snapshot.go`, after `Label`:
+```go
// BoxID is the private, collision-resistant identity of a platform box:
// the first 16 bytes of sha256(machineID), lower-case hex, 32 characters.
// Label is drawn from the first 8 bytes of the same digest, so a label
-// collision (1 in 1,600) is never a BoxID collision (1 in 2^128). It is
-// bucket-private: BuildFleet never copies it, and the dry run's needles
-// prove it (dryrun.go). "" for an empty machine id: a box that cannot read
-// /etc/machine-id has no identity and publishes nothing keyed by it.
+// collision (1 in 1,600) is never a BoxID collision. Bucket-private:
+// BuildFleet never copies it, and dryrun.go's needles prove it. "" for an
+// empty machine id: such a box has no identity (spec 2.1).
func BoxID(machineID string) string {
if machineID == "" {
return ""
@@ -150,34 +152,13 @@ func BoxID(machineID string) string {
}
```
-Add to `Box` after `Label`:
+`Box` gains, after `Label`: `BoxID string json:"box_id,omitempty"` with a comment "the machine's private identity; omitted for a box with no machine id and for fleet boxes; never copied into FleetBox". `CollectBox` sets `BoxID: BoxID(p.MachineID)`. Import `encoding/hex`.
-```go
- // BoxID is the machine's private identity (BoxID). Omitted for a box
- // with no machine id and for fleet boxes. Never copied into FleetBox.
- BoxID string `json:"box_id,omitempty"`
-```
-
-In `CollectBox` set `BoxID: BoxID(p.MachineID),` beside `Label:`. Add `"encoding/hex"` to the imports.
-
-`internal/edgesfile/edgesfile.go`, in `File` after `Box`:
-
-```go
- BoxID string `json:"box_id,omitempty"`
-```
-
-`internal/host/edgesfile.go`, in `edgesFile` after `Box`:
-
-```go
- // BoxID is status.BoxID of this machine, so a reader can tell whether
- // this file and the label's snapshot were written by one machine. Part
- // of the change gate like every field but Generation.
- BoxID string `json:"box_id,omitempty"`
-```
+`internal/edgesfile/edgesfile.go` `File`: `BoxID string json:"box_id,omitempty"` after `Box`. `internal/host/edgesfile.go` `edgesFile`: the same field, with "part of the change gate like every field but Generation".
- [ ] **Step 4: Run the tests to verify they pass**
-Run: the same command. Expected: PASS. Also run `go test ./internal/status/` whole: the existing `TestWriteBoxPutsUnderLabeledKey` still passes (the key is still the label).
+Run: the Step 2 command, then `GOFLAGS=-buildvcs=false go test ./internal/status/`. Expected: PASS; `TestWriteBoxPutsUnderLabeledKey` still passes (the key is still the label).
- [ ] **Step 5: Commit**
@@ -188,12 +169,12 @@ git commit -m "status, edgesfile: carry a private box_id on both per-box objects
---
-### Task 2: `status.DecideLabel` and a `WriteBox` that never overwrites another machine
+### Task 2: `status.DecideLabel`, and `WriteBox` that never overwrites another machine
**Files:**
-- Create: `internal/status/decide.go`
+- Create: `internal/status/decide.go`, `internal/status/decide_test.go`
- Modify: `internal/status/snapshot.go:742-765` (WriteBox)
-- Test: `internal/status/decide_test.go`, `internal/status/status_test.go:225` (`TestWriteBoxOverwritesOnSecondTick` stays green)
+- Test: `internal/status/status_test.go:225` (`TestWriteBoxOverwritesOnSecondTick` stays green)
**Interfaces:**
- Produces:
@@ -201,29 +182,29 @@ git commit -m "status, edgesfile: carry a private box_id on both per-box objects
```go
type Verdict string
const (
- VerdictOwned Verdict = "owned"
- VerdictForeign Verdict = "foreign"
- VerdictTwin Verdict = "twin"
- VerdictUnproven Verdict = "unproven"
- VerdictUnknown Verdict = "unknown"
+ VerdictOwned Verdict = "owned"
+ VerdictForeign Verdict = "foreign"
+ VerdictTwin Verdict = "twin"
+ VerdictUnproven Verdict = "unproven"
+ VerdictUnknown Verdict = "unknown"
)
-// Decision is one reading of the label's object.
type Decision struct {
Verdict Verdict
- Label string
- BoxID string // ours
- HolderBoxID string // the object's, "" if absent
- HolderInstanceID string // the object's metrics.instance_id, "" if absent
+ Label, BoxID string // ours
+ HolderBoxID string // the object's, "" if absent
+ HolderInstanceID string // the object's metrics.instance_id, "" if absent
HolderFresh bool
DecidedAt time.Time
}
-// DecideLabel reads status/boxes/<b.Label>.json and, when the verdict is
-// owned, writes b there. Spec 2.1 table. It never writes otherwise.
-func DecideLabel(ctx context.Context, s release.Store, b Box, now time.Time) (Decision, error)
var ErrNotOwned = errors.New("status: label is not this machine's")
+// DecideLabel: spec 2.1. Writes b only on owned; a fresh Decision per attempt;
+// owned only after the Put succeeded; any Put failure is unknown.
+func DecideLabel(ctx context.Context, s release.Store, b Box, now time.Time) (Decision, error)
+// WriteFleetBox: a fleet box's label-keyed replace, no decision (spec 2.1).
+func WriteFleetBox(ctx context.Context, s release.Store, b Box) error
```
-- `WriteBox(ctx, s, b)` becomes `_, err := DecideLabel(...)`, returning `ErrNotOwned` (wrapped with the decision) when not owned. Callers that only need the old behaviour keep calling `WriteBox`.
+- `WriteBox(ctx, s, b)` calls `DecideLabel` and returns `ErrNotOwned` (wrapped with the verdict) when not owned.
- [ ] **Step 1: Write the failing tests**
@@ -238,6 +219,7 @@ import (
"testing"
"time"
+ "github.com/carlosframework/platform/internal/registry"
"github.com/carlosframework/platform/internal/release"
"github.com/carlosframework/platform/internal/status"
)
@@ -246,30 +228,34 @@ func boxFor(machine, instance string, at time.Time) status.Box {
return status.CollectBox(nil, status.Probes{MachineID: machine, InstanceID: instance, Region: "x"}, at)
}
-// put writes a snapshot body under the label without any decision, standing
-// in for an old-release writer or a hand-placed object.
-func put(t *testing.T, s release.Store, label string, body any) {
+// putRaw writes a body under the label with no decision: an old-release
+// writer, a hand-placed object, or a corrupt one.
+func putRaw(t *testing.T, s release.Store, label string, body []byte) {
t.Helper()
- raw, _ := json.Marshal(body)
key := "status/boxes/" + label + ".json"
_, etag, _ := s.Get(context.Background(), key)
- if _, err := s.Put(context.Background(), key, raw, etag); err != nil {
+ if _, err := s.Put(context.Background(), key, body, etag); err != nil {
t.Fatal(err)
}
}
+func putBox(t *testing.T, s release.Store, b status.Box) {
+ t.Helper()
+ raw, _ := json.Marshal(b)
+ putRaw(t, s, b.Label, raw)
+}
+
func TestDecideLabelTable(t *testing.T) {
now := time.Unix(1_700_000_000, 0)
fresh, stale := now.Add(-time.Minute), now.Add(-time.Hour)
me := boxFor("mine", "i-me", now)
- label := me.Label
type object struct {
boxID, instance string
at time.Time
}
cases := []struct {
name string
- obj *object // nil = not found
+ obj *object
want status.Verdict
written bool
}{
@@ -279,7 +265,7 @@ func TestDecideLabelTable(t *testing.T) {
{"own id, other instance is a twin", &object{me.BoxID, "i-other", fresh}, status.VerdictTwin, false},
{"foreign fresh", &object{"ffffffffffffffffffffffffffffffff", "i-other", fresh}, status.VerdictForeign, false},
{"foreign stale stays foreign", &object{"ffffffffffffffffffffffffffffffff", "i-other", stale}, status.VerdictForeign, false},
- {"foreign zero generated_at", &object{"ffffffffffffffffffffffffffffffff", "i-other", time.Time{}}, status.VerdictForeign, false},
+ {"foreign, zero generated_at", &object{"ffffffffffffffffffffffffffffffff", "i-other", time.Time{}}, status.VerdictForeign, false},
{"malformed box_id is foreign", &object{"not-hex", "i-me", fresh}, status.VerdictForeign, false},
{"old writer, same instance, owned", &object{"", "i-me", fresh}, status.VerdictOwned, true},
{"old writer, other instance, foreign", &object{"", "i-other", fresh}, status.VerdictForeign, false},
@@ -290,11 +276,11 @@ func TestDecideLabelTable(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
s := release.NewDirStore(t.TempDir())
if tc.obj != nil {
- prior := status.Box{GeneratedAt: tc.obj.at, Label: label, BoxID: tc.obj.boxID}
+ prior := status.Box{GeneratedAt: tc.obj.at, Label: me.Label, BoxID: tc.obj.boxID}
if tc.obj.instance != "" {
prior.Metrics = &status.BoxMetrics{InstanceID: tc.obj.instance}
}
- put(t, s, label, prior)
+ putBox(t, s, prior)
}
d, err := status.DecideLabel(context.Background(), s, me, now)
if err != nil {
@@ -303,38 +289,116 @@ func TestDecideLabelTable(t *testing.T) {
if d.Verdict != tc.want {
t.Fatalf("verdict = %s, want %s", d.Verdict, tc.want)
}
- body, _, gerr := s.Get(context.Background(), "status/boxes/"+label+".json")
+ body, _, gerr := s.Get(context.Background(), "status/boxes/"+me.Label+".json")
var got status.Box
if gerr == nil {
_ = json.Unmarshal(body, &got)
}
- if tc.written && !got.GeneratedAt.Equal(now) {
- t.Fatalf("owned must have written our snapshot; object generated_at = %v", got.GeneratedAt)
- }
- if !tc.written && gerr == nil && got.GeneratedAt.Equal(now) {
- t.Fatal("a non-owned verdict must not write")
+ if tc.written != (gerr == nil && got.GeneratedAt.Equal(now)) {
+ t.Fatalf("written = %v, want %v", !tc.written, tc.written)
}
})
}
}
+// failingPut is a store whose Put fails with a non-CAS error after n
+// successes, standing in for an IAM deny or an outage on the write path.
+type failingPut struct {
+ release.Store
+ allow int
+ puts int
+}
+
+func (f *failingPut) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ f.puts++
+ if f.puts > f.allow {
+ return "", errors.New("put: access denied")
+ }
+ return f.Store.Put(ctx, key, body, ifMatch)
+}
+
+func TestDecideLabelFailedReplaceIsUnknownNotOwned(t *testing.T) {
+ inner := release.NewDirStore(t.TempDir())
+ now := time.Now()
+ me := boxFor("mine", "i-me", now)
+ if _, err := status.DecideLabel(context.Background(), inner, me, now); err != nil {
+ t.Fatal(err)
+ }
+ s := &failingPut{Store: inner, allow: 0}
+ d, err := status.DecideLabel(context.Background(), s, boxFor("mine", "i-me", now.Add(time.Minute)), now.Add(time.Minute))
+ if err == nil || d.Verdict != status.VerdictUnknown {
+ t.Fatalf("a failed replace must be unknown with an error, got %s %v", d.Verdict, err)
+ }
+}
+
+// casOnce is a store whose first replace Put loses a CAS race and, in the
+// same breath, swaps the object for a legacy body with no instance id: the
+// retry must decide the new body on its own merits.
+type casOnce struct {
+ release.Store
+ label string
+ fired bool
+}
+
+func (c *casOnce) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ if !c.fired && ifMatch != "" {
+ c.fired = true
+ legacy, _ := json.Marshal(status.Box{GeneratedAt: time.Now(), Label: c.label})
+ if _, err := c.Store.Put(ctx, key, legacy, ifMatch); err != nil {
+ return "", err
+ }
+ return "", release.ErrCASConflict
+ }
+ return c.Store.Put(ctx, key, body, ifMatch)
+}
+
+func TestDecideLabelRetryDoesNotInheritTheFirstRead(t *testing.T) {
+ inner := release.NewDirStore(t.TempDir())
+ now := time.Now()
+ me := boxFor("mine", "i-me", now)
+ putBox(t, inner, status.Box{GeneratedAt: now, Label: me.Label, Metrics: &status.BoxMetrics{InstanceID: "i-me"}}) // old writer, same instance: owned on first read
+ s := &casOnce{Store: inner, label: me.Label}
+ d, err := status.DecideLabel(context.Background(), s, me, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if d.Verdict != status.VerdictUnproven {
+ t.Fatalf("the retry saw a legacy object with no instance id and must be unproven, got %s (holder instance %q carried over?)", d.Verdict, d.HolderInstanceID)
+ }
+}
+
func TestDecideLabelMalformedBodyIsUnknownAndWritesNothing(t *testing.T) {
s := release.NewDirStore(t.TempDir())
me := boxFor("mine", "i-me", time.Now())
- key := "status/boxes/" + me.Label + ".json"
- if _, err := s.Put(context.Background(), key, []byte("{not json"), ""); err != nil {
- t.Fatal(err)
- }
+ putRaw(t, s, me.Label, []byte("{not json"))
d, err := status.DecideLabel(context.Background(), s, me, time.Now())
if err == nil || d.Verdict != status.VerdictUnknown {
t.Fatalf("want unknown with an error, got %s %v", d.Verdict, err)
}
- body, _, _ := s.Get(context.Background(), key)
+ body, _, _ := s.Get(context.Background(), "status/boxes/"+me.Label+".json")
if string(body) != "{not json" {
t.Fatal("unknown must leave the object alone")
}
}
+// interleave lands writer B's create between A's read and A's Put.
+type interleave struct {
+ release.Store
+ other status.Box
+ done bool
+}
+
+func (i *interleave) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ if !i.done && ifMatch == "" {
+ i.done = true
+ raw, _ := json.Marshal(i.other)
+ if _, err := i.Store.Put(ctx, key, raw, ""); err != nil {
+ return "", err
+ }
+ }
+ return i.Store.Put(ctx, key, body, ifMatch)
+}
+
func TestDecideLabelSimultaneousStartOneWins(t *testing.T) {
now := time.Now()
a := boxFor("00000000000000000000000000000018", "i-a", now)
@@ -342,27 +406,64 @@ func TestDecideLabelSimultaneousStartOneWins(t *testing.T) {
if a.Label != b.Label {
t.Fatal("fixture: the pair must share a label")
}
+ // Sequential, both orders.
for _, order := range [][2]status.Box{{a, b}, {b, a}} {
s := release.NewDirStore(t.TempDir())
- first, err := status.DecideLabel(context.Background(), s, order[0], now)
- if err != nil || first.Verdict != status.VerdictOwned {
- t.Fatalf("first: %s %v", first.Verdict, err)
- }
- second, err := status.DecideLabel(context.Background(), s, order[1], now)
- if err != nil || second.Verdict != status.VerdictForeign {
- t.Fatalf("second: %s %v, want foreign", second.Verdict, err)
+ first, _ := status.DecideLabel(context.Background(), s, order[0], now)
+ second, _ := status.DecideLabel(context.Background(), s, order[1], now)
+ if first.Verdict != status.VerdictOwned || second.Verdict != status.VerdictForeign {
+ t.Fatalf("sequential: %s then %s", first.Verdict, second.Verdict)
}
if second.HolderBoxID != order[0].BoxID || second.HolderInstanceID == "" {
t.Fatalf("the decision must name the holder: %+v", second)
}
}
+ // Interleaved: B's create lands between A's not-found read and A's create.
+ inner := release.NewDirStore(t.TempDir())
+ s := &interleave{Store: inner, other: b}
+ d, err := status.DecideLabel(context.Background(), s, a, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if d.Verdict != status.VerdictForeign || d.HolderBoxID != b.BoxID {
+ t.Fatalf("A lost the create race and must be foreign naming B, got %+v", d)
+ }
+}
+
+func TestDecideLabelExhaustedCASIsUnknown(t *testing.T) {
+ s := &alwaysConflict{Store: release.NewDirStore(t.TempDir())}
+ d, err := status.DecideLabel(context.Background(), s, boxFor("mine", "i-me", time.Now()), time.Now())
+ if err == nil || d.Verdict != status.VerdictUnknown {
+ t.Fatalf("three conflicts are unknown, got %s %v", d.Verdict, err)
+ }
+}
+
+type alwaysConflict struct{ release.Store }
+
+func (alwaysConflict) Put(context.Context, string, []byte, string) (string, error) {
+ return "", release.ErrCASConflict
+}
+
+func TestDecideLabelWritesTheCompleteSnapshot(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ now := time.Now()
+ routes := []registry.Route{{Host: "a.example", Kind: "instance", Status: "ready", Version: "abc1234"}}
+ b := status.CollectBox(routes, status.Probes{MachineID: "mine", InstanceID: "i-me"}, now)
+ if _, err := status.DecideLabel(context.Background(), s, b, now); err != nil {
+ t.Fatal(err)
+ }
+ body, _, _ := s.Get(context.Background(), "status/boxes/"+b.Label+".json")
+ var got status.Box
+ _ = json.Unmarshal(body, &got)
+ if got.Routes.Resident+got.Routes.Static+got.Routes.Hibernating != 1 || got.Metrics == nil || got.Metrics.RouteVersions["a.example"] != "abc1234" {
+ t.Fatalf("the object must be the complete collected snapshot, got %+v", got)
+ }
}
func TestWriteBoxRefusesAForeignLabel(t *testing.T) {
s := release.NewDirStore(t.TempDir())
now := time.Now()
- holder := boxFor("00000000000000000000000000000018", "i-a", now)
- if err := status.WriteBox(context.Background(), s, holder); err != nil {
+ if err := status.WriteBox(context.Background(), s, boxFor("00000000000000000000000000000018", "i-a", now)); err != nil {
t.Fatal(err)
}
err := status.WriteBox(context.Background(), s, boxFor("0000000000000000000000000000003c", "i-b", now))
@@ -405,15 +506,13 @@ const (
)
// ErrNotOwned is WriteBox's refusal: the label's object was written by
-// another machine (or cannot be proved ours), and this box must not
-// overwrite it. Spec 2026-10-06-box-identity-detect-design.md 2.1.
+// another machine, or cannot be proved ours, and this box must not
+// overwrite it (spec 2026-10-06-box-identity-detect-design.md 2.1).
var ErrNotOwned = errors.New("status: label is not this machine's")
-// Decision is one reading of status/boxes/<label>.json against this box.
type Decision struct {
Verdict Verdict
- Label string
- BoxID string
+ Label, BoxID string
HolderBoxID string
HolderInstanceID string
HolderFresh bool
@@ -424,37 +523,40 @@ var boxIDRe = regexp.MustCompile(`^[0-9a-f]{32}$`)
const decideAttempts = 3
-// DecideLabel is the spec's 2.1 table. It reads the label's object, decides,
-// and writes b only when the verdict is owned: a create-only Put when the
-// key is absent, a replace on the etag read when the object is proved
-// ours. A CAS conflict re-reads and re-decides; it never re-Puts on a
-// fresh etag without re-reading the body. Every other verdict writes
-// nothing. Unknown (unreadable, unparsable, CAS exhausted) returns an
-// error alongside the decision.
+// DecideLabel is the spec's 2.1 table. Each attempt reads the object and
+// builds a fresh Decision from that read alone; owned is returned only
+// after the create-only or replace Put succeeded, and any Put failure is
+// unknown, so a box whose writes fail cannot renew its own authority.
func DecideLabel(ctx context.Context, s release.Store, b Box, now time.Time) (Decision, error) {
- d := Decision{Verdict: VerdictUnknown, Label: b.Label, BoxID: b.BoxID, DecidedAt: now}
+ unknown := Decision{Verdict: VerdictUnknown, Label: b.Label, BoxID: b.BoxID, DecidedAt: now}
if b.BoxID == "" {
- return d, fmt.Errorf("%w: this box has no machine id", ErrNotOwned)
+ return unknown, fmt.Errorf("%w: this box has no machine id", ErrNotOwned)
}
key := "status/boxes/" + b.Label + ".json"
body, err := json.MarshalIndent(b, "", " ")
if err != nil {
- return d, err
+ return unknown, err
+ }
+ myInstance := ""
+ if b.Metrics != nil {
+ myInstance = b.Metrics.InstanceID
}
var lastErr error
for attempt := 0; attempt < decideAttempts; attempt++ {
+ d := Decision{Verdict: VerdictUnknown, Label: b.Label, BoxID: b.BoxID, DecidedAt: now}
raw, etag, err := s.Get(ctx, key)
switch {
case errors.Is(err, os.ErrNotExist):
- if _, err := s.Put(ctx, key, body, ""); err != nil {
- if errors.Is(err, release.ErrCASConflict) {
- lastErr = err
- continue
- }
- return d, err
+ _, err := s.Put(ctx, key, body, "")
+ if err == nil {
+ d.Verdict = VerdictOwned
+ return d, nil
}
- d.Verdict = VerdictOwned
- return d, nil
+ if errors.Is(err, release.ErrCASConflict) {
+ lastErr = err
+ continue
+ }
+ return d, fmt.Errorf("status: %s: create: %w", key, err)
case err != nil:
return d, err
}
@@ -467,43 +569,40 @@ func DecideLabel(ctx context.Context, s release.Store, b Box, now time.Time) (De
d.HolderInstanceID = prior.Metrics.InstanceID
}
d.HolderFresh = !BoxStale(prior, now, SnapshotPeriod)
- myInstance := ""
- if b.Metrics != nil {
- myInstance = b.Metrics.InstanceID
+ if v := classify(b.BoxID, myInstance, prior.BoxID, d.HolderInstanceID); v != VerdictOwned {
+ d.Verdict = v
+ return d, nil
}
- d.Verdict = classify(b.BoxID, myInstance, prior.BoxID, d.HolderInstanceID)
- if d.Verdict != VerdictOwned {
+ _, err = s.Put(ctx, key, body, etag)
+ if err == nil {
+ d.Verdict = VerdictOwned
return d, nil
}
- if _, err := s.Put(ctx, key, body, etag); err != nil {
- if errors.Is(err, release.ErrCASConflict) {
- lastErr = err
- d.Verdict = VerdictUnknown
- continue
- }
- return d, err
+ if errors.Is(err, release.ErrCASConflict) {
+ lastErr = err
+ continue
}
- return d, nil
+ return d, fmt.Errorf("status: %s: replace: %w", key, err)
}
- return d, fmt.Errorf("status: %s: lost the CAS race %d times: %w", key, decideAttempts, lastErr)
+ return unknown, fmt.Errorf("status: %s: lost the CAS race %d times: %w", key, decideAttempts, lastErr)
}
-// classify is the 2.1 table without I/O, so every row is a unit test.
+// classify is the 2.1 table without I/O.
func classify(myID, myInstance, theirID, theirInstance string) Verdict {
- bothInstances := myInstance != "" && theirInstance != ""
+ both := myInstance != "" && theirInstance != ""
switch {
case theirID == "":
switch {
- case bothInstances && myInstance == theirInstance:
+ case both && myInstance == theirInstance:
return VerdictOwned
- case bothInstances:
+ case both:
return VerdictForeign
default:
return VerdictUnproven
}
case !boxIDRe.MatchString(theirID) || theirID != myID:
return VerdictForeign
- case bothInstances && myInstance != theirInstance:
+ case both && myInstance != theirInstance:
return VerdictTwin
default:
return VerdictOwned
@@ -511,7 +610,7 @@ func classify(myID, myInstance, theirID, theirInstance string) Verdict {
}
```
-Replace `WriteBox`'s body:
+Replace `WriteBox`:
```go
func WriteBox(ctx context.Context, s release.Store, b Box) error {
@@ -524,77 +623,76 @@ func WriteBox(ctx context.Context, s release.Store, b Box) error {
}
return nil
}
-```
-Keep `currentEtag` only if `putOverwrite` still uses it (it does, for `PublishFleet`); otherwise delete it. Fleet boxes: `statusTick` sets `box.Label = labelOverride` and must also clear `box.BoxID = ""`; with an empty `BoxID`, `WriteBox` must still write for them. Add this branch at the top of `WriteBox`, before `DecideLabel`:
-
-```go
- if b.BoxID == "" && b.Label != "" && isFleetLabel(b) {
- return putOverwrite(ctx, s, "status/boxes/"+b.Label+".json", mustJSON(b))
+// WriteFleetBox is the fleet-box path: the label is the operator's and
+// unique by the registry, so there is no decision, only a replace.
+func WriteFleetBox(ctx context.Context, s release.Store, b Box) error {
+ body, err := json.MarshalIndent(b, "", " ")
+ if err != nil {
+ return err
}
+ return putOverwrite(ctx, s, "status/boxes/"+b.Label+".json", body)
+}
```
-where `isFleetLabel` is `b.Fleet` ... there is no such field; instead give `WriteBox` the explicit seam: add `func WriteFleetBox(ctx, s, b) error { return putOverwrite(...) }` and have `statusTick` call it when `labelOverride != ""`. Do not let `WriteBox` guess.
+`putOverwrite` is in `fleet.go:590` and uses `currentEtag`; keep both.
- [ ] **Step 4: Run the tests to verify they pass**
Run: `GOFLAGS=-buildvcs=false go test ./internal/status/ -v`
-Expected: PASS, including `TestWriteBoxOverwritesOnSecondTick` (same machine, same `BoxID`, so owned) and the dry run test (one box, not found then owned).
-
-- [ ] **Step 5: Add the zero-`generated_at` and malformed-`box_id` rows (Review Focus 1, 2)**
+Expected: PASS, including `TestWriteBoxOverwritesOnSecondTick` (same machine, owned) and the dry-run tests (one box: not found, then owned).
-They are already rows in `TestDecideLabelTable` ("foreign zero generated_at", "malformed box_id is foreign"). Confirm both pass; if `classify` let `not-hex` through as owned, the regexp check is wrong.
-
-- [ ] **Step 6: Commit**
+- [ ] **Step 5: Commit**
```bash
-git add internal/status/decide.go internal/status/decide_test.go internal/status/snapshot.go internal/status/status_test.go
+git add internal/status/decide.go internal/status/decide_test.go internal/status/snapshot.go
git commit -m "status: DecideLabel refuses to overwrite another machine's snapshot (#212)"
```
---
-### Task 3: the identity file, `Publishable`, `gatedStore`, `ClaimEligible`
+### Task 3: the identity file, `Publishable`, `Gated`, `ClaimEligible`
**Files:**
- Create: `internal/host/identity.go`, `internal/host/identity_test.go`
-- Modify: `internal/host/metrics_loop.go:729-743` (`metricsBoxLabel`, `BoxLabel` keep working; add `boxIdentity`)
+- Modify: `internal/host/metrics_loop.go:729-743` (`metricsBoxLabel`, `BoxLabel` become wrappers)
**Interfaces:**
- Produces:
```go
-// StoreTarget names the arbitration namespace: the status store a verdict
-// was decided against. Bucket is "" and Dir set for a directory deployment.
-type StoreTarget struct {
+type StoreTarget struct { // the publication namespace: the deployment bucket or directory, at the root
Bucket string `json:"bucket"`
- Prefix string `json:"prefix"`
Dir string `json:"dir,omitempty"`
}
+func DeployTarget(getenv func(string) string) StoreTarget // CARLOS_DEPLOYMENT_BUCKET, else CARLOS_DEPLOYMENT_DIR
type BoxIdentity struct {
- ID string // status.BoxID(machineID); "" when unidentified; the fleet label on a fleet box
- Label string
- Fleet bool
+ ID, Label string // fleet box: both the fleet label; platform: BoxID and Label of the machine id; unidentified: both ""
+ Fleet bool
}
func boxIdentity(getenv func(string) string, machineID string) BoxIdentity
+func Identity() BoxIdentity // boxIdentity(os.Getenv, readMachineID())
+const (
+ VerdictFleet status.Verdict = "fleet"
+ VerdictConfigured status.Verdict = "configured"
+ mutationTTL = 60 * time.Second
+)
type identityFile struct {
V int `json:"v"`
BoxID string `json:"box_id"`
Label string `json:"label"`
- Verdict status.Verdict `json:"verdict"` // plus "fleet"
+ Verdict status.Verdict `json:"verdict"`
+ Reason string `json:"reason,omitempty"`
DecidedAt time.Time `json:"decided_at"`
Store StoreTarget `json:"store"`
}
-const VerdictFleet status.Verdict = "fleet"
func writeIdentity(dir string, f identityFile) error
func readIdentity(dir string) (identityFile, error)
-// Publishable: spec 2.2. myID is status.BoxID(readMachineID()) of the CALLER.
-func publishable(dir string, myID, label string, target StoreTarget, now time.Time) bool
-func Publishable(label string, target StoreTarget) bool // dataDir(), readMachineID(), time.Now()
+func publishable(dir string, me BoxIdentity, label string, target StoreTarget, now time.Time) bool
+func Publishable(label string) bool // dataDir(), Identity(), DeployTarget(os.Getenv), time.Now()
var ErrNotPublishable = errors.New("host: not publishable under this label")
-// gatedStore refuses Put and Delete unless publishable; Get/List pass.
-func gated(s release.Store, label string, target StoreTarget) release.Store
-func ClaimEligible() bool // identified && verdict != twin && Publishable(label, target)
+func Gated(s release.Store, label string) release.Store // refuses Put/Delete/DeleteIfMatch unless Publishable; mutationTTL deadline
+func ClaimEligible() bool
```
- [ ] **Step 1: Write the failing tests**
@@ -606,6 +704,8 @@ package host
import (
"context"
"errors"
+ "os"
+ "path/filepath"
"testing"
"time"
@@ -613,7 +713,7 @@ import (
"github.com/carlosframework/platform/internal/status"
)
-func TestBoxIdentityPlatformAndFleet(t *testing.T) {
+func TestBoxIdentityPlatformFleetAndUnidentified(t *testing.T) {
p := boxIdentity(envFunc(map[string]string{}), "m")
if p.ID != status.BoxID("m") || p.Label != status.Label("m") || p.Fleet {
t.Fatalf("platform identity = %+v", p)
@@ -627,72 +727,87 @@ func TestBoxIdentityPlatformAndFleet(t *testing.T) {
}
}
-func TestPublishableBindsMachineLabelStoreAndAge(t *testing.T) {
+func TestDeployTarget(t *testing.T) {
+ if got := DeployTarget(envFunc(map[string]string{"CARLOS_DEPLOYMENT_BUCKET": "d"})); got != (StoreTarget{Bucket: "d"}) {
+ t.Fatalf("bucket target = %+v", got)
+ }
+ if got := DeployTarget(envFunc(map[string]string{"CARLOS_DEPLOYMENT_DIR": "/x"})); got != (StoreTarget{Dir: "/x"}) {
+ t.Fatalf("dir target = %+v", got)
+ }
+}
+
+func TestPublishableBindsMachineLabelStoreVerdictAndAge(t *testing.T) {
dir := t.TempDir()
now := time.Now()
- target := StoreTarget{Bucket: "b", Prefix: ""}
- me := status.BoxID("m")
+ target := StoreTarget{Bucket: "b"}
+ me := BoxIdentity{ID: status.BoxID("m"), Label: "annie-welding"}
write := func(f identityFile) {
t.Helper()
if err := writeIdentity(dir, f); err != nil {
t.Fatal(err)
}
}
- base := identityFile{V: 1, BoxID: me, Label: "annie-welding", Verdict: status.VerdictOwned, DecidedAt: now, Store: target}
+ base := identityFile{V: 1, BoxID: me.ID, Label: me.Label, Verdict: status.VerdictOwned, DecidedAt: now, Store: target}
write(base)
- if !publishable(dir, me, "annie-welding", target, now) {
+ if !publishable(dir, me, me.Label, target, now) {
t.Fatal("owned, fresh, same machine, label and store must be publishable")
}
- if publishable(dir, status.BoxID("other"), "annie-welding", target, now) {
+ if publishable(dir, BoxIdentity{ID: status.BoxID("other"), Label: me.Label}, me.Label, target, now) {
t.Fatal("another machine's file must not authorise this one")
}
if publishable(dir, me, "bea-sailing", target, now) {
t.Fatal("another label must not be authorised")
}
- if publishable(dir, me, "annie-welding", StoreTarget{Bucket: "c"}, now) {
+ if publishable(dir, me, me.Label, StoreTarget{Bucket: "c"}, now) {
t.Fatal("another store must not be authorised")
}
- if publishable(dir, me, "annie-welding", target, now.Add(3*status.SnapshotPeriod+time.Second)) {
- t.Fatal("a decision older than three periods must not be publishable")
+ if publishable(dir, me, me.Label, target, now.Add(3*status.SnapshotPeriod+time.Second)) {
+ t.Fatal("an owned decision older than three periods must not be publishable")
}
for _, v := range []status.Verdict{status.VerdictForeign, status.VerdictTwin, status.VerdictUnproven, status.VerdictUnknown} {
f := base
f.Verdict = v
write(f)
- if publishable(dir, me, "annie-welding", target, now) {
+ if publishable(dir, me, me.Label, target, now) {
t.Fatalf("%s must not be publishable", v)
}
}
- fleet := base
- fleet.Verdict, fleet.BoxID, fleet.Label = VerdictFleet, "desk", "desk"
+ fleet := identityFile{V: 1, BoxID: "desk", Label: "desk", Verdict: VerdictFleet, DecidedAt: now.Add(-48 * time.Hour), Store: target}
write(fleet)
- if !publishable(dir, "desk", "desk", target, now) {
- t.Fatal("a fleet box is publishable for its fleet label")
+ if !publishable(dir, BoxIdentity{ID: "desk", Label: "desk", Fleet: true}, "desk", target, now) {
+ t.Fatal("a fleet box is publishable for its fleet label, and the fleet verdict does not expire")
+ }
+ configured := identityFile{V: 1, BoxID: me.ID, Label: me.Label, Verdict: VerdictConfigured, DecidedAt: now.Add(-48 * time.Hour), Store: StoreTarget{Dir: "/x"}}
+ write(configured)
+ if !publishable(dir, me, me.Label, StoreTarget{Dir: "/x"}, now) {
+ t.Fatal("a directory deployment's configured verdict does not expire")
}
}
-func TestPublishableRefusesMissingOrForeignShapedFile(t *testing.T) {
+func TestPublishableRefusesMissingOrOlderShapedFile(t *testing.T) {
dir := t.TempDir()
- if publishable(dir, status.BoxID("m"), "annie-welding", StoreTarget{Bucket: "b"}, time.Now()) {
+ me := BoxIdentity{ID: status.BoxID("m"), Label: "annie-welding"}
+ if publishable(dir, me, me.Label, StoreTarget{Bucket: "b"}, time.Now()) {
t.Fatal("no file must mean not publishable")
}
if err := os.WriteFile(filepath.Join(dir, "identity.json"), []byte(`{"v":0,"label":"annie-welding","verdict":"owned"}`), 0o600); err != nil {
t.Fatal(err)
}
- if publishable(dir, status.BoxID("m"), "annie-welding", StoreTarget{Bucket: "b"}, time.Now()) {
- t.Fatal("an older-shaped file (v 0, no store, no box_id) must not be publishable")
+ if publishable(dir, me, me.Label, StoreTarget{Bucket: "b"}, time.Now()) {
+ t.Fatal("an older-shaped file must not be publishable")
}
}
-func TestGatedStoreRefusesWritesAndPassesReads(t *testing.T) {
+func TestGatedStoreRefusesWritesPassesReadsAndBoundsMutations(t *testing.T) {
dir := t.TempDir()
- t.Setenv("CARLOS_DATA_DIR", dir)
- inner := release.NewDirStore(t.TempDir())
+ inner := &deadlineSpy{Store: release.NewDirStore(t.TempDir())}
ctx := context.Background()
- if _, err := inner.Put(ctx, "k", []byte("x"), ""); err != nil {
+ if _, err := inner.Store.Put(ctx, "k", []byte("x"), ""); err != nil {
t.Fatal(err)
}
- g := gated(inner, "annie-welding", StoreTarget{Bucket: "b"})
+ me := BoxIdentity{ID: status.BoxID("m"), Label: "annie-welding"}
+ target := StoreTarget{Bucket: "b"}
+ g := &gatedStore{Store: inner, label: me.Label, me: func() BoxIdentity { return me }, target: func() StoreTarget { return target }, dir: func() string { return dir }, now: time.Now}
if _, _, err := g.Get(ctx, "k"); err != nil {
t.Fatalf("Get must pass: %v", err)
}
@@ -702,29 +817,57 @@ func TestGatedStoreRefusesWritesAndPassesReads(t *testing.T) {
if err := g.Delete(ctx, "k"); !errors.Is(err, ErrNotPublishable) {
t.Fatalf("Delete without a verdict = %v", err)
}
- if _, _, err := inner.Get(ctx, "k"); err != nil {
+ if err := release.DeleteIfMatch(ctx, g, "k", `"e"`); !errors.Is(err, ErrNotPublishable) {
+ t.Fatalf("DeleteIfMatch without a verdict = %v", err)
+ }
+ if _, _, err := inner.Store.Get(ctx, "k"); err != nil {
t.Fatal("the refused Delete must not have deleted")
}
+ if err := writeIdentity(dir, identityFile{V: 1, BoxID: me.ID, Label: me.Label, Verdict: status.VerdictOwned, DecidedAt: time.Now(), Store: target}); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := g.Put(ctx, "k2", []byte("y"), ""); err != nil {
+ t.Fatalf("Put with an owned verdict = %v", err)
+ }
+ if d, ok := inner.lastDeadline.Deadline(); !ok || time.Until(d) > mutationTTL+time.Second {
+ t.Fatalf("every gated mutation carries a %s deadline, got %v %v", mutationTTL, d, ok)
+ }
}
-```
-`gated` needs the caller's machine id; in tests `readMachineID()` reads `/etc/machine-id`, so give `gatedStore` an injectable `myID func() string` and have `gated` pass `func() string { return status.BoxID(readMachineID()) }`. In this test, write an identity file for `status.BoxID(readMachineID())` to prove the positive path too:
+// deadlineSpy records the context the last Put received.
+type deadlineSpy struct {
+ release.Store
+ lastDeadline context.Context
+}
-```go
- me := status.BoxID(readMachineID())
- if me != "" {
- if err := writeIdentity(dir, identityFile{V: 1, BoxID: me, Label: "annie-welding", Verdict: status.VerdictOwned, DecidedAt: time.Now(), Store: StoreTarget{Bucket: "b"}}); err != nil {
- t.Fatal(err)
- }
- if _, err := g.Put(ctx, "k2", []byte("y"), ""); err != nil {
- t.Fatalf("Put with an owned verdict = %v", err)
+func (d *deadlineSpy) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ d.lastDeadline = ctx
+ return d.Store.Put(ctx, key, body, ifMatch)
+}
+
+func TestClaimEligible(t *testing.T) {
+ dir := t.TempDir()
+ t.Setenv("CARLOS_DATA_DIR", dir)
+ t.Setenv("CARLOS_DEPLOYMENT_BUCKET", "b")
+ me := Identity()
+ if me.ID == "" {
+ t.Skip("no /etc/machine-id on this host")
+ }
+ if ClaimEligible() {
+ t.Fatal("no identity file: not eligible")
+ }
+ for v, want := range map[status.Verdict]bool{status.VerdictOwned: true, status.VerdictTwin: false, status.VerdictForeign: false, status.VerdictUnproven: false} {
+ _ = writeIdentity(dir, identityFile{V: 1, BoxID: me.ID, Label: me.Label, Verdict: v, DecidedAt: time.Now(), Store: StoreTarget{Bucket: "b"}})
+ if got := ClaimEligible(); got != want {
+ t.Fatalf("%s: eligible = %v, want %v", v, got, want)
}
}
+}
```
- [ ] **Step 2: Run the tests to verify they fail**
-Run: `GOFLAGS=-buildvcs=false go test ./internal/host/ -run 'BoxIdentity|Publishable|GatedStore' -v`
+Run: `GOFLAGS=-buildvcs=false go test ./internal/host/ -run 'BoxIdentity|DeployTarget|Publishable|GatedStore|ClaimEligible' -v`
Expected: FAIL, undefined symbols.
- [ ] **Step 3: Implement `identity.go`**
@@ -748,20 +891,28 @@ import (
type StoreTarget struct {
Bucket string `json:"bucket"`
- Prefix string `json:"prefix"`
Dir string `json:"dir,omitempty"`
}
+// DeployTarget is the publication namespace: every label-keyed key lives
+// in the deployment bucket (or directory), and spec 2.6 requires the label
+// to be decided in that same bucket at the root.
+func DeployTarget(getenv func(string) string) StoreTarget {
+ if b := getenv("CARLOS_DEPLOYMENT_BUCKET"); b != "" {
+ return StoreTarget{Bucket: b}
+ }
+ return StoreTarget{Dir: getenv("CARLOS_DEPLOYMENT_DIR")}
+}
+
type BoxIdentity struct {
- ID string
- Label string
- Fleet bool
+ ID, Label string
+ Fleet bool
}
-// boxIdentity replaces metricsBoxLabel as the one selector. On a fleet box
-// both fields are the operator's fleet_boxes.label (fleetidentity.go); on
-// a platform box ID is status.BoxID and Label status.Label of the machine
-// id; an unreadable machine id gives neither.
+// boxIdentity is the one selector (it replaces metricsBoxLabel). A fleet
+// box is its operator-chosen fleet_boxes.label on both sides; a platform
+// box is status.BoxID and status.Label of its machine id; an unreadable
+// machine id gives neither.
func boxIdentity(getenv func(string) string, machineID string) BoxIdentity {
if identity, err := resolveFleetIdentity(getenv); err == nil {
return BoxIdentity{ID: identity.Label, Label: identity.Label, Fleet: true}
@@ -772,19 +923,25 @@ func boxIdentity(getenv func(string) string, machineID string) BoxIdentity {
return BoxIdentity{ID: status.BoxID(machineID), Label: status.Label(machineID)}
}
-const VerdictFleet status.Verdict = "fleet"
+func Identity() BoxIdentity { return boxIdentity(os.Getenv, readMachineID()) }
+
+const (
+ VerdictFleet status.Verdict = "fleet"
+ VerdictConfigured status.Verdict = "configured"
+ identityFileV = 1
+ mutationTTL = 60 * time.Second
+)
type identityFile struct {
V int `json:"v"`
BoxID string `json:"box_id"`
Label string `json:"label"`
Verdict status.Verdict `json:"verdict"`
+ Reason string `json:"reason,omitempty"`
DecidedAt time.Time `json:"decided_at"`
Store StoreTarget `json:"store"`
}
-const identityFileV = 1
-
func identityPath(dir string) string { return filepath.Join(dir, "identity.json") }
func writeIdentity(dir string, f identityFile) error {
@@ -806,30 +963,32 @@ func readIdentity(dir string) (identityFile, error) {
if err != nil {
return f, err
}
- if err := json.Unmarshal(body, &f); err != nil {
- return f, err
- }
- return f, nil
+ return f, json.Unmarshal(body, &f)
}
-// publishable is the gate: this machine's file, this exact label, this
-// store, owned (or a fleet box's own label), decided within three periods.
-func publishable(dir, myID, label string, target StoreTarget, now time.Time) bool {
+// publishable: this machine's file, this exact label, this namespace,
+// and a verdict that permits writing: owned within three periods, or the
+// non-expiring fleet and configured verdicts.
+func publishable(dir string, me BoxIdentity, label string, target StoreTarget, now time.Time) bool {
f, err := readIdentity(dir)
if err != nil || f.V != identityFileV {
return false
}
- if myID == "" || f.BoxID != myID || f.Label == "" || f.Label != label || f.Store != target {
+ if me.ID == "" || f.BoxID != me.ID || label == "" || f.Label != label || f.Store != target {
return false
}
- if f.Verdict != status.VerdictOwned && f.Verdict != VerdictFleet {
+ switch f.Verdict {
+ case VerdictFleet, VerdictConfigured:
+ return true
+ case status.VerdictOwned:
+ return now.Sub(f.DecidedAt) <= 3*status.SnapshotPeriod
+ default:
return false
}
- return now.Sub(f.DecidedAt) <= 3*status.SnapshotPeriod
}
-func Publishable(label string, target StoreTarget) bool {
- return publishable(dataDir(), status.BoxID(readMachineID()), label, target, time.Now())
+func Publishable(label string) bool {
+ return publishable(dataDir(), Identity(), label, DeployTarget(os.Getenv), time.Now())
}
var ErrNotPublishable = errors.New("host: not publishable under this label (identity.json says this machine does not hold it)")
@@ -837,144 +996,234 @@ var ErrNotPublishable = errors.New("host: not publishable under this label (iden
type gatedStore struct {
release.Store
label string
- target StoreTarget
- myID func() string
+ me func() BoxIdentity
+ target func() StoreTarget
dir func() string
now func() time.Time
}
-func gated(s release.Store, label string, target StoreTarget) release.Store {
- return &gatedStore{Store: s, label: label, target: target,
- myID: func() string { return status.BoxID(readMachineID()) }, dir: dataDir, now: time.Now}
+// Gated wraps s so every mutation under label is refused unless
+// Publishable, and runs under a mutationTTL deadline (spec 2.2).
+func Gated(s release.Store, label string) release.Store {
+ return &gatedStore{Store: s, label: label, me: Identity,
+ target: func() StoreTarget { return DeployTarget(os.Getenv) }, dir: dataDir, now: time.Now}
}
-func (g *gatedStore) ok() bool { return publishable(g.dir(), g.myID(), g.label, g.target, g.now()) }
+func (g *gatedStore) admit(ctx context.Context) (context.Context, context.CancelFunc, error) {
+ if !publishable(g.dir(), g.me(), g.label, g.target(), g.now()) {
+ return nil, nil, ErrNotPublishable
+ }
+ ctx, cancel := context.WithTimeout(ctx, mutationTTL)
+ return ctx, cancel, nil
+}
func (g *gatedStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
- if !g.ok() {
- return "", ErrNotPublishable
+ ctx, cancel, err := g.admit(ctx)
+ if err != nil {
+ return "", err
}
+ defer cancel()
return g.Store.Put(ctx, key, body, ifMatch)
}
func (g *gatedStore) Delete(ctx context.Context, key string) error {
- if !g.ok() {
- return ErrNotPublishable
+ ctx, cancel, err := g.admit(ctx)
+ if err != nil {
+ return err
}
+ defer cancel()
return g.Store.Delete(ctx, key)
}
-// ClaimEligible is the host lifecycle claim's gate (spec 4): identified,
-// not a twin, and publishable for this machine's label in the store the
-// agent decided against.
+func (g *gatedStore) DeleteIfMatch(ctx context.Context, key, etag string) error {
+ ctx, cancel, err := g.admit(ctx)
+ if err != nil {
+ return err
+ }
+ defer cancel()
+ return release.DeleteIfMatch(ctx, g.Store, key, etag)
+}
+
+// ClaimEligible is the host lifecycle claim's gate (spec 4).
func ClaimEligible() bool {
+ me := Identity()
+ if me.ID == "" {
+ return false
+ }
f, err := readIdentity(dataDir())
if err != nil || f.Verdict == status.VerdictTwin {
return false
}
- return Publishable(f.Label, f.Store)
+ return Publishable(me.Label)
}
```
-The fleet-box case in the test writes `BoxID: "desk"`; `publishable` compares `f.BoxID != myID` where the caller passes `"desk"` as `myID` for a fleet box, so `gated` on a fleet box must pass the fleet label as `myID`. Make `gated` take the identity: `gated(s, id BoxIdentity, target)` and use `id.ID` as `myID` and `id.Label` as the label. Update the test accordingly (`gated(inner, BoxIdentity{ID: me, Label: "annie-welding"}, ...)`).
-
-`release.Store` also has `ListDirs`; the embedded `Store` supplies it and `List` and `Get`. If `release.ConditionalDeleter` is checked by type assertion anywhere a gated store is passed, add a `DeleteIfMatch` passthrough with the same gate.
+`metricsBoxLabel(getenv, machineID)` becomes `return boxIdentity(getenv, machineID).Label` and `BoxLabel()` becomes `return Identity().Label`; the existing `TestMetricsBoxLabelUsesFleetIdentity` (`metrics_loop_test.go:879`) keeps passing.
- [ ] **Step 4: Run the tests to verify they pass**
-Run: the same command. Expected: PASS. Then `go vet ./internal/host/` and `GOOS=darwin go build ./...` (nothing Linux-only was added).
+Run: the Step 2 command; then `go vet ./internal/host/` and `GOOS=darwin GOARCH=arm64 go build ./...`. Expected: PASS, both build.
- [ ] **Step 5: Commit**
```bash
-git add internal/host/identity.go internal/host/identity_test.go
-git commit -m "host: identity.json, Publishable and a gated store for label-keyed writes (#212)"
+git add internal/host/identity.go internal/host/identity_test.go internal/host/metrics_loop.go
+git commit -m "host: identity.json, Publishable, a gated store with a mutation deadline, ClaimEligible (#212)"
```
---
-### Task 4: decide at arm, gate every publisher, in every process
+### Task 4: decide at arm with the complete snapshot, gate every publisher in every process
**Files:**
-- Modify: `internal/host/agent.go` (RunAgent arm order near `:482-500`; `statusLoop` `:2681-2882`; `statusTick` `:2905-2935`; `configAckFunc` `:948-960`; `adopt` `:3486`; sinks at `:244,245,477,480,569`; `edge.WithBoxLabel` `:542`)
-- Modify: `internal/host/edgesfile.go:529-545` (seed), `:626-700` (loop)
-- Modify: `internal/host/metrics_loop.go:799` and `:729-743`, `internal/host/snapshot_loop.go:205`, `internal/host/usage_loop.go:72`, `internal/host/logship.go:39-95`
-- Modify: `relay.go:125-145`, `internal/relay/serve.go:144-215`
-- Test: `internal/host/agent_test.go`, `internal/host/edgesfile_test.go`, `internal/host/snapshot_loop_test.go`, `internal/relay/serve_test.go`
+- Modify: `internal/host/agent.go` (RunAgent arm order `:482-500`; `statusLoop` `:2731-2930`; `statusTick` `:2955-2985`; `configAckFunc` `:953-962`; sinks `:244,245,477,480,569`; `edge.WithBoxLabel` `:542`)
+- Modify: `internal/host/edgesfile.go:545` (EIP hold), `:598-615` (seed), `:696-720` (loop)
+- Modify: `internal/host/metrics_loop.go:799`, `internal/host/snapshot_loop.go:194-220`, `internal/host/usage_loop.go:72`, `internal/host/logship.go:39-95`
+- Modify: `relay.go:125-145`
+- Test: `internal/host/agent_test.go`, `internal/host/edgesfile_test.go`, `internal/host/snapshot_loop_test.go`, `internal/relay/serve_test.go`, `internal/relay/meter_test.go`
**Interfaces:**
-- Consumes: Task 2 `status.DecideLabel`, `status.WriteFleetBox`; Task 3 `boxIdentity`, `writeIdentity`, `gated`, `StoreTarget`, `Publishable`.
-- Produces: `func decideIdentity(ctx, store release.Store, target StoreTarget, id BoxIdentity, probes status.Probes, now time.Time) (status.Decision, error)` which calls `DecideLabel` and `writeIdentity`; `func statusStoreTarget(getenv func(string) string) StoreTarget`; `relay.Config.Publishable func(label string) bool` (nil means always, for tests).
+- Consumes: Task 2 `status.DecideLabel`, `status.WriteFleetBox`; Task 3 everything.
+- Produces:
+
+```go
+// collectSnapshot builds the complete Box statusTick publishes: routes and
+// aliases from reg, probes p (agent.go:2955-2975 today).
+func collectSnapshot(reg *registry.Registry, p status.Probes, now time.Time) (status.Box, []registry.Route, error)
+// decideIdentityIn decides the label against s with the complete snapshot and
+// writes identity.json in dir. Spec 2.1, 2.5, 2.6.
+func decideIdentityIn(ctx context.Context, dir string, s release.Store, target StoreTarget, id BoxIdentity, b status.Box, now time.Time) (status.Decision, error)
+// statusNamespaceOK: spec 2.6. A platform box's status store must be the
+// deployment bucket at the root.
+func statusNamespaceOK(getenv func(string) string) (ok bool, reason string)
+```
- [ ] **Step 1: Write the failing tests**
```go
// internal/host/agent_test.go
func TestDecideIdentityWritesTheFileAndRefusesAForeignLabel(t *testing.T) {
- dir := t.TempDir()
s := release.NewDirStore(t.TempDir())
target := StoreTarget{Bucket: "b"}
now := time.Now()
- a := BoxIdentity{ID: status.BoxID("00000000000000000000000000000018"), Label: status.Label("00000000000000000000000000000018")}
- b := BoxIdentity{ID: status.BoxID("0000000000000000000000000000003c"), Label: a.Label}
- da, err := decideIdentityIn(context.Background(), dir, s, target, a, status.Probes{MachineID: "00000000000000000000000000000018", InstanceID: "i-a"}, now)
- if err != nil || da.Verdict != status.VerdictOwned {
+ const m1, m2 = "00000000000000000000000000000018", "0000000000000000000000000000003c"
+ a, b := boxIdentity(envFunc(map[string]string{}), m1), boxIdentity(envFunc(map[string]string{}), m2)
+ dirA, dirB := t.TempDir(), t.TempDir()
+ snapA := status.CollectBox(nil, status.Probes{MachineID: m1, InstanceID: "i-a"}, now)
+ da, err := decideIdentityIn(context.Background(), dirA, s, target, a, snapA, now)
+ if err != nil || da.Verdict != status.VerdictOwned || !publishable(dirA, a, a.Label, target, now) {
t.Fatalf("a: %s %v", da.Verdict, err)
}
- if !publishable(dir, a.ID, a.Label, target, now) {
- t.Fatal("a must be publishable after an owned decision")
- }
- dirB := t.TempDir()
- db, err := decideIdentityIn(context.Background(), dirB, s, target, b, status.Probes{MachineID: "0000000000000000000000000000003c", InstanceID: "i-b"}, now)
- if err != nil || db.Verdict != status.VerdictForeign {
+ snapB := status.CollectBox(nil, status.Probes{MachineID: m2, InstanceID: "i-b"}, now)
+ db, err := decideIdentityIn(context.Background(), dirB, s, target, b, snapB, now)
+ if err != nil || db.Verdict != status.VerdictForeign || publishable(dirB, b, b.Label, target, now) {
t.Fatalf("b: %s %v", db.Verdict, err)
}
- if publishable(dirB, b.ID, b.Label, target, now) {
- t.Fatal("b must not be publishable")
- }
f, _ := readIdentity(dirB)
if f.Verdict != status.VerdictForeign || f.Store != target {
t.Fatalf("b's identity file = %+v", f)
}
}
-func TestStatusStoreTargetNamesBucketPrefixOrDir(t *testing.T) {
- got := statusStoreTarget(envFunc(map[string]string{"CARLOS_DEPLOYMENT_BUCKET": "d", "CARLOS_STATUS_PREFIX": "fleets/pi/"}))
- if got != (StoreTarget{Bucket: "d", Prefix: "fleets/pi/"}) {
- t.Fatalf("target = %+v", got)
+func TestDecideIdentityFailedWriteDoesNotRenewAuthority(t *testing.T) {
+ inner := release.NewDirStore(t.TempDir())
+ dir := t.TempDir()
+ target := StoreTarget{Bucket: "b"}
+ now := time.Now()
+ me := boxIdentity(envFunc(map[string]string{}), "m")
+ snap := status.CollectBox(nil, status.Probes{MachineID: "m", InstanceID: "i-m"}, now)
+ if _, err := decideIdentityIn(context.Background(), dir, inner, target, me, snap, now); err != nil {
+ t.Fatal(err)
+ }
+ later := now.Add(2 * status.SnapshotPeriod)
+ failing := &failingPutStore{Store: inner} // every Put fails with a non-CAS error
+ d, _ := decideIdentityIn(context.Background(), dir, failing, target, me, status.CollectBox(nil, status.Probes{MachineID: "m", InstanceID: "i-m"}, later), later)
+ if d.Verdict != status.VerdictUnknown {
+ t.Fatalf("a failed replace is unknown, got %s", d.Verdict)
+ }
+ f, _ := readIdentity(dir)
+ if f.Verdict != status.VerdictUnknown {
+ t.Fatalf("identity.json must record unknown, not renew owned: %+v", f)
+ }
+ if publishable(dir, me, me.Label, target, now.Add(3*status.SnapshotPeriod+time.Second)) {
+ t.Fatal("authority expires three periods after the last OWNED decision")
+ }
+}
+
+func TestStatusNamespaceRule(t *testing.T) {
+ if ok, _ := statusNamespaceOK(envFunc(map[string]string{"CARLOS_DEPLOYMENT_BUCKET": "d"})); !ok {
+ t.Fatal("one bucket at the root is the supported shape")
+ }
+ if ok, _ := statusNamespaceOK(envFunc(map[string]string{"CARLOS_DEPLOYMENT_BUCKET": "d", "CARLOS_STATUS_BUCKET": "d"})); !ok {
+ t.Fatal("an explicit status bucket equal to the deployment bucket is fine")
+ }
+ if ok, reason := statusNamespaceOK(envFunc(map[string]string{"CARLOS_DEPLOYMENT_BUCKET": "d", "CARLOS_STATUS_BUCKET": "s"})); ok || !strings.Contains(reason, "CARLOS_STATUS_BUCKET") {
+ t.Fatalf("a split status bucket is unsupported: %v %q", ok, reason)
+ }
+ if ok, _ := statusNamespaceOK(envFunc(map[string]string{"CARLOS_DEPLOYMENT_BUCKET": "d", "CARLOS_STATUS_PREFIX": "x/"})); ok {
+ t.Fatal("a prefixed status store is unsupported on a platform box")
+ }
+ if ok, _ := statusNamespaceOK(envFunc(map[string]string{"CARLOS_DEPLOYMENT_BUCKET": "d", "CARLOS_FLEET": "pi", "CARLOS_FLEET_BOX": "desk", "CARLOS_STATUS_PREFIX": "fleets/pi/"})); !ok {
+ t.Fatal("fleet boxes are exempt")
+ }
+}
+
+func TestDirectoryDeploymentIsConfiguredAndDoesNotExpire(t *testing.T) {
+ dir := t.TempDir()
+ me := boxIdentity(envFunc(map[string]string{}), "m")
+ target := StoreTarget{Dir: "/deploy"}
+ if err := writeConfiguredIdentity(dir, me, target, time.Now()); err != nil {
+ t.Fatal(err)
}
- got = statusStoreTarget(envFunc(map[string]string{"CARLOS_STATUS_BUCKET": "s", "CARLOS_DEPLOYMENT_BUCKET": "d"}))
- if got.Bucket != "s" {
- t.Fatalf("CARLOS_STATUS_BUCKET must win: %+v", got)
+ if !publishable(dir, me, me.Label, target, time.Now().Add(72*time.Hour)) {
+ t.Fatal("configured authority must survive three days without a status loop")
}
}
```
```go
// internal/host/edgesfile_test.go
-func TestSeedEdgesFileGateDiscardsAForeignSeedAndItsEIP(t *testing.T) {
+func TestSeedEdgesFileGateKeepsOnlyOurOwnSeed(t *testing.T) {
s := release.NewDirStore(t.TempDir())
ctx := context.Background()
- foreign, _ := json.Marshal(edgesFile{V: 1, Box: "annie", BoxID: "ffffffffffffffffffffffffffffffff", EIP: "9.9.9.9", Hosts: map[string]edgeHost{}})
- if _, err := s.Put(ctx, edgesFileKey("annie"), foreign, ""); err != nil {
- t.Fatal(err)
+ const mine = "0123456789abcdef0123456789abcdef"
+ write := func(f edgesFile) {
+ t.Helper()
+ body, _ := json.Marshal(f)
+ _, etag, _ := s.Get(ctx, edgesFileKey("annie"))
+ if _, err := s.Put(ctx, edgesFileKey("annie"), body, etag); err != nil {
+ t.Fatal(err)
+ }
}
- if got := seedEdgesFileGate(ctx, s, "annie", "0123456789abcdef0123456789abcdef"); got != nil {
- t.Fatalf("a seed written by another machine must be discarded, got %s", got)
+ write(edgesFile{V: 1, Box: "annie", BoxID: "ffffffffffffffffffffffffffffffff", EIP: "9.9.9.9", Hosts: map[string]edgeHost{}})
+ if got := seedEdgesFileGate(ctx, s, "annie", mine); got != nil {
+ t.Fatalf("another machine's seed must be discarded, got %s", got)
}
- own, _ := json.Marshal(edgesFile{V: 1, Box: "annie", BoxID: "0123456789abcdef0123456789abcdef", EIP: "1.2.3.4", Hosts: map[string]edgeHost{}})
- _, etag, _ := s.Get(ctx, edgesFileKey("annie"))
- if _, err := s.Put(ctx, edgesFileKey("annie"), own, etag); err != nil {
- t.Fatal(err)
+ write(edgesFile{V: 1, Box: "annie", EIP: "9.9.9.9", Hosts: map[string]edgeHost{}})
+ if got := seedEdgesFileGate(ctx, s, "annie", mine); got != nil {
+ t.Fatalf("a seed with no box_id is unidentifiable and must be discarded, got %s", got)
}
- if got := seedEdgesFileGate(ctx, s, "annie", "0123456789abcdef0123456789abcdef"); got == nil {
+ write(edgesFile{V: 1, Box: "annie", BoxID: mine, EIP: "1.2.3.4", Hosts: map[string]edgeHost{}})
+ if got := seedEdgesFileGate(ctx, s, "annie", mine); got == nil {
t.Fatal("our own seed must be kept")
}
- legacy, _ := json.Marshal(edgesFile{V: 1, Box: "annie", EIP: "1.2.3.4", Hosts: map[string]edgeHost{}})
- _, etag, _ = s.Get(ctx, edgesFileKey("annie"))
- _, _ = s.Put(ctx, edgesFileKey("annie"), legacy, etag)
- if got := seedEdgesFileGate(ctx, s, "annie", "0123456789abcdef0123456789abcdef"); got == nil {
- t.Fatal("a pre-box_id seed (our own old file) is kept; the status decision already proved the label")
+}
+
+func TestEdgesFilePassNeverInheritsAForeignEIP(t *testing.T) {
+ // A blank probe with a discarded (nil) seed publishes "" rather than
+ // another machine's address.
+ s := release.NewDirStore(t.TempDir())
+ reg := edgesTestRegistry(t)
+ var last []byte // nil: the seed was discarded
+ ident := edgesFile{Box: "annie", BoxID: "0123456789abcdef0123456789abcdef"}
+ if err := edgesFilePass(context.Background(), s, reg, ident, "", &last); err != nil {
+ t.Fatal(err)
+ }
+ body, _, _ := s.Get(context.Background(), edgesFileKey("annie"))
+ var got edgesFile
+ _ = json.Unmarshal(body, &got)
+ if got.EIP != "" {
+ t.Fatalf("published EIP = %q, want none", got.EIP)
}
}
```
@@ -982,18 +1231,19 @@ func TestSeedEdgesFileGateDiscardsAForeignSeedAndItsEIP(t *testing.T) {
```go
// internal/host/snapshot_loop_test.go
func TestRegistrySnapshotAbortsMidSequenceWhenTheGateCloses(t *testing.T) {
- // A fake store that flips a shared *bool after the first Put; the
- // gatedStore's ok() reads the same bool through an injected dir whose
- // identity.json the test rewrites to foreign on that flip.
dir := t.TempDir()
- me := status.BoxID("m")
+ me := BoxIdentity{ID: status.BoxID("m"), Label: "annie"}
target := StoreTarget{Bucket: "b"}
- _ = writeIdentity(dir, identityFile{V: 1, BoxID: me, Label: "annie", Verdict: status.VerdictOwned, DecidedAt: time.Now(), Store: target})
+ open := func() {
+ _ = writeIdentity(dir, identityFile{V: 1, BoxID: me.ID, Label: me.Label, Verdict: status.VerdictOwned, DecidedAt: time.Now(), Store: target})
+ }
+ open()
inner := &flipStore{Store: release.NewDirStore(t.TempDir()), onFirstPut: func() {
- _ = writeIdentity(dir, identityFile{V: 1, BoxID: me, Label: "annie", Verdict: status.VerdictForeign, DecidedAt: time.Now(), Store: target})
+ _ = writeIdentity(dir, identityFile{V: 1, BoxID: me.ID, Label: me.Label, Verdict: status.VerdictForeign, DecidedAt: time.Now(), Store: target})
}}
- g := &gatedStore{Store: inner, label: "annie", target: target, myID: func() string { return me }, dir: func() string { return dir }, now: time.Now}
- err := snapshotPass(context.Background(), g, newTestRegistry(t), "annie", time.Now()) // the existing per-tick function
+ g := &gatedStore{Store: inner, label: me.Label, me: func() BoxIdentity { return me }, target: func() StoreTarget { return target }, dir: func() string { return dir }, now: time.Now}
+ var fp string
+ err := registrySnapshotPass(context.Background(), g, snapTestRegistry(t), "annie", time.Now(), &fp)
if !errors.Is(err, ErrNotPublishable) {
t.Fatalf("the second write must be refused: %v", err)
}
@@ -1001,67 +1251,107 @@ func TestRegistrySnapshotAbortsMidSequenceWhenTheGateCloses(t *testing.T) {
t.Fatalf("exactly one write before the gate closed, got %d", inner.puts)
}
}
-```
-(`flipStore` is a ten-line test type: embeds `release.Store`, counts `puts`, calls `onFirstPut` after the first successful `Put`. `snapshotPass` is whatever `registrySnapshotLoop` calls per tick; if the loop body is inline, extract it as part of this task.)
+// flipStore counts Puts and runs onFirstPut after the first succeeds.
+type flipStore struct {
+ release.Store
+ puts int
+ onFirstPut func()
+}
+
+func (f *flipStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ etag, err := f.Store.Put(ctx, key, body, ifMatch)
+ if err == nil {
+ f.puts++
+ if f.puts == 1 && f.onFirstPut != nil {
+ f.onFirstPut()
+ }
+ }
+ return etag, err
+}
+```
```go
-// internal/relay/serve_test.go
-func TestServeRefusesAMismatchedRelayBox(t *testing.T) {
- cfg := testConfig(t) // the existing helper
- cfg.Box = "bea-sailing"
- cfg.Publishable = func(label string) bool { return label == "annie-welding" }
- var logged []string
- err := Serve(ctx(t), cfg, func(f string, a ...any) { logged = append(logged, fmt.Sprintf(f, a...)) })
- // Serve returns when ctx ends; assert no heartbeat object was written
- // and the mismatch was logged once.
- if _, _, err := cfg.Store.Get(context.Background(), HeartbeatKey("bea-sailing")); !errors.Is(err, os.ErrNotExist) {
- t.Fatal("a relay whose box label this machine does not hold must publish no heartbeat")
- }
- if n := countContaining(logged, "not publishing"); n != 1 {
- t.Fatalf("mismatch logged %d times, want once", n)
+// internal/relay/meter_test.go
+func TestMeterFlushRetriesAreRefusedWhenTheStoreRefuses(t *testing.T) {
+ // The relay never sees host's gate; it sees a release.Store whose Put
+ // refuses. Flip refusal on between the read and the write.
+ inner := release.NewDirStore(t.TempDir())
+ m := NewMeter("annie", func(string, ...any) {})
+ m.SetAllocsAll(map[Identity]int{{Account: "a", App: "x", Incarnation: "1"}: 1})
+ s := &refuseAfterGet{Store: inner}
+ err := m.Flush(context.Background(), s)
+ if !errors.Is(err, errRefused) {
+ t.Fatalf("flush = %v, want the refusal surfaced", err)
+ }
+ keys, _ := inner.List(context.Background(), "metrics/relay/")
+ if len(keys) != 0 {
+ t.Fatalf("nothing may be written after refusal, got %v", keys)
+ }
+}
+
+var errRefused = errors.New("refused")
+
+type refuseAfterGet struct {
+ release.Store
+ got bool
+}
+
+func (r *refuseAfterGet) Get(ctx context.Context, key string) ([]byte, string, error) {
+ r.got = true
+ return r.Store.Get(ctx, key)
+}
+
+func (r *refuseAfterGet) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ if r.got {
+ return "", errRefused
}
+ return r.Store.Put(ctx, key, body, ifMatch)
}
```
+(`NewMeter` and `SetAllocsAll` are the meter's existing constructors; use whatever `serve.go:354` builds `meter` with.)
+
- [ ] **Step 2: Run the tests to verify they fail**
-Run: `GOFLAGS=-buildvcs=false go test ./internal/host/ ./internal/relay/ -run 'DecideIdentity|StatusStoreTarget|DiscardsAForeignSeed|AbortsMidSequence|MismatchedRelayBox' -v`
-Expected: FAIL, undefined symbols and signature mismatches.
+Run: `GOFLAGS=-buildvcs=false go test ./internal/host/ ./internal/relay/ -run 'DecideIdentity|StatusNamespace|DirectoryDeployment|KeepsOnlyOurOwnSeed|NeverInheritsAForeignEIP|AbortsMidSequence|RetriesAreRefused' -v`
+Expected: FAIL, undefined symbols and the seed test keeping a `box_id`-less seed.
- [ ] **Step 3: Implement, in this order**
-a. `statusStoreTarget(getenv)` in `agent.go` next to `ResolveStatusBucket` (`:2602`): `Bucket: ResolveStatusBucket(getenv)`, `Prefix: resolveStatusPrefix(getenv)`, `Dir: getenv("CARLOS_DEPLOYMENT_DIR")` when the bucket is empty.
+a. `statusNamespaceOK(getenv)` in `agent.go` beside `ResolveStatusBucket` (`:2602`): fleet boxes (`resolveFleetIdentity` ok) return true; otherwise ok when `CARLOS_STATUS_PREFIX == ""` and (`CARLOS_STATUS_BUCKET == ""` or equals `CARLOS_DEPLOYMENT_BUCKET`); the reason is the spec §2.6 log text with both values.
+
+b. `collectSnapshot(reg, p, now)` extracted from `statusTick` (`:2955-2975`: `reg.List`, `reg.ListAliases` folded into `p.Aliases`, `status.CollectBox(routes, p, now)`); `statusTick` calls it.
-b. `decideIdentityIn(ctx, dir, store, target, id, probes, now)` in `agent.go`: builds `b := status.CollectBox(nil, probes, now)` with `b.Label, b.BoxID = id.Label, id.ID`, calls `status.DecideLabel`, then `writeIdentity(dir, identityFile{BoxID: id.ID, Label: id.Label, Verdict: d.Verdict, DecidedAt: now, Store: target})` whatever the verdict (unknown included, so a stale file goes stale honestly), and returns the decision. `decideIdentity` is the `dataDir()` wrapper. For `id.Fleet` it writes `VerdictFleet` and skips `DecideLabel`. For `id.ID == ""` it writes nothing and returns `status.VerdictUnknown` with an error naming the unreadable machine id.
+c. `decideIdentityIn(ctx, dir, s, target, id, b, now)`: `id.Fleet` writes `VerdictFleet` and returns owned without deciding; `id.ID == ""` writes nothing and returns unknown with an error; otherwise `b.Label, b.BoxID = id.Label, id.ID`, `d, err := status.DecideLabel(ctx, s, b, now)`, then `writeIdentity(dir, identityFile{BoxID: id.ID, Label: id.Label, Verdict: d.Verdict, DecidedAt: now, Store: target})` whatever the verdict (unknown included: the file must say unknown so `publishable` falls back on the last owned `decided_at`; **do not** rewrite `DecidedAt` on unknown; keep the previous file's `DecidedAt` when the previous verdict was owned and the new one is unknown, so the three-period clock runs from the last owned decision). `writeConfiguredIdentity(dir, id, target, now)` writes `VerdictConfigured`.
-c. **Arm order** in `RunAgent`: before `go statusLoop(reg)` (`:482`), open the status store once (the same `openStatusStore` call `statusLoop` makes), compute `id := boxIdentity(os.Getenv, readMachineID())` and `target := statusStoreTarget(os.Getenv)`, and call `decideIdentity` synchronously with up to three attempts a second apart. Log the verdict once. A directory deployment (`errNoStatusStore`) writes `identityFile{Verdict: status.VerdictOwned, Store: target}` directly with a log line `agent: no status store, so this is a one-box deployment: label <label> owned by configuration`. Only then start the loops.
+d. **Arm order** in `RunAgent`, before `go statusLoop(reg)` (`:482`): `id := Identity()`, `target := DeployTarget(os.Getenv)`. If `statusNamespaceOK` is false: write `identityFile{Verdict: unknown, Reason: reason}` and log the §2.6 line once; proceed (every gated publisher stays closed). Else open the status store once; on `errNoStatusStore` call `writeConfiguredIdentity` and log `agent: no status store, so this is a one-box deployment: label <label> owned by configuration`; else `b, _, _ := collectSnapshot(reg, probes, now)` with the same probes `statusLoop` builds, and `decideIdentityIn` with three attempts a second apart, logging the verdict once. Only then start the loops.
-d. **`statusLoop`**: replace the `lease.Acquire` then `statusTick` order (`:2865-2882`) with: `d, err := decideIdentity(...)` first; if `d.Verdict != status.VerdictOwned` (and not fleet), skip the election and the tick, call `refuseTransition(d)` (Task 5), and continue. Otherwise acquire the lease and run `statusTick` as today, but `statusTick` no longer calls `WriteBox` for a platform box: the decision already wrote the snapshot. Give `statusTick` a `wrote bool` parameter: `false` on fleet boxes, where it calls `status.WriteFleetBox` after `box.Label = labelOverride; box.BoxID = ""`.
+e. **`statusLoop`** (`:2731`): each tick, `b, routes, err := collectSnapshot(...)`, `d, _ := decideIdentityIn(...)`, `refuser.observe(d)` (Task 5). If `d.Verdict != owned` and not fleet: skip the election and `statusTick`; continue. Otherwise acquire the lease (`:2915`) and call `statusTick(ctx, s, reg, b, routes, publish, labelOverride)`, whose signature now takes the collected box and routes instead of collecting; for a fleet box it sets `box.Label = labelOverride; box.BoxID = ""` and calls `status.WriteFleetBox`; for a platform box the decision already wrote the snapshot and `statusTick` only publishes the fleet when `publish`.
-e. **Edges loop** (`edgesFileLoop`, `:626`): `id := boxIdentity(...)`; `ident := edgesFile{Box: id.Label, BoxID: id.ID, ...}`; the store becomes `gated(openDeployStore(...), id, target)`; `seedEdgesFileGate(ctx, s, box, boxID string)` returns nil when the seeded file carries a non-empty `BoxID != boxID`. In the loop, keep `prevPublishable bool`; when `Publishable` flips false to true, set `last = nil` (republish next pass). An `ErrNotPublishable` from `edgesFilePass` is logged once per transition, not per pass.
+f. **Edges loop** (`:696`): `id := Identity()`; `ident := edgesFile{Box: id.Label, BoxID: id.ID, ...}`; `store = Gated(store, id.Label)`; `seedEdgesFileGate(ctx, s, box, boxID string)` returns nil unless the seeded file's `BoxID == boxID` (absent included); in the loop keep `wasPublishable bool` and when `Publishable(id.Label)` flips false to true set `last = nil`. `ErrNotPublishable` from a pass is logged once per transition.
-f. **Metrics, alerts, registry snapshot, usage, logship, errship, analytics**: at each arm (`metrics_loop.go:799`, `snapshot_loop.go:205`, `usage_loop.go:72`, `logship.go:39` and its errship twin, the analytics flush), replace `box := metricsBoxLabel(...)` with `id := boxIdentity(os.Getenv, readMachineID()); box := id.Label` and wrap the store: `s = gated(s, id, statusStoreTarget(os.Getenv))`. The `agent.go:244,245,477,480,569` sinks take `boxIdentity(...).Label` instead of `status.Label(readMachineID())`. `metricsBoxLabel` and `BoxLabel` stay as thin wrappers returning `boxIdentity(...).Label` so nothing else breaks; `edge.WithBoxLabel(id.Label)` at `:542`.
+g. **Metrics, alerts, registry snapshot, usage, logship, errship, analytics** (`metrics_loop.go:799`, `snapshot_loop.go:205`, `usage_loop.go:72`, `logship.go:39` and errship, the analytics flush): `id := Identity(); box := id.Label` and `s = Gated(s, id.Label)` for the store each writes through; a refused write is logged once per transition and the loop keeps its state. The sinks at `agent.go:244,245,477,480,569` take `Identity().Label`; `edge.WithBoxLabel(Identity().Label)` at `:542`.
-g. **Config acks**: `configAckFunc(ctx, store, prefix)` wraps `store` with `gated(store, boxIdentity(...), statusStoreTarget(os.Getenv))`; `adopt` (`:3486`) gets the gate for free through the same function.
+h. **Config acks**: `configAckFunc(ctx, store, prefix)` (`:953`) wraps `store` with `Gated(store, Identity().Label)`; `adopt` (`:3536`) gets the gate through the same function.
-h. **Relay**: `relay.Config` gains `Publishable func(label string) bool`; `Serve` checks it before `WriteHeartbeat` (`serve.go:343`) and `meter.Flush` (`:354`), skipping both and logging `relay: box %s: not publishing (this machine does not hold that label)` once per transition. `relay.go:138` sets `cfg.Publishable = func(label string) bool { return host.Publishable(label, host.StatusStoreTarget(os.Getenv)) }` (export `StatusStoreTarget`), and `cfg.Box = host.BoxLabel()` as today when unset.
+i. **Relay**: `relay.go:138` after `cfg.Box` is resolved: `cfg.Store = host.Gated(cfg.Store, cfg.Box)`. Nothing in `internal/relay` changes except that `Serve` logs a refused heartbeat or flush once per transition rather than every tick (a small `sayOnce` keyed by the error string).
- [ ] **Step 4: Run the tests to verify they pass**
Run: `GOFLAGS=-buildvcs=false go test ./internal/host/ ./internal/relay/ ./internal/status/`
-Expected: PASS. The existing `TestMetricsBoxLabelUsesFleetIdentity` (`metrics_loop_test.go:879`) still passes through the wrapper.
+Expected: PASS, including `TestMetricsBoxLabelUsesFleetIdentity`.
-- [ ] **Step 5: Cross-compile check**
+- [ ] **Step 5: Cross-compile and vet**
-Run: `GOOS=darwin GOARCH=arm64 go build ./... && GOOS=windows go build ./...`
-Expected: both build.
+Run: `go vet ./... && GOOS=darwin GOARCH=arm64 go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...`
+Expected: all clean.
- [ ] **Step 6: Commit**
```bash
git add internal/host relay.go internal/relay
-git commit -m "host, relay: decide the label at arm and gate every label-keyed publisher (#212)"
+git commit -m "host, relay: decide the label at arm with the full snapshot; gate every label-keyed publisher (#212)"
```
---
@@ -1069,8 +1359,7 @@ git commit -m "host, relay: decide the label at arm and gate every label-keyed p
### Task 5: evidence, the refusal log lines, and the webhook event
**Files:**
-- Create: `internal/status/identity.go` (evidence type and key), `internal/host/refuse.go`
-- Test: `internal/status/identity_test.go`, `internal/host/refuse_test.go`
+- Create: `internal/status/identity.go`, `internal/status/identity_test.go`, `internal/host/refuse.go`, `internal/host/refuse_test.go`
**Interfaces:**
- Produces:
@@ -1078,7 +1367,8 @@ git commit -m "host, relay: decide the label at arm and gate every label-keyed p
```go
// internal/status
const IdentityPrefix = "status/identity/"
-func IdentityKey(boxID string) string // IdentityPrefix + boxID + ".json"
+const FleetObservationKey = IdentityPrefix + "fleet.json" // Task 7's shape; LoadEvidence skips it by name
+func IdentityKey(boxID string) string
type Evidence struct {
V int `json:"v"`
Label string `json:"label"`
@@ -1089,9 +1379,9 @@ type Evidence struct {
HolderInstanceID string `json:"holder_instance_id,omitempty"`
SeenAt time.Time `json:"seen_at"`
}
-func WriteEvidence(ctx, s release.Store, e Evidence) error // putOverwrite, not gated: the refuser must be able to say so
+func WriteEvidence(ctx, s release.Store, e Evidence) error // putOverwrite on the UNGATED status store
func LoadEvidence(ctx, s release.Store, now time.Time) ([]Evidence, bool, error) // unexpired only; complete=false on any skipped object
-func EvidenceExpired(e Evidence, now time.Time) bool // now - SeenAt > 3*SnapshotPeriod
+func EvidenceExpired(e Evidence, now time.Time) bool // now - SeenAt > 3*SnapshotPeriod
// internal/host
type refuser struct { prev status.Verdict; deliver func(metrics.Event); logf func(string, ...any) }
func (r *refuser) observe(ctx, s release.Store, d status.Decision, instanceID string, now time.Time)
@@ -1101,7 +1391,7 @@ func (r *refuser) observe(ctx, s release.Store, d status.Decision, instanceID st
```go
// internal/status/identity_test.go
-func TestEvidenceRoundTripAndExpiry(t *testing.T) {
+func TestEvidenceRoundTripExpiryAndSkipsTheFleetFile(t *testing.T) {
s := release.NewDirStore(t.TempDir())
now := time.Now()
e := status.Evidence{V: 1, Label: "edsger-knitting", BoxID: status.BoxID("b"), InstanceID: "i-b",
@@ -1109,6 +1399,12 @@ func TestEvidenceRoundTripAndExpiry(t *testing.T) {
if err := status.WriteEvidence(context.Background(), s, e); err != nil {
t.Fatal(err)
}
+ // The lease holder's observation file lives under the same prefix with
+ // a different shape; the evidence loader must not try to read it as
+ // evidence and must not count it against completeness.
+ if _, err := s.Put(context.Background(), status.FleetObservationKey, []byte(`{"v":1,"labels":{}}`), ""); err != nil {
+ t.Fatal(err)
+ }
got, complete, err := status.LoadEvidence(context.Background(), s, now)
if err != nil || !complete || len(got) != 1 || got[0].HolderBoxID != e.HolderBoxID {
t.Fatalf("load = %+v %v %v", got, complete, err)
@@ -1120,8 +1416,7 @@ func TestEvidenceRoundTripAndExpiry(t *testing.T) {
if _, err := s.Put(context.Background(), status.IdentityPrefix+"junk.json", []byte("{"), ""); err != nil {
t.Fatal(err)
}
- _, complete, _ = status.LoadEvidence(context.Background(), s, now)
- if complete {
+ if _, complete, _ = status.LoadEvidence(context.Background(), s, now); complete {
t.Fatal("an unparsable evidence object makes the load incomplete")
}
}
@@ -1140,7 +1435,7 @@ func TestRefuserLogsAndDeliversOncePerTransition(t *testing.T) {
HolderBoxID: status.BoxID("a"), HolderInstanceID: "i-a", HolderFresh: true, DecidedAt: now}
r.observe(context.Background(), s, foreign, "i-b", now)
r.observe(context.Background(), s, foreign, "i-b", now.Add(time.Minute))
- if len(logs) != 1 || !strings.Contains(logs[0], "held by another machine") || !strings.Contains(logs[0], "i-a") {
+ if len(logs) != 1 || !strings.Contains(logs[0], "held by another machine") || !strings.Contains(logs[0], "i-a") || !strings.Contains(logs[0], "carlos status --check-label") {
t.Fatalf("logs = %q", logs)
}
if len(events) != 1 || events[0].Rule != "identity" || !events[0].Firing || events[0].Box != "edsger-knitting" {
@@ -1153,29 +1448,29 @@ func TestRefuserLogsAndDeliversOncePerTransition(t *testing.T) {
owned := foreign
owned.Verdict = status.VerdictOwned
r.observe(context.Background(), s, owned, "i-b", now.Add(2*time.Minute))
- if len(events) != 2 || events[1].Firing {
- t.Fatalf("clearing is a delivered transition: %+v", events)
- }
- if len(logs) != 2 || !strings.Contains(logs[1], "now holds") {
- t.Fatalf("clearing is logged: %q", logs)
+ if len(events) != 2 || events[1].Firing || len(logs) != 2 || !strings.Contains(logs[1], "now holds") {
+ t.Fatalf("clearing is logged and delivered: %+v %q", events, logs)
}
twin := foreign
twin.Verdict = status.VerdictTwin
r.observe(context.Background(), s, twin, "i-b", now.Add(3*time.Minute))
- if !strings.Contains(logs[2], "share this machine id") {
- t.Fatalf("twin text: %q", logs[2])
- }
unproven := foreign
unproven.Verdict = status.VerdictUnproven
r.observe(context.Background(), s, unproven, "i-b", now.Add(4*time.Minute))
- if !strings.Contains(logs[3], "carlos fleet retire edsger-knitting") {
- t.Fatalf("unproven text: %q", logs[3])
+ if !strings.Contains(logs[2], "share this machine id") || !strings.Contains(logs[3], "carlos status --retire-label edsger-knitting") {
+ t.Fatalf("texts: %q", logs[2:])
}
for _, l := range logs {
if strings.Contains(l, status.BoxID("a")) || strings.Contains(l, status.BoxID("b")) {
t.Fatalf("log lines name instances and labels, never box ids: %q", l)
}
}
+ unknown := foreign
+ unknown.Verdict = status.VerdictUnknown
+ r.observe(context.Background(), s, unknown, "i-b", now.Add(5*time.Minute))
+ if len(events) != 3 {
+ t.Fatalf("unknown is not an alert transition on its own; events = %+v", events)
+ }
}
```
@@ -1186,44 +1481,44 @@ Expected: FAIL, undefined.
- [ ] **Step 3: Implement**
-`internal/status/identity.go`: the type, `IdentityKey`, `WriteEvidence` (via `putOverwrite`), `LoadEvidence` (list the prefix, skip non-`.json`, `Get`, unmarshal, drop expired, `complete=false` on read or parse failure, cap at `maxFleetBoxKeys` like `loadBoxesReport`), `EvidenceExpired`.
+`internal/status/identity.go`: the constants, the type, `WriteEvidence` via `putOverwrite`, `LoadEvidence` (list `IdentityPrefix`, skip `FleetObservationKey` and any key not ending `.json`, `Get`, unmarshal, drop expired, `complete=false` on a read or parse failure, cap at `maxFleetBoxKeys`), `EvidenceExpired`.
`internal/host/refuse.go`:
```go
package host
-// The refusing side of spec 2.3: say it once per transition, write the
-// evidence every tick, and fire the webhook through the metrics sinks.
-
type refuser struct {
prev status.Verdict
deliver func(metrics.Event)
logf func(string, ...any)
}
+// observe is spec 2.3: evidence every tick while not owned; one log line
+// and one delivered event per transition between firing and clear.
func (r *refuser) observe(ctx context.Context, s release.Store, d status.Decision, instanceID string, now time.Time) {
- if d.Verdict != status.VerdictOwned {
+ if d.Verdict != status.VerdictOwned && d.Verdict != status.VerdictUnknown {
_ = status.WriteEvidence(ctx, s, status.Evidence{V: 1, Label: d.Label, BoxID: d.BoxID, InstanceID: instanceID,
Verdict: d.Verdict, HolderBoxID: d.HolderBoxID, HolderInstanceID: d.HolderInstanceID, SeenAt: now})
}
- if d.Verdict == r.prev {
+ if d.Verdict == r.prev || d.Verdict == status.VerdictUnknown {
return
}
switch d.Verdict {
case status.VerdictForeign:
- r.logf("agent: label %s is held by another machine (instance %s); this box (instance %s) is not publishing. Fix the box that is new (box-identity spec 2.3; carlos fleet check-label shows both).", d.Label, orUnknown(d.HolderInstanceID), orUnknown(instanceID))
+ r.logf("agent: label %s is held by another machine (instance %s); this box (instance %s) is not publishing. Fix the box that is new (box-identity spec 2.3; carlos status --check-label shows both).", d.Label, orUnknown(d.HolderInstanceID), orUnknown(instanceID))
case status.VerdictTwin:
r.logf("agent: label %s: two machines share this machine id (instances %s and %s); fix the one that is new.", d.Label, orUnknown(d.HolderInstanceID), orUnknown(instanceID))
case status.VerdictUnproven:
- r.logf("agent: cannot prove label %s is ours (no instance id to compare); if it is, carlos fleet retire %s frees it.", d.Label, d.Label)
+ r.logf("agent: cannot prove label %s is ours (no instance id to compare); if it is, carlos status --retire-label %s frees it.", d.Label, d.Label)
case status.VerdictOwned:
if r.prev != "" {
r.logf("agent: label %s: this box now holds it.", d.Label)
}
}
- firing := d.Verdict != status.VerdictOwned && d.Verdict != status.VerdictUnknown
- if r.prev != "" || firing {
+ firing := d.Verdict != status.VerdictOwned
+ wasFiring := r.prev != "" && r.prev != status.VerdictOwned
+ if firing != wasFiring {
r.deliver(metrics.Event{Rule: "identity", Firing: firing, Box: d.Label})
}
r.prev = d.Verdict
@@ -1237,11 +1532,11 @@ func orUnknown(s string) string {
}
```
-Wire it: `statusLoop` owns one `refuser{deliver: func(e metrics.Event) { sinks.deliver(ctx, rules, time.Now(), e) }, logf: log.Printf}` built from `buildAlertSinks(ctx, os.Getenv, ...)` (`alertsink.go:209`) and `metrics.RulesFromEnv`, and calls `observe` after every decision with `cachedInstanceID()`. The evidence store is the status store **ungated** (the refuser must be able to write evidence).
+`statusLoop` owns one `refuser{deliver: func(e metrics.Event) { sinks.deliver(ctx, rules, time.Now(), e) }, logf: log.Printf}` built from `buildAlertSinks(ctx, os.Getenv, &processAlertCounters, newSNSClient)` (`alertsink.go:209`) and `metrics.RulesFromEnv(os.Getenv)`, and calls `observe` after every decision with `cachedInstanceID()`. The evidence store is the **ungated** status store.
- [ ] **Step 4: Run the tests to verify they pass**
-Run: the same command. Expected: PASS.
+Run: the Step 2 command. Expected: PASS.
- [ ] **Step 5: Commit**
@@ -1252,42 +1547,62 @@ git commit -m "status, host: evidence objects and the refusal transition (#212)"
---
-### Task 6: console readers, eligibility only
+### Task 6: console readers, eligibility only, completeness monotonic
**Files:**
-- Modify: `internal/console/instancesplacement.go:38-60` (view gains `BoxID`), `:93-168` (`readLiveEdgesSnapshot`)
+- Modify: `internal/console/instancesplacement.go:38-60` (view gains `BoxID`), `:103-180` (`readLiveEdgesSnapshot`)
- Test: `internal/console/instancesplacement_test.go`, `internal/edgesfile/fold_test.go`
**Interfaces:**
-- Consumes: Task 1 `edgesfile`/`status` `BoxID`; Task 5 `status.LoadEvidence`.
-- Produces: `edgesFilePlacementView.BoxID string json:"box_id"`; `edgesSnapshot.Contested []string` (labels), folded into `BoxesComplete`.
+- Consumes: Task 1 `BoxID` fields; Task 5 `status.LoadEvidence`; Task 7 `status.LoadObservations` (declare it here as a stub returning an empty record so this task compiles alone; Task 7 fills it).
+- Produces: `edgesFilePlacementView.BoxID string json:"box_id"`; `edgesSnapshot.Contested []string`.
- [ ] **Step 1: Write the failing tests**
```go
// internal/console/instancesplacement_test.go
+func writeSnapshot(t *testing.T, s release.Store, b status.Box) {
+ t.Helper()
+ raw, _ := json.Marshal(b)
+ key := "status/boxes/" + b.Label + ".json"
+ _, etag, _ := s.Get(context.Background(), key)
+ if _, err := s.Put(context.Background(), key, raw, etag); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func writeEdges(t *testing.T, s release.Store, label, body string) {
+ t.Helper()
+ key := edgesFilePrefix + label + ".json"
+ _, etag, _ := s.Get(context.Background(), key)
+ if _, err := s.Put(context.Background(), key, []byte(body), etag); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func edgesBody(label, boxID string) string {
+ return `{"v":1,"box":"` + label + `","box_id":"` + boxID + `","eip":"1.2.3.4","placement":"uk","hosts":{}}`
+}
+
func TestLiveEdgesContestedPairIsIncompleteEvidence(t *testing.T) {
s := release.NewDirStore(t.TempDir())
now := time.Now()
writeSnapshot(t, s, status.Box{GeneratedAt: now, Label: "edsger-knitting", BoxID: status.BoxID("a")})
- writeEdges(t, s, "edsger-knitting", `{"v":1,"box":"edsger-knitting","box_id":"`+status.BoxID("b")+`","eip":"1.2.3.4","placement":"uk","hosts":{}}`)
+ writeEdges(t, s, "edsger-knitting", edgesBody("edsger-knitting", status.BoxID("b")))
snap, err := readLiveEdgesSnapshot(context.Background(), s, now)
if err != nil {
t.Fatal(err)
}
- if snap.BoxesComplete {
- t.Fatal("a snapshot and an edges file written by two machines must be incomplete evidence")
- }
- if len(snap.Contested) != 1 || snap.Contested[0] != "edsger-knitting" {
- t.Fatalf("contested = %v", snap.Contested)
+ if snap.BoxesComplete || len(snap.Contested) != 1 || snap.Contested[0] != "edsger-knitting" {
+ t.Fatalf("a pair written by two machines must be incomplete and contested: %+v", snap)
}
}
-func TestLiveEdgesAgreeingPairIsLive(t *testing.T) {
+func TestLiveEdgesAgreeingPairIsLiveAndComplete(t *testing.T) {
s := release.NewDirStore(t.TempDir())
now := time.Now()
writeSnapshot(t, s, status.Box{GeneratedAt: now, Label: "annie-welding", BoxID: status.BoxID("a")})
- writeEdges(t, s, "annie-welding", `{"v":1,"box":"annie-welding","box_id":"`+status.BoxID("a")+`","eip":"1.2.3.4","placement":"ie","hosts":{}}`)
+ writeEdges(t, s, "annie-welding", edgesBody("annie-welding", status.BoxID("a")))
snap, _ := readLiveEdgesSnapshot(context.Background(), s, now)
if !snap.BoxesComplete || !snap.Files[0].Fresh {
t.Fatalf("agreeing pair must be live and complete: %+v", snap)
@@ -1305,26 +1620,58 @@ func TestLiveEdgesOldObjectsJoinByLabelAsToday(t *testing.T) {
}
}
+func TestLiveEdgesFreshEvidenceWithMatchingFilesIsIncomplete(t *testing.T) {
+ // The ordinary collision shape: the holder's two objects agree, and the
+ // refuser's evidence says a second machine claims the label.
+ s := release.NewDirStore(t.TempDir())
+ now := time.Now()
+ writeSnapshot(t, s, status.Box{GeneratedAt: now, Label: "edsger-knitting", BoxID: status.BoxID("a")})
+ writeEdges(t, s, "edsger-knitting", edgesBody("edsger-knitting", status.BoxID("a")))
+ _ = status.WriteEvidence(context.Background(), s, status.Evidence{V: 1, Label: "edsger-knitting", BoxID: status.BoxID("b"), Verdict: status.VerdictForeign, SeenAt: now})
+ snap, _ := readLiveEdgesSnapshot(context.Background(), s, now)
+ if snap.BoxesComplete {
+ t.Fatal("fresh foreign evidence must make eligibility incomplete even when the files match")
+ }
+ if !slices.Contains(snap.Contested, "edsger-knitting") {
+ t.Fatalf("contested = %v", snap.Contested)
+ }
+}
+
func TestLiveEdgesEvidenceContestsALabelEvenWithNoFiles(t *testing.T) {
s := release.NewDirStore(t.TempDir())
now := time.Now()
- _ = status.WriteEvidence(context.Background(), s, status.Evidence{V: 1, Label: "edsger-knitting", BoxID: status.BoxID("b"),
- Verdict: status.VerdictForeign, SeenAt: now})
+ _ = status.WriteEvidence(context.Background(), s, status.Evidence{V: 1, Label: "edsger-knitting", BoxID: status.BoxID("b"), Verdict: status.VerdictForeign, SeenAt: now})
snap, _ := readLiveEdgesSnapshot(context.Background(), s, now)
if snap.BoxesComplete {
t.Fatal("evidence must be read before the empty-files early return")
}
}
-func TestLiveEdgesUnreadableEvidenceIsIncomplete(t *testing.T) {
+func TestLiveEdgesUnreadableEvidenceStaysIncompleteAfterTheSnapshotRead(t *testing.T) {
s := release.NewDirStore(t.TempDir())
now := time.Now()
writeSnapshot(t, s, status.Box{GeneratedAt: now, Label: "annie-welding", BoxID: status.BoxID("a")})
- writeEdges(t, s, "annie-welding", `{"v":1,"box":"annie-welding","box_id":"`+status.BoxID("a")+`","eip":"1.2.3.4","hosts":{}}`)
+ writeEdges(t, s, "annie-welding", edgesBody("annie-welding", status.BoxID("a")))
_, _ = s.Put(context.Background(), status.IdentityPrefix+"junk.json", []byte("{"), "")
snap, _ := readLiveEdgesSnapshot(context.Background(), s, now)
if snap.BoxesComplete {
- t.Fatal("a failed evidence read is incomplete evidence")
+ t.Fatal("a failed evidence read is incomplete evidence and a later complete snapshot read must not overwrite it")
+ }
+}
+
+func TestLiveEdgesObservationOnlyContentionIsIncomplete(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ now := time.Now()
+ writeSnapshot(t, s, status.Box{GeneratedAt: now, Label: "annie-welding", BoxID: status.BoxID("a")})
+ writeEdges(t, s, "annie-welding", edgesBody("annie-welding", status.BoxID("a")))
+ obs := status.FleetObservations{V: 1, Labels: map[string]status.LabelObservation{"annie-welding": {BoxID: status.BoxID("a"), GeneratedAt: now, ContestedUntil: now.Add(time.Hour)}}}
+ raw, _ := json.Marshal(obs)
+ if _, err := s.Put(context.Background(), status.FleetObservationKey, raw, ""); err != nil {
+ t.Fatal(err)
+ }
+ snap, _ := readLiveEdgesSnapshot(context.Background(), s, now)
+ if snap.BoxesComplete {
+ t.Fatal("an active contested_until from the lease holder is incomplete evidence")
}
}
```
@@ -1332,10 +1679,10 @@ func TestLiveEdgesUnreadableEvidenceIsIncomplete(t *testing.T) {
```go
// internal/edgesfile/fold_test.go
func TestEdgeIPsAndDeploymentRoutesIgnoreBoxIDDisagreement(t *testing.T) {
- s := release.NewDirStore(t.TempDir())
+ s := foldTestStore(t)
now := time.Now()
- writeBox(t, s, status.Box{GeneratedAt: now, Label: "edsger-knitting", BoxID: status.BoxID("a")}) // the existing helper
- writeFile(t, s, "edsger-knitting", `{"v":1,"box":"edsger-knitting","box_id":"`+status.BoxID("b")+`","eip":"5.5.5.5","hosts":{"shop.example":{"role":"home"}}}`)
+ putFreshBoxWithID(t, s, "edsger-knitting", status.BoxID("a"), now)
+ putEdgesFile(t, s, "edsger-knitting", `{"v":1,"box":"edsger-knitting","box_id":"`+status.BoxID("b")+`","eip":"5.5.5.5","hosts":{"shop.example":{"role":"home"}}}`)
if ips := edgesfile.EdgeIPs(context.Background(), s, nil, now); !slices.Contains(ips, "5.5.5.5") {
t.Fatal("a contested file's EIP is still an edge: the union only widens")
}
@@ -1343,58 +1690,109 @@ func TestEdgeIPsAndDeploymentRoutesIgnoreBoxIDDisagreement(t *testing.T) {
t.Fatal("a contested file's hostnames stay taken: occupancy is never weakened")
}
}
+
+// putFreshBoxWithID is putFreshBox (fold_test.go:129) with a box_id.
+func putFreshBoxWithID(t *testing.T, s release.Store, label, boxID string, at time.Time) {
+ t.Helper()
+ raw, _ := json.Marshal(status.Box{GeneratedAt: at, Label: label, BoxID: boxID})
+ if _, err := s.Put(context.Background(), "status/boxes/"+label+".json", raw, ""); err != nil {
+ t.Fatal(err)
+ }
+}
```
- [ ] **Step 2: Run the tests to verify they fail**
Run: `GOFLAGS=-buildvcs=false go test ./internal/console/ ./internal/edgesfile/ -run 'LiveEdges|IgnoreBoxIDDisagreement' -v`
-Expected: the console tests FAIL (`Contested` undefined; the evidence test finds `BoxesComplete` true); the fold test PASSES already (it pins behaviour that must not change).
+Expected: the console tests FAIL (`Contested` undefined, completeness wrong); the fold test PASSES (it pins behaviour that must not change).
- [ ] **Step 3: Implement**
-In `edgesFilePlacementView` add `BoxID string json:"box_id"`. In `readLiveEdgesSnapshot`, before `if len(snap.Files) == 0`:
+Add `BoxID string json:"box_id"` to `edgesFilePlacementView` and `Contested []string` to `edgesSnapshot`. Restructure `readLiveEdgesSnapshot` so completeness is monotonic:
```go
- evidence, evComplete, evErr := status.LoadEvidence(ctx, s, now)
- if evErr != nil || !evComplete {
+func readLiveEdgesSnapshot(ctx context.Context, s release.Store, now time.Time) (edgesSnapshot, error) {
+ snap, err := readEdgesSnapshot(ctx, s)
+ if err != nil {
+ return snap, err
+ }
+ snap.BoxesComplete = true
+ incomplete := func(key, format string, args ...any) {
snap.BoxesComplete = false
- if _, said := livenessSaid.LoadOrStore("evidence", true); !said {
- log.Printf("console: could not read every object under %s (%v), so no region outside the home is automatically eligible until it reads; said once", status.IdentityPrefix, evErr)
+ if _, said := livenessSaid.LoadOrStore(key, true); !said {
+ log.Printf(format, args...)
}
}
+ contest := func(label string) {
+ if !slices.Contains(snap.Contested, label) {
+ snap.Contested = append(snap.Contested, label)
+ }
+ snap.BoxesComplete = false
+ }
+
+ // Rules 2 and 3 run first, before the empty-files return: a collision is
+ // evidence about the fleet whether or not any file is published.
+ evidence, evComplete, evErr := status.LoadEvidence(ctx, s, now)
+ if evErr != nil || !evComplete {
+ incomplete("evidence", "console: could not read every object under %s (%v), so no region outside the home is automatically eligible until it reads; said once", status.IdentityPrefix, evErr)
+ }
for _, e := range evidence {
- snap.Contested = append(snap.Contested, e.Label)
+ contest(e.Label)
+ }
+ obs, obsErr := status.LoadObservations(ctx, s)
+ if obsErr != nil && !errors.Is(obsErr, os.ErrNotExist) {
+ incomplete("observations", "console: could not read %s (%v); said once", status.FleetObservationKey, obsErr)
}
+ for label, o := range obs.Labels {
+ if o.ContestedUntil.After(now) {
+ contest(label)
+ }
+ }
+
if len(snap.Files) == 0 {
- snap.BoxesComplete = snap.BoxesComplete && evErr == nil && evComplete && len(evidence) == 0
return snap, nil
}
-```
-
-(Initialise `snap.BoxesComplete = true` before this block rather than inside the early return.) Then, after the `known`/`live` maps are built, also build `idByLabel := map[string]string{}` from `b.Label -> b.BoxID` for boxes with a `BoxID`, and in the files loop:
-
-```go
- if f := snap.Files[i]; f.BoxID != "" && idByLabel[f.Box] != "" && f.BoxID != idByLabel[f.Box] {
- snap.Files[i].Fresh = false
- snap.BoxesComplete = false
- snap.Contested = append(snap.Contested, f.Box)
- if _, said := livenessSaid.LoadOrStore("contested:"+f.Box, true); !said {
- log.Printf("console: box %q: its edges file and its status snapshot were written by two machines, so the label is contested and no region outside the home is automatically eligible; said once", f.Box)
- }
+ boxes, boxesComplete, err := status.LoadBoxesComplete(ctx, s)
+ if err != nil {
+ incomplete("load", "console: could not load box snapshots (%v), so no box outside the home counts for automatic region placement until they load", err)
+ return snap, nil
+ }
+ if !boxesComplete {
+ snap.BoxesComplete = false
+ }
+ known, live, idByLabel := map[string]bool{}, map[string]bool{}, map[string]string{}
+ for _, b := range boxes {
+ known[b.Label] = true
+ if !status.BoxStale(b, now, status.SnapshotPeriod) {
+ live[b.Label] = true
+ }
+ if b.BoxID != "" {
+ idByLabel[b.Label] = b.BoxID
}
+ }
+ // ... the existing published/joined loop, with one addition per file:
+ // if f.BoxID != "" && idByLabel[f.Box] != "" && f.BoxID != idByLabel[f.Box] {
+ // snap.Files[i].Fresh = false
+ // contest(f.Box) // rule 1
+ // incomplete("contested:"+f.Box, "console: box %q: its edges file and its status snapshot were written by two machines; said once", f.Box)
+ // }
+ // ... and the existing !known / unmatched checks call incomplete(...) rather than assigning.
+ return snap, nil
+}
```
-`edgesfile.EdgeIPs` and `DeploymentRoutes` are not touched.
+`status.LoadObservations(ctx, s) (FleetObservations, error)` and the `FleetObservations`/`LabelObservation` types are declared in this task in `internal/status/fleetidentity.go` as a plain `Get` + unmarshal of `FleetObservationKey` (Task 7 adds the writer side).
- [ ] **Step 4: Run the tests to verify they pass**
-Run: the same command plus `go test ./internal/console/ -run Placement`. Expected: PASS.
+Run: `GOFLAGS=-buildvcs=false go test ./internal/console/ -run 'LiveEdges|Placement' && go test ./internal/edgesfile/`
+Expected: PASS.
- [ ] **Step 5: Commit**
```bash
-git add internal/console/instancesplacement.go internal/console/instancesplacement_test.go internal/edgesfile/fold_test.go
-git commit -m "console: a contested label is incomplete evidence for automatic eligibility (#212)"
+git add internal/console/instancesplacement.go internal/console/instancesplacement_test.go internal/edgesfile/fold_test.go internal/status/fleetidentity.go
+git commit -m "console: collisions are incomplete evidence for automatic eligibility; completeness is monotonic (#212)"
```
---
@@ -1402,17 +1800,17 @@ git commit -m "console: a contested label is incomplete evidence for automatic e
### Task 7: the lease holder's observation file and the per-label fleet alert
**Files:**
-- Create: `internal/status/fleetidentity.go`, `internal/status/fleetidentity_test.go`
-- Modify: `internal/host/agent.go` (`statusTick` publish branch, `:2938` after `BuildFleet`), `internal/console/alertwatch.go:97-145`
-- Test: `internal/console/alertwatch_test.go`
+- Modify: `internal/status/fleetidentity.go` (from Task 6), `internal/status/fleetidentity_test.go`
+- Modify: `internal/host/agent.go` (`statusTick` publish branch, after `BuildFleet`), `internal/console/alertwatch.go:103-146`
+- Test: `internal/host/agent_test.go`, `internal/console/alertwatch_test.go`
**Interfaces:**
- Produces:
```go
// internal/status
-const FleetObservationKey = "status/identity/fleet.json"
const FleetAlertKey = "alerts/fleet/identity.json"
+const clearedRuleRetention = 24 * time.Hour
type LabelObservation struct {
BoxID string `json:"box_id"`
GeneratedAt time.Time `json:"generated_at"`
@@ -1422,42 +1820,43 @@ type FleetObservations struct {
V int `json:"v"`
Labels map[string]LabelObservation `json:"labels"`
}
-// ObserveWriters applies rule 3 to boxes against prev and returns the next
-// record; it never shortens an existing ContestedUntil.
+func LoadObservations(ctx, s release.Store) (FleetObservations, error) // Task 6
+func LoadObservationsWithETag(ctx, s release.Store) (FleetObservations, string, error) // "" etag when absent
func ObserveWriters(prev FleetObservations, boxes []Box, now time.Time) FleetObservations
-// IdentityAlert builds alerts/fleet/identity.json: one rule per label and
-// condition: "foreign:<label>", "twin:<label>", "unproven:<label>", "contested:<label>".
-func IdentityAlert(ev []Evidence, obs FleetObservations, contested []string, now time.Time) metrics.AlertState
+// IdentityAlert folds all three rules into one AlertState with per-label rules,
+// keeping any rule in prev that stopped firing as Firing:false for clearedRuleRetention.
+func IdentityAlert(prev metrics.AlertState, ev []Evidence, obs FleetObservations, mismatched []string, now time.Time) metrics.AlertState
+// internal/host
+// mismatchedLabels: rule 1 computed on the box (host imports edgesfile and status).
+func mismatchedLabels(ctx, s release.Store, boxes []status.Box) []string
```
- [ ] **Step 1: Write the failing tests**
```go
// internal/status/fleetidentity_test.go
-func TestObserveWritersFlagsAChangeWithinFreshnessOnly(t *testing.T) {
+func TestObserveWritersFlagsAChangeWithinFreshnessOnlyAndNeverShortens(t *testing.T) {
now := time.Now()
prev := status.FleetObservations{V: 1, Labels: map[string]status.LabelObservation{
"edsger-knitting": {BoxID: status.BoxID("a"), GeneratedAt: now.Add(-time.Minute)},
"annie-welding": {BoxID: status.BoxID("c"), GeneratedAt: now.Add(-2 * time.Hour)},
}}
next := status.ObserveWriters(prev, []status.Box{
- {GeneratedAt: now, Label: "edsger-knitting", BoxID: status.BoxID("b")}, // changed while fresh
- {GeneratedAt: now, Label: "annie-welding", BoxID: status.BoxID("d")}, // previous was stale: a legitimate successor? No: still a change, but after staleness, so not contested
+ {GeneratedAt: now, Label: "edsger-knitting", BoxID: status.BoxID("b")},
+ {GeneratedAt: now, Label: "annie-welding", BoxID: status.BoxID("d")},
}, now)
if cu := next.Labels["edsger-knitting"].ContestedUntil; !cu.Equal(now.Add(3 * status.SnapshotPeriod)) {
- t.Fatalf("contested_until = %v", cu)
+ t.Fatalf("a change while the previous object was fresh: contested_until = %v", cu)
}
if !next.Labels["annie-welding"].ContestedUntil.IsZero() {
t.Fatal("a change after the previous object went stale is not a writer change")
}
- // never shortened
far := now.Add(24 * time.Hour)
prev.Labels["edsger-knitting"] = status.LabelObservation{BoxID: status.BoxID("a"), GeneratedAt: now, ContestedUntil: far}
next = status.ObserveWriters(prev, []status.Box{{GeneratedAt: now.Add(time.Minute), Label: "edsger-knitting", BoxID: status.BoxID("b")}}, now.Add(time.Minute))
if !next.Labels["edsger-knitting"].ContestedUntil.Equal(far) {
t.Fatal("an existing contested_until is never shortened")
}
- // disappearance (an old sibling overwrote): box_id "" while fresh
next = status.ObserveWriters(status.FleetObservations{V: 1, Labels: map[string]status.LabelObservation{
"edsger-knitting": {BoxID: status.BoxID("a"), GeneratedAt: now}}},
[]status.Box{{GeneratedAt: now.Add(time.Minute), Label: "edsger-knitting"}}, now.Add(time.Minute))
@@ -1466,13 +1865,16 @@ func TestObserveWritersFlagsAChangeWithinFreshnessOnly(t *testing.T) {
}
}
-func TestIdentityAlertIsPerLabel(t *testing.T) {
+func TestIdentityAlertIsPerLabelAndRetainsClears(t *testing.T) {
now := time.Now()
- st := status.IdentityAlert(
+ st := status.IdentityAlert(metrics.AlertState{},
[]status.Evidence{{Label: "edsger-knitting", Verdict: status.VerdictForeign, SeenAt: now}, {Label: "bea-sailing", Verdict: status.VerdictUnproven, SeenAt: now}},
- status.FleetObservations{V: 1, Labels: map[string]status.LabelObservation{"annie-welding": {ContestedUntil: now.Add(time.Hour)}}},
+ status.FleetObservations{V: 1, Labels: map[string]status.LabelObservation{
+ "annie-welding": {ContestedUntil: now.Add(time.Hour)},
+ "old-label": {ContestedUntil: now.Add(-time.Hour)},
+ }},
[]string{"cal-hiking"}, now)
- if st.Box != "fleet" || st.V != 1 {
+ if st.Box != "fleet" || st.V != 1 || !st.UpdatedAt.Equal(now) {
t.Fatalf("state = %+v", st)
}
for _, rule := range []string{"foreign:edsger-knitting", "unproven:bea-sailing", "contested:annie-welding", "contested:cal-hiking"} {
@@ -1480,36 +1882,99 @@ func TestIdentityAlertIsPerLabel(t *testing.T) {
t.Fatalf("rule %s must fire: %+v", rule, st.Rules)
}
}
+ if _, ok := st.Rules["contested:old-label"]; ok {
+ t.Fatal("a contested_until in the past does not fire")
+ }
if len(st.Rules) != 4 {
t.Fatalf("exactly four rules, got %d", len(st.Rules))
}
- clear := status.IdentityAlert(nil, status.FleetObservations{V: 1}, nil, now)
- if len(clear.Rules) != 0 || clear.UpdatedAt.IsZero() {
- t.Fatal("no conditions is an all-clear object with a fresh UpdatedAt")
+ // Everything clears: the rules stay, as Firing:false, so the watcher sees the transition.
+ cleared := status.IdentityAlert(st, nil, status.FleetObservations{V: 1}, nil, now.Add(time.Minute))
+ if len(cleared.Rules) != 4 {
+ t.Fatalf("cleared rules are retained: %+v", cleared.Rules)
+ }
+ for name, rs := range cleared.Rules {
+ if rs.Firing {
+ t.Fatalf("%s must be Firing:false", name)
+ }
+ }
+ // ...and are dropped after the retention window.
+ gone := status.IdentityAlert(cleared, nil, status.FleetObservations{V: 1}, nil, now.Add(25*time.Hour))
+ if len(gone.Rules) != 0 {
+ t.Fatalf("cleared rules are dropped after a day: %+v", gone.Rules)
}
}
```
```go
// internal/console/alertwatch_test.go
-func TestAlertPassReadsTheFleetIdentityObject(t *testing.T) {
- fx := newAlertFixture(t, nil) // the existing fixture
+func TestAlertPassFleetIdentityFiresClearsAndFiresAgain(t *testing.T) {
+ fx := newCapturedFixture(t)
now := time.Now()
- st := status.IdentityAlert([]status.Evidence{{Label: "edsger-knitting", Verdict: status.VerdictForeign, SeenAt: now}}, status.FleetObservations{V: 1}, nil, now)
- body, _ := json.Marshal(st)
- if _, err := fx.store.Put(context.Background(), status.FleetAlertKey, body, ""); err != nil {
- t.Fatal(err)
+ put := func(st metrics.AlertState) {
+ t.Helper()
+ body, _ := json.Marshal(st)
+ _, etag, _ := fx.store.Get(context.Background(), status.FleetAlertKey)
+ if _, err := fx.store.Put(context.Background(), status.FleetAlertKey, body, etag); err != nil {
+ t.Fatal(err)
+ }
}
+ ev := []status.Evidence{{Label: "edsger-knitting", BoxID: status.BoxID("b"), InstanceID: "i-4242", Verdict: status.VerdictForeign, HolderBoxID: status.BoxID("a"), HolderInstanceID: "i-4343", SeenAt: now}}
+ firing := status.IdentityAlert(metrics.AlertState{}, ev, status.FleetObservations{V: 1}, nil, now)
+ put(firing)
if err := fx.srv.alertPass(context.Background()); err != nil {
t.Fatal(err)
}
if len(fx.sent) != 1 || !strings.Contains(fx.sent[0].subject, "foreign:edsger-knitting") {
t.Fatalf("sent = %+v", fx.sent)
}
- if strings.Contains(fx.sent[0].body, status.BoxID("a")) {
- t.Fatal("the mail must not carry a box id")
+ for _, needle := range []string{status.BoxID("a"), status.BoxID("b"), "i-4242", "i-4343", "box_id", "instance_id"} {
+ if strings.Contains(fx.sent[0].body, needle) || strings.Contains(fx.sent[0].subject, needle) {
+ t.Fatalf("the mail must not carry %q", needle)
+ }
+ }
+ cleared := status.IdentityAlert(firing, nil, status.FleetObservations{V: 1}, nil, now.Add(time.Minute))
+ put(cleared)
+ if err := fx.srv.alertPass(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if len(fx.sent) != 2 {
+ t.Fatalf("the clear must be mailed: %+v", fx.sent)
+ }
+ again := status.IdentityAlert(cleared, ev, status.FleetObservations{V: 1}, nil, now.Add(2*time.Minute))
+ put(again)
+ if err := fx.srv.alertPass(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if len(fx.sent) != 3 {
+ t.Fatalf("a second collision after a clear must be mailed: %+v", fx.sent)
+ }
+ // A second label while the first still fires is its own transition.
+ two := status.IdentityAlert(again, append(ev, status.Evidence{Label: "bea-sailing", Verdict: status.VerdictTwin, SeenAt: now}), status.FleetObservations{V: 1}, nil, now.Add(3*time.Minute))
+ put(two)
+ if err := fx.srv.alertPass(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if len(fx.sent) != 4 || !strings.Contains(fx.sent[3].subject, "twin:bea-sailing") {
+ t.Fatalf("sent = %+v", fx.sent)
}
- // a stale object is all-clear
+}
+
+func TestAlertPassReadsTheFleetObjectWithNoBoxAlerts(t *testing.T) {
+ fx := newCapturedFixture(t)
+ now := time.Now()
+ st := status.IdentityAlert(metrics.AlertState{}, []status.Evidence{{Label: "edsger-knitting", Verdict: status.VerdictForeign, SeenAt: now}}, status.FleetObservations{V: 1}, nil, now)
+ body, _ := json.Marshal(st)
+ if _, err := fx.store.Put(context.Background(), status.FleetAlertKey, body, ""); err != nil {
+ t.Fatal(err)
+ }
+ if err := fx.srv.alertPass(context.Background()); err != nil {
+ t.Fatal(err)
+ }
+ if len(fx.sent) != 1 {
+ t.Fatalf("the fleet object is read even when no box has published an alert: %+v", fx.sent)
+ }
+ // A stale fleet object clears every rule it carries.
st.UpdatedAt = now.Add(-4 * status.SnapshotPeriod)
body, _ = json.Marshal(st)
_, etag, _ := fx.store.Get(context.Background(), status.FleetAlertKey)
@@ -1517,167 +1982,314 @@ func TestAlertPassReadsTheFleetIdentityObject(t *testing.T) {
if err := fx.srv.alertPass(context.Background()); err != nil {
t.Fatal(err)
}
- if len(fx.sent) != 2 || fx.sent[1].subject == fx.sent[0].subject {
- t.Fatalf("a stale fleet object clears the rule: %+v", fx.sent)
+ if len(fx.sent) != 2 {
+ t.Fatalf("a stale fleet object clears: %+v", fx.sent)
}
}
```
+```go
+// internal/host/agent_test.go
+func TestPublishIdentityFilesLoseToASuccessor(t *testing.T) {
+ // Publisher A reads both fleet files, pauses; B publishes; A resumes
+ // with its stale etags and must write neither file.
+ s := release.NewDirStore(t.TempDir())
+ ctx := context.Background()
+ now := time.Now()
+ a := newIdentityPublisher(s)
+ if err := a.begin(ctx); err != nil { // reads observation and alert etags
+ t.Fatal(err)
+ }
+ b := newIdentityPublisher(s)
+ if err := b.begin(ctx); err != nil {
+ t.Fatal(err)
+ }
+ ev := []status.Evidence{{Label: "edsger-knitting", Verdict: status.VerdictForeign, SeenAt: now}}
+ if err := b.finish(ctx, nil, ev, nil, now, func() error { return nil }); err != nil {
+ t.Fatal(err)
+ }
+ err := a.finish(ctx, nil, nil, nil, now, func() error { return nil })
+ if err == nil || !errors.Is(err, release.ErrCASConflict) {
+ t.Fatalf("A's stale write must lose: %v", err)
+ }
+ body, _, _ := s.Get(ctx, status.FleetAlertKey)
+ var st metrics.AlertState
+ _ = json.Unmarshal(body, &st)
+ if !st.Rules["foreign:edsger-knitting"].Firing {
+ t.Fatal("B's alert must stand")
+ }
+ // A lost lease skips the writes entirely.
+ c := newIdentityPublisher(s)
+ _ = c.begin(ctx)
+ if err := c.finish(ctx, nil, nil, nil, now, func() error { return lease.ErrLost }); !errors.Is(err, lease.ErrLost) {
+ t.Fatalf("a lost lease must skip: %v", err)
+ }
+}
+```
+
+(`identityPublisher` is the small type Step 3 adds: `begin` reads both files with etags, `finish(boxes, evidence, mismatched, now, leaseErr)` computes and writes both conditionally. `lease.ErrLost` is whatever `Held.Err()` returns on loss; use the real sentinel.)
+
- [ ] **Step 2: Run the tests to verify they fail**
-Run: `GOFLAGS=-buildvcs=false go test ./internal/status/ ./internal/console/ -run 'ObserveWriters|IdentityAlert|FleetIdentityObject' -v`
+Run: `GOFLAGS=-buildvcs=false go test ./internal/status/ ./internal/console/ ./internal/host/ -run 'ObserveWriters|IdentityAlert|FleetIdentity|FleetObject|LoseToASuccessor' -v`
Expected: FAIL, undefined.
- [ ] **Step 3: Implement**
-`internal/status/fleetidentity.go`: `ObserveWriters` iterates `boxes`; for each label with a previous observation, `changed := prev.BoxID != b.BoxID` and `prevFresh := b.GeneratedAt.Sub(prev.GeneratedAt) <= 3*SnapshotPeriod`; when both, `ContestedUntil = max(prev.ContestedUntil, now.Add(3*SnapshotPeriod))`; otherwise carry `prev.ContestedUntil` forward; always set `BoxID`, `GeneratedAt` to the current object's. Labels not seen this tick are carried forward unchanged. `IdentityAlert` builds `metrics.AlertState{V: 1, Box: "fleet", UpdatedAt: now, Rules: map[string]metrics.RuleState{}}` with `Firing: true, FiredAt: seenAt` per condition; `contested:<label>` for every `obs.Labels[l].ContestedUntil.After(now)` and every entry of `contested`.
+`internal/status/fleetidentity.go`: `ObserveWriters` as the test pins (carry forward unseen labels; `prevFresh := b.GeneratedAt.Sub(prev.GeneratedAt) <= 3*SnapshotPeriod`; changed = `prev.BoxID != b.BoxID`; `ContestedUntil = max(prev.ContestedUntil, now+3 periods)` when both). `IdentityAlert(prev, ev, obs, mismatched, now)`: start from `prev.Rules`, mark every rule `Firing:false` (keeping its `FiredAt`), then set `Firing:true, FiredAt: seenAt` for each condition (`foreign:`/`twin:`/`unproven:` from evidence, `contested:` from `obs` entries with a future `ContestedUntil` and from `mismatched`), then drop rules that are `Firing:false` and whose `Since` (set to `now` when a rule first becomes false) is older than `clearedRuleRetention`. `LoadObservationsWithETag`.
-`statusTick`'s publish branch (`agent.go:2938`, after `BuildFleet`): read `FleetObservationKey` (`Get`; `os.ErrNotExist` is an empty record; any other error or parse failure logs once and skips rule 3 this tick, `prev` empty and **no write**), `next := ObserveWriters(prev, boxes, now)` where `boxes` is `status.LoadBoxes`'s result (reuse the slice `BuildFleet` loaded: add a `BuildFleetFrom(boxes, ...)` or have `BuildFleet` return the boxes; smallest change is to call `LoadBoxes` once in `statusTick` and pass the slice to a new `BuildFleetFromBoxes`). Load evidence (`LoadEvidence`), compute `contested` from the same `box_id` mismatch rule as the console (extract `status.ContestedLabels(boxes, files)` is more than needed; the holder reads edges files only through `edgesfile`, which `internal/status` cannot import; so the holder's `contested` input is evidence plus observations only, and the console's rule 1 stays console-side). Then `if held.Err() == nil { Put(FleetObservationKey, next, etag) }` ignoring `ErrCASConflict` with a log line, and `putOverwrite(FleetAlertKey, IdentityAlert(...))`. Transitions in the alert object compared with the previous object's `Rules` are delivered through the same `sinks.deliver` as Task 5, one `metrics.Event{Rule: name, Firing, Box: "fleet"}` each.
+`internal/host/identitypublish.go`: `identityPublisher{s, obsETag, alertETag, prevObs, prevAlert}`; `begin` reads both (absent is `""` etag and zero value); `finish(boxes, ev, mismatched, now, leaseErr func() error)`: if `leaseErr() != nil` return it without writing; `next := status.ObserveWriters(prevObs, boxes, now)`; `Put(FleetObservationKey, next, obsETag)` (a conflict returns `ErrCASConflict` and stops); `alert := status.IdentityAlert(prevAlert, ev, next, mismatched, now)`; `Put(FleetAlertKey, alert, alertETag)`; deliver one `metrics.Event{Rule: name, Firing, Box: "fleet"}` per rule whose `Firing` differs from `prevAlert`. `mismatchedLabels(ctx, s, boxes)` lists `edgesfile.Prefix`, parses each file with `edgesfile.Parse`, and returns labels where the file's `BoxID` and the snapshot's `BoxID` are both set and differ. In `statusTick`'s publish branch (after `BuildFleet`): `p := newIdentityPublisher(s); p.begin(ctx)` at the top of the publish branch, and after `PublishFleet`: `ev, _, _ := status.LoadEvidence(ctx, s, now)`; `p.finish(ctx, boxes, ev, mismatchedLabels(ctx, deployStore, boxes), now, held.Err)`; a `ErrCASConflict` is logged once, not returned.
-`alertwatch.go:alertPass`: after the `alerts/boxes/` loop, `Get(status.FleetAlertKey)`; `os.ErrNotExist` is nothing to do; parse; if `now.Sub(st.UpdatedAt) > 3*status.SnapshotPeriod` treat every rule as `Firing: false`; then run the same per-rule `alertRule` loop with `box = "fleet"`. `alertMessage` already names `box` and `rule` only; assert in the test that the body has no `box_id`.
+`alertwatch.go:alertPass`: move the fleet read **above** the `len(keys) == 0` return (`:111`): `Get(status.FleetAlertKey)`; absent is nothing; parse; if `now.Sub(st.UpdatedAt) > 3*status.SnapshotPeriod` set every rule's `Firing=false`; run the same per-rule `alertRule` loop with `box = "fleet"`. Then the existing box loop.
-- [ ] **Step 4: Add the far-future `contested_until` case (Review Focus 4)**
-
-Already in `TestObserveWritersFlagsAChangeWithinFreshnessOnly` ("never shortened"). Add one assertion that `IdentityAlert` with `ContestedUntil` in the past does not fire `contested:` for that label.
-
-- [ ] **Step 5: Run the tests to verify they pass**
+- [ ] **Step 4: Run the tests to verify they pass**
Run: `GOFLAGS=-buildvcs=false go test ./internal/status/ ./internal/console/ ./internal/host/`
Expected: PASS.
-- [ ] **Step 6: Commit**
+- [ ] **Step 5: Commit**
```bash
-git add internal/status/fleetidentity.go internal/status/fleetidentity_test.go internal/host/agent.go internal/console/alertwatch.go internal/console/alertwatch_test.go
-git commit -m "status, console: the lease holder's writer observations and a per-label identity alert (#212)"
+git add internal/status/fleetidentity.go internal/status/fleetidentity_test.go internal/host/identitypublish.go internal/host/agent.go internal/host/agent_test.go internal/console/alertwatch.go internal/console/alertwatch_test.go
+git commit -m "status, host, console: writer observations and a per-label fleet identity alert, written conditionally under the lease (#212)"
```
---
-### Task 8: `carlos fleet check-label` and `carlos fleet retire`
+### Task 8: `carlos status --check-label` and `--retire-label`, the contested chip
**Files:**
-- Modify: `fleets.go:30-121` (usage and dispatch)
-- Create: `fleets_identity.go`, `fleets_identity_test.go`
-- Modify: `cli.go:3418-3440` (`printLiveBoxes` shows `contested`), `internal/console/view.go:830-877` and `templates/fleet.html:100-120` (a `contested` chip on the Hosts table)
+- Modify: `cli.go:3345-3440` (`RunStatus`, `statusUsage`, `printLiveBoxes`)
+- Create: `status_identity.go`, `status_identity_test.go`
+- Modify: `internal/console/view.go:830-877` (`HostRow` gains `Contested bool`, `HolderInstance`, `RefuserInstance string`), `internal/console/templates/fleet.html:100-120`
+- Modify: `docs/site/cli.md` (`### carlos status`, `:1561`)
**Interfaces:**
-- Consumes: Task 2 `status.BoxID`, `status.Label`; Task 5 `status.LoadEvidence`; `release.DeleteIfMatch` (`internal/release/conditional_delete.go:28`); `host.ResolveStatusBucket`.
-- Produces: `func checkLabel(ctx, s release.Store, machineID string, now time.Time, w io.Writer) (exit int)`; `func retireLabel(ctx, s release.Store, label string, now time.Time, w io.Writer) error`; `HostRow.Contested bool`.
+- Consumes: Task 2 `status.BoxID`, `status.Label`; Task 5 `status.LoadEvidence`; `release.DeleteIfMatch`; `host.ResolveStatusBucket`, `host.DeployTarget`.
+- Produces:
+
+```go
+// Both objects' locations: under the supported shape one store, but each is
+// resolved by its own rule and printed (spec 2.3 step 4).
+type labelStores struct {
+ status, deploy release.Store
+ statusWhere, deployWhere string // "s3://bucket/prefix" or "dir:/path", for printing
+}
+func checkLabel(ctx, st labelStores, machineID string, now time.Time, w io.Writer) (exit int)
+func retireLabel(ctx, st labelStores, label string, now time.Time, w io.Writer) error
+const retireAfter = 3*status.SnapshotPeriod + 60*time.Second
+```
- [ ] **Step 1: Write the failing tests**
```go
-// fleets_identity_test.go
-func TestCheckLabelFreeOwnAndForeign(t *testing.T) {
+// status_identity_test.go
+func oneStore(t *testing.T) (labelStores, release.Store) {
s := release.NewDirStore(t.TempDir())
+ return labelStores{status: s, deploy: s, statusWhere: "dir:a", deployWhere: "dir:a"}, s
+}
+
+func TestCheckLabelFreeOwnForeignAndUnproven(t *testing.T) {
+ st, s := oneStore(t)
now := time.Now()
+ const m1, m2 = "00000000000000000000000000000018", "0000000000000000000000000000003c"
var out bytes.Buffer
- if code := checkLabel(context.Background(), s, "00000000000000000000000000000018", now, &out); code != 0 {
+ if code := checkLabel(context.Background(), st, m1, now, &out); code != 0 {
t.Fatalf("a free label exits 0, got %d: %s", code, out.String())
}
- holder := status.CollectBox(nil, status.Probes{MachineID: "00000000000000000000000000000018", InstanceID: "i-a"}, now)
+ holder := status.CollectBox(nil, status.Probes{MachineID: m1, InstanceID: "i-a"}, now)
if err := status.WriteBox(context.Background(), s, holder); err != nil {
t.Fatal(err)
}
out.Reset()
- if code := checkLabel(context.Background(), s, "00000000000000000000000000000018", now, &out); code != 0 {
+ if code := checkLabel(context.Background(), st, m1, now, &out); code != 0 {
t.Fatalf("our own label exits 0, got %d: %s", code, out.String())
}
out.Reset()
- if code := checkLabel(context.Background(), s, "0000000000000000000000000000003c", now, &out); code != 1 {
+ if code := checkLabel(context.Background(), st, m2, now, &out); code != 1 {
t.Fatalf("a label held by another machine exits 1, got %d: %s", code, out.String())
}
- if !strings.Contains(out.String(), "edsger-knitting") || !strings.Contains(out.String(), "i-a") || !strings.Contains(out.String(), "fresh") {
- t.Fatalf("the report names the label, the holder's instance and freshness: %s", out.String())
+ for _, want := range []string{"edsger-knitting", "i-a", "fresh", "dir:a"} {
+ if !strings.Contains(out.String(), want) {
+ t.Fatalf("the report must name %q: %s", want, out.String())
+ }
+ }
+ // unproven: an old-release snapshot with no box_id and no instance id
+ raw, _ := json.Marshal(status.Box{GeneratedAt: now, Label: holder.Label})
+ _, etag, _ := s.Get(context.Background(), "status/boxes/"+holder.Label+".json")
+ _, _ = s.Put(context.Background(), "status/boxes/"+holder.Label+".json", raw, etag)
+ out.Reset()
+ if code := checkLabel(context.Background(), st, m1, now, &out); code != 1 || !strings.Contains(out.String(), "unproven") {
+ t.Fatalf("an object with no box_id is unproven and exits 1: %d %s", code, out.String())
}
}
-func TestRetireLabelRefusesFreshAndDeletesStaleConditionally(t *testing.T) {
- s := release.NewDirStore(t.TempDir())
- now := time.Now()
+func TestRetireLabelRefusesYoungAndDeletesOldConditionally(t *testing.T) {
+ st, s := oneStore(t)
ctx := context.Background()
+ now := time.Now()
holder := status.CollectBox(nil, status.Probes{MachineID: "a", InstanceID: "i-a"}, now)
if err := status.WriteBox(ctx, s, holder); err != nil {
t.Fatal(err)
}
- edges := []byte(`{"v":1,"box":"` + holder.Label + `","eip":"1.2.3.4","hosts":{}}`)
- if _, err := s.Put(ctx, edgesfile.Prefix+holder.Label+".json", edges, ""); err != nil {
+ if _, err := s.Put(ctx, edgesfile.Prefix+holder.Label+".json", []byte(`{"v":1,"box":"`+holder.Label+`","eip":"1.2.3.4","hosts":{}}`), ""); err != nil {
t.Fatal(err)
}
var out bytes.Buffer
- if err := retireLabel(ctx, s, holder.Label, now, &out); err == nil {
- t.Fatal("a fresh snapshot must refuse")
+ if err := retireLabel(ctx, st, holder.Label, now.Add(3*status.SnapshotPeriod+30*time.Second), &out); err == nil {
+ t.Fatal("a snapshot younger than retireAfter must refuse (a mutation may be in flight)")
}
- later := now.Add(4 * status.SnapshotPeriod)
- if err := retireLabel(ctx, s, holder.Label, later, &out); err != nil && !errors.Is(err, release.ErrConditionalDeleteUnsupported) {
- t.Fatalf("stale: %v", err)
- }
- // DirStore has no conditional delete: the verb must refuse rather than
- // fall back, and leave both objects. The S3 path is exercised by the
- // fake below.
- if _, _, err := s.Get(ctx, "status/boxes/"+holder.Label+".json"); err != nil {
- t.Fatal("without conditional delete nothing is deleted")
- }
- fake := &condStore{Store: s} // implements DeleteIfMatch by comparing the etag it is given with Get's
+ later := now.Add(retireAfter + time.Second)
out.Reset()
- if err := retireLabel(ctx, fake, holder.Label, later, &out); err != nil {
+ if err := retireLabel(ctx, st, holder.Label, later, &out); err != nil {
t.Fatalf("retire: %v\n%s", err, out.String())
}
- if _, _, err := s.Get(ctx, "status/boxes/"+holder.Label+".json"); !errors.Is(err, os.ErrNotExist) {
- t.Fatal("snapshot must be gone")
+ for _, key := range []string{"status/boxes/" + holder.Label + ".json", edgesfile.Prefix + holder.Label + ".json"} {
+ if _, _, err := s.Get(ctx, key); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("%s must be gone", key)
+ }
}
- if _, _, err := s.Get(ctx, edgesfile.Prefix+holder.Label+".json"); !errors.Is(err, os.ErrNotExist) {
- t.Fatal("edges file must be gone")
+ if !strings.Contains(out.String(), "i-a") || !strings.Contains(out.String(), "dir:a") {
+ t.Fatalf("retire prints what it deleted and where: %s", out.String())
}
}
-func TestRetireLabelWithNoEdgesFileDeletesTheSnapshotAndSaysSo(t *testing.T) {
- s := release.NewDirStore(t.TempDir())
+func TestRetireLabelRefusesWhenTheObjectChangedUnderIt(t *testing.T) {
+ st, s := oneStore(t)
+ ctx := context.Background()
now := time.Now()
holder := status.CollectBox(nil, status.Probes{MachineID: "a", InstanceID: "i-a"}, now)
- _ = status.WriteBox(context.Background(), s, holder)
+ _ = status.WriteBox(ctx, s, holder)
+ racing := &putBetween{Store: s, key: "status/boxes/" + holder.Label + ".json"} // rewrites the object after Get, before DeleteIfMatch
+ st.status, st.deploy = racing, racing
+ var out bytes.Buffer
+ err := retireLabel(ctx, st, holder.Label, now.Add(retireAfter+time.Second), &out)
+ if err == nil {
+ t.Fatal("an etag change between read and delete must refuse")
+ }
+ if _, _, gerr := s.Get(ctx, "status/boxes/"+holder.Label+".json"); gerr != nil {
+ t.Fatal("nothing may be deleted on a lost race")
+ }
+}
+
+func TestRetireLabelLoneEdgesFileIsDeletedWithAWarning(t *testing.T) {
+ st, s := oneStore(t)
+ ctx := context.Background()
+ if _, err := s.Put(ctx, edgesfile.Prefix+"annie-welding.json", []byte(`{"v":1,"box":"annie-welding","eip":"1.2.3.4","hosts":{}}`), ""); err != nil {
+ t.Fatal(err)
+ }
var out bytes.Buffer
- if err := retireLabel(context.Background(), &condStore{Store: s}, holder.Label, now.Add(4*status.SnapshotPeriod), &out); err != nil {
+ if err := retireLabel(ctx, st, "annie-welding", time.Now(), &out); err != nil {
t.Fatal(err)
}
- if !strings.Contains(out.String(), "edges file: already gone") {
+ if !strings.Contains(out.String(), "snapshot: already gone") || !strings.Contains(out.String(), "edges file has no freshness of its own") {
t.Fatalf("out = %s", out.String())
}
+ if _, _, err := s.Get(ctx, edgesfile.Prefix+"annie-welding.json"); !errors.Is(err, os.ErrNotExist) {
+ t.Fatal("the lone edges file must be gone")
+ }
+}
+
+func TestRetireLabelResolvesSplitStores(t *testing.T) {
+ // Two stores: the snapshot in one, the edges file in the other. The
+ // supported production shape is one bucket, but the command resolves
+ // each object by its own rule and must find both.
+ statusStore, deployStore := release.NewDirStore(t.TempDir()), release.NewDirStore(t.TempDir())
+ st := labelStores{status: statusStore, deploy: deployStore, statusWhere: "dir:status", deployWhere: "dir:deploy"}
+ ctx := context.Background()
+ now := time.Now()
+ holder := status.CollectBox(nil, status.Probes{MachineID: "a", InstanceID: "i-a"}, now)
+ _ = status.WriteBox(ctx, statusStore, holder)
+ _, _ = deployStore.Put(ctx, edgesfile.Prefix+holder.Label+".json", []byte(`{"v":1,"box":"`+holder.Label+`","eip":"1.2.3.4","hosts":{}}`), "")
+ var out bytes.Buffer
+ if err := retireLabel(ctx, st, holder.Label, now.Add(retireAfter+time.Second), &out); err != nil {
+ t.Fatal(err)
+ }
+ if _, _, err := deployStore.Get(ctx, edgesfile.Prefix+holder.Label+".json"); !errors.Is(err, os.ErrNotExist) {
+ t.Fatal("the edges file in the deployment store must be gone")
+ }
+ if !strings.Contains(out.String(), "dir:status") || !strings.Contains(out.String(), "dir:deploy") {
+ t.Fatalf("both locations are printed: %s", out.String())
+ }
+}
+
+// putBetween rewrites key after the first Get so the etag no longer matches.
+type putBetween struct {
+ release.Store
+ key string
+ done bool
+}
+
+func (p *putBetween) Get(ctx context.Context, key string) ([]byte, string, error) {
+ body, etag, err := p.Store.Get(ctx, key)
+ if err == nil && key == p.key && !p.done {
+ p.done = true
+ _, _ = p.Store.Put(ctx, key, append(body, '\n'), etag)
+ }
+ return body, etag, err
+}
+```
+
+```go
+// internal/console/view_test.go (or the existing HostRows test file)
+func TestHostRowsCarryContestedAndBothInstances(t *testing.T) {
+ s := release.NewDirStore(t.TempDir())
+ now := time.Now()
+ raw, _ := json.Marshal(status.Box{GeneratedAt: now, Label: "edsger-knitting", BoxID: status.BoxID("a"), Metrics: &status.BoxMetrics{InstanceID: "i-a"}})
+ _, _ = s.Put(context.Background(), "status/boxes/edsger-knitting.json", raw, "")
+ _ = status.WriteEvidence(context.Background(), s, status.Evidence{V: 1, Label: "edsger-knitting", BoxID: status.BoxID("b"), InstanceID: "i-b", Verdict: status.VerdictForeign, HolderInstanceID: "i-a", SeenAt: now})
+ rows, err := HostRows(context.Background(), s, now)
+ if err != nil || len(rows) != 1 {
+ t.Fatalf("rows = %+v %v", rows, err)
+ }
+ if !rows[0].Contested || rows[0].HolderInstance != "i-a" || rows[0].RefuserInstance != "i-b" {
+ t.Fatalf("row = %+v", rows[0])
+ }
}
```
- [ ] **Step 2: Run the tests to verify they fail**
-Run: `GOFLAGS=-buildvcs=false go test . -run 'CheckLabel|RetireLabel' -v`
+Run: `GOFLAGS=-buildvcs=false go test . ./internal/console/ -run 'CheckLabel|RetireLabel|HostRowsCarry' -v`
Expected: FAIL, undefined.
-- [ ] **Step 3: Implement `fleets_identity.go`**
+- [ ] **Step 3: Implement `status_identity.go`**
+
+`resolveLabelStores(ctx, getenv)`: the status store is `host.ResolveStatusBucket` plus `CARLOS_STATUS_PREFIX` (`release.NewPrefixStore` when set), `statusWhere = "s3://<bucket>/<prefix>"`; the deployment store is `CARLOS_DEPLOYMENT_BUCKET` or `release.NewDirStore(CARLOS_DEPLOYMENT_DIR)`, `deployWhere` likewise. Print both before acting.
-`checkLabel`: `label, id := status.Label(machineID), status.BoxID(machineID)`; print `label <label>, box_id <id>`; `Get` the snapshot and the edges file; print for each: absent, or `box_id`, `instance_id` (snapshot only), `fresh`/`stale` (snapshot, via `BoxStale`), `eip` (edges only); load evidence naming the label and print each entry's verdict and instance ids. Exit 0 when both objects are absent or every present object's `box_id` equals `id`; exit 1 otherwise (foreign, or an object with no `box_id`: unproven). Store resolution for the real command is `host.ResolveStatusBucket` and `release.NewS3Store`, the `statusLive` recipe (`cli.go:3389`).
+`checkLabel`: `label, id := status.Label(machineID), status.BoxID(machineID)`; print `label <label>, box_id <id>`; `Get` the snapshot from `st.status` and the edges file from `st.deploy`; for each print the location and either `absent` or `box_id`, `instance_id` (snapshot), `fresh`/`stale` (snapshot, `BoxStale`), `eip` (edges); print each unexpired evidence entry naming the label (`verdict`, both instance ids); print `claim references: not checked (hostclaim is not on main)`. Exit 0 when both are absent or every present object's `box_id == id`; exit 1 otherwise, printing `verdict: foreign` or `verdict: unproven` (an object with no `box_id`).
-`retireLabel`: `Get` both keys, keeping etags; if the snapshot parses and `!BoxStale(..., now, SnapshotPeriod)` return an error `label <label> has a fresh snapshot (instance <id>); a live writer holds it, refusing`; print both objects' `box_id`/`instance_id`; `release.DeleteIfMatch` each present object on its etag, snapshot first; print `edges file: already gone` when absent; an `ErrConditionalDeleteUnsupported` is returned as is (never a plain `Delete`). The claim check (spec §4) is a comment and a TODO-free note: `internal/hostclaim` is not on main; when it lands, `retireLabel` gains `hostclaim.References(ctx, s, boxID)` before deleting. Write that sentence in the function's doc comment, not as a code placeholder.
+`retireLabel`: read the snapshot (`st.status`) with etag; if present and parses and `now.Sub(GeneratedAt) < retireAfter`, return `label <label>: snapshot is <age> old, younger than retireAfter (<retireAfter>); a mutation may still be in flight, refusing`; read the edges file (`st.deploy`) with etag; print both objects' `box_id`/`instance_id`/`eip` and locations; `release.DeleteIfMatch(ctx, st.status, snapshotKey, etag)` then `release.DeleteIfMatch(ctx, st.deploy, edgesKey, etag)`; absent prints `snapshot: already gone` or `edges file: already gone`; a lone edges file is deleted with `edges file has no freshness of its own (generation is a change stamp); deleting it because its snapshot is gone`; `ErrConditionalDeleteUnsupported` and a failed condition are returned as they are. The doc comment states the claim dependency (spec §4).
-Dispatch in `fleets.go`: `case "check-label": runFleetsCheckLabel(args[1:])`, `case "retire": runFleetsRetire(args[1:])`; usage lines:
+`RunStatus`: `checkFlag := fs.String("check-label", "", "...")`, `retireFlag := fs.String("retire-label", "", "...")`; both resolve stores and dispatch; `statusUsage` gains the two lines. `printLiveBoxes(w, boxes, evidence, now)`: a label named by unexpired evidence prints ` CONTESTED (holder <instance>, refuser <instance>)`.
+`HostRows` loads evidence (a nil store or failed load leaves every row uncontested) and sets `Contested`, `HolderInstance` (from `Metrics.InstanceID`) and `RefuserInstance` (from the evidence); `fleet.html`'s Host cell renders `{{if .Contested}}<span class="chip chip-stale">contested</span> holder {{.HolderInstance}}, refuser {{.RefuserInstance}}{{end}}`.
+
+`docs/site/cli.md` `### carlos status`: add after the `--live` example:
+
+````markdown
```
- check-label <machine-id>
- show who holds the label this machine id gives; exit 1 if another machine does
- retire <label> delete a stale label's snapshot and edges file so a refused box can take it
+carlos status --check-label $(ssh box cat /etc/machine-id)
+carlos status --retire-label edsger-knitting
```
-`printLiveBoxes`: load evidence in `statusLive` and pass a `contested map[string]bool`; a contested row prints ` CONTESTED` after the label. `HostRows` gains `Contested bool` from the same evidence load (nil store or a failed load leaves it false), and `fleet.html`'s Hosts row gets `{{if .Contested}}<span class="chip chip-stale">contested</span>{{end}}` in the Host cell.
+`--check-label` prints the label a machine id gives, who holds it, and
+whether that is this machine; exit 1 when another machine holds it. Run
+it for a new box before bootstrapping the agent. `--retire-label` frees a
+label after a collision, once its snapshot is sixteen minutes old; the
+steps are in the README's fleet status section.
+````
- [ ] **Step 4: Run the tests to verify they pass**
-Run: `GOFLAGS=-buildvcs=false go test . ./internal/console/ -run 'CheckLabel|RetireLabel|HostRows|Fleet'`
-Expected: PASS.
+Run: `GOFLAGS=-buildvcs=false go test . ./internal/console/ ./internal/docsite/`
+Expected: PASS (docsite: the heading exists and both flags are under it).
- [ ] **Step 5: Commit**
```bash
-git add fleets.go fleets_identity.go fleets_identity_test.go cli.go internal/console/view.go internal/console/templates/fleet.html
-git commit -m "carlos fleet check-label and retire; contested chip on the operator fleet page (#212)"
+git add cli.go status_identity.go status_identity_test.go internal/console/view.go internal/console/view_test.go internal/console/templates/fleet.html docs/site/cli.md
+git commit -m "carlos status --check-label and --retire-label; contested rows name both machines (#212)"
```
---
@@ -1685,14 +2297,14 @@ git commit -m "carlos fleet check-label and retire; contested chip on the operat
### Task 9: privacy fixtures
**Files:**
-- Modify: `internal/status/dryrun.go:62-70` (needles), `:182` (fixture), `:336-456` (scan)
-- Test: `internal/status/dryrun_internal_test.go`, `internal/status/status_test.go` (FleetBox), `internal/console/apiinstances_test.go`, `internal/console/alertwatch_test.go`
+- Modify: `internal/status/dryrun.go:62-70` (needles), `:182` (fixture), `:309-456` (scan)
+- Test: `internal/status/dryrun_internal_test.go`, `internal/status/status_test.go`, `internal/console/apiinstances_test.go`
- [ ] **Step 1: Write the failing tests**
```go
// internal/status/status_test.go
-func TestFleetBoxNeverCarriesBoxID(t *testing.T) {
+func TestFleetBoxNeverCarriesBoxIDOrInstanceID(t *testing.T) {
s := release.NewDirStore(t.TempDir())
now := time.Now()
b := status.CollectBox(nil, status.Probes{MachineID: "m", InstanceID: "i-4242"}, now)
@@ -1714,21 +2326,15 @@ func TestFleetBoxNeverCarriesBoxID(t *testing.T) {
```go
// internal/status/dryrun_internal_test.go
-func TestDryRunNeedlesIncludeIdentity(t *testing.T) {
- // The fixture machine id must produce a BoxID the scan can look for,
- // and the dry run must create one evidence object so status/identity/
- // is exercised as a private home.
- if !slices.Contains(dryRunMetricsNeedles, BoxID(dryRunMachineID)) {
- t.Fatal("the fixture BoxID must be a needle")
- }
- if !slices.Contains(dryRunMetricsNeedles, dryRunInstanceID) {
- t.Fatal("the fixture instance id must be a needle")
+func TestDryRunNeedlesIncludeIdentityAndTheRunPublishesEvidence(t *testing.T) {
+ if !slices.Contains(dryRunMetricsNeedles, BoxID(dryRunMachineID)) || !slices.Contains(dryRunMetricsNeedles, dryRunInstanceID) {
+ t.Fatal("the fixture BoxID and instance id must both be needles")
}
var out bytes.Buffer
if err := DryRun(t.TempDir(), &out); err != nil {
t.Fatalf("dry run: %v\n%s", err, out.String())
}
- if !strings.Contains(out.String(), "status/identity/") {
+ if !strings.Contains(out.String(), IdentityPrefix) {
t.Fatal("the dry run must publish and scan an evidence object")
}
}
@@ -1736,13 +2342,17 @@ func TestDryRunNeedlesIncludeIdentity(t *testing.T) {
```go
// internal/console/apiinstances_test.go
-func TestInstanceListNeverCarriesBoxID(t *testing.T) {
- // Use the existing list fixture that seeds one record and one box
- // snapshot; add BoxID and Metrics.InstanceID to the seeded snapshot.
- fx := newInstancesFixture(t)
- fx.seedBox(status.Box{GeneratedAt: time.Now(), Label: "annie-welding", BoxID: status.BoxID("m"), Metrics: &status.BoxMetrics{InstanceID: "i-4242"}})
- body := fx.list(t)
- for _, needle := range []string{status.BoxID("m"), "i-4242", "box_id"} {
+func TestInstanceListNeverCarriesBoxIDOrInstanceID(t *testing.T) {
+ fx, acct := instanceFixture(t, "ie")
+ enableInstances(t, fx, "jam", titoManifest())
+ // Seed the box snapshot the staged_version join reads, carrying both private ids.
+ raw, _ := json.Marshal(status.Box{GeneratedAt: time.Now(), Label: "annie-welding", BoxID: status.BoxID("m"),
+ Metrics: &status.BoxMetrics{InstanceID: "i-4242", RouteVersions: map[string]string{"x.jam." + acct + ".oncarlos.com": "abc1234"}}})
+ if _, err := fx.store.Put(context.Background(), "status/boxes/annie-welding.json", raw, ""); err != nil {
+ t.Fatal(err)
+ }
+ _, _, body := rawReq(t, http.MethodGet, instancesURL(fx, acct, "jam"), fx.bearer, "")
+ for _, needle := range []string{status.BoxID("m"), "i-4242", "box_id", "instance_id"} {
if strings.Contains(body, needle) {
t.Fatalf("instances response carries %q", needle)
}
@@ -1750,36 +2360,36 @@ func TestInstanceListNeverCarriesBoxID(t *testing.T) {
}
```
-The alert mail assertion is already in Task 7's `TestAlertPassReadsTheFleetIdentityObject`.
+(`instanceFixture`, `enableInstances`, `titoManifest`, `instancesURL`, `rawReq` are the existing helpers at `apiinstances_test.go:42-128`; the fixture's store and bearer fields are whatever `consoleFixture` names them.)
- [ ] **Step 2: Run the tests to verify they fail**
-Run: `GOFLAGS=-buildvcs=false go test ./internal/status/ ./internal/console/ -run 'NeverCarriesBoxID|NeedlesIncludeIdentity' -v`
-Expected: `TestFleetBoxNeverCarriesBoxID` PASSES (pins), the dry run test FAILS (no such needles, no evidence object), the instances test FAILS only if the fixture helpers are missing (write them from the existing list test's shape).
+Run: `GOFLAGS=-buildvcs=false go test ./internal/status/ ./internal/console/ -run 'NeverCarries|NeedlesInclude' -v`
+Expected: `TestFleetBoxNeverCarriesBoxIDOrInstanceID` PASSES (pins); the dry-run test FAILS (no needles, no evidence); the instances test PASSES or FAILS only on fixture names.
- [ ] **Step 3: Implement**
-`dryrun.go`: `const dryRunMachineID = "ip-10-0-0-7"`, `const dryRunInstanceID = "i-42424242424242424"`; set `InstanceID: dryRunInstanceID` in the fixture `Probes`; append `BoxID(dryRunMachineID)` and `dryRunInstanceID` to `dryRunMetricsNeedles`. In `DryRun`, after `WriteBox`, write one evidence object: `WriteEvidence(ctx, s, Evidence{V: 1, Label: box.Label, BoxID: box.BoxID, InstanceID: dryRunInstanceID, Verdict: VerdictForeign, HolderBoxID: BoxID("other"), HolderInstanceID: "i-other", SeenAt: now})`. In the scan loop, treat `strings.HasPrefix(key, IdentityPrefix)` like `status/boxes/` for the two identity needles (required in `status/boxes/`, allowed in `status/identity/`, banned everywhere else), and add a `identityKeySeen` positive check mirroring `boxKeySeen`.
+`dryrun.go`: `const dryRunMachineID = "ip-10-0-0-7"`, `const dryRunInstanceID = "i-42424242424242424"`; `Probes{MachineID: dryRunMachineID, InstanceID: dryRunInstanceID, ...}`; append `BoxID(dryRunMachineID)` and `dryRunInstanceID` to `dryRunMetricsNeedles`. After `WriteBox`, `WriteEvidence(ctx, s, Evidence{V: 1, Label: box.Label, BoxID: box.BoxID, InstanceID: dryRunInstanceID, Verdict: VerdictForeign, HolderBoxID: BoxID("other"), HolderInstanceID: "i-other", SeenAt: now})`. In the scan, the two identity needles are REQUIRED in `status/boxes/*.json`, ALLOWED under `IdentityPrefix`, BANNED everywhere else; add `identityKeySeen` mirroring `boxKeySeen` so a run that never scanned an evidence object fails.
- [ ] **Step 4: Run the tests to verify they pass**
Run: `GOFLAGS=-buildvcs=false go test ./internal/status/ ./internal/console/`
-Expected: PASS, including the existing `TestDryRun*`.
+Expected: PASS, including every existing `TestDryRun*`.
- [ ] **Step 5: Commit**
```bash
git add internal/status/dryrun.go internal/status/dryrun_internal_test.go internal/status/status_test.go internal/console/apiinstances_test.go
-git commit -m "status: the dry run proves box_id and instance_id stay out of fleet.json and the page (#212)"
+git commit -m "status: the dry run proves box_id and instance_id stay in the private objects (#212)"
```
---
-### Task 10: simulation and the operator runbook
+### Task 10: simulation, the runbook, and the gate
**Files:**
- Create: `internal/status/decide_sim_test.go`
-- Modify: `README.md` (the "Fleet status" section, after the `CARLOS_FLEET_BOX` bullet near `:614`)
+- Modify: `README.md` (fleet status section, after the `CARLOS_FLEET_BOX` bullet near `:614`, and the `CARLOS_STATUS_BUCKET` bullet at `:578-592`)
- [ ] **Step 1: Write the simulation**
@@ -1787,11 +2397,10 @@ git commit -m "status: the dry run proves box_id and instance_id stay out of fle
// internal/status/decide_sim_test.go
package status_test
-// Spec 7: random fleets with injected collisions, simultaneous starts and
-// clones, asserting that at most one machine writes a label while its
-// object is fresh and that no label is ever reassigned without an
-// operator. Old-release writers (WriteBox bypassed with a plain put) are
-// the stated exception and are asserted to be detected, not refused.
+// Spec 7: random fleets with injected collisions, clones, old-release
+// writers (which bypass the decision), release changes mid-run, store
+// failures, and retirement. Asserts spec 5's first bullet for writers on
+// this release and detection-not-refusal for old ones.
func TestDecideLabelSimulation(t *testing.T) {
seeds := 200
if testing.Short() {
@@ -1800,41 +2409,69 @@ func TestDecideLabelSimulation(t *testing.T) {
for seed := 0; seed < seeds; seed++ {
t.Run(fmt.Sprint(seed), func(t *testing.T) {
rng := rand.New(rand.NewSource(int64(seed)))
- s := release.NewDirStore(t.TempDir())
+ inner := release.NewDirStore(t.TempDir())
+ flaky := &flakyStore{Store: inner, rng: rng}
now := time.Unix(1_700_000_000, 0)
- type machine struct{ id, instance string; old bool }
+ type machine struct {
+ id, instance string
+ old bool // an old-release writer this tick
+ }
n := 2 + rng.Intn(6)
- var ms []machine
- for i := 0; i < n; i++ {
- ms = append(ms, machine{id: fmt.Sprintf("m%d-%d", seed, i), instance: fmt.Sprintf("i-%d-%d", seed, i), old: rng.Intn(5) == 0})
+ ms := make([]machine, n)
+ for i := range ms {
+ ms[i] = machine{id: fmt.Sprintf("m%d-%d", seed, i), instance: fmt.Sprintf("i-%d-%d", seed, i), old: rng.Intn(5) == 0}
}
- // force a collision: give the last machine the first's label by reusing the known pair
ms[0].id, ms[n-1].id = "00000000000000000000000000000018", "0000000000000000000000000000003c"
- if rng.Intn(4) == 0 { // a clone
- ms[n-1].id = ms[0].id
+ if rng.Intn(4) == 0 {
+ ms[n-1].id = ms[0].id // a clone
}
- holder := map[string]string{} // label -> instance that currently holds it (new writers only)
- for tick := 0; tick < 40; tick++ {
+ holder := map[string]string{} // label -> instance owning it among new writers
+ lastOwnedAt := map[string]time.Time{}
+ for tick := 0; tick < 60; tick++ {
now = now.Add(time.Minute)
- rng.Shuffle(len(ms), func(i, j int) { ms[i], ms[j] = ms[j], ms[i] })
+ flaky.failRate = 0
+ if rng.Intn(10) == 0 {
+ flaky.failRate = 0.5 // a store outage window
+ }
+ if rng.Intn(15) == 0 { // a roll: one machine changes release
+ i := rng.Intn(n)
+ ms[i].old = !ms[i].old
+ }
+ if rng.Intn(40) == 0 { // an operator retires a stale label
+ for label, at := range lastOwnedAt {
+ if now.Sub(at) > 3*status.SnapshotPeriod+60*time.Second {
+ key := "status/boxes/" + label + ".json"
+ if _, etag, err := inner.Get(context.Background(), key); err == nil {
+ _ = release.DeleteIfMatch(context.Background(), inner, key, etag)
+ delete(holder, label)
+ delete(lastOwnedAt, label)
+ }
+ }
+ }
+ }
+ rng.Shuffle(n, func(i, j int) { ms[i], ms[j] = ms[j], ms[i] })
for _, m := range ms {
b := boxFor(m.id, m.instance, now)
if m.old {
- put(t, s, b.Label, status.Box{GeneratedAt: now, Label: b.Label, Metrics: &status.BoxMetrics{InstanceID: m.instance}})
+ raw, _ := json.Marshal(status.Box{GeneratedAt: now, Label: b.Label, Metrics: &status.BoxMetrics{InstanceID: m.instance}})
+ key := "status/boxes/" + b.Label + ".json"
+ _, etag, _ := inner.Get(context.Background(), key)
+ _, _ = inner.Put(context.Background(), key, raw, etag) // old WriteBox: blind replace
continue
}
- d, _ := status.DecideLabel(context.Background(), s, b, now)
+ d, _ := status.DecideLabel(context.Background(), flaky, b, now)
switch d.Verdict {
case status.VerdictOwned:
if h, ok := holder[b.Label]; ok && h != m.instance {
t.Fatalf("tick %d: label %s moved from %s to %s without an operator", tick, b.Label, h, m.instance)
}
holder[b.Label] = m.instance
+ lastOwnedAt[b.Label] = now
case status.VerdictForeign, status.VerdictTwin, status.VerdictUnproven:
- raw, _, _ := s.Get(context.Background(), "status/boxes/"+b.Label+".json")
+ raw, _, _ := inner.Get(context.Background(), "status/boxes/"+b.Label+".json")
var cur status.Box
_ = json.Unmarshal(raw, &cur)
- if cur.Metrics != nil && cur.Metrics.InstanceID == m.instance && cur.BoxID == b.BoxID {
+ if cur.BoxID == b.BoxID && cur.Metrics != nil && cur.Metrics.InstanceID == m.instance {
t.Fatalf("tick %d: %s refused its own object", tick, m.instance)
}
}
@@ -1843,46 +2480,92 @@ func TestDecideLabelSimulation(t *testing.T) {
})
}
}
+
+// flakyStore fails reads and writes at failRate with a non-CAS error.
+type flakyStore struct {
+ release.Store
+ rng *rand.Rand
+ failRate float64
+}
+
+func (f *flakyStore) Get(ctx context.Context, key string) ([]byte, string, error) {
+ if f.rng.Float64() < f.failRate {
+ return nil, "", errors.New("flaky: read failed")
+ }
+ return f.Store.Get(ctx, key)
+}
+
+func (f *flakyStore) Put(ctx context.Context, key string, body []byte, ifMatch string) (string, error) {
+ if f.rng.Float64() < f.failRate {
+ return "", errors.New("flaky: write failed")
+ }
+ return f.Store.Put(ctx, key, body, ifMatch)
+}
```
- [ ] **Step 2: Run it**
Run: `GOFLAGS=-buildvcs=false go test ./internal/status/ -run Simulation -v`
-Expected: PASS. If a seed fails, the failure names the tick and the machines; fix `classify`, not the test.
+Expected: PASS. A failing seed names the tick and machines; fix `DecideLabel`, not the test.
- [ ] **Step 3: README**
-Add under the fleet status section, after the `CARLOS_FLEET_BOX` bullet:
+Under the fleet status section, after the `CARLOS_FLEET_BOX` bullet:
```markdown
-- Box identity (#212). Every platform box carries a private `box_id`
- (a digest of its machine id) in its snapshot and its edges file, and
- refuses to publish under a label another machine holds. A refusing box
- logs `label <label> is held by another machine`, publishes evidence
- under `status/identity/`, and the operator fleet page shows the label
- as contested. To fix a collision: stop the new box's agent, run
- `rm /etc/machine-id && systemd-machine-id-setup && reboot` on it, wait
- fifteen minutes for the old objects to go stale, then
- `carlos fleet retire <label>`. Before a new box joins, run
- `carlos fleet check-label $(cat /etc/machine-id)` against the status
- bucket. Design: docs/superpowers/specs/2026-10-06-box-identity-detect-design.md.
+- Box identity (#212, design
+ docs/superpowers/specs/2026-10-06-box-identity-detect-design.md). Every
+ platform box carries a private `box_id` (a digest of its machine id) in
+ its snapshot and its edges file, and refuses to publish anything under
+ a label another machine holds. A refusing box logs
+ `label <label> is held by another machine`, publishes evidence under
+ `status/identity/`, and the operator fleet page shows the label as
+ contested with both machines' instance ids.
+ - Before a new box joins: `carlos status --check-label <machine-id>`
+ (read the id with `cat /etc/machine-id` over SSM after first boot,
+ before bootstrapping the agent). Exit 1 means another machine holds
+ that label; give the box a new machine id before it serves anything.
+ - To fix a collision: `carlos status --check-label` on the newcomer
+ (stop if it names an open claim operation); `systemctl stop
+ carlos-edge.service` on the newcomer, and its relay unit if one is
+ installed; `rm /etc/machine-id && systemd-machine-id-setup && reboot`
+ on the newcomer; wait sixteen minutes; `carlos status --retire-label
+ <label>`. The refused box takes the label on its next tick.
+ - The status store must be the deployment bucket at the root; a
+ separate `CARLOS_STATUS_BUCKET` or a `CARLOS_STATUS_PREFIX` on a
+ platform box stops it publishing (fleet boxes are exempt).
+```
+
+And in the `CARLOS_STATUS_BUCKET` bullet (`:578-592`) replace the sentence suggesting a separate status bucket with: "A separate status bucket is not supported with box identity (above); leave `CARLOS_STATUS_BUCKET` unset on platform boxes."
+
+- [ ] **Step 4: The full gate**
+
+Run, from the worktree, with the repo's `-exec` wrapper for the socket tests:
+
+```bash
+GOFLAGS=-buildvcs=false go build ./... && go vet ./... && gofmt -l . && go test ./... -count=1
+GOOS=darwin GOARCH=arm64 go build ./... && GOOS=windows go build ./... && GOOS=linux GOARCH=arm64 go build ./...
```
-- [ ] **Step 4: Commit**
+Expected: all clean, `gofmt -l .` prints nothing.
+
+- [ ] **Step 5: Commit**
```bash
git add internal/status/decide_sim_test.go README.md
-git commit -m "status: label-ownership simulation; README: the collision runbook (#212)"
+git commit -m "status: label-ownership simulation with rolls, outages and retirement; README: the collision runbook (#212)"
```
+Then push and read the CI verdict: `git push origin design-box-identity && amadan ci status carlos/platform -branch design-box-identity`.
+
---
## Self-review
-**Spec coverage.** §2.1 table: Task 2. §2.2 identity file, `Publishable`, `gatedStore`, entry points, fleet verdict, `CARLOS_RELAY_BOX`, edges gate reset and foreign seed: Tasks 3 and 4. §2.3 log texts, evidence, webhook, transitions: Task 5; freeing a label: Task 8. §2.4 evidence shape and expiry: Task 5. §2.5 race: Task 2 (`TestDecideLabelSimultaneousStartOneWins`) and directory deployments: Task 4 step 3c. §2.6: no code; Task 6 pins `EdgeIPs`/`DeploymentRoutes` unchanged. §2.7 `check-label`: Task 8. §3 rules 1 and 2: Task 6; rule 3 and the alert object: Task 7. §4 `ClaimEligible`: Task 3 (the claim itself is not on main; `retire`'s claim check is a documented follow-up in Task 8). §7 privacy fixtures: Task 9; simulation: Task 10. §8 runbook: Task 10. Gap: none found.
+**Spec coverage.** §2.1 table, fresh decision per attempt, owned after Put, complete snapshot: Task 2 (`TestDecideLabelFailedReplaceIsUnknownNotOwned`, `TestDecideLabelRetryDoesNotInheritTheFirstRead`, `TestDecideLabelWritesTheCompleteSnapshot`) and Task 4 (`collectSnapshot`, `TestDecideIdentityFailedWriteDoesNotRenewAuthority`). §2.2 identity file, one identity path, `fleet`/`configured` never expiring, `mutationTTL`, every mutation gated including relay retries, entry points, `CARLOS_RELAY_BOX`, edges gate reset and seed discard: Tasks 3 and 4. §2.3 texts, evidence, webhook: Task 5; the freeing procedure: Task 8 (`retireAfter`, both locations, lone edges file) and Task 10 (README). §2.4 evidence and the `fleet.json` skip: Task 5. §2.5 race (sequential and interleaved), CAS exhaustion, directory deployments: Tasks 2 and 4. §2.6 namespace rule: Task 4 (`statusNamespaceOK`); routing unchanged: Task 6 pins `EdgeIPs`/`DeploymentRoutes`. §2.7 flags on `carlos status`, cli.md: Task 8. §3 rules 1 to 3, monotonic completeness, reads before the early return, the AND at `:123`: Task 6; observation file under the lease, conditional writes, lost lease, mismatch rule computed on the box, cleared rules retained, watcher reads the fleet object first: Task 7. §4 `ClaimEligible`: Task 3; the claim dependency: Task 8's doc comment and printed line. §7 privacy fixtures with both identities seeded: Tasks 7 and 9. Simulation with rolls, outages and retirement: Task 10. §8 squash landing: the header; runbook and the status-bucket line: Task 10; cli.md: Task 8; the carloku-infrastructure README is a change in that repository after this lands (noted in the spec, not a task here).
-**Placeholders.** `retireLabel`'s claim check is a doc-comment sentence, not code; `snapshotPass` and the fixture helpers (`writeSnapshot`, `writeEdges`, `writeBox`, `writeFile`, `newTestRegistry`, `newAlertFixture`, `newInstancesFixture`, `condStore`, `flipStore`, `envFunc`, `testConfig`) are named after the helpers the existing test files already contain; where a name differs, use the existing one, and write `condStore` and `flipStore` (ten lines each) in the test file that uses them.
+**Placeholders.** Every code step has code. Helper names are the existing ones (`edgesTestRegistry`, `snapTestRegistry`, `foldTestStore`, `putEdgesFile`, `putFreshBox`, `newCapturedFixture`, `alertFixture`, `instanceFixture`, `enableInstances`, `titoManifest`, `instancesURL`, `rawReq`, `envFunc`); the new test doubles (`failingPut`, `casOnce`, `interleave`, `alwaysConflict`, `deadlineSpy`, `flipStore`, `refuseAfterGet`, `putBetween`, `flakyStore`) are written out in the tests that use them. `identityPublisher` is specified by its two methods and their test.
-**Type consistency.** `status.Decision`, `status.Verdict`, `host.StoreTarget`, `host.BoxIdentity`, `host.gated(s, id BoxIdentity, target)`, `status.Evidence`, `status.FleetObservations`, `status.IdentityAlert` are used with the same names and parameter orders in Tasks 2 to 9. Task 3's test first wrote `gated(inner, "annie-welding", ...)` and was corrected to the `BoxIdentity` form; the implementation uses that form.
+**Type consistency.** `status.Decision`, `status.Verdict`, `host.StoreTarget{Bucket, Dir}`, `host.BoxIdentity`, `host.Gated(s, label)`, `host.Publishable(label)`, `status.Evidence`, `status.FleetObservations`, `status.LabelObservation`, `status.IdentityAlert(prev, ev, obs, mismatched, now)`, `labelStores`, `retireAfter` are used with the same names and parameter orders throughout. Task 4's `decideIdentityIn` takes the collected `status.Box`, not probes, everywhere it appears.
-**Review Focus.** 1 and 2 are rows in Task 2's table; 3 is `TestPublishableRefusesMissingOrForeignShapedFile` in Task 3; 4 is the "never shortened" assertion plus the past-`ContestedUntil` assertion in Task 7; 5 is `TestRetireLabelWithNoEdgesFileDeletesTheSnapshotAndSaysSo` in Task 8.
+**Review Focus.** 1 and 2 are rows in Task 2's table; 3 is `TestPublishableRefusesMissingOrOlderShapedFile` in Task 3; 4 is the "never shortened" and "past contested_until" assertions in Task 7; 5 is `TestRetireLabelLoneEdgesFileIsDeletedWithAWarning` in Task 8.
diff --git a/docs/superpowers/specs/2026-10-06-box-identity-detect-design.md b/docs/superpowers/specs/2026-10-06-box-identity-detect-design.md
index f67ce24f6..ba32538c3 100644
--- a/docs/superpowers/specs/2026-10-06-box-identity-detect-design.md
+++ b/docs/superpowers/specs/2026-10-06-box-identity-detect-design.md
@@ -1,24 +1,28 @@
# Box identity, Option B: detect and refuse, no key migration
**Date:** 2026-10-06
-**Revision:** 3
+**Revision:** 4
-- Rev 1 (7ff690c5) went to Astra (GPT-6, via codex); the findings are in
+- Rev 1 (7ff690c5) went to Astra (GPT-6, via codex); findings in
`CODEX-FINDINGS-BOX-IDENTITY-B-R1.md`: seven P1s, five P2s, accepted.
- Changes marked "(codex B-N)".
-- Rev 2 (8c362f12) went back; the findings are in
- `CODEX-FINDINGS-BOX-IDENTITY-B-R2.md`: four P1s, six P2s, one P3,
- accepted. Changes marked "(codex B2-N)". §10 records every disposition.
-- **Rev 3 records Paul's choice of Option B (2026-10-06) and four rulings
- taken by default for him to override (§9).** Rev 3 has not been
- reviewed.
+- Rev 2 (8c362f12) went back; findings in `CODEX-FINDINGS-BOX-IDENTITY-B-R2.md`:
+ four P1s, six P2s, one P3, accepted.
+- Rev 3 (2b7641ec) recorded Paul's choice of Option B and four default
+ rulings (§9), and went back with the plan; findings in
+ `CODEX-FINDINGS-BOX-IDENTITY-B-R3.md`: three P1s and seven P2s, most of
+ them in the plan, plus two spec requirements (a retirement barrier and
+ an enforceable arbitration namespace). All accepted; changes are marked
+ "(codex B3-N)" and §10 records every disposition.
+- **Rev 4 has not been reviewed.** Code line references in this spec are
+ against main at `04cd2d78`; the branch has since merged main (`fb59cb23`)
+ and a few line numbers have moved by tens of lines.
**Discussion:** amadan carlos/platform #212
**Sibling of:** `2026-10-06-box-identity-collisions-design.md` (Option A,
migrate the keys), superseded by this spec as the recommendation on
2026-10-06; it stays as the record and its round-3 P1s stay open.
**Plan:** `docs/superpowers/plans/2026-10-06-box-identity-detect.md`.
-**Status:** chosen by Paul; rev 3 awaits Astra's third round.
+**Status:** chosen by Paul; rev 4 awaits Astra's fourth round.
## 1. What Option B is
@@ -46,33 +50,37 @@ an operator acts.
The arbiter is the status snapshot: `statusTick` writes it unconditionally
every tick (`internal/host/agent.go:2905`) and `status.BoxStale` defines
-its freshness. `status.DecideLabel` is called by the agent at arm, before
-any per-box publisher starts, and again on every status tick. It reads
-the label's object and returns one verdict:
+its freshness. `status.DecideLabel(ctx, s, b, now)` is called by the agent
+at arm, before any per-box publisher starts, and again on every status
+tick, **with the complete collected snapshot** `b` (routes, aliases,
+probes: `status.CollectBox(routes, probes, now)` exactly as `statusTick`
+builds it today, `agent.go:2905-2926`), never a routeless stand-in (codex
+B3-3). It reads the label's object and returns one verdict:
| Object under our label | Verdict |
|---|---|
| not found | **owned**, after a create-only `Put` (`ifMatch ""`) succeeds. A CAS conflict re-reads and re-decides; it never re-Puts with a fresh etag without re-reading the body (that re-Put, `snapshot.go:749-760`, is the overwrite this spec removes) |
-| `box_id` equals ours, and `metrics.instance_id` equals ours or either is empty | **owned**, replace with the etag |
+| `box_id` equals ours, and `metrics.instance_id` equals ours or either is empty | **owned**, after a replace with the etag succeeds |
| `box_id` equals ours, both instance ids non-empty and different | **twin**: two machines with one machine id. Refuse |
| `box_id` non-empty and different, fresh or stale | **foreign**. Refuse. Staleness does not free a label |
-| `box_id` absent (an old-release writer, or this box before it rolled), instance ids equal and non-empty | **owned** |
+| `box_id` absent (an old-release writer, or this box before it rolled), instance ids equal and non-empty | **owned**, after a replace with the etag succeeds |
| `box_id` absent, both instance ids non-empty and different | **foreign**. Refuse |
-| `box_id` absent, either instance id empty, fresh or stale | **unproven**. Refuse. There is no machine-bound evidence to promote it: an EIP is a separate resource that a replacement instance can inherit (`infra/modules/region-host/main.tf:503`) and an edges file persists without refresh (`edgesfile.go:491`), so neither proves the object is ours (codex B2-1). An operator frees the label (§2.3) |
-| body does not parse, read fails, CAS exhausted (three attempts) | **unknown**. No write; retry next tick |
+| `box_id` absent, either instance id empty, fresh or stale | **unproven**. Refuse. There is no machine-bound evidence to promote it: an EIP is a separate resource that a replacement instance can inherit (`infra/modules/region-host/main.tf:503`) and an edges file persists without refresh (`edgesfile.go:491`), so neither proves the object is ours. An operator frees the label (§2.3) |
+| body does not parse, read fails, **any `Put` fails**, CAS exhausted (three attempts) | **unknown**. Retry next tick |
-Nothing is ever promoted to owned by reading back its own bytes: the only
-writes are the create-only `Put` and a replace of an object already proved
-ours. `owned` is recorded only after that write succeeds.
+Each attempt starts from a fresh decision built from that attempt's read;
+nothing carries over from a previous attempt (codex B3-1). `owned` is the
+verdict **only after** the create or replace `Put` returned success: a
+read that classifies the object as ours followed by a failed `Put` is
+unknown, not owned, so a box whose writes fail cannot renew its own
+authority while its snapshot goes stale (codex B3-1).
Fleet boxes (`agent.go:2926`, operator-chosen label, unique per fleet by
`registry.ErrFleetBoxActive`) publish no `box_id` and skip the decision;
-their identity is the fleet label, not the machine, and §2.2 says what
-the gate does for them (codex B2-9). A box whose machine id is unreadable
-(`probes.go:153`) has no `box_id` and no identity: it is **unidentified**,
-publishes nothing keyed by its label, takes no claim work, and logs once
-an hour. `Label("")` is no longer a thing any platform box publishes
-under.
+their identity is the fleet label, not the machine (§2.2). A box whose
+machine id is unreadable (`probes.go:153`) has no `box_id` and no
+identity: it is **unidentified**, publishes nothing keyed by its label,
+takes no claim work, and logs once an hour.
On the flagship, Tito and dev every box has an instance id, so the
unproven row is reached only by hardware no cloud owns.
@@ -85,42 +93,54 @@ because not every publisher lives in the agent process:
```json
{"v": 1, "box_id": "...", "label": "edsger-knitting", "verdict": "owned",
"decided_at": "2026-10-06T10:00:00Z",
- "store": {"bucket": "carlos-flagship-deploy", "prefix": ""}}
+ "store": {"bucket": "carlos-flagship-deploy", "dir": ""}}
```
-`host.Publishable(label string, target StoreTarget) bool` reads it and
-answers true only when (codex B2-2):
+`host.Publishable(label string) bool` reads it and answers true only when:
-- the file's `box_id` equals `status.BoxID(readMachineID())` of the
- **calling** process (the file is this machine's, not a copy);
+- the file's `box_id` equals `boxIdentity(os.Getenv, readMachineID()).ID`
+ of the **calling** process, resolved through the one identity selector
+ (so on a fleet box both are the fleet label, and on a platform box both
+ are the machine's `BoxID`; codex B3-10);
- the file's `label` equals the label the caller is about to publish
under, exactly;
-- the file's `store` equals the caller's bucket and prefix (one
- arbitration namespace per publication namespace, §2.6);
-- the verdict is owned and `decided_at` is within three snapshot periods.
+- the file's `store` equals the caller's publication namespace (§2.6);
+- the verdict is `owned` and `decided_at` is within three snapshot
+ periods; or the verdict is `fleet` (a fleet box, whose uniqueness is the
+ registry's) or `configured` (a directory deployment, one box by
+ construction, §2.5), neither of which expires (codex B3-4).
A missing, unparsable or stale file answers false. A box that cannot
re-prove its label for fifteen minutes stops writing under it, so an
outage on one box cannot leave two machines writing one label. **Ruling
R3** (§9): this is deliberate, and metrics and logs stop too.
-Fleet boxes write the same file with `verdict: "fleet"` and their fleet
-label; `Publishable` treats `fleet` as owned for that label. Their
-uniqueness is the registry's, not the machine's.
-
-The gate is enforced at **every mutation**, not at loop entry (codex
-B2-5): `host.gatedStore` wraps a `release.Store` and checks `Publishable`
-before each `Put` and `Delete`, returning `ErrNotPublishable`, so a
-multi-write operation (the registry snapshot's generation put, current
-put, sidecar put and prune, `snapshot_loop.go:104-130`; `putReplacing`'s
-retries, `:221`) aborts at the next call when the verdict changes. Reads
-and lists are not gated. In-flight mutations are seconds long; a label is
-freed by an operator only after fifteen minutes of staleness (§2.3), so
-none can land after reassignment.
+The gate is enforced at **every mutation**: `host.Gated(s, label)` wraps a
+`release.Store` and checks `Publishable` before each `Put`, `Delete` and
+`DeleteIfMatch`, returning `ErrNotPublishable`, so a multi-write
+operation (the registry snapshot's generation put, current put, sidecar
+put and prune, `snapshot_loop.go:104-130`; `putReplacing`'s retries,
+`:221`; the relay meter's three CAS attempts and its corrupt-chunk
+preservation write, `internal/relay/meter.go:167-190`) aborts at the next
+call when the verdict changes (codex B3-5). Reads and lists are not gated.
+
+**Bounded mutations, and the retirement barrier** (codex B3, spec
+requirement 1). Every gated mutation runs under a context deadline of
+`mutationTTL = 60s`, applied by `Gated` itself, so a mutation admitted at
+the last publishable instant (`decided_at + 3 periods`) has ended, one way
+or the other, by `decided_at + 3 periods + mutationTTL`. A label's objects
+may be deleted only when its snapshot's `generated_at` is older than
+`retireAfter = 3 * SnapshotPeriod + mutationTTL` (sixteen minutes): the
+snapshot is written on every owned decision, so `generated_at` is the
+last `decided_at` any process on that machine could have read, and
+`retireAfter` is the drain interval past that. Standalone processes
+(`relay serve`, root-side `adopt`) read the same file and expire at the
+same instant, so stopping the agent is not what drains them; the barrier
+is.
The entry points, each of which today derives the label from
`metricsBoxLabel`, `BoxLabel` or `status.Label(readMachineID())`, and
-each of which publishes through a `gatedStore`:
+each of which publishes through `Gated`:
| Process | Entry point | Key |
|---|---|---|
@@ -131,12 +151,11 @@ each of which publishes through a `gatedStore`:
| agent | `usageFlush` (`usage_loop.go:72`) | usage objects stamped with the box |
| agent | `logshipLoop` and `chunkPrune`, `errshipLoop`, the analytics flush (`agent.go:244,245,477,480,569`; `logship.go:92,161`) | `apps/<sqid>/<app>/{logs,errors,analytics}/<label>/` |
| agent, and root-side `adopt` | `configAckFunc` (`agent.go:954`, `agent.go:3486`) | `control/config-ack/<label>/<host>.json` |
-| `relay serve` | heartbeat and meter flush (`relay.go:138`, `internal/relay/serve.go:343,354`) | `control/relay-ack/<label>.json`, `metrics/relay/<label>/` |
+| `relay serve` | its whole `cfg.Store`, wrapped in `relay.go` before `Serve` (`relay.go:138`), so the heartbeat (`serve.go:343`) and every meter attempt (`meter.go:167-190`) are gated per mutation | `control/relay-ack/<label>.json`, `metrics/relay/<label>/` |
-`relay serve` accepts `CARLOS_RELAY_BOX` (`serve.go:152`). It publishes
-only when `Publishable(cfg.Box, target)` is true, which requires the
-override to equal the label this machine owns; a mismatched override is
-logged once and the relay publishes nothing (codex B2-2).
+`relay serve` accepts `CARLOS_RELAY_BOX` (`serve.go:152`). Its store is
+gated for `cfg.Box`, so an override that is not the label this machine
+owns publishes nothing; the refusal is logged once per transition.
A publisher that is not publishable skips its write and keeps its state:
the sinks buffer to their caps, the sampler keeps its ring, the edges
@@ -146,9 +165,12 @@ untouched: the edge, the activator and the reconcile pass do not depend
on these publishers.
Transitions **to** owned reset the edges gate: `*last` is cleared so the
-next pass republishes, and `seedEdgesFileGate` (`edgesfile.go:529`)
-discards a seeded file whose `box_id` is not ours and never inherits its
-EIP for the blank-probe fallback (`edgesfile.go:478`).
+next pass republishes. `seedEdgesFileGate` (`edgesfile.go:529`) keeps a
+seed only when its `box_id` equals ours; a seed with a different **or
+absent** `box_id` is discarded and its EIP is never inherited for the
+blank-probe fallback (`edgesfile.go:478`), because snapshot ownership
+proves nothing about who wrote the edges object (codex B3-7). The cost is
+one extra republish per box on the first roll.
### 2.3 Refusal, and how a label is freed
@@ -156,11 +178,11 @@ On foreign, twin or unproven:
- the box logs, once per transition, with the instructions. Foreign:
`label <label> is held by another machine (instance <id>); this box
- (instance <id>) is not publishing. Fix the box that is new (§2.3 of the
- box-identity spec; carlos fleet check-label shows both).` Twin: `two
- machines share this machine id; fix the one that is new.` Unproven:
- `cannot prove label <label> is ours (no instance id to compare); if it
- is, carlos fleet retire <label> frees it.`
+ (instance <id>) is not publishing. Fix the box that is new (box-identity
+ spec 2.3; carlos status --check-label shows both).` Twin: `two machines
+ share this machine id; fix the one that is new.` Unproven: `cannot prove
+ label <label> is ours (no instance id to compare); if it is,
+ carlos status --retire-label <label> frees it.`
- the box writes the evidence object (§2.4) every tick while refusing, and
delivers one `metrics.Event{Rule: "identity"}` through the metrics
loop's sinks (`metrics_loop.go:510`) on the transition, so
@@ -169,32 +191,42 @@ On foreign, twin or unproven:
- every tick re-runs §2.1. The verdict changes only when the object
changes or this box's label changes.
-**Freeing a label** is the operator's, in this order (codex B2-4). The
-alert names the holder (the object's writer) and the refuser; the
+**Freeing a label** is the operator's, in this order (codex B2-4, B3-6).
+The alert names the holder (the object's writer) and the refuser; the
operator decides which machine is the newcomer, normally the one with no
history under the label:
-1. Stop the newcomer's agent (`systemctl stop carlos-agent`), so nothing
- writes under the label from it. If the newcomer holds the label and
- the established box is the refuser, this is the step that lets the
- object go stale.
-2. Regenerate the newcomer's machine id: `rm /etc/machine-id &&
+1. `carlos status --check-label <newcomer machine id>` prints both sides
+ (§2.7). Once the claim lands (§4), it also prints any open claim
+ operation naming the newcomer's `box_id`; **stop here** while one is
+ open and let the claim's own recovery finish it, because regenerating
+ the machine id afterwards would orphan the reference.
+2. Stop the newcomer's publishers: `systemctl stop carlos-edge.service`
+ (the agent's unit, which runs `carlos agent`), and the relay's unit
+ where one is installed. This lets the label's snapshot start going
+ stale; it is not the barrier (§2.2 is).
+3. Regenerate the newcomer's machine id: `rm /etc/machine-id &&
systemd-machine-id-setup && reboot`. It comes back under a new label
and a new `box_id`, decides that label at arm, and publishes.
-3. Once the contested label's objects are stale (fifteen minutes after
- step 1), `carlos fleet retire <label>` deletes `status/boxes/<label>.json`
- and `control/edges/<label>.json` with `release.DeleteIfMatch` on the
- etags it read, after printing both objects' `box_id` and `instance_id`.
- It refuses while either object is fresh (no `--force`: a fresh object
- has a live writer), and it refuses while the claim names the holder's
- `box_id` in any open operation, which is the claim's own retirement to
- resolve first.
-4. The refuser's next tick finds not-found, creates, and is owned.
+4. Once the contested label's snapshot is older than `retireAfter`
+ (sixteen minutes after step 2), `carlos status --retire-label <label>`
+ resolves **each object's actual location** (the snapshot in the status
+ store, the edges file in the deployment bucket; under the supported
+ shape, §2.6, they are one bucket at the root, and the command prints
+ both locations before acting), deletes both with
+ `release.DeleteIfMatch` on the etags it read, and prints each object's
+ `box_id` and `instance_id`. It refuses while the snapshot is younger
+ than `retireAfter` (no `--force`: a younger snapshot may still have a
+ mutation in flight), and it refuses while the claim names the holder's
+ `box_id` in any open operation. An edges file has no freshness of its
+ own (`generation` is a change stamp, not a heartbeat, `edgesfile.go:491`);
+ it is deleted with its snapshot or, when the snapshot is already gone,
+ on its own with a printed warning.
+5. The refuser's next tick finds not-found, creates, and is owned.
The holder is never told to regenerate: an established box's `box_id` is
in its claim history and its label is in its objects' history; the
-newcomer has neither. `carlos fleet check-label <machine-id>` (§2.7)
-prints what each side holds so the operator can tell them apart.
+newcomer has neither.
### 2.4 Evidence
@@ -207,11 +239,12 @@ prints what each side holds so the operator can tell them apart.
```
Keyed by the writer's own id, so two colliding boxes never share an
-evidence key, written in the status store beside the snapshots (§2.6).
-Written every tick while the verdict is not owned; an entry older than
-three snapshot periods is expired and ignored. The prefix is under
-`status/` but is not one of the three keys the edge serves
-(`internal/edge/status.go:54`).
+evidence key, written in the status store beside the snapshots. Written
+every tick while the verdict is not owned; an entry older than three
+snapshot periods is expired and ignored. The prefix also holds the lease
+holder's `fleet.json` (§3), which has its own shape; the evidence loader
+skips that key by name (codex B3-2). The prefix is under `status/` but is
+not one of the three keys the edge serves (`internal/edge/status.go:54`).
### 2.5 Two new boxes start at once with an empty key
@@ -220,105 +253,147 @@ Both read not-found and both `Put` with `ifMatch ""`. On S3 that is
succeeds; the other gets `ErrCASConflict`, re-reads, finds a fresh object
with a foreign `box_id`, and refuses. The decision happens at arm, before
any other publisher runs, so the loser writes nothing at all. S3 is the
-arbitrating backend. `DirStore` enforces CAS under a mutex **per handle**
-(`store.go:258`; codex B2-11), so it arbitrates writers that share one
-handle, which is what the tests use, and a one-box deployment needs no
-arbitration. A deployment with no status store (`CARLOS_DEPLOYMENT_DIR`,
-`agent.go:2717`) is one box by construction: its verdict is owned by
-configuration, logged once, and the gate is open.
-
-### 2.6 One arbitration namespace, and what refusal does not do
-
-Every writer that shares a publication namespace (the deployment bucket's
-label-keyed keys) must decide its label in the **same status store**
-(bucket and prefix), or two boxes could each be owned in their own
-namespace and still collide in the shared one (codex B2-6). The identity
-file records the store (§2.2); `carlos fleet check-label` prints it. The
-console reads snapshots, evidence and the fleet observation file from
-`cfg.Releases` (`instancesplacement.go:85`, `alertwatch.go:106`), so the
-supported shape for §3's rules is the one every deployment runs today:
-status store equals deployment bucket at an empty prefix. A split status
-bucket (`CARLOS_STATUS_BUCKET`) keeps today's degradation (the console
-sees no snapshots and no evidence, says so once, and automatic
-eligibility counts nothing outside the home); this spec does not wire a
-second store into the console.
+arbitrating backend. `DirStore` enforces CAS under a mutex per handle
+(`store.go:258`), so it arbitrates writers that share one handle, which
+is what the tests use, and a one-box deployment needs no arbitration. A
+deployment with no status store (`CARLOS_DEPLOYMENT_DIR`, `agent.go:2717`)
+is one box by construction: the agent writes `verdict: "configured"`
+once at arm, logs once, and that verdict does not expire (codex B3-4).
+
+### 2.6 One arbitration namespace, enforced
+
+A deployment's label-keyed keys live in its deployment bucket (or
+directory). The arbiter for those keys must be the snapshot in **that same
+bucket at the root**; two machines each owned in a private status
+namespace would still collide in the shared one. Recording a chosen
+namespace does not make two machines agree, so the agent **rejects the
+unsupported combination** instead (codex B3, spec requirement 2): a
+platform box whose `CARLOS_STATUS_BUCKET` names a bucket other than
+`CARLOS_DEPLOYMENT_BUCKET`, or whose `CARLOS_STATUS_PREFIX` is non-empty,
+does not decide, writes `verdict: "unknown"` with the reason, closes
+every gated publisher, and logs once:
+`agent: box identity needs the status store to be the deployment bucket
+at the root (CARLOS_STATUS_BUCKET=%q, CARLOS_STATUS_PREFIX=%q); not
+publishing under label %s until that is so`. Fleet boxes are exempt:
+their prefix is `fleets/<name>/` by design and their uniqueness is the
+registry's. The identity file's `store` is therefore the deployment
+bucket (or directory) itself, and `Publishable` compares it with the
+caller's deployment store.
+
+The flagship, Tito and dev all run that shape today (empty prefix, one
+bucket); README's suggestion of a separate status bucket becomes "not
+with box identity", and the plan updates that line. The console reads
+snapshots, evidence and the fleet observation file from `cfg.Releases`
+(`instancesplacement.go:85`, `alertwatch.go:106`), which under this rule
+is the same bucket the boxes decide in.
Refusal does not withdraw routing (codex B-7, B2-10). Steering
(`steering.go:119`) and the home map (`homemap.go:273`) read edges files
as they do today and consume none of §3's verdicts; Option B changes
-nothing there. The edges file in the bucket is the holder's: a refuser
-has written nothing under the label since it refused, and before that
-only as owner. Exceptions, which are costs and not protocol (§6.2): an
-old-release sibling cannot refuse and overwrites the holder's objects
-with its own during a roll, and routing then follows the old box; and
-when a label's host set flips between two machines the steering
-converger's shrink guard (`route53.go:209`) may refuse a pass, which is
-today's collision behaviour, now shorter.
-
-### 2.7 `carlos fleet check-label` (ruling R1, §9)
-
-`carlos fleet check-label <machine-id>` runs from an operator's machine
-with bucket credentials, like `carlos status -live`. It prints the label
-and `box_id` the machine id gives, then reads `status/boxes/<label>.json`
-and `control/edges/<label>.json` and prints each object's `box_id`,
-`instance_id`, freshness and store, and any evidence under
-`status/identity/` naming the label. Exit 0 when the label is free or
-held by this machine id; exit 1 when it is held by another machine or
-unproven. The provisioning recipe runs it after a new box's first boot
-(the machine id exists then; `cat /etc/machine-id` over SSM) and before
-the agent is bootstrapped, so a collision is found before the box serves
-anything.
+nothing there. A refuser has written nothing under the label since it
+refused, and before that only as owner, so the edges file in the bucket
+is the holder's. Exceptions, which are costs (§6.2): an old-release
+sibling cannot refuse and overwrites the holder's objects during a roll,
+and routing then follows the old box; and when a label's host set flips
+between two machines the steering converger's shrink guard
+(`route53.go:209`) may refuse a pass, which is today's collision
+behaviour, now shorter.
+
+### 2.7 `carlos status --check-label` and `--retire-label` (ruling R1, §9)
+
+Both are flags of `carlos status`, the verb that already reads the status
+bucket with the operator's own credentials (`--live`, `cli.go:3340-3372`),
+rather than `carlos fleets` verbs, which are console-API-only by this
+repository's rule (`fleets.go:122`) (codex B3, plan quality).
+
+`carlos status --check-label <machine-id>` prints the label and `box_id`
+the machine id gives; the resolved location of each object; for the
+snapshot its `box_id`, `instance_id` and `fresh`/`stale`; for the edges
+file its `box_id` and `eip`; every unexpired evidence entry naming the
+label with both machines' instance ids; and, once the claim lands, open
+claim operations naming the `box_id`. Exit 0 when the label is free or
+every present object's `box_id` equals this machine id's; exit 1 when
+another machine holds it or an object has no `box_id` (unproven).
+
+`carlos status --retire-label <label>` is §2.3 step 4.
+
+The provisioning recipe runs `--check-label` after a new box's first boot
+(`cat /etc/machine-id` over SSM) and before the agent is bootstrapped, so
+a collision is found before the box serves anything. The recipe lives in
+this repository's README (fleet status section) and the flagship root's
+operator notes (`carloku-infrastructure/README.md`), which the rollout
+updates (§8).
## 3. Readers
-Readers add two stateless rules and one durable one. Keys and joins do
-not change, and **hostname occupancy is never weakened** (codex B2-3).
+Readers add three rules. Keys and joins do not change, and **hostname
+occupancy is never weakened**.
1. **Disagreeing objects, for eligibility only.** Where a snapshot and an
edges file share a label and both carry `box_id`, they must be equal.
A mismatch makes the pair **contested** for `readLiveEdgesSnapshot`
- (`instancesplacement.go:93`): `BoxesComplete=false`, so no region
- outside the home is automatically eligible. `edgesfile.EdgeIPs` and
+ (`instancesplacement.go:93`). `edgesfile.EdgeIPs` and
`DeploymentRoutes` (`fold.go:46,125`) are **unchanged**: a contested
file whose label has a fresh snapshot still contributes its EIP (the
- union only widens, which is safe) and still reserves its hostnames
+ union only widens) and still reserves its hostnames
(`deploymentRouteTaken`, `apidomains.go:1282`, and the alias pass,
- `aliases.go:117`, read absence as "not taken", so dropping a contested
- file would let an attach steal an operator-minted route). Objects
- without `box_id` join by label as today.
-2. **Evidence.** `readLiveEdgesSnapshot` reads `status/identity/` from
- `cfg.Releases` **before** its empty-files early return
- (`instancesplacement.go:98`); a failed read is incomplete evidence. An
- unexpired entry naming a label makes that label contested.
-3. **Writer change.** The status lease holder keeps
- `status/identity/fleet.json`: per label, the last `box_id` and
- `generated_at` it saw and a `contested_until`. At the start of each
- publish it reads the file (a failed or unparsable read means no
- writer-change detection this tick, logged once; rules 1 and 2 still
- run); at the end it writes it with `ifMatch` on the etag it read, after
- `held.Err()` says the lease is still held (`lease.go:170`), and a CAS
- conflict or a lost lease skips the write, leaving the other
- publisher's record standing (codex B2-7). A `box_id` that changes,
- appears or disappears while the previous object was still fresh at the
- new object's `generated_at` sets `contested_until` three periods
- ahead; a later write never shortens an existing `contested_until`. The
- console reads the same file under rule 2.
-
-The lease holder folds all three into one singly-owned
-`alerts/fleet/identity.json` (`metrics.AlertState`, `Box: "fleet"`),
-written every publish tick, firing or not. Rules are **per label**
-(codex B2-8): `foreign:<label>`, `twin:<label>`, `unproven:<label>`,
-`contested:<label>`, so a second label going foreign while the first is
-still foreign is its own transition for the watcher's `(box, rule)`
-tracking (`alertwatch.go:187`). The console's alert watcher
-(`alertwatch.go:97`) reads this one key alongside `alerts/boxes/*.json`,
-treats an object older than three periods as clear, and mails operators
-holding a grant; with none the transition is logged and dropped
+ `aliases.go:117`, read absence as "not taken"). Objects without
+ `box_id` join by label as today.
+2. **Evidence.** `readLiveEdgesSnapshot` reads `status/identity/` (the
+ evidence objects) from `cfg.Releases`.
+3. **Writer change.** `readLiveEdgesSnapshot` also reads the lease
+ holder's `status/identity/fleet.json` (below) with its own loader; a
+ label whose `contested_until` is in the future is contested.
+
+**Completeness is monotonic** (codex B3-2): `BoxesComplete` starts true
+and is only ever set false, by any of: a failed or partial evidence read,
+a failed or unparsable observation read, any contested label from any of
+the three rules, a skipped snapshot read, an edges file with no snapshot,
+or a fresh snapshot with no file. The three reads happen **before** the
+empty-files early return (`instancesplacement.go:98`), and the existing
+`snap.BoxesComplete = boxesComplete` assignment (`:113`) becomes an AND.
+A contested label with perfectly matching files is still incomplete: the
+evidence says two machines claim it.
+
+**The lease holder's observation file**, `status/identity/fleet.json`:
+per label, the last `box_id` and `generated_at` seen and a
+`contested_until`. The holder reads it at the start of each publish (a
+failed or unparsable read means no writer-change detection this tick,
+logged once) and writes it at the end with `ifMatch` on the etag it read,
+after `held.Err()` says the lease is still held (`lease.go:170`); a CAS
+conflict or a lost lease skips the write. A `box_id` that changes, appears
+or disappears while the previous object was still fresh at the new
+object's `generated_at` sets `contested_until` three periods ahead; a
+later write never shortens an existing `contested_until`.
+
+**The fleet alert**, `alerts/fleet/identity.json` (`metrics.AlertState`,
+`Box: "fleet"`), is built by the lease holder from all three rules: the
+evidence (`foreign:<label>`, `twin:<label>`, `unproven:<label>`), the
+observation file (`contested:<label>`), and the snapshot/edges `box_id`
+mismatch, which the agent computes itself from `status.LoadBoxes` and
+`edgesfile` (the agent imports both; `internal/status` cannot import
+`edgesfile`) and passes in (codex B3, missing requirement). Rules are per
+label. A rule that stops firing is **kept with `Firing: false` for
+twenty-four hours** so the console watcher sees the clear transition and
+can mail the next firing (codex B3-8); after that it is dropped. The
+object is written **conditionally** on the etag read at the start of the
+tick, after `held.Err()`, like the observation file, so a former holder
+that resumes after losing the lease cannot overwrite its successor's
+alert (codex B3-9); a conflict skips the write.
+
+The console's alert watcher (`alertwatch.go:97`) reads the fleet object
+**before** its "no box has published an alert" early return
+(`alertwatch.go:110`) (codex B3-8), treats an object older than three
+periods as clear for every rule it carries, and runs its per-rule
+transition logic unchanged, so a retained `Firing: false` rule clears the
+`alert_seen` row and the next `Firing: true` mails. Operators holding a
+grant are mailed; with none the transition is logged and dropped
(`alertwatch.go:194`, amadan #192). The mail body (`alertMessage`,
`alertwatch.go:285`) names the label and the rule, never a `box_id` or
`instance_id`; those are on the operator fleet page and in
-`carlos status -live`, which show a `contested` chip and both machines'
-instance ids. **Ruling R4** (§9): fleet.json says nothing.
+`carlos status --live`, which show a `contested` chip, the holder's
+instance id (from the snapshot) and the refuser's (from the evidence).
+**Ruling R4** (§9): fleet.json says nothing.
## 4. The host lifecycle claim
@@ -349,17 +424,24 @@ cannot prove uniqueness, is this design's job. The answer:
- fleet boxes are outside the claim today; if that changes, their
reference is `fleet/<fleet>/<label>`, unique by the registry.
-`carlos fleet retire <label>` refuses while any open claim operation
-names the holder's `box_id`.
+**Cross-branch dependency** (codex B3, missing requirement): `internal/hostclaim`
+is on `feat/hostclaim-library`, not main. `--check-label` and
+`--retire-label` gain their open-operation check (`hostclaim` lists
+`control/hosts/*.json` whose `op.exec`, `op.participants` or open
+obligations name the `box_id`) in the claim's first plan that lands on
+main, and no claim operation may be armed on a box before that plan and
+this spec's step 1 are both on main. Until then the two commands print
+`claim references: not checked (hostclaim is not on main)`.
## 5. Mixed versions and rollback
-Guarantees, stated per state (codex B2-10):
+Guarantees, stated per state:
- **Every writer on this release.** At most one machine writes under a
label while its object is fresh; no label is reassigned without an
- operator; a non-owner writes nothing under any label-keyed key; every
- collision is in the evidence within one period. Clones with no
+ operator; a non-owner writes nothing under any label-keyed key; a
+ machine whose writes fail loses its authority within three periods;
+ every collision is in the evidence within one period. Clones with no
instance ids are the stated exception.
- **Old writer, new reader.** No `box_id`: join by label, as today. Rule 3
sees `box_id` disappear if an old sibling overwrites a new box's object,
@@ -370,15 +452,13 @@ Guarantees, stated per state (codex B2-10):
and a different `instance_id`, refuses and alerts. Detection is on the
new side only. When the old box rolls, both run §2.1: the object in the
bucket is the old box's and now carries its `box_id`, so the old box is
- owned and the new box stays foreign. The repair is §2.3, and it needs
- all four steps: regenerating the newcomer's machine id alone leaves its
- old object under the label, which the other box refuses forever
- (codex B2-4).
+ owned and the new box stays foreign. The repair is §2.3, all five
+ steps.
- **An established holder during the roll.** The one way a production
holder loses its label is an old-release sibling overwriting it. It
then refuses, withdraws its publishers, keeps serving, keeps its claim
- references (they are `box_id`), and its routing follows the sibling
- until §2.3 is run.
+ references (they are `box_id`), and its existing routing records
+ persist while new ones follow the sibling, until §2.3 is run.
- **Rollback.** An old-release box has no check and overwrites blindly, as
today. No object moved, so there is nothing to sweep. `identity.json`,
the evidence prefix and the fleet files are additive; an old reader
@@ -392,7 +472,7 @@ Guarantees, stated per state (codex B2-10):
| Finding | Under Option B |
|---|---|
| R1-1, R2-3, R3-4: steering `SetIdentifier`, renames, shrink guard | **gone.** No identifier changes |
-| R1-2, R2-1, R3-1: cleanup or sweep deletes a live writer's object | **gone.** Nothing is deleted except by an operator, after staleness |
+| R1-2, R2-1, R3-1: cleanup or sweep deletes a live writer's object | **gone.** Nothing is deleted except by an operator, past the barrier |
| R1-3, R2-2, R3-2, R3-3, R3-5: rollback duplicates, aliases, effective-file choice, orphans | **gone.** One key shape, one object per label |
| R1-4: the mixed-shape join | **gone** |
| R1-5, R3-6: the other label-keyed families | **covered by the gate**: a non-owner writes none of them |
@@ -404,40 +484,41 @@ Guarantees, stated per state (codex B2-10):
### 6.2 Residual risk and cost under Option B
-- A colliding box is out of multi-region routing, fleet status, metrics,
- logs and claim work until §2.3 is run. Its own edge serves its hosts
- locally. On the flagship a new box collides with one of four labels
- with probability 4 in 1,600 per box added; Tito 6 in 1,600. With
- `check-label` in the provisioning recipe, it is found before the box
- serves anything.
+- A colliding box publishes nothing under its label until §2.3 is run: no
+ snapshot, no new edges file, no metrics, no logs, no claim work. Its
+ existing routing records, if it ever published any, persist (§2.6);
+ its own edge serves its hosts locally. On the flagship a new box
+ collides with one of four labels with probability 4 in 1,600 per box
+ added; Tito 6 in 1,600. With `--check-label` in the provisioning
+ recipe, it is found before the box serves anything.
- During a roll, an old-release sibling can displace an established
holder (§5). The holder keeps serving and its claim references; its
- routing follows the sibling until §2.3 is run, which includes stopping
- and re-identifying the newcomer.
+ routing follows the sibling until §2.3 is run.
- An unproven label needs an operator to free it. On the flagship, Tito
and dev every box has an instance id; this is hardware no cloud owns.
- Retired boxes' objects still linger, as today; a new box that collides
- with a retired label is refused until `carlos fleet retire`.
-- A box that cannot read its status store for fifteen minutes stops
- every label-keyed publisher (ruling R3).
+ with a retired label is refused until `--retire-label`.
+- A box that cannot read or write its status store for fifteen minutes
+ stops every label-keyed publisher (ruling R3).
- Steering and the home map do not consume the verdicts (§2.6).
-- A split status bucket is outside §3's supported shape (§2.6).
+- A platform box with a split or prefixed status store does not arm
+ (§2.6); none exists today.
-Operator cost: one alert; the four steps of §2.3 for a collision; one
-`check-label` per new box. No runbook step on roll, none on rollback.
+Operator cost: one alert; the five steps of §2.3 for a collision; one
+`--check-label` per new box. No runbook step on roll, none on rollback.
### 6.3 Comparison
| | Option A: migrate the keys | Option B: detect and refuse |
|---|---|---|
| What it fixes | Both colliding boxes keep working; the label becomes cosmetic | The collision is loud and deterministic; one box works, the other is withdrawn until fixed |
-| Residual risk | Four open P1s after three rounds, all protocol. Deletes production objects. Rollback needs a runbook | A withdrawn box; a displaced holder during a roll; unproven labels off-cloud; routing not withdrawn; a four-step repair |
+| Residual risk | Four open P1s after three rounds, all protocol. Deletes production objects. Rollback needs a runbook | A withdrawn box; a displaced holder during a roll; unproven labels off-cloud; routing not withdrawn; a five-step repair |
| Changes to objects, keys, DNS | Two key shapes for two releases, a sweep, a DNS identifier flip, step 2 for six more families | Two additive fields, one private prefix, two fleet files, one local file |
-| Code touched | `status`, `edgesfile`, `host`, `console`, `edge`, `steering`, CLI; two releases minimum | `status` (decide), `host` (gate at twelve entry points, identity file), `relay` and `adopt` (read the file), `console` (two reader rules, two fleet files, alert), CLI (`check-label`, `retire`); one release |
-| Review | Astra twice asked for a migration state machine and the DNS boundary; both open | Two rounds; rev 3 unreviewed; remaining risk is operational |
+| Code touched | `status`, `edgesfile`, `host`, `console`, `edge`, `steering`, CLI; two releases minimum | `status` (decide), `host` (gate at every entry point, identity file), `relay.go` and `adopt` (gated stores), `console` (three reader rules, two fleet files, alert), CLI (`carlos status` flags); one release |
+| Review | Astra twice asked for a migration state machine and the DNS boundary; both open | Three rounds; rev 4 unreviewed; remaining risk is operational |
| Flagship (4 boxes, no collision today) | roll with sweep, N+1 DNS flip, rollback runbook | a roll; nothing moves; an alert and a withdrawn box if a future box collides |
| Tito (6 boxes) | the same, on a customer's fleet | the same as the flagship |
-| dev (1 box) | the flip and sweep run for nothing | owned by configuration; nothing happens |
+| dev (1 box) | the flip and sweep run for nothing | `configured`; nothing happens |
### 6.4 Decision
@@ -447,51 +528,56 @@ withdrawn box is not an acceptable answer.
## 7. Test plan
-- `DecideLabel`: every row of §2.1's table, including stale foreign and
- stale unproven (both still refuse), twin by instance id, malformed body
- and CAS exhaustion (unknown, no write); a CAS conflict re-reads and
- re-decides; owned is recorded only after a successful conditional
- write. Two writers against one `DirStore` handle from one start:
- exactly one owns, in both orders.
+The plan carries the exact tests; these are the sequences they must pin.
+
+- `DecideLabel`: every row of §2.1's table; a failed replace `Put` is
+ unknown and no identity renewal follows; a CAS conflict whose re-read
+ shows a legacy object with no instance id is unproven, not owned (no
+ state carried between attempts); CAS exhaustion is unknown; the full
+ collected snapshot (routes, a sidecar, aliases, versions, database
+ sizes) is what lands in the bucket on an owned tick. Two writers
+ against one `DirStore` handle, including a forced interleaving (writer
+ B's `Put` lands between A's read and A's `Put`), exactly one owns.
- The gate: `Publishable` false on a missing, stale, other-machine,
- other-label or other-store file; `gatedStore` refuses every `Put` and
- `Delete` and passes `Get` and `List`; every entry point in §2.2's table
- publishes through it, in the agent, `adopt` and `relay serve`; a
- mismatched `CARLOS_RELAY_BOX` publishes nothing; a one-box directory
- deployment is open; a fleet box is open for its fleet label; the
- registry snapshot aborts mid-sequence when the verdict changes;
- transitions to owned reset the edges gate and `seedEdgesFileGate`
- discards a foreign seed and its EIP.
-- An unidentified box publishes nothing and refuses claim work.
-- Evidence: written each tick while not owned, keyed by the writer's id,
- expiring after three periods.
-- The fleet observation file: survives a lease move; a conditional write
- loses to a concurrent writer and the record stands; a lost lease skips
- the write; `contested_until` is never shortened; an unreadable file
- disables rule 3 for the tick and nothing else.
-- Readers: rule 1 makes a disagreeing pair incomplete for eligibility and
- leaves `EdgeIPs` and `DeploymentRoutes` unchanged (a contested file's
- EIP is in the set and its hostnames are taken); rule 2 runs before the
- empty-files return and a failed read is incomplete; rule 3 fires on a
- change within freshness, not after staleness; the fleet alert object
- carries per-label rules and the watcher mails each label's transition;
- the mail body carries no `box_id` or `instance_id`.
-- Privacy fixtures: distinct sentinels for `box_id` and `instance_id` in
- the dry run (`dryrun.go:62`), a collision evidence object created in the
- run, positive checks in `status/boxes/` and `status/identity/`,
- negative in fleet.json and index.html; unit tests at `fleet.go:172`
- (`FleetBox`), `apiinstances.go:327` (the instances response) and
- `alertwatch.go:285` (the mail) asserting neither field appears.
-- CLI: `check-label` exits 0 for free and own, 1 for foreign and
- unproven, and prints both sides; `retire` refuses a fresh object,
- deletes stale ones conditionally, refuses on an etag change.
-- Claim: `ClaimEligible` false for unidentified, twin and non-publishable.
-- Mixed versions: old-shape objects join as today; new-shape objects are
- byte-compatible for the fields an old reader uses.
-- Simulation: random fleets with injected collisions, simultaneous starts,
- clones, rolls, rollbacks and one-box store outages, asserting §5's
- first bullet for every writer on this release and the stated exceptions
- for the others.
+ other-label, other-store or older-shaped file; true for `fleet` and
+ `configured` regardless of age; a directory deployment stays open past
+ three periods; `Gated` refuses `Put`, `Delete` and `DeleteIfMatch`,
+ passes `Get` and `List`, and applies `mutationTTL`; the registry
+ snapshot aborts mid-sequence; the relay meter's retry and
+ corrupt-preservation writes are refused after an ownership flip between
+ its read and its write; a mismatched `CARLOS_RELAY_BOX` publishes
+ nothing; a split or prefixed status store on a platform box is
+ `unknown` with the log line.
+- Edges: transitions to owned reset the gate; a seed with a different or
+ absent `box_id` is discarded and its EIP never inherited under a blank
+ probe.
+- Readers: fresh foreign evidence **with matching snapshot and edges
+ objects present** is incomplete; an unreadable evidence or observation
+ read is incomplete and is not overwritten by a later complete snapshot
+ read; an observation-only `contested_until` is incomplete; `EdgeIPs`
+ and `DeploymentRoutes` are unchanged for a contested pair.
+- The fleet files: a delayed former holder's conditional writes of both
+ files lose to its successor's; an unreadable observation file disables
+ rule 3 only; `contested_until` never shortens.
+- Alerts: fire, clear, fire again mails three times; a second label
+ firing while the first still fires mails; a fleet object with no box
+ alert objects is read; the mail fixture seeds both a `box_id` and an
+ `instance_id` and asserts neither appears.
+- CLI: `--check-label` exits 0 for free and own, 1 for foreign and
+ unproven, and prints both locations and both instance ids;
+ `--retire-label` refuses a snapshot younger than `retireAfter`, deletes
+ older objects conditionally, refuses on an etag change, deletes a lone
+ edges file with a warning, and resolves a prefixed status store
+ (fleet-shaped fixture) and a split bucket (two `DirStore`s) to the
+ right objects.
+- Privacy: distinct `box_id` and `instance_id` sentinels in the dry run,
+ an evidence object created in the run, each required positively in the
+ private objects and absent from fleet.json and index.html; the
+ `FleetBox`, instances-response and mail projections asserted.
+- Simulation: machines change release mid-run (old writers that bypass
+ the decision), stores fail reads and writes for random windows, and
+ retirement is exercised; the assertions are §5's first bullet for new
+ writers and detection-not-refusal for old ones.
## 8. Prerequisite and rollout
@@ -503,24 +589,34 @@ boxes from `infra/modules/region-host`, whose AMI is Debian's own
`debian-13-arm64-*` image from account `136693071363`
(`variables.tf:131-141`, `main.tf:14-24`); no root overrides
`ami_owner` or `ami_name_pattern`, and no root builds a custom AMI or a
-snapshot-derived one. Their `host-init.sh` first-boot scripts (byte-
-identical between IE and the templated regions, differing only in the
-SSM region) never touch `/etc/machine-id`. Debian's cloud images ship
-`/etc/machine-id` empty and systemd generates it on first boot, so every
-instance launched from them has its own. The flagship's four distinct
-labels are the observable consequence. The HA ladder's instance recovery
-(`main.tf:507`) keeps the root volume and so the machine id, which is
-correct: it is the same machine. An EBS-snapshot clone would carry the
-machine id with it; no root does that today, and §2.1's twin row is the
-detection if one ever does. The rollout step is to keep it so: any future
-custom AMI must clear `/etc/machine-id`.
-
-1. Land on main; cut a release; roll boxes in any order; roll the
- console. Nothing moves, so there is no order and no sweep.
-2. Add `carlos fleet check-label` to the provisioning recipe, and §2.3's
- four steps and the alert texts to the operator runbook.
-3. Record §4 on the claim branch as the answer to rev 6.16's open
- reference: `box_id`, and `host.ClaimEligible()` as the gate.
+snapshot-derived one. Their `host-init.sh` first-boot scripts never touch
+`/etc/machine-id`. Debian's cloud images ship `/etc/machine-id` empty and
+systemd generates it on first boot, so every instance launched from them
+has its own. The flagship's four distinct labels are the observable
+consequence. The HA ladder's instance recovery (`main.tf:507`) keeps the
+root volume and so the machine id, which is correct: it is the same
+machine. An EBS-snapshot clone would carry the machine id with it; no
+root does that today, and §2.1's twin row is the detection if one ever
+does. The rollout step is to keep it so: any future custom AMI must clear
+`/etc/machine-id`.
+
+1. Land the completed, reviewed branch on main as **one squash**
+ (`amadan branch merge -squash`): the plan's task commits are not
+ independently shippable (snapshot refusal arrives before the other
+ publishers are gated), and the branch is reviewed whole (codex B3,
+ plan quality).
+2. Cut a release; roll boxes in any order; roll the console. Nothing
+ moves, so there is no order and no sweep.
+3. README (fleet status section): `--check-label` in the provisioning
+ recipe, §2.3's five steps, the alert texts, and "a separate status
+ bucket is not supported with box identity"; `docs/site/cli.md`'s
+ `### carlos status` gains both flags. The flagship root's
+ `carloku-infrastructure/README.md` operator notes get the
+ `--check-label` step (a change in that repository, after this one
+ lands).
+4. Record §4 on the claim branch: `box_id` is the reference,
+ `host.ClaimEligible()` is the gate, and the `--check-label` and
+ `--retire-label` claim checks belong to the claim's first plan on main.
## 9. Rulings
@@ -529,7 +625,7 @@ override any of them.
| | Ruling | Why |
|---|---|---|
-| R1 | `carlos fleet check-label <machine-id>` is added and the provisioning recipe runs it before bootstrapping the agent (§2.7) | a collision found before the box serves anything costs nothing |
+| R1 | `carlos status --check-label <machine-id>` is added and the provisioning recipe runs it before bootstrapping the agent (§2.7) | a collision found before the box serves anything costs nothing |
| R2 | a withdrawn box buffers logs, errors and analytics to its caps and ships nothing; no temporary `<label>-<4 hex>` segment (§2.2) | a temporary segment is a second identity to migrate off later; the box is fixed in minutes |
| R3 | a box that cannot re-prove its label for fifteen minutes stops every label-keyed publisher, metrics and logs included (§2.2) | the conservative choice: one machine writing a label at a time is the invariant, and a box that cannot read the store cannot know it still holds the label |
| R4 | fleet.json stays silent on contested labels (§3) | every other operator-only condition is silent there, and the public feed carries no condition a reader could act on |
@@ -539,25 +635,29 @@ override any of them.
| Finding | Severity | Disposition |
|---|---|---|
| B-1 takeover violates the claim's one-machine contract | P1 | accepted. No automatic reassignment; the claim references `box_id` (§1, §2.1, §4) |
-| B-2 twin detection bypassed; empty machine id manufactures an identity | P1 | accepted. Equal `box_id` with differing instance ids is a twin; an unreadable machine id publishes nothing (§2.1) |
+| B-2 twin detection bypassed; empty machine id manufactures an identity | P1 | accepted (§2.1) |
| B-3 unproven becomes owned by reading its own write | P1 | accepted. Unproven never writes (§2.1) |
-| B-4 startup and expiry gating | P1 | accepted. Decision at arm before any publisher; publishable expires after three periods (§2.1, §2.2) |
-| B-5 cross-process publishers | P1 | accepted. `<dataDir>/identity.json`; the entry-point inventory (§2.2) |
-| B-6 stale edges after regaining ownership | P1 | accepted. Transitions to owned reset the gate; a foreign seed and its EIP are discarded (§2.2) |
+| B-4 startup and expiry gating | P1 | accepted (§2.1, §2.2) |
+| B-5 cross-process publishers | P1 | accepted. `identity.json`; the entry-point inventory (§2.2) |
+| B-6 stale edges after regaining ownership | P1 | accepted (§2.2) |
| B-7 refusal does not withdraw routing | P1 | accepted as a stated limit (§2.6) |
-| B-8 remaining branches and the CAS guarantee | P2 | accepted (§2.1, §2.5) |
-| B-9 directory deployments and split status stores | P2 | accepted (§2.5, §2.6) |
-| B-10 writer-change evidence does not survive a publisher change | P2 | accepted. Durable fleet observation file (§3) |
-| B-11 privacy fixtures | P2 | accepted (§7) |
-| B-12 the comparison understates B's costs | P2 | accepted (§5, §6.2, §6.3) |
-| B2-1 EIP equality does not establish ownership | P1 | accepted. The EIP branch is removed; stale unproven stays unproven; an operator frees it (§2.1, §2.3) |
-| B2-2 the gate can authorise publication under another label | P1 | accepted. `Publishable(label, target)` binds the calling machine's `box_id`, the exact label and the store; a mismatched `CARLOS_RELAY_BOX` publishes nothing (§2.2) |
-| B2-3 excluding contested routes disables hostname occupancy | P1 | accepted. Rule 1 applies to eligibility only; `EdgeIPs` and `DeploymentRoutes` are unchanged (§3) |
-| B2-4 the mixed-version repair leaves the displaced holder refused | P1 | accepted. The four-step freeing procedure: stop, regenerate, `retire` after staleness, reclaim (§2.3, §5) |
-| B2-5 in-flight revocation | P2 | accepted. `gatedStore` checks every `Put` and `Delete`; the staleness barrier before freeing (§2.2) |
-| B2-6 split stores | P2 | accepted. One arbitration namespace per publication namespace, recorded in the identity file; the supported console shape stated (§2.6) |
-| B2-7 fleet observation file failure and lease loss | P2 | accepted. Conditional write after `held.Err()`, skip on conflict or loss, never shorten `contested_until`, unreadable disables rule 3 only (§3) |
-| B2-8 the fleet alert loses the label | P2 | accepted. Per-label rule names (§3) |
-| B2-9 fleet boxes and the gate | P2 | accepted. `verdict: "fleet"`, open for the fleet label; the claim reference if ever needed (§2.2, §4) |
-| B2-10 limits and guarantees disagree | P2 | accepted. §5 states guarantees per state; §4 no longer claims unreachability (§4, §5) |
+| B-8, B-9, B-10, B-11, B-12 | P2 | accepted (§2.1, §2.5, §2.6, §3, §5, §6) |
+| B2-1 EIP equality does not establish ownership | P1 | accepted. The EIP branch is removed (§2.1) |
+| B2-2 the gate can authorise publication under another label | P1 | accepted (§2.2) |
+| B2-3 excluding contested routes disables hostname occupancy | P1 | accepted (§3) |
+| B2-4 the mixed-version repair leaves the displaced holder refused | P1 | accepted. The freeing procedure (§2.3) |
+| B2-5 to B2-10 | P2 | accepted (§2.2, §2.6, §3, §4, §5) |
| B2-11 DirStore mutex is per handle | P3 | accepted (§2.5) |
+| B3 spec requirement 1: retirement needs a real quiescence barrier | spec | accepted. `mutationTTL` on every gated mutation; `retireAfter = 3 periods + mutationTTL`; `--retire-label` refuses a younger snapshot; standalone processes expire with the same file (§2.2, §2.3) |
+| B3 spec requirement 2: enforceable arbitration namespace | spec | accepted. A platform box with a split or prefixed status store does not arm; the identity file's store is the deployment bucket (§2.6) |
+| B3-1 a failed replacement grants fresh ownership; retry state carries over | P1 | accepted. Fresh decision per attempt; owned only after a successful `Put`; a `Put` failure is unknown (§2.1) |
+| B3-2 collided evidence accepted as complete; rule 3 unwired; loaders confused | P1 | accepted. Monotonic completeness; three reads before the early return; the AND at `:113`; an observation loader of its own; the evidence loader skips `fleet.json` (§2.4, §3) |
+| B3-3 every snapshot loses its route data | P1 | accepted. `DecideLabel` takes the complete collected snapshot at arm and on every tick (§2.1) |
+| B3-4 a directory deployment closes its gate after fifteen minutes | P2 | accepted. `configured` never expires (§2.2, §2.5) |
+| B3-5 relay retries bypass the gate | P2 | accepted. The relay's whole store is wrapped in `relay.go` (§2.2) |
+| B3-6 retirement addresses the wrong objects; wrong unit name | P2 | accepted. Each object's location resolved and printed; `carlos-edge.service`; edges freshness defined (§2.3) |
+| B3-7 an unidentifiable legacy edges seed is trusted | P2 | accepted. A seed without our `box_id` is discarded; the EIP is never inherited (§2.2) |
+| B3-8 cleared alerts never reset mail state; fleet read after the early return | P2 | accepted. Cleared rules kept as `Firing: false` for a day; the watcher reads the fleet object first (§3) |
+| B3-9 a former holder overwrites its successor's alert | P2 | accepted. Conditional write after `held.Err()` for the alert too (§3) |
+| B3-10 the exported gate never accepts the fleet verdict | P2 | accepted. One identity-resolution path in `Publishable` (§2.2) |
+| B3 plan quality: unshippable task commits; mismatch rule missing from the fleet alert; both instance ids on the page; provisioning recipe, CLI reference, full gate, claim dependency, API-backed administration | plan | accepted. Squash landing (§8); the agent computes the mismatch rule (§3); both ids shown (§3); recipe and `cli.md` in the rollout (§8); `carlos status` flags, not `fleets` verbs (§2.7); the claim dependency stated (§4); the plan carries the gate |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-BOX-IDENTITY-B-R3.md b/docs/superpowers/specs/CODEX-FINDINGS-BOX-IDENTITY-B-R3.md
new file mode 100644
index 000000000..418047ae5
--- /dev/null
+++ b/docs/superpowers/specs/CODEX-FINDINGS-BOX-IDENTITY-B-R3.md
@@ -0,0 +1,180 @@
+**SPEC: NOT READY. PLAN: NOT READY TO BUILD.**
+
+Reviewed `2b7641ec` against local `origin/main` at `eef3e121`, including B-R1 and B-R2. No files changed, tests run, or hosts contacted. The collision fixture does produce two distinct IDs with the same `edsger-knitting` label.
+
+The remaining problems fit within Option B; none requires key migration.
+
+**B-R2 dispositions**
+
+“Resolved” here means resolved by the proposed design, not implemented or verified.
+
+| B-R2 | Assessment | Reason |
+|---|---|---|
+| 1 — EIP ownership proof | **Resolved** | The EIP promotion branch is removed; unresolved legacy ownership remains unproven. |
+| 2 — Gate authorizes another label | **Partly** | Exact label and identity comparisons are specified. However, the plan supplies the configured **status** target to gates around deployment-store writers, rather than establishing that the actual publication namespace has one arbiter. Fleet handling also differs between the two gate helpers. |
+| 3 — Contested routes lose hostname protection | **Resolved** | `EdgeIPs` and `DeploymentRoutes` remain unchanged; exclusion applies only to placement eligibility. |
+| 4 — Displaced holder cannot recover | **Partly** | Explicit retirement addresses the abandoned snapshot. The stop command names the wrong service, retirement does not resolve split/prefixed stores correctly, and claim obligations are checked after the prescribed identity regeneration. |
+| 5 — In-flight revocation | **Partly** | Per-mutation wrapping is specified. The assertion that fifteen minutes of snapshot staleness drains mutations is unsupported; relay implementation still checks only operation entry. |
+| 6 — Split stores | **Partly** | Console degradation is now explicit. A common arbitration namespace remains an operational requirement without enforcement: two status targets can authorize writes into one deployment namespace. |
+| 7 — History failures and lease loss | **Partly** | Conditional observation updates and lease checks improve the design. The plan omits the console’s observation reader and allows a former publisher to overwrite the fleet alert after losing its lease. |
+| 8 — Alert loses affected label | **Resolved** | Per-label rule names fix the original problem. A separate clearing/re-firing defect remains below. |
+| 9 — Fleet boxes have no gate verdict | **Partly** | A fleet verdict is defined, but `Publishable` still hashes the machine ID while fleet identity files contain the fleet label. |
+| 10 — Guarantees contradict limits | **Partly** | §5 distinguishes rollout states. [Spec:407](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/specs/2026-10-06-box-identity-detect-design.md:407) still says a colliding box is out of multi-region routing despite retained routing records and §2.6’s explicit non-withdrawal. |
+| 11 — DirStore mutex scope | **Resolved** | The specification correctly says “per handle.” |
+
+Two unresolved specification requirements are material:
+
+- **Retirement needs a real quiescence barrier.** [Spec:117](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/specs/2026-10-06-box-identity-detect-design.md:117) assumes mutations finish before fifteen-minute staleness permits reassignment. A cached verdict permits a mutation to *start near that deadline*. Stopping the agent also does not stop standalone relay or root-side publishers. No bounded mutation lifetime plus drain interval is specified.
+- **Publication namespaces need enforceable arbitration.** [Spec:232](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/specs/2026-10-06-box-identity-detect-design.md:232) requires one status namespace, but recording each machine’s chosen namespace does not establish agreement between machines. Either reject unsupported combinations or define how the deployment namespace fixes its arbiter.
+
+**New findings**
+
+1. **P1 — A failed snapshot replacement grants fresh ownership.**
+
+ At [Plan:474](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:474), `DecideLabel` sets `d.Verdict = owned` before attempting the replacement. A non-CAS `Put` error returns that owned decision. [Plan:1036](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:1036) then writes its verdict with a fresh timestamp, and the loop checks the verdict.
+
+ **Failure:** snapshot reads succeed but writes fail. Every tick renews local publication authority while the authoritative snapshot becomes stale and eligible for retirement. Another machine can subsequently own the label while the first continues publishing.
+
+ The retry loop also retains `HolderInstanceID` when a subsequent snapshot has no `Metrics`. After a CAS conflict, a legacy snapshot without identity evidence can inherit the previous attempt’s matching instance ID and be classified owned.
+
+ **Fix/test:** construct a fresh decision per attempt; assign owned only after successful `Put`. Inject replacement errors and a CAS retry that changes the object to legacy/no-instance evidence.
+
+2. **P1 — The reader implementation accepts collided evidence as complete.**
+
+ [Plan:1358](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:1358) appends evidence labels to `Contested`, but only closes `BoxesComplete` for evidence when there are **no edges files**. With ordinary files present, collision evidence never closes it. Furthermore, `origin/main:internal/console/instancesplacement.go:113` subsequently assigns `snap.BoxesComplete = boxesComplete`, overwriting an earlier evidence-read failure.
+
+ Rule 3 is also unwired: Task 7 never adds a console reader for `FleetObservationKey`. Task 5’s generic evidence loader would deserialize that different shape with zero `SeenAt` and discard it as expired.
+
+ **Failure:** matching snapshot/edges IDs plus a fresh foreign/twin report—or an active durable `contested_until`—still permit automatic non-home placement.
+
+ **Fix/test:** combine completeness conditions monotonically, explicitly load observations, and test fresh evidence **with matching existing files**, unreadable evidence, and observation-only contention.
+
+3. **P1 — Every platform snapshot loses its route data.**
+
+ [Plan:1036](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:1036) writes `CollectBox(nil, probes, now)`. Line 1040 removes the later platform snapshot write.
+
+ Today, `origin/main:internal/host/agent.go:2968` loads aliases and line 2974 collects actual routes. `internal/status/snapshot.go:560` derives route counts, versions, database sizes, and route DNS/certificate readings from them.
+
+ **Failure:** healthy boxes continually publish zero route counts and omit route-derived operational data; the complete snapshot assembled later is never written.
+
+ **Fix/test:** pass the complete collected `Box` into arbitration. Exercise the actual agent tick with routes, a sidecar, aliases, versions, and database readings.
+
+4. **P2 — A directory deployment permanently closes its gate after fifteen minutes.**
+
+ [Plan:1038](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:1038) grants configuration ownership once at startup. [Plan:828](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:828) expires every verdict after three periods.
+
+ `origin/main:internal/host/agent.go:2764` returns from `statusLoop` when no status store exists, so nothing refreshes that file.
+
+ **Failure:** edges and other gated deployment-store publication stop until restart, despite no collision.
+
+ **Fix/test:** explicitly represent configuration authority or refresh it independently; advance a directory-deployment test beyond three periods.
+
+5. **P2 — Relay retries bypass the promised mutation gate.**
+
+ [Plan:1048](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:1048) checks `Publishable` before heartbeat and meter operations but passes raw `cfg.Store`.
+
+ `origin/main:internal/relay/meter.go:167` performs three CAS attempts, including a corrupt-object preservation write at line 180 and the meter write at line 188.
+
+ **Failure:** ownership changes after the entry check; preservation writes and retries continue under the refused label.
+
+ **Fix/test:** wrap the relay’s actual store with a per-mutation gate. Flip ownership between a meter read and write, and between CAS attempts. The proposed test only verifies initial heartbeat refusal.
+
+6. **P2 — Retirement commands address the wrong objects in supported configurations.**
+
+ [Plan:1657](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:1657) copies `statusLive`’s bucket-only resolution and uses one store for both objects. But `origin/main:internal/host/agent.go:2778` prefixes snapshots, while `internal/host/edgesfile.go:632` opens the deployment store.
+
+ **Failure:** with a status prefix, retirement misses the blocking snapshot. With separate status and deployment buckets, it misses the edges object. It can report absence/success without freeing the actual label.
+
+ The runbook’s [Spec:177](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/specs/2026-10-06-box-identity-detect-design.md:177) also stops `carlos-agent`; the installed unit is `carlos-edge.service`, whose line 19 executes `carlos agent`.
+
+ **Fix/test:** resolve both actual object locations, print them, use the real services, and test prefixed/split configurations. Define edges freshness explicitly: its `Generation` is a change timestamp, not a heartbeat (`origin/main:internal/host/edgesfile.go:491`).
+
+7. **P2 — The plan deliberately trusts an unidentifiable legacy edges seed.**
+
+ [Plan:976](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:976) requires retaining a seed without `box_id`, claiming the snapshot decision proves it belongs to this machine.
+
+ That proof establishes only snapshot ownership. A different legacy writer can have written the edges object. The fallback at `origin/main:internal/host/edgesfile.go:478` inherits its EIP.
+
+ **Failure:** during upgrade, a legitimate snapshot owner with an unavailable address probe republishes another machine’s address under its own new `box_id`.
+
+ **Fix/test:** do not use an unproven seed as address authority. Test independently authored legacy snapshot and edges objects with a blank address probe.
+
+8. **P2 — Cleared identity alerts never reset mail delivery state.**
+
+ [Plan:1486](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:1486) defines all-clear as an empty rule map. The planned watcher processes only present rules.
+
+ `origin/main:internal/console/alertwatch.go:142` visits those rules; line 243 suppresses a firing state already recorded as firing.
+
+ **Failure:** `foreign:L` fires, disappears on recovery, and later fires again. The console never records the clear, so the second collision produces no mail.
+
+ Also, inserting the fleet read “after the loop” leaves the existing empty-box-alert early return at line 110 ahead of it.
+
+ **Fix/test:** deliver missing-rule clears or retain explicit false states. Test fire → clear → fire, a second simultaneous label, and a fleet alert with no box-alert objects.
+
+9. **P2 — A former lease holder can overwrite its successor’s identity alert.**
+
+ [Plan:1535](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:1535) guards the observation write with `held.Err()`, then calls `putOverwrite` for the fleet alert regardless of lease loss or observation conflict.
+
+ `origin/main:internal/status/fleet.go:590` implements last-writer-wins replacement with a fresh-ETag retry.
+
+ **Failure:** publisher A pauses, B acquires the lease and publishes a collision alert, then A resumes with older all-clear inputs and overwrites B’s alert.
+
+ **Fix/test:** apply lease-loss handling and conditional publication to the observation/alert sequence. Test a delayed former publisher against its successor.
+
+10. **P2 — The exported fleet gate never accepts the fleet verdict.**
+
+ [Plan:879](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:879) repairs `gated` to use `BoxIdentity.ID`, which is a fleet label. It leaves [Plan:831](/home/paulca/amadan.net/carlos/platform-design-box-identity/docs/superpowers/plans/2026-10-06-box-identity-detect.md:831)’s exported `Publishable` passing the machine-ID digest.
+
+ **Failure:** the same fleet identity file is accepted by `gatedStore` and refused by `Publishable`, including the relay callback using that helper. The low-level test passes `"desk"` directly and hides the discrepancy.
+
+ **Fix/test:** use one identity-resolution path and test the exported helper for platform and fleet configurations.
+
+I found **no concrete public `box_id` disclosure** in the proposed projections. That is a static-review result; the mail privacy assertion currently checks an ID never put into its fixture and does not exercise `instance_id`.
+
+**Plan quality**
+
+| Task | Do its tests pin the required behavior? |
+|---|---|
+| 1 — Wire fields | **Mostly.** Good additive parsing/hash checks; privacy verification comes later. |
+| 2 — Arbitration | **No.** “Simultaneous” starts are sequential. Missing forced CAS interleaving, exhaustion, read/write failures, and retry-state reset. |
+| 3 — Shared gate | **Partly.** Useful negative comparisons; no `ClaimEligible` test, exported fleet-helper test, or isolated version/store-shape checks. |
+| 4 — Integration | **No.** Missing full snapshot preservation, startup ordering, directory expiry, positive publication followed by revocation at each entry point, and relay mutation retries. |
+| 5 — Evidence | **Partly.** Round-trip and transitions are useful. Missing mixed evidence/observation-prefix handling and delivery through actual sinks. |
+| 6 — Eligibility | **No.** Missing the ordinary “existing files plus evidence” case. Its unreadable-evidence test should expose the completeness overwrite. |
+| 7 — Observations/alerts | **No.** Pure folds do not test lease transfer, CAS loss, unreadable history, console observation consumption, or alert recurrence. |
+| 8 — Operator commands | **No.** Missing namespace resolution, changed-ETag races, unproven checks, and interrupted retirement. The test incorrectly assumes DirStore lacks conditional delete: `origin/main:internal/release/conditional_delete.go:62` implements it. |
+| 9 — Privacy | **Partly.** Good projection boundaries, but seed both identities in mail tests and require each private sentinel positively. |
+| 10 — Simulation/runbook | **No.** Random iteration order is not concurrency. Machines never change release version, stores never fail, and retirement/recovery is not exercised. |
+
+The task commits are **not independently shippable**. Task 2 activates snapshot refusal before other publishers are gated or recovery tooling exists. Task 4 calls refusal handling introduced in Task 5. The README/runbook arrives only in Task 10. Landing the completed, reviewed branch as one squash avoids exposing intermediate commits, but the plan should state that explicitly.
+
+Missing or reduced spec requirements include:
+
+- Task 7 explicitly drops snapshot/edges mismatch detection from fleet alerts, although §3 requires all three detection rules.
+- Task 8 adds a contested chip but does not provide both machines’ instance IDs on the fleet page and `status -live`.
+- Provisioning-recipe integration is required by §8 but assigned to no task.
+- Claim retirement integration is deferred to a comment; it needs an explicit cross-branch dependency before claim work can ship.
+- CLI reference documentation and the repository’s complete build/vet/test/format gate are not included.
+- The new bucket-credential CLI design needs reconciliation with this repository’s API-backed administration requirement.
+
+**Verdict reasons:** the spec still lacks enforceable publication scope and a sound retirement barrier. The plan additionally contains direct authorization, placement, snapshot, recovery, and alerting defects. Correct those contracts first, then revise the tests around the concrete failure sequences above.
+
+---
+
+**FIRST PROMPT:**
+Independent read-only review of Option B rev 3 and its implementation plan for #212.
+
+**WHAT:**
+Check prior findings, new failures, test coverage, and build readiness.
+
+**STATUS:**
+Review complete; both documents need revision. Workspace unchanged.
+
+**NEEDED:**
+Nothing for this review.
+
+**RECOMMENDED:**
+Resolve ownership, eligibility, and retirement defects before implementation.
+
+**SESSION:**
+Safe to close; nothing running.
\ No newline at end of file