| new file mode 100644 |
| index 000000000..e78067685 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-06-box-identity-detect-design.md |
| @@ -0,0 +1,359 @@ |
| +# Box identity, Option B: detect and refuse, no key migration |
| + |
| +**Date:** 2026-10-06 |
| +**Revision:** 1 |
| +**Discussion:** amadan carlos/platform #212 |
| +**Sibling of:** `2026-10-06-box-identity-collisions-design.md` (Option A, |
| +migrate the keys), rev 3, whose third Astra round |
| +(`CODEX-FINDINGS-BOX-IDENTITY-R3.md`) still found four P1s in the |
| +key-migration protocol. This spec is the alternative, and §6 compares the |
| +two and recommends one. |
| +**Status:** draft, for Astra's review, then Paul's. |
| + |
| +## 1. What Option B is |
| + |
| +Nothing moves. `status/boxes/<label>.json` and `control/edges/<label>.json` |
| +keep their keys; Route53 `SetIdentifier` stays the label; every other |
| +per-box key stays as it is. Two things are added: |
| + |
| +1. Both objects carry `box_id = hex(sha256(machineID)[0:16])`, 32 hex, |
| + bucket-private, never in fleet.json or any public or member-facing |
| + surface. It is the proof of which machine wrote the object. |
| +2. A writer that finds its label's object **fresh and written by another |
| + machine** does not overwrite it. It refuses to publish, latches, alerts |
| + with instructions, and takes no claim work. Readers treat a label whose |
| + writer changed, or whose two objects disagree about the writer, as |
| + incomplete evidence. |
| + |
| +A collision therefore costs the colliding box, not the fleet: the first |
| +writer keeps the label and keeps working, the second is loud and |
| +withdrawn until an operator regenerates its machine id. That is the trade |
| +against Option A, where both boxes keep working and the migration carries |
| +the risk. |
| + |
| +## 2. The writer rule |
| + |
| +The arbiter is the status snapshot, because `statusTick` writes it |
| +unconditionally every tick (`internal/host/agent.go:2905`, five minutes |
| +plus jitter) and `status.BoxStale` already defines its freshness. The |
| +edges file follows the snapshot's verdict; it does not arbitrate. |
| + |
| +### 2.1 Ownership of a label |
| + |
| +`status.WriteBox` (`snapshot.go:742`) replaces its read-etag-then-Put with |
| +a decide loop, three attempts: |
| + |
| +1. `Get` the label's object. Not found: `Put` with `ifMatch ""`, |
| + create-only. Success: **owned**. `ErrCASConflict`: another writer got |
| + there first; go to 1 (this is the simultaneous-start race, §2.4). |
| +2. Found: parse it. Then: |
| + - its `box_id` equals ours: **owned**, `Put` with the etag. |
| + - its `box_id` is non-empty and differs, and the object is fresh |
| + (`BoxStale` false): **foreign**. Do not write. Refuse (§2.2). |
| + - its `box_id` is non-empty and differs, and the object is stale: |
| + **takeover**. `Put` with the etag; log once that the label was taken |
| + from a dark writer. Its `box_id` is recorded in the evidence (§2.3) |
| + so readers can see the change. |
| + - its `box_id` is absent (an old-release writer, or this box before it |
| + rolled): compare `metrics.instance_id`. Equal and non-empty: owned. |
| + Both non-empty and different, object fresh: foreign; refuse. Either |
| + empty: **unproven**. Write, as today, and log once. An unproven |
| + label is not a proof of uniqueness (§4). |
| +3. A `Put` that loses the CAS race goes back to 1 and re-decides. It |
| + never re-Puts with a fresh etag without re-reading the body; that |
| + re-Put is the overwrite this spec removes. |
| + |
| +`CollectBox` sets `Box.BoxID` from `Probes.MachineID`. Fleet boxes |
| +(`agent.go:2926`, operator-chosen label, unique per fleet by |
| +`registry.ErrFleetBoxActive`) publish no `box_id` and skip the check; |
| +nothing changes for them. |
| + |
| +The verdict is held in a process-wide `labelOwnership` value: owned, |
| +foreign, unproven, or unknown (no successful read yet), with the time of |
| +the last successful decision. |
| + |
| +### 2.2 Refusal |
| + |
| +On foreign: |
| + |
| +- `WriteBox` returns `ErrLabelForeign`, carrying the foreign `box_id` and |
| + `instance_id`. `statusTick` does not publish; the lease election is not |
| + entered (a refusing box must not become the fleet's publisher). |
| +- The **twin latch** is set. While set: `edgesFilePass` publishes nothing |
| + and deletes nothing; the metrics, alerts, registry-snapshot, config-ack, |
| + relay-ack and usage loops skip their writes (they all key by the same |
| + label, `metricsBoxLabel`, `metrics_loop.go:729`, and would overwrite the |
| + owner's objects); the log, error and analytics sinks keep buffering to |
| + their caps and ship nothing (the chunk prefix is the label segment and |
| + the hourly prune would delete the owner's chunks, `storedoor.go:482`); |
| + the claim's arm and every claim step refuse (§4). Serving traffic is |
| + untouched: the edge, the activator and the reconcile pass do not depend |
| + on these publishers. |
| +- The box logs, once per transition, with the instructions: |
| + `box <label> is already held by another machine (instance <id>); this |
| + box (instance <id>) will not publish. Regenerate this box's machine id: |
| + rm /etc/machine-id && systemd-machine-id-setup && reboot. Its label |
| + will change.` |
| +- The box writes the evidence object (§2.3) every tick while latched, and |
| + delivers one `metrics.Event{Rule: "identity"}` through the metrics |
| + loop's sinks (`metrics_loop.go:510`) on the transition, so |
| + `CARLOS_ALERT_WEBHOOK` fires from the box that knows. |
| +- Every tick it re-runs §2.1. The latch clears when the verdict is owned |
| + (the holder went dark and this box took over) or when this box's own |
| + label changes (machine id regenerated, process restarted). Clearing is |
| + a transition and is logged and delivered the same way. |
| + |
| +### 2.3 Evidence |
| + |
| +`status/identity/<own box_id>.json`: |
| + |
| +```json |
| +{"v": 1, "label": "edsger-knitting", "box_id": "...", "instance_id": "i-...", |
| + "verdict": "foreign", "holder_box_id": "...", "holder_instance_id": "i-...", |
| + "seen_at": "2026-10-06T10:00:00Z"} |
| +``` |
| + |
| +Keyed by the writer's own id, so two colliding boxes never share an |
| +evidence key. Written every tick while the verdict is foreign or a |
| +takeover happened within the last three periods; an entry older than |
| +three snapshot periods is expired and ignored. The status lease holder |
| +lists the prefix on its publish tick. The prefix is under `status/` but |
| +is not one of the three keys the edge serves (`internal/edge/status.go:54`), |
| +and the leak scan (`dryrun.go`) scopes it like `status/boxes/`: private |
| +values allowed there, forbidden in fleet.json and index.html. |
| + |
| +### 2.4 Two new boxes start at once with an empty key |
| + |
| +Both read not-found and both `Put` with `ifMatch ""`. `release.Store`'s |
| +Put with an empty etag is create-only (the contract `WriteBox`'s own doc |
| +describes; `s3store` uses `If-None-Match`, `dirstore` the same rule), so |
| +exactly one succeeds. The other gets `ErrCASConflict`, re-reads, finds a |
| +fresh object with a foreign `box_id`, and refuses. The edges loop on both |
| +boxes waits for an owned verdict before its first publish (it starts a |
| +minute after arm, `edgesfile.go:749`, and the status loop's first tick is |
| +also at a minute, `agent.go:2769`; the wait makes the order irrelevant), |
| +so the loser never writes an edges file. If the label's edges file is a |
| +retired box's leftover with another `box_id`, the owner overwrites it: |
| +the snapshot settled the label. |
| + |
| +The one store whose create-only semantics are not the backend's own is |
| +the fleet store door; fleet boxes skip the check, so it does not matter. |
| + |
| +### 2.5 Takeover and flapping |
| + |
| +A holder whose snapshot goes stale (three missed periods, fifteen |
| +minutes) can be taken over by a latched sibling. When the holder's next |
| +write succeeds it finds a fresh foreign object and latches itself. One |
| +swap per staleness window at most, each logged, delivered and recorded in |
| +evidence; readers see the `box_id` change (§3). A transient bucket outage |
| +on one box can cause one swap; the alert then names both machines and |
| +the operator fixes the one that is new. |
| + |
| +## 3. Readers |
| + |
| +Readers add two stateless rules and one stateful one. Nothing about keys |
| +or joins changes. |
| + |
| +1. **Disagreeing objects.** Where a snapshot and an edges file share a |
| + label and both carry `box_id`, they must be equal. A mismatch is a |
| + sibling's leftover or a race, and the pair is **contested**: not live |
| + for `edgesfile.EdgeIPs` and `DeploymentRoutes` (`fold.go:84,158`), |
| + `BoxesComplete=false` for `readLiveEdgesSnapshot` |
| + (`instancesplacement.go:93`). Objects without `box_id` join by label |
| + as today. |
| +2. **Evidence.** An unexpired `status/identity/*` entry naming a label |
| + makes that label contested for `readLiveEdgesSnapshot` and |
| + `BoxesComplete=false`. This is how the console, which has no memory |
| + between requests, learns of a collision the writers saw. |
| +3. **Writer change.** The status lease holder keeps, in process memory, |
| + the last `box_id` and `generated_at` it saw per label. A `box_id` that |
| + changes, or appears or disappears, while the previous object was still |
| + fresh at the new object's `generated_at`, is a writer change: two |
| + machines are writing one label (an old-release sibling, which cannot |
| + refuse, or a takeover). It is reported as contested for three periods. |
| + A console restart forgets this; rules 1 and 2 do not. |
| + |
| +The lease holder folds all three into one singly-owned |
| +`alerts/fleet/identity.json` (`metrics.AlertState`, `Box: "fleet"`, rules |
| +`foreign`, `contested`, `changed`), written every publish tick, firing or |
| +not, so a lease transfer cannot strand a firing rule. The console's alert |
| +watcher (`alertwatch.go:97`) reads it alongside `alerts/boxes/*.json`, |
| +treats an object older than three periods as clear, and mails operators |
| +holding a grant; with none the transition is logged and dropped |
| +(`alertwatch.go:194`, amadan #192). The fleet page and |
| +`carlos status -live` show a `contested` chip and the holder's and the |
| +refuser's instance ids, so the operator can tell which box is new. |
| + |
| +## 4. The host lifecycle claim |
| + |
| +Claim rev 6.16 (`design-replica-ownership`, |
| +`2026-10-02-host-lifecycle-claim-design.md:238-246`) says box references |
| +are opaque identifiers compared by exact equality, each naming exactly one |
| +machine, and that how a box obtains a unique reference and refuses claim |
| +work when it cannot prove uniqueness is this design's job. Under Option B |
| +the reference stays the label, and the proof is label ownership: |
| + |
| +- a box takes or continues claim work only while `labelOwnership` is |
| + **owned** and its last successful decision is within three snapshot |
| + periods. Foreign, unproven, unknown, or a stale decision all refuse, |
| + the same way `resolveFleetIdentity` refuses an invalid fleet identity; |
| +- a box whose verdict turns foreign mid-operation drops its execution |
| + lease and lets the claim's own recovery owner take over, as it would |
| + for any lost lease; |
| +- an **unproven** label (an old-release or off-cloud object under our |
| + label with no instance id to compare) is exactly the case rev 6.16 |
| + names: the box cannot prove uniqueness and does not take claim work. |
| + On the flagship every box has an instance id, so this bites only during |
| + the roll (an old-release sibling's object) and on hardware no cloud |
| + owns. |
| + |
| +No claim field changes. Twins (one machine id on two machines) overwrite |
| +one key and cannot be told apart by `box_id`; `instance_id` tells them |
| +apart, and the rule in §2.1 refuses the second. Machine-id uniqueness |
| +stays a prerequisite the image recipe enforces, and this spec makes no |
| +clone-safety promise beyond that detection. |
| + |
| +## 5. Mixed versions and rollback |
| + |
| +- **Old writer, new reader.** No `box_id`: join by label, as today. Rule 3 |
| + sees `box_id` disappear if an old sibling overwrites a new box's object, |
| + and reports it. |
| +- **New writer, old reader.** The field is additive and ignored. |
| +- **Old sibling colliding with a new box.** The old box cannot refuse and |
| + overwrites every tick; the new box sees a fresh object with no `box_id` |
| + and a different `instance_id`, refuses, latches, alerts. The old box |
| + holds the label until the roll reaches it; then both run §2.1, and |
| + whichever object is fresh at the other's next tick holds. Today both |
| + overwrite silently; this is strictly better from the first rolled box. |
| +- **Rollback.** An old-release box has no check and overwrites blindly, as |
| + today. No object moved, so there is nothing to sweep, no key shape to |
| + reconcile, no DNS identifier to rename. The evidence prefix and the |
| + fleet alert object are additive; an old reader never lists them. The |
| + cost of rolling back is losing the detection. |
| +- **Steering** is untouched: `SetIdentifier` stays the label, and a |
| + refusing box publishes no edges file, so there is nothing to dedupe. |
| + Its records are simply absent, which is §1's cost. |
| + |
| +## 6. Findings, costs and the comparison |
| + |
| +### 6.1 Which findings disappear, which remain |
| + |
| +| Finding | Under Option B | |
| +|---|---| |
| +| R1-1, R2-3, R3-4: steering `SetIdentifier` is the label; renames; shrink guard | **gone.** No identifier changes. The refusing box has no records, which is a cost, not a hole | |
| +| R1-2, R2-1, R3-1: cleanup or sweep deletes a live writer's object | **gone.** Nothing is deleted | |
| +| R1-3, R2-2, R3-2, R3-3, R3-5: rollback duplicates, aliases, effective-file choice, orphan edges files | **gone.** One key shape, one object per label, no supersession | |
| +| R1-4: the mixed-shape join | **gone** | |
| +| R1-5, R3-6: the other label-keyed families (registry snapshots, acks, metrics, alerts, logs, `alert_seen`) | **covered by refusal**, not fixed: the latched box suspends those publishers, so the owner's objects are not overwritten. Their collisions are not removed, they are prevented from happening twice | |
| +| R1-6, R2-7: architecture page | **gone.** One box per label publishes. The refuser's machine appears as an unclaimed cloud instance, which the page already renders | |
| +| R1-7, R2-6: alert object ownership and webhook delivery | **remain**, same design: singly-owned fleet object, box-side webhook (§2.2, §3) | |
| +| R1-8: step 2 privacy in `apps/` keys | **gone.** No key changes; `box_id` is in bodies only | |
| +| R1-9, R2-4: claim references and twins | **remain**, simpler: the label stays the reference, ownership is the proof, unproven refuses (§4) | |
| +| R2-5: twin evidence reaching the publisher | **remains**, same mechanism (§2.3) | |
| +| R1-10: three facts | corrected in Option A rev 2; nothing to carry | |
| + |
| +### 6.2 Residual risk under Option B |
| + |
| +- A colliding box is out of multi-region routing, fleet status, metrics, |
| + logs and claim work until an operator regenerates its machine id. Its |
| + own edge serves its hosts locally. On the flagship a new box collides |
| + with one of four existing labels with probability 4 in 1,600, 0.25%, |
| + per box added; Tito 6 in 1,600. The first start of a new box is where |
| + it fires, so the cost lands in provisioning, not in production. |
| +- An old-release sibling cannot refuse. During the roll a collision |
| + between an old and a new box is detected and alerted but the old box |
| + keeps overwriting until it rolls. |
| +- An off-cloud box (no `instance_id`) under an old-release object is |
| + unproven: it publishes as today and takes no claim work. |
| +- Retired boxes' objects still linger (nothing deletes them today either); |
| + `EdgeIPs` and eligibility already handle that through liveness. |
| +- The detection trusts the status store: a deployment whose status bucket |
| + is split from its deployment bucket (`CARLOS_STATUS_BUCKET`) still |
| + arbitrates correctly, because both colliding boxes use the same |
| + configuration; the console's rule 2 then reads evidence from the status |
| + store it already reads snapshots from. |
| + |
| +### 6.3 Operator cost |
| + |
| +One alert, one `rm /etc/machine-id && systemd-machine-id-setup && reboot`, |
| +about fifteen minutes, on a box that has done no claim work and published |
| +nothing under the label. Its label changes; since it never held the label |
| +it has no history under it to lose. No runbook step on roll, none on |
| +rollback. |
| + |
| +### 6.4 Comparison |
| + |
| +| | Option A: migrate the keys | Option B: detect and refuse | |
| +|---|---|---| |
| +| Residual risk | Three review rounds in, four open P1s in the migration protocol (writer retirement, cross-object cleanup, effective-file choice for steering, DNS rename in N). Deletes production objects. Rollback has a runbook and known costs | A colliding box is withdrawn until fixed by hand (§6.2). No deletes, no key moves, no DNS changes. Old-release siblings undetected until they roll | |
| +| What it fixes | Both colliding boxes keep working, now and forever; the label is purely cosmetic | The collision is loud and deterministic; one box works, the other is fixed in minutes | |
| +| Implementation size | `status`, `edgesfile` (`Effective`), `host` (writer, guard, sweep, gate verify), `console` (join, arch page, alerts, CLI `fleet sweep`/`retire`), `edge` (homemap), `steering` (two releases), plus step 2 for six more key families. Two releases minimum, three with step 2 | `status.WriteBox` decide loop and `box_id`; a latch in `host` consulted by eight loops; the evidence object; one fleet alert object written by the lease holder; two console rules; the claim arm gate. One release | |
| +| Review rounds still likely | Two to three: the migration state machine and the DNS boundary are protocol decisions Astra has asked for twice | One to two: the writer rule and the race are small enough to simulate | |
| +| Flagship (4 boxes, no collision today) | A roll with a sweep, an N+1 DNS flip, a rollback runbook with a named steering cost; nothing visible if it goes well | A roll; nothing moves; a chip and an alert if a future box collides | |
| +| Tito (6 boxes) | the same, on a customer's fleet, with `alert_seen` migration for step 2 | the same as the flagship | |
| +| dev (1 box) | the key flip and sweep run for nothing | nothing happens | |
| + |
| +### 6.5 Recommendation |
| + |
| +**Option B now.** It removes the silent failure #212 describes, costs one |
| +release and no object moves, and its one residual cost is a quarter of a |
| +percent per box added, paid at provisioning in fifteen minutes. Option A |
| +buys two colliding boxes both working, which at this fleet size is not |
| +worth a deletion protocol that has needed three rounds and is not yet |
| +closed. Option A's spec stays on the branch as the record of why, and as |
| +the design to return to if a fleet ever grows to where regenerating a |
| +machine id is not an acceptable answer. |
| + |
| +## 7. Test plan |
| + |
| +- `WriteBox` decide loop: not-found creates; owned replaces; foreign |
| + fresh refuses with the foreign ids in the error; foreign stale takes |
| + over and logs; absent `box_id` with equal, differing and empty |
| + `instance_id`; CAS conflict re-reads and re-decides, never re-Puts |
| + blind. Two writers against one DirStore from one start: exactly one |
| + owns, the other refuses, in both orders. |
| +- The latch: every listed publisher skips while set; the edges loop |
| + waits for an owned verdict before its first publish; the sinks buffer |
| + to cap; the claim gate refuses on foreign, unproven, unknown and stale |
| + decisions; clear on takeover and on label change. |
| +- Evidence: written each tick while latched, expires after three periods, |
| + keyed by the writer's id; the dry run allows private values there and |
| + forbids them in fleet.json and index.html. |
| +- Readers: rule 1 (disagreeing `box_id`) makes a pair not live and |
| + incomplete; rule 2 (evidence) makes a label contested for the console; |
| + rule 3 (writer change) fires on a change within freshness and not after |
| + staleness; the fleet alert object is written firing and clear; the |
| + watcher mails and clears. |
| +- Mixed versions: an old-shape object (no `box_id`) under each prefix |
| + joins as today; a new-shape object asserted byte-compatible for the |
| + fields an old reader uses. |
| +- Simulation: random fleets with injected collisions, simultaneous |
| + starts, rolls, rollbacks and one-box bucket outages, asserting that at |
| + most one machine writes a label while its object is fresh, that a |
| + latched box takes no claim work, and that every collision is in the |
| + evidence within one period. |
| + |
| +## 8. Rollout |
| + |
| +1. Confirm the image recipes clear `/etc/machine-id`. |
| +2. Land on main; cut a release; roll boxes in any order; roll the |
| + console. Nothing moves, so there is no order and no sweep. |
| +3. Add the alert's instruction text to the operator runbook, and |
| + `rm /etc/machine-id && systemd-machine-id-setup && reboot` to the |
| + provisioning recipe's "if the box reports a foreign label" note. |
| +4. Record the §4 gate on the claim branch as the answer to rev 6.16's |
| + open reference. |
| + |
| +## 9. Open questions for Paul |
| + |
| +1. Option B or Option A (§6.5)? |
| +2. Under B, should the first start of a new box check its label against |
| + the fleet before joining (a `carlos fleet check-label` in the |
| + provisioning recipe), so the collision is found before the box is |
| + built rather than at its first tick? |
| +3. A latched box suspends log shipping and buffers to cap. Is losing a |
| + colliding box's logs for the minutes until it is fixed acceptable, or |
| + should it ship under a temporary `<label>-<4 hex>` segment? |
| +4. Should the public fleet.json say anything when a label is contested |
| + (a `contested: true` on the row), or stay silent as it does for every |
| + other operator-only condition? This spec keeps it silent. |
amadan