amadan
Box identity spec rev 2: answer Astra's round 1
Four P1s: steering keys Route53 SetIdentifier by label (the converger learns renames exempt from the shrink count); legacy cleanup becomes a conditional DeleteIfMatch on non-empty evidence; edgesfile.Effective picks one file per box for every reader and the rollback runbook gains carlos fleet sweep; the join is one explicit three-step algorithm with an unambiguous-only label fallback. Five P2s and the P3 accepted too: registry snapshots, config-ack and relay-ack added to the inventory, the architecture page keyed by id from HostRows, a singly-owned alerts/fleet/identity.json, step 2 gated on a privacy ruling, the claim ruling widened with a twin check, and three facts corrected. Findings committed unedited as CODEX-FINDINGS-BOX-IDENTITY-R1.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+464
−271
docs/superpowers/specs/2026-10-06-box-identity-collisions-design.md+439 −271docs/superpowers/specs/CODEX-FINDINGS-BOX-IDENTITY-R1.md+25 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-10-06-box-identity-collisions-design.md b/docs/superpowers/specs/2026-10-06-box-identity-collisions-design.md| index 7f2ad9d1d..f478a11a1 100644 |
| --- a/docs/superpowers/specs/2026-10-06-box-identity-collisions-design.md |
| +++ b/docs/superpowers/specs/2026-10-06-box-identity-collisions-design.md |
| @@ -1,11 +1,18 @@ |
| # Box identity: key box objects by a collision-resistant id, keep the label for people |
| **Date:** 2026-10-06 |
| -**Revision:** 1 |
| +**Revision:** 2 |
| + |
| +- Rev 1 (71112c9e) went to Astra (GPT-6, via codex); the findings are in |
| + `CODEX-FINDINGS-BOX-IDENTITY-R1.md`. Four P1s, five P2s, one P3, all |
| + checked against the code and accepted. Changes are marked "(codex N)" and |
| + §11 records each disposition. |
| + |
| **Discussion:** amadan carlos/platform #212 |
| -**Status:** draft, for Paul's review after Astra's. |
| +**Status:** draft, for Paul's review after Astra's second round. |
| **Builds on:** the fleet status page (`internal/status`), the multi-region |
| -ingress edges file (2026-08-20, Part 3), nearest-region placement |
| +ingress edges file (2026-08-20, Part 3), the steering converger |
| +(`internal/steering`), nearest-region placement |
| (`2026-10-01-nearest-region-default-design.md`), the host lifecycle claim |
| (`2026-10-02-host-lifecycle-claim-design.md`, on `design-replica-ownership`). |
| @@ -18,22 +25,24 @@ two per-box objects every platform box writes: |
| - `status/boxes/<label>.json` (`snapshot.go:747`) |
| - `control/edges/<label>.json` (`internal/host/edgesfile.go:36`) |
| -Two boxes with one label overwrite each other. Birthday odds are about 0.4% |
| -for four boxes, 1% for six, 6% for twenty. Astra found a concrete pair: |
| -machine ids `00000000000000000000000000000018` and |
| -`0000000000000000000000000000003c` both give `edsger-knitting`. |
| +and the Route53 `SetIdentifier` of every steered record |
| +(`internal/steering/route53.go:601`). Two boxes with one label overwrite |
| +each other. Birthday odds are about 0.4% for four boxes, 0.9% for six, 11% |
| +for twenty (codex 10). Astra found a concrete pair: machine ids |
| +`00000000000000000000000000000018` and `0000000000000000000000000000003c` |
| +both give `edsger-knitting`. |
| When it happens: `edgesfile.EdgeIPs` loses one box's address; the fleet |
| page shows one box where there are two; nearest-region eligibility joins |
| edges files to snapshots by label and can read a keyless box as eligible; |
| -the edges writer's unchanged-file gate (`edgesfile.go:491`) can leave the |
| -wrong box's file in place indefinitely, because the gate compares against |
| -memory, not the bucket. |
| +steering keeps one of the two boxes' records; the edges writer's |
| +unchanged-file gate (`edgesfile.go:491`) can leave the wrong box's file in |
| +place indefinitely, because it compares against memory, not the bucket. |
| The label is public (fleet.json, the status page) and must stay |
| -non-reversible from the machine id. That contract is kept. The label is |
| -not changed. What changes is what the objects are keyed by, and how readers |
| -join them. |
| +non-reversible from the machine id. That contract is kept. The label is not |
| +changed. What changes is what the objects are keyed by, how readers match |
| +them, and what names a box in steering. |
| ## 2. Every consumer of the label-keyed objects |
| @@ -47,46 +56,48 @@ join them. |
| | `internal/host/metrics_loop.go:729` `metricsBoxLabel`, `:743` `BoxLabel` | the one selector: fleet label on a fleet box, else `status.Label` | |
| | `internal/host/agent.go:542` `edge.WithBoxLabel` | the edge's notion of self for the home map | |
| -The same selector names the box in other per-box keys. They are not the |
| -subject of #212 but collide the same way and are listed so the fix lands in |
| -one place (§3.5, step 2): |
| +The same selector names the box in other per-box keys. They collide the |
| +same way. They are listed so one identity fixes all of them, in two steps |
| +(§3.6). (codex 5: the first three were missing in rev 1.) |
| -| Key | Where | |
| -|---|---| |
| -| `metrics/boxes/<box>/raw/<hour>.json`, `/days/<date>.json` | `internal/metrics/chunk.go:25,31`; writer `metrics_loop.go:136` | |
| -| `alerts/boxes/<box>.json` | `internal/metrics/alert.go:120` | |
| -| `metrics/relay/<box>/<hour>.json` | `internal/relay/meter.go:26`; `relay.go:138` | |
| -| `apps/<sqid>/<app>/{logs,errors,analytics}/<box>/...` | `internal/applog/chunk.go:58`; sinks at `agent.go:244,245,477,480,569` use `status.Label` directly | |
| -| usage flush | `internal/host/usage_loop.go:72` | |
| +| Key | Where | Readers | |
| +|---|---|---| |
| +| `control/registry-snapshot/<box>.db`, `.json`; `control/registry-snapshots/<box>/` | `internal/host/snapshot_loop.go:34-37`, `:205` | the DR restore (README), `internal/dbverify/dbverify.go:28`. A collision here hands DR the wrong box's registry | |
| +| `control/config-ack/<box>/<host>.json` | `internal/appconfig/configack.go:116`; `agent.go:954` uses `BoxLabel()` | `internal/console/apirelay.go:151` | |
| +| `control/relay-ack/<box>.json` | `internal/relay/authority.go:71`; `relay.go:138` | `apirelay.go:163` | |
| +| `metrics/boxes/<box>/raw/<hour>.json`, `/days/<date>.json` | `internal/metrics/chunk.go:25,31`; `metrics_loop.go:136` | `internal/console/metricsview.go:140`, which displays the key segment (`:547`), not the body's `Box` | |
| +| `alerts/boxes/<box>.json` | `internal/metrics/alert.go:120` | `internal/console/alertwatch.go:97` | |
| +| `metrics/relay/<box>/<hour>.json` | `internal/relay/meter.go:26` | the relay status page | |
| +| `apps/<sqid>/<app>/{logs,errors,analytics}/<box>/...` | `internal/applog/chunk.go:58`; sinks at `agent.go:244,245,477,480,569` call `status.Label` directly | `carlos logs`; the cursor orders by `Box` (`internal/applog/cursor.go:24`); readable by the account's fleet credentials (`storedoor.go:403`) | |
| +| usage flush | `internal/host/usage_loop.go:72` | billing passes | |
| ### 2.2 Readers of `status/boxes/` |
| | Where | Join or key | |
| |---|---| |
| | `internal/status/fleet.go:238` `loadBoxesReport` | the ONE parser; `:150` `LoadBoxes`, `:234` `LoadBoxesComplete` | |
| -| `fleet.go:157` `BuildFleet` | public feed; `:173` copies `Label` into `FleetBox`; `:563` `PublishFleet` writes `status/fleet.json` | |
| +| `fleet.go:157` `BuildFleet` | public feed; `:173` copies `Label` into `FleetBox`; `:272` `sumInstances` counts stale boxes too; `:563` `PublishFleet` writes `status/fleet.json` | |
| | `internal/edgesfile/fold.go:46` `EdgeIPs`, `:125` `DeploymentRoutes` | join `live[b.Label]` to `f.Box` (`:64,84,137,158`) | |
| -| `internal/console/instancesplacement.go:93` `readLiveEdgesSnapshot` | `known[b.Label]`, `live[b.Label]` (`:120,122`) joined to `f.Box`; feeds automatic eligibility | |
| +| `internal/console/instancesplacement.go:93` `readLiveEdgesSnapshot` | `known[b.Label]`, `live[b.Label]` (`:120,122`) joined to `f.Box`; feeds automatic eligibility; `:138` a fresh snapshot with no file is incomplete evidence | |
| | `internal/console/view.go:854` `HostRows` | operator fleet page rows by label; `templates/fleet.html:47,109` | |
| -| `internal/console/apiinstances.go:368` `loadBoxJoin` | staged_version, upstreams, db_bytes: joined by hostname inside `Metrics`, label incidental; callers `apiinstances.go:422`, `apiroutes.go:151`, `apisidecars.go:150` | |
| +| `internal/console/apiinstances.go:368` `loadBoxJoin` | staged_version, upstreams, db_bytes: joined by hostname inside `Metrics`; callers `apiinstances.go:422`, `apiroutes.go:151`, `apisidecars.go:150` | |
| | `internal/console/apidomains.go:1144` | alias DNS and cert readings, by hostname | |
| | `internal/console/apisystemupdate.go:233,258,764` | `carlos system` rows by label; CLI `system.go:207,438` | |
| -| `internal/console/archgraph.go:561` `buildBoxCards` | keyed by label (`:574,580`), node id `"box:"+label` (`:807`), machine join by `CloudID` (`:271`); `archregion.go:59` | |
| +| `internal/console/archgraph.go:561` `buildBoxCards` | unions public `FleetBox` rows and private `HostRow`s by label (`:574,580`); node id `"box:"+label` (`:807`); machine join `claimed[CloudID]=label` (`:271`) and `targetOf` (`:993`); `archregion.go:59` | |
| | `cli.go:3389` `statusLive`, `:3418` | `carlos status -live`, label-sorted | |
| | `internal/status/dryrun.go` | the leak scan; `:182` fixture machine id `ip-10-0-0-7` | |
| -| `internal/console/fleets.go:1012,1030` | fleet heartbeats under `fleets/<name>/status/boxes/<label>.json`, joined by the operator's `fleet_boxes.label`. Out of scope (§3.6) | |
| +| `internal/console/fleets.go:1012,1030` | fleet heartbeats under `fleets/<name>/status/boxes/<label>.json`, joined by the operator's `fleet_boxes.label`. Out of scope (§3.7) | |
| ### 2.3 Readers of `control/edges/` |
| | Where | Join or key | |
| |---|---| |
| -| `internal/edgesfile/edgesfile.go:68` `File`, `:81` `Parse` | the shared reader | |
| -| `internal/edge/homemap.go:130` `edgesFileView`; `:273` `refreshHosts`; `:538` `foldHomes` | skips the file whose `Box == self` (`:547`); keys the file map by object key | |
| -| `internal/host/steering_loop.go:50` `loadEdgesFileBodies`; `internal/steering/deployment.go:56` | fails closed on a short read; folds by EIP and hosts, not by box | |
| +| `internal/edgesfile/edgesfile.go:68` `File`, `:81` `Parse` | the shared leaf package every other reader imports | |
| +| `internal/edge/homemap.go:130` `edgesFileView`; `:273` `refreshHosts`; `:538` `foldHomes` | skips the file whose `Box == self` (`:547`); on a host conflict the dial target is the first file in sorted **object key** order | |
| +| `internal/host/steering_loop.go:50` `loadEdgesFileBodies`; `internal/steering/steering.go:73` view, `:349` `desiredState`, `:310` `dedupeRecords` by `(Name, Box)`, `route53.go:601` `SetIdentifier: rec.Box` | fails closed on a short read; **keys DNS records by the label** (codex 1) | |
| | `internal/console/instancesplacement.go:38` view, `:175` `loadEdgesJoin` (`served_by.box`, `:179`), `:232` `readEdgesSnapshot`, `:320` `placements` | the create door and the instances API | |
| | `internal/host/movespass.go:335` `moveTargets` | one target per placement, tie broken by `Box` (`:346`) | |
| | `internal/console/server.go:906` `edgeFold`; `internal/host/agent.go:133` | `EdgeIPs` for the custom-domain DNS check | |
| -| `internal/edge/storedoor.go:411` | everything under `control/` is readable by fleet boxes through the store door | |
| ### 2.4 The public fleet feed and its privacy contract |
| @@ -97,11 +108,16 @@ README "Fleet status"). `BuildFleet` copies `Box` fields into `FleetBox` |
| one by one and never names `Metrics`; `dryrun.go` asserts a raw value |
| survives into the box object and never into fleet.json or index.html. |
| -The contract this spec keeps: fleet.json carries the label only. The new |
| -id never appears in fleet.json, index.html, or any account-facing page |
| -(`accountfleet.go:18` reads fleet.json only). Both label-keyed objects are |
| -bucket-private. `control/edges/` is readable by customer fleet boxes |
| -through the store door; that is already true of the label and the EIP. |
| +The contract this spec keeps: fleet.json carries the label only. The id |
| +never appears in fleet.json, index.html, any account-facing page |
| +(`accountfleet.go:18` reads fleet.json only), the instances API |
| +(`served_by.box` stays the label), or an alert mail (§3.4). `status/boxes/` |
| +and `control/edges/` are bucket-private: the store door grants fleet boxes |
| +`accounts/<sqid>/`, `apps/<sqid>/`, `fleets/<fleet>/`, `domains/` and two |
| +exact `control/system/` and `bootstrap/router/` shapes |
| +(`storedoor.go:403-425`), nothing else under `control/` (codex 10 corrects |
| +rev 1 here). The one place the id would become member-visible is the |
| +`apps/` subtree in step 2, and that is held for a ruling (§3.6, §10). |
| ## 3. The design |
| @@ -117,23 +133,35 @@ func LabelForID(id string) (label string, ok bool) |
| ``` |
| - 128 bits. A collision needs about 2^64 boxes. The label's 10.6 bits are |
| - the whole problem; 64 bits would already do, 128 costs nothing and makes |
| - the hex id visibly not a label. Fixed length, lower-case hex, so it is a |
| - valid key segment in S3, the DirStore (`store.go:269`) and the store |
| - door, and it can never be confused with a `name-action` label or a fleet |
| - label (`registry.ValidTunnelName`). |
| + the whole problem; 64 bits would already do, 128 costs nothing. Fixed |
| + length, lower-case hex: a valid key segment in S3, the DirStore |
| + (`store.go:269`) and the store door, and a valid Route53 |
| + `SetIdentifier` (limit 128). It is told from a `name-action` label by |
| + having no hyphen. It is **not** syntactically disjoint from a fleet label |
| + (`registry.ValidTunnelName`, `registry.go:337`, accepts 32 hex), so |
| + `AddFleetBox` refuses a label that parses as a box id (codex 10). |
| - Non-reversibility is the same argument the label makes: sha256 over a |
| 128-bit random `/etc/machine-id`. Exposing more digest bits does not |
| - weaken a preimage. The honest difference: for a low-entropy machine id |
| - (the dry-run fixture `ip-10-0-0-7`, a container that hands out a |
| - hostname), the id confirms a dictionary guess where the label only |
| - narrows it to 1 in 1,600. That is why the id stays bucket-private (§2.4) |
| - and the leak scan gets a needle for it (§5). |
| -- An empty machine id (`probes.go:153`, unreadable file) gives no id. The |
| - agent then behaves exactly as today: label key, no `box_id` field. Readers |
| - treat it as an old writer. This is the one case the id cannot be minted, |
| - and it is already the case `Label("")` documents as "this probe never |
| - worked". |
| + weaken a preimage. `LabelForID` reproduces the label from the id and |
| + discloses nothing about the machine id. The honest difference: for a |
| + low-entropy machine id (the dry-run fixture `ip-10-0-0-7`, a container |
| + that hands out a hostname), the id confirms a dictionary guess where the |
| + label only narrows it to 1 in 1,600. That is why the id stays |
| + bucket-private (§2.4) and the leak scan gets a needle for it (§8). |
| +- An empty machine id (`probes.go:153`) gives no id. The agent then behaves |
| + exactly as today: label key, no `box_id` field, and it refuses to arm |
| + anything that needs an identity (§5). Readers treat it as an old writer. |
| +- **Machine ids are assumed unique.** A cloned image that keeps |
| + `/etc/machine-id` produces two boxes with one id, and they overwrite one |
| + key rather than producing two objects, so §3.4's duplicate check cannot |
| + see it (codex 9). The writer checks instead: `WriteBox` already reads the |
| + current object for its etag; it now also parses it, and if that object is |
| + fresh and carries a non-empty `instance_id` different from this box's, |
| + this box has a twin. It logs, publishes the `identity` alert (§3.4) and |
| + keeps writing, so the twin is visible rather than silent. A box with no |
| + instance id (off-cloud) cannot make this check. The image build must |
| + clear `/etc/machine-id` so systemd regenerates it on first boot; the |
| + rollout checks the flagship's AMI recipe does (§9). |
| - The label is unchanged. `Label` keeps its function and its table so the |
| public label of every existing box is stable through the roll. |
| @@ -143,8 +171,9 @@ One selector replaces `metricsBoxLabel`: |
| ```go |
| type BoxIdentity struct{ ID, Label string } |
| -// On a platform box: ID = status.BoxID(machineID), Label = status.Label(machineID). |
| -// On a fleet box: both are the operator-chosen fleet_boxes.label (§3.6). |
| +// Platform box: ID = status.BoxID(machineID), Label = status.Label(machineID). |
| +// Fleet box: both are the operator-chosen fleet_boxes.label (§3.7). |
| +// ID is "" when the machine id is unreadable. |
| func boxIdentity(getenv func(string) string, machineID string) BoxIdentity |
| ``` |
| @@ -155,269 +184,408 @@ label and leaves `box_id` empty. |
| Edges file. `edgesFile` gains `BoxID string json:"box_id,omitempty"` next |
| to `Box`. `edgesFileKey` takes the id when present, else the label. `Box` |
| -keeps the label; every reader that shows a box to a person keeps reading |
| -it. The new field is part of the change gate like every field but |
| -`Generation`. |
| - |
| -Both objects carry both. A reader that has both sides new joins by id. A |
| -reader that has an old side joins by label, as today. |
| - |
| -### 3.3 Readers: one join rule, in one place |
| - |
| -`loadBoxesReport` (`fleet.go:238`) stays the one parser and gains a |
| -dedupe step before returning, reported in a new `Report` for the |
| -completeness readers: |
| - |
| -1. Group objects by `box_id` when present. Two objects (two keys) with one |
| - id is a **duplicate identity**: both are returned but flagged, `complete` |
| - is false, and the condition is reported (§3.4). |
| -2. Among objects that share a label, exactly one of which has no `box_id`, |
| - compare `Metrics.InstanceID`. Same instance id: one box, seen through an |
| - old key and a new key (a roll or a rollback in progress). Keep the |
| - fresher, mark the other **superseded**, do not count it. Different or |
| - absent instance ids: two boxes sharing a label, one old writer. Both |
| - stand; this is the collision the design exists for, and it is reported |
| - while the legacy key exists for that label. |
| +keeps the label for every surface that shows a box to a person. The new |
| +field is part of the change gate like every field but `Generation`. |
| + |
| +Steering. `steering.edgesFileView` gains `BoxID`; `desiredRecord.Box` |
| +becomes `BoxKey(file)` (the id when present, else the label), so the |
| +Route53 `SetIdentifier` and the `dedupeRecords` key become the id (codex 1). |
| +The converger's handling of the identifier change is §3.5. |
| + |
| +A `BoxKey` helper, in `internal/edgesfile` for files and `internal/status` |
| +for snapshots, returns the id if set, else the label. Both objects carry |
| +both names; which one a reader matches on is §3.3. |
| + |
| +### 3.3 Matching: one algorithm, in one place (codex 4) |
| + |
| +Three steps, each in one package, each with one owner. |
| + |
| +**Step A: snapshots** (`status.loadBoxesReport`, `fleet.go:238`). Returns |
| +the boxes plus a `Report{Duplicates, Superseded, Contested []string}`. |
| + |
| +1. Group by `box_id`. Two objects (two keys) with one id is a |
| + **duplicate**: both are returned and flagged; `complete` is false. |
| +2. For each label carried by exactly one id-less object and one or more |
| + id-bearing objects: if the id-less object and an id-bearing one carry |
| + the same non-empty `metrics.instance_id`, they are one box seen through |
| + an old key and a new key (a roll or rollback in progress). The fresher |
| + is kept; the other is **superseded** and dropped from the returned |
| + slice, so nothing downstream counts it. If instance ids differ, or |
| + either is empty, both stand and the label is **contested** (two boxes, |
| + one old writer, or an off-cloud box mid-rollback); `complete` is false |
| + for the completeness readers. Contested is not Duplicate: the public |
| + feed shows both rows, because both are boxes. |
| 3. Everything else passes through unchanged. |
| -A `BoxKey(b Box) string` helper returns `box_id` if set, else the label. |
| -The edges-file join uses it on both sides: |
| - |
| -- `edgesfile.EdgeIPs` and `DeploymentRoutes` (`fold.go:64,84,137,158`) |
| - build `live` over `BoxKey(b)` and also over `b.Label` for boxes with no |
| - id; a file joins on `box_id` when it has one, else on `box`. A file whose |
| - `box_id` is set but whose label does not equal `LabelForID(box_id)` is |
| - not a live edge: it was not written by this code. |
| -- `readLiveEdgesSnapshot` (`instancesplacement.go:120,122`): the same |
| - rule, and the dedupe report folds into `BoxesComplete`. |
| -- `homemap.foldHomes:547` skips self by id when both sides have one, else |
| - by label. Two files with one label and different ids both fold; today |
| - the second would never have existed. |
| -- `moveTargets:346` ties on `BoxKey`. |
| -- `archgraph.go:574,580,807` and `archregion.go:59` key cards and nodes by |
| - `BoxKey`; `HostRow` gains `ID`. The title stays the label; when two cards |
| - share a label the private page appends the first four hex characters of |
| - the id. fleet.json and the public page show two rows with one label, |
| - which is the true count. |
| - |
| -Because every reader lists the prefix and reads identity from the body, |
| -the object key name is not load-bearing for any reader on main today. That |
| -is what makes the migration a one-release change (§4). |
| - |
| -### 3.4 Duplicate detection is incomplete evidence |
| - |
| -Two conditions, both reported by the one parser: |
| - |
| -- **Two objects claim one id.** Impossible for honest writers; it means a |
| - copied disk image, a forged object, or a bug. Eligibility fails closed |
| - for the whole request (`BoxesComplete=false`), the same posture as a |
| - failed read. The fleet page and `carlos status -live` show both rows |
| - with a `duplicate id` chip. |
| -- **One label has two ids while a legacy key exists for that label.** The |
| - label-keyed objects are contested and nothing keyed by label can be |
| - trusted for those boxes (the step-2 keys in §2.1 included). Eligibility |
| - fails closed for the whole request. Once no legacy key remains for the |
| - label it is a display collision only and stops being reported. |
| - |
| -Alerting reuses what exists. The status lease holder evaluates these two |
| -conditions in `BuildFleet` and publishes a rule `identity` in its own |
| -`alerts/boxes/<box>.json` (`metrics.AlertState`), firing while either |
| -holds. The console's alert watcher (`alertwatch.go:97`) then mails |
| -operators as it does for disk or memory, and clears when the condition |
| -clears. One log line per condition per process, as `livenessSaid` does. |
| -Nothing reaches fleet.json. |
| +**Step B: edges files** (`edgesfile.Effective(files []Ident) (keep |
| +map[string]bool, contested []string)`, where `Ident` is `{Key, Box, BoxID, |
| +EIP, Generation}`). Every reader of `control/edges/` applies it to its |
| +listing before folding: homemap, steering's loader, the console's |
| +`readEdgesSnapshot`, `moveTargets`, `EdgeIPs`, `DeploymentRoutes`. This is |
| +what makes the object key not load-bearing; in rev 1 it was asserted, now |
| +it is a function (codex 3). |
| + |
| +1. Two files with one `box_id`: both kept, flagged duplicate. |
| +2. A file with no `box_id` and a file with one, same `box` label and the |
| + same non-empty `eip`: one box, old and new key. Two boxes never share an |
| + EIP. The newer `generation` is effective; the other is dropped. The |
| + dropped file's hosts do not fold, so the effective file alone decides |
| + what the box serves. On rollback the legacy file is rewritten by the |
| + box's first tick (§6), so it is the newer one. |
| +3. Same label, no `box_id` on one side, EIPs differ or either is empty: |
| + both kept, label contested. |
| +4. Everything else kept. |
| + |
| +**Step C: the join** (`EdgeIPs`, `DeploymentRoutes`, |
| +`readLiveEdgesSnapshot`, `foldHomes` self-skip, `moveTargets`). After A and |
| +B, with `snapshots` and `files`: |
| + |
| +1. A file with a `box_id` matches the snapshot with that `box_id`. It never |
| + falls back to the label: a box writes both objects from one process, so |
| + its snapshot carries the id within one tick of its file, and a legacy |
| + snapshot under its label during that minute is either its own |
| + not-yet-deleted old object (superseded or about to be) or a colliding |
| + sibling's, which must not vouch for it. |
| +2. A file with no `box_id` matches by label, and only when exactly one |
| + snapshot carries that label after Step A. Two or more is ambiguous: no |
| + match, label contested. |
| +3. A snapshot with no file, or a file with no snapshot, keeps today's |
| + meaning (`instancesplacement.go:120-150`: incomplete evidence for |
| + eligibility, not live for `EdgeIPs`). |
| +4. A file whose `box_id` is set and whose `box` is not `LabelForID(box_id)` |
| + was not written by this code. It is not live and is flagged. |
| +5. `foldHomes` skips self by `BoxKey` when the edge was given an id, else |
| + by label. `moveTargets` ties on `BoxKey`. |
| + |
| +Contested, duplicate and flagged all make `BoxesComplete` false for |
| +automatic eligibility, which is the posture `readLiveEdgesSnapshot` |
| +already takes for anything it cannot prove. |
| + |
| +**The architecture page** (codex 6). `buildBoxCards` stops unioning |
| +`FleetBox` rows with `HostRow`s. It builds cards from `HostRows` alone |
| +(`Box` carries every field `FleetBox` has, and `Stale` is computed the |
| +same way), keyed by `BoxKey`, node id `"box:"+BoxKey`, `claimed` and |
| +`targetOf` likewise. `HostRow` gains `ID`. When the host read failed |
| +(`ArchBoxesUnread`) the page falls back to `FleetBox` rows keyed by label, |
| +and a label shared by two rows renders one card marked ambiguous. Card |
| +titles are the label; two cards with one label append the first four hex |
| +characters of the id. |
| + |
| +### 3.4 Duplicate detection is incomplete evidence, and how it alerts |
| + |
| +Three conditions, all from §3.3: a duplicate id; a contested label; a |
| +twin (§3.1). The status lease holder evaluates them in `BuildFleet` and |
| +writes one singly-owned object, `alerts/fleet/identity.json`, in |
| +`metrics.AlertState` shape with `Box: "fleet"` and one rule per condition, |
| +**every publish tick**, firing or not (codex 7). Ownership follows the |
| +status lease: whichever box publishes writes it, so a lease transfer |
| +cannot strand a firing rule, and clearing is just the next tick's write. |
| +The alert watcher (`alertwatch.go:97`) reads this one key alongside |
| +`alerts/boxes/*.json`, and treats an object older than three snapshot |
| +periods as all-clear, the same `BoxStale` rule the boxes use. Delivery is |
| +the existing path: operators holding a keymail grant are mailed |
| +(`alertwatch.go:187`), and the webhook when `CARLOS_ALERT_WEBHOOK` is set. |
| +The existing limitation stands and is named: with no grant and no webhook |
| +the transition is logged and dropped (`alertwatch.go:194`, amadan #192). |
| +The fleet page and `carlos status -live` show the condition as a chip on |
| +the rows involved. Nothing reaches fleet.json. |
| ### 3.5 Writers |
| -Step 1 (this release): |
| - |
| -- `WriteBox` and `edgesFilePass` write the id key. |
| -- **No dual write of the legacy key.** No reader needs it (§3.3), and a |
| - second object for the same box would be counted twice by every old |
| - reader: `sumInstances` includes stale boxes, so the public instance |
| - total would double for that box, and `BuildFleet` would show two rows. |
| - Dual-writing would create the symptom we are removing. |
| -- **Own legacy key cleanup, guarded.** After its first successful id-keyed |
| - write, a box deletes `status/boxes/<label>.json` only if that object's |
| - `metrics.instance_id` equals its own, and `control/edges/<label>.json` |
| - only if that object's `eip` equals its own non-empty EIP. Otherwise the |
| - legacy object is a colliding sibling's, or unprovable, and is left for |
| - the sweep. Deleting a sibling's edges file would be the gate bug again |
| - with the roles swapped. The check runs once per process start, so a |
| - box that was rolled back and forward again cleans up after itself. |
| +- `WriteBox` and `edgesFilePass` write the id key. **No dual write of the |
| + legacy key.** No reader needs it (§3.3), and a second object for one box |
| + would be counted twice by every old reader: `sumInstances` includes stale |
| + boxes, so the public instance total would double for that box, and |
| + `BuildFleet` would show two rows. Dual-writing creates the symptom we are |
| + removing. |
| +- **Own legacy key cleanup, conditional** (codex 2). After its first |
| + successful id-keyed write, a box reads `status/boxes/<label>.json` and |
| + `control/edges/<label>.json`. It deletes the snapshot only if that |
| + object's `metrics.instance_id` equals its own and both are non-empty; |
| + the edges file only if that object's `eip` equals its own and both are |
| + non-empty. Each delete is `release.DeleteIfMatch` against the etag it |
| + just read (`internal/release/conditional_delete.go:28`), never a plain |
| + `Delete`: a sibling that overwrites the key between the read and the |
| + delete changes the etag and the delete refuses. A store that cannot |
| + enforce the condition refuses too, and the object is left for the sweep. |
| + Positive, non-empty evidence or nothing: an off-cloud box with no |
| + instance id leaves its legacy snapshot for the sweep. The check runs once |
| + per process start. |
| - **The edges gate verifies against the bucket.** Every tenth tick with |
| - no local change (about ten minutes), `edgesFilePass` reads its own key. |
| - Missing or different from `*last` (Generation aside): republish. Nothing |
| - else should write an id key, so this is defence in depth, and it is also |
| - how a file removed by the sweep or by hand comes back without waiting for |
| - a route change. |
| -- **Retired boxes.** The status lease holder runs an identity sweep on its |
| - publish tick: a snapshot stale for `retiredAfter` (7 days) is deleted |
| - together with the edges file that joins to it (by id, else by label); a |
| - superseded legacy snapshot (§3.3 rule 2) is deleted at once, with its |
| - legacy edges file when the EIP matches the fresh object's. An edges file |
| - with no snapshot at all is reported, never deleted automatically: nothing |
| - can date it. `carlos fleet retire <id|label>` deletes both objects by |
| - hand after confirming what it found. Deletes name the key only; the |
| - deployment bucket is versioned, so a wrong delete is recoverable. |
| - |
| -Step 2 (next release, its own plan): `boxIdentity().ID` replaces the label |
| -in every key of §2.1's second table, with the label carried inside the |
| -object as `box` is today (`RawChunk.Box`, `AlertState.Box`). Readers of |
| -those prefixes (`metricsview.go:140`, `alertwatch.go:131`, the logs API) |
| -take the segment as opaque and display the label from the body. This spec |
| -does not fix those collisions; it fixes the two objects that other |
| -machinery joins on, and gives step 2 its identity. |
| - |
| -### 3.6 Fleet boxes |
| + no local change, `edgesFilePass` reads its own key. Missing, or different |
| + from `*last` with Generation cleared: republish. Nothing else should |
| + write an id key, so this is defence in depth, and it is also how a file |
| + removed by the sweep or by hand comes back without waiting for a route |
| + change. It does not fix the old release's gate; nothing can. |
| +- **The steering identifier change** (codex 1). When `SetIdentifier` moves |
| + from label to id, a box's records are deleted under one identifier and |
| + upserted under another in one pass. The converger learns a **rename**: a |
| + managed-but-unwanted record whose `(Name, Value, Region)` is also being |
| + upserted under a wanted identifier is a rename, applied as upsert then |
| + delete in the same change batch and exempt from the shrink count |
| + (`route53.go:207-214`). Without this a single-armed-box deployment |
| + could never flip: all its records would count as a shrink of 100%. |
| + While old and new boxes alternate holding the housekeeper lease, an old |
| + holder renames back to labels and a new one forward to ids. Each flip is |
| + one upsert-and-delete per record with a 60-second TTL, bounded by the |
| + roll. Rollback is the same flip once, in reverse, by the old release's |
| + converger, which sees the id-identified records as managed and unwanted; |
| + for a single-box deployment the old converger's shrink guard refuses |
| + that, and the records simply stay under id identifiers, still resolving, |
| + until an operator deletes them or the box rolls forward again. |
| +- **The identity sweep** (codex 2, 3). The status lease holder runs it on |
| + its publish tick, and `carlos fleet sweep` runs the same code from an |
| + operator's machine with bucket credentials, for the case where no box is |
| + on the new release (§6). Every delete is `DeleteIfMatch` on the etag |
| + observed in that pass. Rules: a snapshot stale for `retiredAfter` (7 |
| + days) is deleted with the edges file that matches it under §3.3 (by id, |
| + else unambiguous label); a snapshot that Step A marked superseded is |
| + deleted at once, in either direction (legacy superseded by id, or id |
| + superseded by legacy after a rollback), with the edges file Step B |
| + marked non-effective for the same box; an edges file with no snapshot at |
| + all is reported, never deleted automatically, because nothing can date |
| + it. `carlos fleet retire <id|label>` deletes both objects by hand after |
| + printing what it found. The deployment bucket is versioned, which makes |
| + a wrong delete recoverable but is not a substitute for the condition. |
| + |
| +### 3.6 Step 2: the other per-box keys |
| + |
| +`boxIdentity().ID` replaces the label in every key of §2.1's second table, |
| +with the label carried inside the object as `box` is today |
| +(`RawChunk.Box`, `AlertState.Box`, `snapshotMeta.Box`, `Heartbeat.Box`, |
| +`ConfigAck.Box`). It is a separate plan because each key has readers with |
| +their own display and migration questions (codex 8): |
| + |
| +- **Registry snapshots** go first: a collision there hands the DR restore |
| + another box's registry. The restore reads the metadata's `Box` for the |
| + label and the key for the id. Legacy per-label keys are abandoned, not |
| + rewritten, the way the pre-multi-region fixed keys were |
| + (`snapshot_loop.go:31`). |
| +- `config-ack`, `relay-ack`, `metrics`, `alerts`, `relay` meter: readers |
| + take the key segment as opaque and display the label from the body. |
| + `metricsview.go:547` shows the key today and must switch to the body. |
| +- `apps/<sqid>/<app>/{logs,errors,analytics}/<box>/`: the segment is |
| + readable by the account's own fleet credentials (`storedoor.go:403`) and |
| + by `carlos logs`, so an id there is **member-visible**, and the cursor |
| + orders by it (`cursor.go:24`). Three identities need separating: the |
| + storage key, the display label, and what a member may see. Step 2 is |
| + gated on Paul's ruling (§10, question 3); until then those keys keep the |
| + label and its collision, which is inside one account's own data |
| + (`storedoor.go:482-487` records the existing limit). |
| + |
| +### 3.7 Fleet boxes |
| A fleet box's label is operator-chosen and unique within its fleet |
| (`registry/fleets.go:26` `ErrFleetBoxActive`). Its heartbeat lands under |
| `fleets/<name>/status/boxes/<label>.json` and the console joins it by |
| `fleet_boxes.label` (`fleets.go:1012`). Fleet boxes do not run the edges |
| -loop. They are not in #212 and they do not change: `boxIdentity` returns |
| -the fleet label as both ID and Label, `box_id` is omitted, and the key |
| -stays the label. |
| +loop or steering. They are not in #212 and do not change: `boxIdentity` |
| +returns the fleet label as both fields, `box_id` is omitted, the key stays |
| +the label. |
| ## 4. Migration with mixed versions during a roll |
| -The roll order does not matter. Each pair: |
| +Roll order does not matter. Each pair: |
| -| Writer | Reader | Join | |
| +| Writer | Reader | What happens | |
| |---|---|---| |
| -| old box (label key, no `box_id`) | new console or edge | `BoxKey` falls back to label. As today | |
| -| new box (id key, both fields) | old console or edge | lists the prefix, parses `label`/`box`, ignores the unknown field. As today | |
| +| old box (label key, no `box_id`) | new console, edge, housekeeper | Step C rule 2: label match, unambiguous. As today | |
| +| new box (id key, both fields) | old console, edge, housekeeper | lists the prefix, parses `label`/`box`, ignores the unknown field. Steering on an old housekeeper keys by label. As today | |
| | new box | new reader | by id | |
| -| colliding old pair | any | still collides, until both roll; reported once one has | |
| -| colliding pair, one rolled | new reader | the rolled box has its own key; the legacy key is the old sibling's and is left alone (§3.5 guard); reported as a label with two ids while the legacy key exists | |
| +| colliding pair, both old | any | still collides until one rolls | |
| +| colliding pair, one rolled | new reader | the rolled box has its own key and joins by id. The legacy objects are the sibling's: its `instance_id`/`eip` differ, so the guard leaves them and Step A marks the label contested. Eligibility is incomplete and the `identity` alert fires until the sibling rolls | |
| +| colliding pair, one rolled | old reader | the legacy objects are now the sibling's alone, so the old reader sees two boxes correctly for the first time | |
| +| one box, first minutes after roll | new reader | id snapshot and legacy snapshot share `instance_id`: superseded (A2). Legacy file and id file share `eip`: the newer is effective (B2). Then the guard deletes both legacy objects | |
| -One release of readers that accept both shapes is this release. The |
| -fallback to label stays for at least one more release after it, and is |
| -removed only when the sweep reports no legacy key in production. |
| +One release of readers accepting both shapes is this release. The label |
| +path (A2, B2, C2) stays for at least one more release and is removed only |
| +when the sweep reports no legacy key in production. |
| Nothing customer-visible changes. The public label is the same string. |
| fleet.json's shape is unchanged. The instances API's `served_by.box` keeps |
| -the label. Alert mails name the label. |
| +the label. Alert mails name the label. A DNS record's `SetIdentifier` is |
| +not visible to a resolver. |
| ## 5. The host lifecycle claim |
| `internal/hostclaim` is not on main at `04cd2d78`; it is on |
| `feat/hostclaim-library`, and the spec is on `design-replica-ownership`. |
| The claim names boxes in `holder.writer_box`, `op.exec`, |
| -`op.participants`, `touched.boxes`, `obligation.owner = "box:<b>"`, |
| -`admissions[].box`, and keys execution leases at |
| -`control/hosts/<host>/exec/<box>.json`. The spec's example is |
| -`"writer_box": "adi-knitting"`: a label. On the library branch |
| -`configAckFunc` already stamps `BoxLabel()` into config acks |
| -(`agent.go:942` there). Plan 3's box guard checks `c.Op.Exec != in.Box`. |
| +`op.participants`, `touched.boxes`, the `inventory` map's keys, `lost`, |
| +`obligations[].box` and `.owner = "box:<b>"`, `admissions[].box`, |
| +`writes_lost.box` (`claim.go:58-134` on that branch), and keys execution |
| +leases at `control/hosts/<host>/exec/<box>.json`. The spec's example is |
| +`"writer_box": "adi-knitting"`: a label. `generation.go:31` enforces exact |
| +executor equality, `c.Op.Exec != in.Box`. |
| A label collision there is worse than here: a sibling with the same label |
| passes "am I the executor", can renew or take the other's execution lease, |
| -and can complete or park the other's obligations. Ruling proposed: the |
| -claim and every later plan name a box by `boxIdentity().ID`, never by the |
| -label, before any claim-driven operation is armed. The claim carries no |
| -display name; a person reads the label through the fleet page. Step 1 of |
| -this spec lands first so the identity exists when the claim's plans build |
| -on it. No other label-keyed structure was found in the claim spec. |
| +and can complete or park the other's obligations. Ruling proposed (codex 9 |
| +widened it): |
| + |
| +- every box-naming field and key in the claim uses `boxIdentity().ID`, |
| + never the label; the claim carries no display name, a person reads the |
| + label through the fleet page; |
| +- a box whose id is empty (§3.1) refuses to arm any claim-driven work, the |
| + same way `resolveFleetIdentity` refuses an invalid fleet identity; |
| +- the claim relies on machine-id uniqueness, which §3.1 states, the image |
| + recipe enforces, and the twin check detects but cannot prevent. The |
| + claim's own fencing (execution lease, self-fence) is what bounds a twin's |
| + damage, not the id; |
| +- step 1 of this spec lands first so the identity exists when the claim's |
| + plans build on it. |
| ## 6. Rollback safety |
| A box rolled back below this release writes `status/boxes/<label>.json` |
| -and `control/edges/<label>.json` again. Its id-keyed objects remain. |
| - |
| -- A new console dedupes them by `instance_id` (§3.3 rule 2): the legacy |
| - object is fresher, the id-keyed one is superseded, counted for nothing, |
| - and deleted by the sweep on the next publish tick. |
| -- An old console shows the id-keyed snapshot as a stale second row for |
| - that label until the sweep (run by whichever box holds the status lease; |
| - if every box rolled back, 7 days, or by hand). Its instance counts are |
| - included in the public total while it is there. That is the one visible |
| - cost of rollback, bounded by the lease holder's tick when any box is on |
| - the new release. |
| -- The rolled-back box's edges file is readable by every reader: `box`, |
| - `eip`, `hosts` are unchanged. The id-keyed edges file lingers, carrying |
| - the same EIP and hosts; the home map, steering and `EdgeIPs` fold it the |
| - same way they fold the legacy one, with the same content, so nothing is |
| - wrong, only duplicated. The sweep removes it with the superseded snapshot. |
| -- The rolled-back box's own gate (`seedEdgesFileGate`) finds no legacy |
| - file if the forward run deleted it, and republishes on its first tick. |
| - |
| -Nothing on the new release writes a shape the old release cannot parse. |
| - |
| -## 7. Test plan |
| - |
| -Unit, in `internal/status`: |
| - |
| -- `BoxID` vectors: fixed digests; the known pair `...18` and `...3c` share a |
| - label and get different ids; `LabelForID(BoxID(m)) == Label(m)` for a |
| - table of machine ids; `BoxID("") == ""`. |
| -- `WriteBox` keys by id when set, by label when not; the body carries both. |
| -- `loadBoxesReport`: two keys one id (duplicate, incomplete); one label two |
| - ids with a legacy key (reported, incomplete); one label, old and new key, |
| - same instance id (superseded, fresher wins, complete); same with |
| - different instance ids (both stand); old-only and new-only fleets |
| - unchanged against the existing goldens. |
| -- `BuildFleet` with a superseded object does not double count. |
| -- `dryrun`: a needle for the id; it must appear in `status/boxes/*.json` |
| - and never in fleet.json or index.html. |
| - |
| -Unit, in `internal/host` and `internal/edgesfile`: |
| - |
| -- `edgesFilePass` writes the id key; the gate includes `box_id`; the |
| - tenth-tick bucket verify republishes a missing or foreign body. |
| -- Legacy cleanup: deletes only when instance id (status) or EIP (edges) |
| - match; leaves a sibling's object; runs once per start. |
| -- `EdgeIPs` and `DeploymentRoutes` join matrix: old/new snapshot x old/new |
| - file, plus a file whose label does not match its id (not live). |
| -- `moveTargets` with two files sharing a label and different ids. |
| - |
| -Unit, in `internal/console` and `internal/edge`: |
| - |
| -- `readLiveEdgesSnapshot` join matrix as above; the duplicate and |
| - two-ids conditions set `BoxesComplete=false` and log once. |
| -- `foldHomes` self-skip by id, and two same-label files both fold. |
| -- `buildBoxCards` gives two cards for two boxes with one label, titles |
| - disambiguated by id prefix; archregion seeds by key. |
| -- The alert watcher mails on the `identity` rule and clears it. |
| +and `control/edges/<label>.json` again. Its id-keyed objects remain, and |
| +after a while may disagree with the legacy ones (codex 3). The protocol: |
| + |
| +1. Roll the box back. Its first status tick writes the legacy snapshot |
| + (unconditional, `statusTick`). Its first edges tick finds no legacy |
| + file (the forward run deleted it) and publishes one, newer than the |
| + id-keyed file; or finds the undeleted legacy file, seeds its gate from |
| + it, and leaves it, in which case the legacy file is the older one until |
| + a route changes. |
| +2. On any box still on the new release, the next publish tick's sweep |
| + marks the id-keyed snapshot superseded (A2, same `instance_id`, legacy |
| + fresher) and the id-keyed file non-effective (B2, same `eip`) or, in |
| + the seeded-gate case above, keeps the id file effective until the |
| + legacy one is rewritten. Both id-keyed objects are then deleted |
| + conditionally. |
| +3. If every box rolled back, nothing on a box runs the sweep. The rollback |
| + runbook step is `carlos fleet sweep` from the operator's machine, with |
| + the new CLI and bucket credentials, once after the last box. Until it |
| + runs, an old console shows the id-keyed snapshot as a stale second row |
| + and `sumInstances` counts it; an old home map and steering see two |
| + files with one EIP and fold both, and on a host conflict pick by sorted |
| + key, which is wrong only if the two files disagree, which the sweep |
| + step bounds to minutes. |
| +4. An off-cloud box (no `instance_id`) cannot be deduped by A2. Its stale |
| + id-keyed snapshot stays until `retiredAfter` or `carlos fleet retire`. |
| + Its edges file is still deduped by B2 (EIP). This is the one case §6 |
| + does not make automatic, and the runbook says so. |
| + |
| +The rolled-back box's objects are readable by every reader: `label`, |
| +`box`, `eip`, `hosts` are unchanged. Nothing on the new release writes a |
| +shape the old release cannot parse. Steering on rollback is §3.5. |
| + |
| +## 7. Rollback and roll: the invariant |
| + |
| +Across every state in §4 and §6: a live box is counted for at most one |
| +row, its EIP is in the edge set, and its hosts fold from exactly one file; |
| +a retired or superseded object counts for nothing; and anything the |
| +readers cannot prove makes automatic eligibility incomplete rather than |
| +wrong. §8's simulation asserts exactly this. |
| + |
| +## 8. Test plan |
| + |
| +Unit, `internal/status`: |
| + |
| +- `BoxID` vectors; the known pair `...18` and `...3c` share a label and |
| + get different ids; `LabelForID(BoxID(m)) == Label(m)` over a table; |
| + `BoxID("") == ""`. |
| +- `WriteBox` keys by id when set, by label when not; the body carries both; |
| + the twin check fires on a fresh object with a different non-empty |
| + `instance_id` and not on an empty one. |
| +- `loadBoxesReport`: duplicate (two keys, one id); superseded (one label, |
| + old and new key, same instance id, fresher kept, dropped from the slice); |
| + contested (same, different or empty instance ids, both kept); old-only |
| + and new-only fleets byte-identical to the existing goldens. |
| +- `BuildFleet` with a superseded object does not double count; the |
| + `identity` alert object is written every tick, firing and clear. |
| +- `dryrun`: a needle for the id, present in `status/boxes/*.json`, absent |
| + from fleet.json and index.html. |
| +- `AddFleetBox` refuses a 32-hex label. |
| + |
| +Unit, `internal/edgesfile`, `internal/host`: |
| + |
| +- `Effective`: duplicate id; legacy-plus-id same EIP, newer generation |
| + wins in both directions; different EIPs both kept and contested; a |
| + mismatched `box`/`box_id` flagged. |
| +- `edgesFilePass` writes the id key; `box_id` is in the gate; the |
| + tenth-tick verify republishes a missing or foreign body. |
| +- Legacy cleanup: deletes only on non-empty matching evidence; refuses |
| + when the etag changed between read and delete (a fake |
| + `ConditionalDeleter`); refuses on a store without conditional delete; |
| + runs once per start. |
| +- `EdgeIPs`, `DeploymentRoutes`, `moveTargets`: the §4 matrix, plus the |
| + ambiguous-label case (two snapshots, one legacy file: no match). |
| +- Steering: a rename is one upsert-and-delete batch, exempt from the |
| + shrink count; a single-box flip converges; `dedupeRecords` keys by id. |
| + |
| +Unit, `internal/console`, `internal/edge`: |
| + |
| +- `readLiveEdgesSnapshot`: the §4 matrix; duplicate, contested and |
| + flagged each set `BoxesComplete=false` and log once. |
| +- `foldHomes`: self-skip by id; two same-label files with different ids |
| + both fold; a non-effective file does not. |
| +- `buildBoxCards`: two boxes with one label give two cards from |
| + `HostRows`; the `FleetBox` fallback gives one ambiguous card; `targetOf` |
| + and `claimed` resolve to `"box:"+id`. |
| +- The alert watcher reads `alerts/fleet/identity.json`, mails on a |
| + transition, clears on an all-clear or a stale object. |
| Mixed-version goldens: a committed legacy-shape object for each prefix, |
| -parsed by every new reader; and a committed new-shape object asserted to |
| -keep `label`, `box`, `v`, `eip`, `hosts`, `generation` at the same JSON |
| -field names, which is what an old reader needs. |
| +parsed by every new reader; a committed new-shape object asserted to keep |
| +`label`, `box`, `v`, `eip`, `hosts`, `generation` under the same JSON |
| +names, which is all an old reader needs. |
| -Sweep: against a DirStore, retired after 7 days stale, superseded at once, |
| -orphan edges file reported and kept, `carlos fleet retire` removes both. |
| +Sweep: against a DirStore with a fake conditional delete: retired after |
| +7 days, superseded at once in both directions, orphan edges file reported |
| +and kept, `carlos fleet sweep` and `retire` paths. |
| -The simulation habit from the moves work applies to §3.3's dedupe: a |
| -table-driven run over random fleets with injected collisions, rolls and |
| -rollbacks, asserting that a live box is never counted for nothing and a |
| -retired box never counts for eligibility. |
| +Simulation (the moves work's habit): random fleets with injected |
| +collisions, rolls, rollbacks, off-cloud boxes and sweep timing, over |
| +thousands of seeds, asserting §7's invariant at every step. |
| -## 8. Rollout |
| +## 9. Rollout |
| -1. Land step 1 on main; cut a release. |
| -2. Roll the boxes in any order (the platform roll recipe). After each box, |
| +1. Confirm the flagship and Tito image recipes clear `/etc/machine-id` |
| + (§3.1). If not, fix that first; it is a prerequisite, not a step. |
| +2. Land step 1 on main; cut a release. |
| +3. Roll the boxes in any order (the platform roll recipe). After each box, |
| `carlos status -live` shows its id and the fleet page shows one row per |
| - box with no `duplicate` chip. |
| -3. Watch the lease holder's log for the identity sweep's first pass: the |
| - legacy keys of rolled boxes disappear, nothing else does. |
| -4. Roll the console. Nearest-region eligibility keeps reading, now by id. |
| -5. After every box is on the release and the sweep reports no legacy key, |
| - file the step-2 plan (metrics, alerts, relay, logs, usage) and the claim |
| - ruling in §5 on the claim branch. |
| -6. One release later, remove the label fallback from the join (§4). |
| + box, no `duplicate` or `contested` chip once its sibling, if any, has |
| + rolled. |
| +4. Watch the lease holder's log for the sweep's first pass: the rolled |
| + boxes' legacy keys disappear, nothing else does. Watch the steering log |
| + for renames, not shrink refusals. |
| +5. Roll the console. Nearest-region eligibility keeps reading, now by id. |
| +6. After every box is on the release and the sweep reports no legacy key, |
| + file the step-2 plan (registry snapshots first) and record the §5 |
| + ruling on the claim branch. |
| +7. One release later, remove the label path from §3.3. |
| -## 9. Open questions for Paul |
| +Rollback is §6; its runbook step is `carlos fleet sweep`. |
| + |
| +## 10. Open questions for Paul |
| 1. Should fleet.json gain a short, non-reversible disambiguator (the first |
| four hex characters of the id) so two public rows with one label can be |
| told apart, or is two identical rows the honest public answer? This |
| spec keeps fleet.json label-only. |
| -2. `retiredAfter` is 7 days. Shorter makes a box that is down for a long |
| +2. `retiredAfter` is 7 days. Shorter makes a box that is down over a long |
| weekend lose its edges file and its steered records; longer leaves |
| stale rows. Is 7 right for the flagship? |
| -3. Step 2 changes the segment under `apps/<sqid>/<app>/logs/<box>/` from a |
| - label to an id. `carlos logs` shows it today. Is that acceptable for |
| - members, or should the logs API map it back to the label? |
| -4. The claim ruling in §5 is proposed here and belongs on the claim branch. |
| - Do you want it recorded there now, or when step 1 has landed? |
| +3. Step 2's `apps/` keys: is a box id in a member-visible key acceptable |
| + (bucket-private to the account, 128 bits of a digest of the platform's |
| + own machine id), or should those keys use a per-account pseudonym? Step |
| + 2 is gated on this. |
| +4. The §5 ruling belongs on the claim branch. Record it there now, or when |
| + step 1 has landed? |
| + |
| +## 11. Review dispositions |
| + |
| +| Round, finding | Severity | Disposition | |
| +|---|---|---| |
| +| R1-1 steering keys DNS by label | P1 | accepted. `SetIdentifier` becomes `BoxKey`; the converger learns renames exempt from the shrink count; flip-flop under mixed housekeepers bounded and described (§3.2, §3.5, §4) | |
| +| R1-2 cleanup can delete a live sibling's object | P1 | accepted. `DeleteIfMatch` on the observed etag, non-empty evidence only, never a plain delete, for the guard and the sweep (§3.5) | |
| +| R1-3 rollback duplicates not equivalent; key order load-bearing; no sweeper if all roll back | P1 | accepted. `edgesfile.Effective` selects one file per box for every reader; supersession in both directions; `carlos fleet sweep` as the runbook step; off-cloud caveat stated (§3.3 B, §3.5, §6) | |
| +| R1-4 mixed-shape join does not fall back as promised | P1 | accepted. One explicit three-step algorithm; label fallback only when unambiguous; id-bearing files never fall back (§3.3) | |
| +| R1-5 registry snapshots, config-ack, relay-ack missing | P2 | accepted. Added to §2.1 with readers; registry snapshots lead step 2 (§3.6) | |
| +| R1-6 architecture page join not implementable | P2 | accepted. Cards from `HostRows` keyed by id; `FleetBox` fallback marked ambiguous; `claimed`/`targetOf` by id (§3.3) | |
| +| R1-7 identity alert competes with the metrics loop | P2 | accepted. Separate `alerts/fleet/identity.json`, written every tick by the lease holder, stale means clear; delivery limits named (§3.4) | |
| +| R1-8 step 2 contradicts bucket-private | P2 | accepted. Store-door read scope corrected; step 2 split and gated on a ruling; `metricsview` and the cursor named (§2.4, §3.6, §10) | |
| +| R1-9 claim ruling incomplete; clones overwrite | P2 | accepted. Every box-naming field listed; empty id refuses to arm; uniqueness assumed, enforced by the image recipe, detected by the twin check (§3.1, §5, §9) | |
| +| R1-10 store-door readability, `ValidTunnelName`, 11% | P3 | accepted. All three corrected (§1, §2.4, §3.1) | |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-BOX-IDENTITY-R1.md b/docs/superpowers/specs/CODEX-FINDINGS-BOX-IDENTITY-R1.md| new file mode 100644 |
| index 000000000..3d97b6cc5 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-BOX-IDENTITY-R1.md |
| @@ -0,0 +1,25 @@ |
| +Reviewed `71112c9e` against code at `origin/main` (`04cd2d78`), plus both claim branches via `git show`. No files changed. |
| + |
| +1. **P1 — Steering still collapses colliding boxes.** §2.3 says steering folds by EIP and hosts, “not by box.” Actually, `desiredState` copies `e.Box`, `dedupeRecords` drops duplicate `(Name, Box)` pairs, and Route53 uses `Box` as `SetIdentifier` ([steering.go:371](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/steering/steering.go:371), [steering.go:318](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/steering/steering.go:318), [route53.go:601](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/steering/route53.go:601)). Two new files retaining one label still lose one edge. **Fix:** migrate steering identity too, explicitly handling existing DNS identifiers and old housekeepers during rollout and rollback. |
| + |
| +2. **P1 — Cleanup can delete a live sibling’s object.** §3.5 checks ownership, then deletes by key. Between those operations, an old sibling can overwrite that key. The status guard also permits empty instance IDs unless explicitly prohibited; empty is a supported off-cloud state ([snapshot.go:229](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/status/snapshot.go:229)). Losing the sibling’s edges file can persist indefinitely because its old writer checks only memory ([edgesfile.go:491](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/host/edgesfile.go:491)). **Fix:** require positive, non-empty ownership evidence and `release.DeleteIfMatch` against the observed revision, already available at [conditional_delete.go:28](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/release/conditional_delete.go:28). Apply this to cleanup and sweeping; bucket versioning does not prevent the outage. |
| + |
| +3. **P1 — Rollback duplicates are not necessarily equivalent or bounded.** §6 assumes lingering edges files retain identical EIP/hosts. After rollback, a route, replica flag, capability, or address can change only in the legacy file. `foldHomes` retains old-only hosts and chooses conflicting targets by sorted **object keys** ([homemap.go:538](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/edge/homemap.go:538)); steering retains whichever duplicate sorts first. Thus the claim that key names are never load-bearing is too strong. If every box rolls back, no new sweeper exists: stale snapshots remain counted indefinitely ([fleet.go:272](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/status/fleet.go:272)), not seven days. **Fix:** specify an executable rollback cleanup protocol and effective-file selection for all edges consumers. Correct §3.5’s legacy-only supersession cleanup to cover superseded ID objects too. |
| + |
| +4. **P1 — The stated mixed-shape join does not implement the promised fallback.** §3.2 promises label fallback when either side is old; §3.3 indexes an ID-bearing snapshot only by ID and looks up an ID-bearing edge only by ID. Consequently, neither old-status/new-edge nor new-status/old-edge joins. These states occur during independent publication and rollback. Missing joins remove addresses/routes ([fold.go:84](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/edgesfile/fold.go:84)) and make eligibility incomplete ([instancesplacement.go:138](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/console/instancesplacement.go:138)). **Fix:** define one explicit matching and supersession algorithm, permitting fallback only with unambiguous evidence. Preserve superseded identity aliases while reconciling the edges pair. Include missing-instance-ID cases: §3.3 deliberately cannot dedupe them, so §6’s unconditional rollback promise fails. |
| + |
| +5. **P2 — The consumer inventory misses three key families.** Registry backups use `metricsBoxLabel` and overwrite `control/registry-snapshot/<box>.{db,json}` plus generation objects ([snapshot_loop.go:34](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/host/snapshot_loop.go:34), [snapshot_loop.go:205](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/host/snapshot_loop.go:205)). Config acknowledgements already use `BoxLabel` on main and key `control/config-ack/<box>/<host>.json` ([agent.go:954](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/host/agent.go:954), [configack.go:116](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/appconfig/configack.go:116)). Relay heartbeats key `control/relay-ack/<box>.json` ([authority.go:71](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/relay/authority.go:71)). **Fix:** add their writers, readers, restore procedures, and explicit migration scope. Registry backup collisions are particularly consequential. |
| + |
| +6. **P2 — The architecture-page change lacks an implementable join.** §3.3 says to key cards by `BoxKey`, but half the inputs are `FleetBox`, intentionally lacking IDs. Current code unions public fleet rows and private host rows by label ([archgraph.go:561](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/console/archgraph.go:561)). Merely changing private keys creates unmatched or duplicate cards. Machine-to-node links also retain labels at [archgraph.go:271](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/console/archgraph.go:271) and [archgraph.go:993](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/console/archgraph.go:993). **Fix:** build identified cards from private snapshots and define an explicitly ambiguous public-only fallback; migrate every node reference. |
| + |
| +7. **P2 — Identity alerts introduce competing writers and unreliable clearing.** §3.4 writes into the lease holder’s existing alert object. The metrics loop holds separate local state and replaces that entire object on transitions ([metrics_loop.go:496](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/host/metrics_loop.go:496)), erasing independently added rules. Lease transfer can strand a firing rule on the previous holder. The watcher accepts unknown rule names, but mail requires configured delivery and operator grants; transitions without recipients are dropped ([alertwatch.go:103](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/console/alertwatch.go:103), [alertwatch.go:187](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/console/alertwatch.go:187)). **Fix:** give fleet identity alerts a stable, singly owned object with takeover/clear semantics, and specify delivery through the existing operator sinks. |
| + |
| +8. **P2 — Step 2 contradicts “the ID stays bucket-private.”** Customer fleet credentials can read their account’s entire `apps/` subtree ([storedoor.go:403](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/edge/storedoor.go:403)), exposing IDs placed in log/error/analytics keys even if bodies retain labels. Also, metrics views display the key-derived box value, not `RawChunk.Box` ([metricsview.go:547](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/console/metricsview.go:547)). Logs use `Box` for cursor ordering, so preserving a colliding display label is not sufficient identity separation ([cursor.go:24](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/applog/cursor.go:24)). **Fix:** separate storage identity, display label, and member-visible identity; settle the privacy boundary before specifying step 2. |
| + |
| +9. **P2 — The hostclaim ruling is necessary but incomplete.** Exact executor equality is already enforced in `feat/hostclaim-library:internal/hostclaim/generation.go:31`, so replacing labels is correct. However, the inventory also includes `inventory` keys, `lost`, `Obligation.Box`, and `WritesLost.Box` (`claim.go:132`, `134`, `114`, `76` on that branch). More fundamentally, cloned machine IDs produce the **same object key**: they overwrite rather than produce the two objects §3.4 expects to detect. **Fix:** extend the ruling to every authority-bearing box reference, require non-empty identity, and state/enforce the machine-ID uniqueness assumption before claiming clone detection or safe claim admission. |
| + |
| +10. **P3 — Correct two factual details.** `control/edges/` is **not** readable through the store door: only specific system-update shapes are allowed under `control/` ([storedoor.go:408](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/edge/storedoor.go:408)). Also, a 32-character lowercase hex string satisfies `ValidTunnelName` ([registry.go:337](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/registry/registry.go:337)); it is not syntactically disjoint from fleet labels. The twenty-box collision probability is approximately **11.24%**, not 6%. |
| + |
| +The core choices remain sound: avoid legacy dual-writes; retain labels in public projections; use a 128-bit digest for collision resistance. `LabelForID` reproduces the existing digest-derived label and adds no machine-ID disclosure by itself ([snapshot.go:523](/home/paulca/amadan.net/carlos/platform-design-box-identity/internal/status/snapshot.go:523)). Step 1’s explicit `FleetBox` projection, label-only instances response, and alert display fields can preserve privacy; key compatibility alone does not establish migration safety. |
| + |
| +**Verdict: not ready. P1s: steering identity, unsafe deletion, rollback reconciliation, and mixed-shape joins.** |
| \ No newline at end of file |