amadan
spec: instance sizes and egress allows, rev 5 (fleet addresses: grant-time check plus report, Paul's ruling)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+82
−81
docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md+82 −81
diff --git a/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md b/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md| index 528be53ac..00c4fe4a9 100644 |
| --- a/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md |
| +++ b/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md |
| @@ -1,9 +1,9 @@ |
| # Instance sizes and egress allows |
| Amadan #328 (an isolated instance reaching a private service its operator |
| -runs) and #329 (operator-set CPU and memory per instance). Revision 4, |
| -2026-10-21. Revisions 2 to 4 answer the builder check and Astra rounds |
| -1 to 3 (`CODEX-FINDINGS-instance-sizes-egress-R<n>.md`); the |
| +runs) and #329 (operator-set CPU and memory per instance). Revision 5, |
| +2026-10-21. Revisions 2 to 5 answer the builder check, Astra rounds |
| +1 to 4, and Paul's ruling on fleet addresses (`CODEX-FINDINGS-instance-sizes-egress-R<n>.md`); the |
| dispositions are §9. |
| Both are the same mechanism: **an operator-set policy for an app and its |
| @@ -32,6 +32,9 @@ written by root onto each instance's own slice** |
| that is my call, §8.) |
| 7. **Provider-neutral.** Nothing here names a cloud; a self-hosted box |
| with MinIO and plain hosts gets both features. |
| +8. **Fleet box addresses are checked when an allow is granted and |
| + reported afterwards, not enforced continuously by root** (2026-10-21, |
| + §4.2). |
| ## 1. Why |
| @@ -158,70 +161,48 @@ The deny-list ranges come from one table shared with the template test |
| (`internal/host/units_hardening_test.go`), so a range added to the fence |
| is automatically one an allow must fall inside. |
| -### 4.2 Fleet addresses |
| - |
| -Opening another box's address opens every listener on it, including the |
| -private paths boxes use to reach each other. The edges file's |
| -`private_ip` cannot answer this: it holds one IPv4 address and refuses |
| -IPv6 (`internal/edgesfile/private.go:15`) (Astra R1-5). |
| - |
| -- Each box publishes **every** non-loopback address it holds (from the |
| - same procfs sources as `localPrefixes`) in a new edges-file field, |
| - `addrs`. It is part of the write gate (`edgesfile.go:665`), so a change |
| - of address rewrites the file on the next pass. This works with region |
| - HA off: no freshness or timed refresh is relied on, because an address |
| - list does not go stale the way a headroom reading does (Astra R2-2). |
| -- **Age is not evidence of absence.** A file is never ignored for being |
| - old: a dead or replaced box's last file keeps its addresses excluded |
| - until an operator removes the file. That is the conservative direction; |
| - the refusal names the box so the operator knows which file to retire. |
| -- **Inventory completeness** is its own check, independent of liveness |
| - and of the status listing's 24-hour horizon (Astra R3-2; |
| - `BoxesComplete` is a liveness join and does not give it): list |
| - `control/edges/` completely, read and parse every file, and every file |
| - carries `addrs`. Any listing error, unreadable file, or file without |
| - `addrs` (an older box) makes the inventory incomplete. A retired box's |
| - retained file is simply one more readable file; it blocks nothing but |
| - its own addresses. |
| -- **Console:** refuses a grant naming any address in the inventory, and |
| - refuses any grant while the inventory is incomplete, naming the file |
| - that made it so. |
| -- **Root on each box:** before every write (§4.3), drops any entry that |
| - is one of its own addresses (`isLocal`) or appears in the inventory, |
| - and reports the drop (§4.4). **If the inventory is incomplete, root |
| - clears the app's effective allows** (writes the reset) and reports |
| - `inventory incomplete` until a complete read returns, then writes them |
| - again (Astra R3-1, R1-5). A past complete read is not evidence about |
| - now. Its own-address check needs no inventory and always applies. The |
| - cost is stated: a store outage closes granted addresses for its |
| - duration, within one 30-second loop each way. |
| -- **Transport** (Astra R3-3): a box with direct store access reads the |
| - policy object and `control/edges/` itself. A bucketless fleet box reads |
| - through the console's store door (`openFleetDoorStore`, |
| - `internal/host/agent.go:4366`), whose read ACL |
| - (`internal/edge/storedoor.go:431`) grants neither today, and whose |
| - client reads a refusal as an absent object (`fleetapi.go:127`), which |
| - would silently mean "no policy". So: |
| - - the door's `aclRead` grants `control/app-policy/<sqid>/<app>.json` |
| - for the box's own account scope only, read-only (`aclWrite` never |
| - answers true for it); |
| - - the door serves the inventory as one console-computed object, |
| - `control/fleet-addrs.json` (`{"complete": bool, "addrs": [...], |
| - "generation": ...}`), rewritten by the console whenever an edges |
| - file changes, rather than granting fleet boxes a listing of |
| - `control/edges/`. Root fails closed on `complete: false` or on any |
| - read error; |
| - - a fleet-door box tells "refused" from "absent" for these two keys: a |
| - refusal is reported as `policy unreadable`, and root treats it as |
| - incomplete (allows cleared, sizes at the build's tier), never as |
| - "no policy". |
| - Tests prove delivery through the door and that both keys stay |
| - unwritable from a box. |
| -- **Residual, stated:** the inventory is published by each box's agent. |
| - Under claims a compromised agent could leave its own box's addresses |
| - out, so that other boxes' tenants could reach it if an operator also |
| - granted exactly that address. Root on the target box is not exposed by |
| - this: the fence only governs what tenants dial out to. |
| +### 4.2 Fleet addresses: checked at the grant, reported after |
| + |
| +**Ruling (Paul, 2026-10-21): a grant-time check plus a report, not |
| +continuous enforcement.** Revisions 2 to 4 had root enforce a fleet-wide |
| +address inventory continuously; each round found another way the |
| +inventory could look complete when it was not (fleet-door boxes that |
| +publish nothing, a first publication that failed, a cached verdict with |
| +no expiry), and root failing closed on it would close every allow during |
| +a store outage. What the rail protects is narrow: box-to-box hops are |
| +authenticated with the mesh key (`internal/edge/mesh.go`, |
| +`forwardproof.go`), so a tenant that reaches another box's private |
| +address gets what the internet gets, the edge, which refuses an |
| +unsigned hop. So: |
| + |
| +- **Each box publishes its addresses.** Every non-loopback address it |
| + holds (from the same procfs sources as `localPrefixes`), in a new |
| + edges-file field `addrs`, part of the write gate (`edgesfile.go:665`) |
| + so a change of address rewrites the file. No freshness or timed |
| + refresh is relied on, so it works with region HA off. |
| +- **The grant is checked** at the console door, against every readable |
| + file under `control/edges/` regardless of age (a retired box's |
| + retained file keeps excluding its addresses until an operator removes |
| + it). The door refuses an address any file lists, naming the box. It |
| + also refuses while any listed file is unreadable or lacks `addrs` (an |
| + older box). Boxes that publish no edges file at all (bucketless fleet |
| + boxes, `edgesfile.go:803`) cannot be checked; the grant succeeds and |
| + its output names them as `not checked`. |
| +- **Root checks only its own box.** Before every write (§4.3) root drops |
| + any entry that is one of its own addresses (`isLocal`) and reports the |
| + drop (§4.4). It reads no fleet inventory, so a store outage never |
| + closes an allow. |
| +- **The report.** The console re-runs the grant check for every app's |
| + allows on a timer (every 15 minutes) and whenever an edges file |
| + changes. An allowed address that now appears in a box's `addrs` raises |
| + an alert through the existing alert path (`internal/alerting`, as the |
| + no-capacity alert does in `nocapacity.go`), rate-limited per app and |
| + address, and marks the address `!` in `carlos apps show` and the |
| + `EGRESS` column. The allow stays in force: the operator decides. |
| +- **Residual, stated:** an operator-allowed address that later lands on |
| + a fleet box is reachable from that app's instances until the operator |
| + acts on the alert; what it reaches is what the internet reaches. Boxes |
| + that publish nothing are never checked. |
| ### 4.3 How it reaches the instance, and who decides |
| @@ -240,12 +221,23 @@ may run (`validateStagedExe`): |
| (`instanceRegistryHostApp`), so the agent already decides which build |
| an isolated instance runs. The agent converges the policy into a |
| registry column on each of the app's route rows; root reads that |
| - column and validates it, and reads the inventory verdict (§4.2) the |
| - agent converges beside it. This adds no trust the agent does not |
| + column and validates it. This adds no trust the agent does not |
| already hold. **One extra rail:** a box whose layout is not wholly nested (any |
| flat exec child, which runs as the agent's user and may be able to |
| write `registry.db`) gets no allow written at all, and says so. |
| +**Transport.** A box with direct store access reads the policy object |
| +itself. A bucketless fleet box reads through the console's store door |
| +(`openFleetDoorStore`, `internal/host/agent.go:4366`), whose read ACL |
| +(`internal/edge/storedoor.go:431`) does not grant it today, and whose |
| +client reads a refusal as an absent object (`fleetapi.go:127`), which |
| +would silently mean "no policy" (Astra R3-3). So the door's `aclRead` |
| +grants `control/app-policy/<sqid>/<app>.json` for the box's own account |
| +scope only, read-only (`aclWrite` never answers true for it), and a |
| +fleet-door box tells a refusal from an absence for that key: a refusal |
| +is reported as `policy unreadable`, with no allow written and sizes at |
| +the build's tier, never treated as "no policy". |
| + |
| **What root writes:** `systemctl set-property --runtime <slice> |
| IPAddressAllow=` (reset) then `IPAddressAllow=<a>/32 <b>/128 ...`, the |
| reset-then-write shape of `allowResolvers`. An empty list writes only |
| @@ -463,12 +455,10 @@ default and the overrides. |
| - The four sizes and their numbers (§5.1). |
| - Shrinking below the build's tier is allowed with a warning (§5.2). |
| - Memory shrinks wait for a cold start; a swap never lowers memory (§5.4). |
| -- An incomplete address inventory clears every allow until it is |
| - complete again (§4.2). |
| +- Fleet addresses: checked at the grant, reported after, never enforced |
| + by root (§4.2; Paul's ruling on my proposal). |
| - The quarter-of-a-box refusal (§5.5). |
| -- Public addresses are refused as pointless; every fleet box address is |
| - refused as dangerous, and an incomplete inventory refuses the grant |
| - (§4.1, §4.2). |
| +- Public addresses are refused as pointless (§4.1). |
| - The audit write is mandatory for the policy door. |
| - Before claims, root takes the allow from the agent-written registry, |
| and writes none on a box that is not wholly nested; with claims, root |
| @@ -515,6 +505,13 @@ does exist, at `internal/activator/`. |
| | new 2 | `BoxesComplete` is a liveness join, not inventory completeness | Accepted: its own completeness check (§4.2) | |
| | new 3 | Fleet-door boxes cannot read the policy or the inventory | Accepted: account-scoped door read, a console-computed inventory object, refusal is not absence (§4.2) | |
| +**Astra round 4** (REJECT; R1-3, R2-1, R3-1, R3-3 and the rest resolved): |
| + |
| +| # | Finding | Disposition | |
| +|---|---|---| |
| +| R1-5 / R3-2 / new 1 | The inventory can look complete with boxes missing | Superseded: Paul ruled for a grant-time check plus a report; root no longer depends on the inventory (§4.2) | |
| +| new 2 | A cached completeness verdict never expires | Superseded the same way: no cached verdict exists (§4.2) | |
| + |
| ## 10. Left to the builder |
| - Column names, the call shapes, and the status snapshot fields. |
| @@ -543,13 +540,17 @@ does exist, at `internal/activator/`. |
| - `small` and `xlarge` through the complete claim launch. |
| - A live growth updates the captured tier; a partly applied change |
| blocks the next swap until a read-back settles it. |
| -- Root with an incomplete inventory (listing error, unreadable file, a |
| - file without `addrs`, `complete: false` through the door) clears the |
| - allows and restores them on the next complete read; an old edges file |
| - still excludes its box's addresses. |
| -- A bucketless fleet box receives the policy and the inventory through |
| - the store door, cannot write either, and treats a refusal as |
| - incomplete, not absent. |
| +- The grant check refuses another box's address (including one listed |
| + only in an old edges file) and refuses while a listed file is |
| + unreadable or lacks `addrs`; a box that publishes nothing is named |
| + `not checked`. |
| +- The report: an allowed address that a box later publishes raises one |
| + alert and shows `!`, and the allow stays in force. |
| +- Root with the store unreachable keeps writing the allows it holds; |
| + only its own addresses are ever dropped. |
| +- A bucketless fleet box receives the policy through the store door, |
| + cannot write it, and treats a refusal as `policy unreadable`, not as |
| + absent. |
| - A host name deleted and recreated does not inherit its override. |
| - Another app's instance on the same box cannot reach the allowed |
| address; the probe cannot either. |