carlos / platform Public

spec: instance sizes and egress allows, rev 3 (Astra R2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 3 days ago 4af48baa3e9e7d8b18424e2f23e5cb2c32d842ce parent 8930cb2

2 files changed, +106 −18

docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md +81 −18
docs/superpowers/specs/CODEX-FINDINGS-instance-sizes-egress-R2.md +25 −0

CI cancelled — superseded by a newer push — run details

CI log

=== gate ===
go: downloading github.com/aws/aws-sdk-go-v2 v1.44.0
go: downloading github.com/aws/aws-sdk-go-v2/config v1.32.6
go: downloading github.com/aws/aws-sdk-go-v2/service/route53 v1.64.1
go: downloading github.com/benbjohnson/litestream v0.5.14
go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0
go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0
go: downloading github.com/aws/aws-sdk-go-v2/service/sns v1.43.0
go: downloading github.com/superfly/ltx v0.5.1
go: downloading golang.org/x/sys v0.46.0
go: downloading github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.68.0
go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0
go: downloading modernc.org/sqlite v1.55.0
go: downloading golang.org/x/crypto v0.53.0
go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2
go: downloading github.com/oschwald/maxminddb-golang v1.13.1
go: downloading github.com/aws/aws-sdk-go-v2/service/ssm v1.74.0
go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.19.6
go: downloading github.com/aws/smithy-go v1.28.1
go: downloading github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi v1.37.0
go: downloading github.com/aws/aws-sdk-go-v2/service/autoscaling v1.74.0
go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.45.6
go: downloading github.com/klauspost/compress v1.18.0
go: downloading github.com/pion/logging v0.2.4
go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6
go: downloading github.com/pion/stun/v3 v3.0.1
go: downloading golang.org/x/net v0.56.0
go: downloading github.com/pion/turn/v4 v4.1.4
go: downloading github.com/aws/aws-sdk-go-v2/service/costexplorer v1.67.5
go: downloading github.com/pierrec/lz4/v4 v4.1.22
go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.40
go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.38
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.40
go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16
go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.30.8
go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4
go: downloading github.com/aws/aws-sdk-go-v2/service/signin v1.0.4
go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12
go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31
go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.40
go: downloading github.com/prometheus/client_golang v1.17.0
go: downloading golang.org/x/sync v0.21.0
go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18
go: downloading github.com/lmittmann/tint v1.1.3
go: downloading github.com/mattn/go-isatty v0.0.20
go: downloading github.com/pion/dtls/v3 v3.0.7
go: downloading github.com/pion/transport/v3 v3.0.8
go: downloading github.com/pion/randutil v0.1.0
go: downloading github.com/pion/transport/v4 v4.0.1
go: downloading github.com/wlynxg/anet v0.0.5
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/prometheus/common v0.45.0
go: downloading google.golang.org/protobuf v1.36.10
go: downloading github.com/prometheus/client_model v0.5.0
go: downloading github.com/cespare/xxhash/v2 v2.3.0
go: downloading github.com/prometheus/procfs v0.12.0
go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0
go: downloading golang.org/x/text v0.38.0
go: downloading modernc.org/libc v1.74.1
go: downloading modernc.org/memory v1.11.0
go: downloading github.com/dustin/go-humanize v1.0.1
go: downloading modernc.org/mathutil v1.7.1
go: downloading github.com/google/uuid v1.6.0
go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
go: downloading golang.org/x/tools v0.47.0
go: downloading golang.org/x/mod v0.37.0
gofmt -l .: clean
go test ./... -count=1 -timeout 30m

amadan: job cancelled — a newer push superseded it

diff --git a/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md b/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md

index ce860fe47..bb37d8e2a 100644

--- a/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md

+++ b/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md

@@ -1,9 +1,10 @@

# Instance sizes and egress allows

Amadan #328 (an isolated instance reaching a private service its operator

-runs) and #329 (operator-set CPU and memory per instance). Revision 2,

-2026-10-21. Revision 2 answers the builder check and Astra round 1

-(`CODEX-FINDINGS-instance-sizes-egress-R1.md`); the disposition is §9.

+runs) and #329 (operator-set CPU and memory per instance). Revision 3,

+2026-10-21. Revisions 2 and 3 answer the builder check and Astra rounds

+1 and 2 (`CODEX-FINDINGS-instance-sizes-egress-R<n>.md`); the

+dispositions are §9.

Both are the same mechanism: **an operator-set policy for an app and its

instances, kept in one operator-only object, delivered to the box, and

@@ -107,7 +108,7 @@ ETag:

"v": 1,

"egress_allow": ["10.0.5.20", "fd00:5::20"],

"default_size": "large",

- "sizes": { "shop-a.example.com": "xlarge" }

+ "sizes": { "shop-a.example.com": { "size": "xlarge", "created_at": "..." } }

}

Not on `instances.json` or the instance records: those are rewritten by

@@ -117,9 +118,14 @@ silently erase an operator's policy during a mixed-version roll

(Astra R1-7). A separate object no older binary writes cannot be erased

that way.

-- Deleting an instance removes its `sizes` entry (same door path as the

- delete). A stale entry is otherwise inert, but a host name can be

- reused, and a new instance must not inherit an old one's size.

+- An override is bound to the instance it was set on, not just its host

+ name: `"sizes": {"shop-a.example.com": {"size": "xlarge",

+ "created_at": "<the record's CreatedAt>"}}` (`provision.go:53`). An

+ entry whose `created_at` does not match the host's current record is

+ ignored, so a host name reused after a delete (including a delete by an

+ older console that never touches this object) does not inherit the old

+ instance's size (Astra R2-3). Deleting an instance through a new

+ console also removes the entry, for tidiness.

- **Precondition the builder must verify and record in the plan:** the

key is under the store's control boundary (`hostops/mutation_boundary.go`

and the bucket policy), so no app, build, instance or member

@@ -161,15 +167,33 @@ IPv6 (`internal/edgesfile/private.go:15`) (Astra R1-5).

- Each box publishes **every** non-loopback address it holds (from the

same procfs sources as `localPrefixes`) in a new edges-file field,

- `addrs`, refreshed whenever it writes the file.

+ `addrs`. It is part of the write gate (`edgesfile.go:665`), so a change

+ of address rewrites the file on the next pass. This works with region

+ HA off: no freshness or timed refresh is relied on, because an address

+ list does not go stale the way a headroom reading does (Astra R2-2).

+- **Age is not evidence of absence.** A file is never ignored for being

+ old: a dead or replaced box's last file keeps its addresses excluded

+ until an operator removes the file. That is the conservative direction;

+ the refusal names the box so the operator knows which file to retire.

- **Console:** refuses a grant naming any address in any box's `addrs`.

- If any box's edges file is missing, older than the 15-minute freshness

- bound `boxchoice` uses, or has no `addrs` field (an older box), the

- grant is refused, naming the box: the inventory cannot vouch for it.

+ It also refuses unless the inventory is complete: the edges snapshot's

+ `BoxesComplete` (`internal/edgesfile/snapshot.go:198`, every box with a

+ status snapshot has a readable edges file and the reverse), and every

+ file carries `addrs` (an older box does not). The refusal names the box

+ the inventory cannot vouch for.

- **Root on each box:** before every write (§4.3), drops any entry that

- is one of its own addresses (`isLocal`) or appears in any fresh edges

- file's `addrs`, and reports the drop (§4.4). This catches a box that

- joins, or an address that moves, after the grant.

+ is one of its own addresses (`isLocal`) or appears in any edges file's

+ `addrs`, and reports the drop (§4.4). If it cannot read the edges

+ prefix completely, it **fails closed**: it keeps the previous

+ exclusions and writes no newly-allowed address until it can (Astra

+ R2-5). An address already held stays only if it was already checked

+ against a complete read. This catches a box that joins, or an address

+ that moves, after the grant.

+- **Residual, stated:** the inventory is published by each box's agent.

+ Under claims a compromised agent could leave its own box's addresses

+ out, so that other boxes' tenants could reach it if an operator also

+ granted exactly that address. Root on the target box is not exposed by

+ this: the fence only governs what tenants dial out to.

### 4.3 How it reaches the instance, and who decides

@@ -318,6 +342,24 @@ are: they protect the overlap allowance (Astra R1-1).

`POST /size?host=` call. Root sets the slice first, then the serving

generation unit (`set-property --runtime` on the unit), and reads both

back.

+- **The activator's captured tier follows the read-back** (Astra R2-1).

+ A swap sizes its overlap from the incumbent's captured `Gen.Tier`

+ (`swap.go:498`), so a live growth that left it at the old figure would

+ raise the parent too little. Under the same serialization, the

+ activator sets `Gen.Tier` from root's read-back of the serving unit

+ after a live change. If a change is partly applied (slice written, unit

+ not, or the reply lost), the host is marked unsettled: no swap starts

+ until a fresh read-back settles it, and the captured tier is then taken

+ from that read-back, never from the request.

+- **A lowering restore waits for the incumbent to be gone** (Astra

+ R2 new 1). Today a swap restores the parent (`dropMember`) before it

+ replaces a legacy co-spawned sidecar (`replaceSidecar`, `swap.go:678`),

+ and the old sidecar shares the slice. When the successor's size is

+ smaller than the incumbent's, the restore to the smaller figure happens

+ only after the retiring generation and the old sidecar (and their

+ descendants) are proven stopped: units inactive and cgroups empty. Until

+ then the parent keeps the overlap figure, and a failed restore retries

+ with the same proof. An equal-size restore keeps today's order.

- **Shrinking memory never touches a running process** (Astra R1-3: a

usage sample cannot make lowering `memory.max` safe, because the

process can allocate between the sample and the write). The new size

@@ -336,10 +378,14 @@ caps on a box is routinely above its memory. So the console does not

refuse on a sum. It does:

- **Refuse** a size whose memory max exceeds a quarter of the smallest

- total memory among the boxes the app's instances can be placed on

- (`Headroom.MemTotal` from the edges file), naming the box and its

- total. (My call: one instance should never be able to take a box's

- whole headroom.)

+ total memory among the boxes the app's instances can be placed on,

+ naming the box and its total. (My call: one instance should never be

+ able to take a box's whole headroom.) The total cannot come from

+ `Headroom`, which is nil with region HA off (`headroom.go:10`, Astra

+ R2-2). Each box publishes `mem_total_mib` as its own edges-file field,

+ in the write gate like `addrs`. A box whose total is missing or

+ unreadable (0) makes the console refuse any size above `standard`,

+ naming the box.

- **Report**, on every size change, the box's committed figure: the sum

of the effective sizes of its awake instances against its total and

free memory, so the operator sees how much is left. Information, not a

@@ -418,6 +464,16 @@ does exist, at `internal/activator/`.

| 8 | Size authority comes from request values | Declined: sizes keep the tiers' existing trust; the anti-goal is now worded for the egress allow, which is the isolation boundary (§2, §5.3) |

| 9 | Proposed highs are not byte-identical | Accepted: `HighOf` stays authoritative (§5.1) |

+**Astra round 2** (REJECT; R1-1, 4, 6, 7, 8, 9 resolved):

+| # | Finding | Disposition |

+|---|---|---|

+| R1-2 | Captured `Gen.Tier` not updated on growth | Accepted: set from read-back under the serialization; unsettled hosts start no swap (§5.4) |

+| R1-3 / new 1 | A lowering restore can squeeze the old sidecar | Accepted: a lowering restore waits for the incumbent and old sidecar to be proven stopped (§5.4) |

+| R1-5 | Root re-admits an address when a file expires | Accepted: no expiry; age is not absence; root fails closed on an incomplete read (§4.2) |

+| new 2 | Freshness and totals depend on region HA | Accepted: `addrs` and `mem_total_mib` are gated fields, no freshness relied on (§4.2, §5.5) |

+| new 3 | A reused host name inherits an override | Accepted: overrides bound to the record's `CreatedAt` (§3) |

## 10. Left to the builder

- Column names, the call shapes, and the status snapshot fields.

@@ -443,6 +499,13 @@ does exist, at `internal/activator/`.

own cap; a memory shrink leaves the running process's limits alone and

applies to the next generation.

- `small` and `xlarge` through the complete claim launch.

+- A live growth updates the captured tier; a partly applied change

+ blocks the next swap until a read-back settles it.

+- A shrinking swap keeps the overlap figure until the old generation and

+ old sidecar are proven stopped.

+- Root with an unreadable edges prefix writes no newly-allowed address;

+ an old edges file still excludes its box's addresses.

+- A host name deleted and recreated does not inherit its override.

- Another app's instance on the same box cannot reach the allowed

address; the probe cannot either.

- An instance with no policy: units and slice properties byte-identical

diff --git a/docs/superpowers/specs/CODEX-FINDINGS-instance-sizes-egress-R2.md b/docs/superpowers/specs/CODEX-FINDINGS-instance-sizes-egress-R2.md

new file mode 100644

index 000000000..943121125

--- /dev/null

+++ b/docs/superpowers/specs/CODEX-FINDINGS-instance-sizes-egress-R2.md

@@ -0,0 +1,25 @@

+R1-1: **RESOLVED** — Keeping populated-slice skips and serializing resizing with swaps addresses the reported overlap overwrite.

+R1-2: **PARTLY (P1)** — Generation limits now grow, but §5.4 still omits updating the activator’s captured `Gen.Tier`. The next swap uses that cached value ([swap.go:498](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/instance-egress-sizes-328-329/internal/activator/swap.go:498)). After growth from 512 MiB to 2 GiB, a 256 MiB successor could therefore “raise” the parent to only 768 MiB. Require captured-tier updates under the same serialization, including recovery from partial writes.

+R1-3: **PARTLY (P1)** — The usage-sampling race is removed, but shrinking at the next generation still endangers the old sidecar; new finding 1 below.

+R1-4: **RESOLVED** — Startup reconciliation and periodic read-back retries address failed revocation writes.

+R1-5: **PARTLY (P1)** — §4.2 makes only the console fail closed on incomplete inventory. Root excludes addresses from **fresh** edges files only. An existing grant dropped because another box acquired its address becomes eligible again when that box’s file expires. Require root to fail closed on incomplete inventory and retain known box-address exclusions until their removal is verified.

+R1-6: **RESOLVED** — §5.3 explicitly includes all three claim validators and complete-launch tests.

+R1-7: **RESOLVED** — A separate policy object prevents older manifest writers from erasing policy. Lifecycle compatibility remains a separate issue below.

+R1-8: **RESOLVED** — The declined finding matches the revised trust model and the existing agent-supplied tier mechanism.

+R1-9: **RESOLVED** — Keeping `HighOf` authoritative preserves existing memory-high values.

+1. **P1 — Next-generation shrinking lowers the limit beneath an old sidecar.** §5.4 permits applying a smaller parent after generation retirement. The swap calls `dropMember`, which restores the parent, **before** `replaceSidecar` ([swap.go:678](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/instance-egress-sizes-328-329/internal/activator/swap.go:678)). That old sidecar shares the host slice ([carlos-instance-sidecar@.service:33](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/instance-egress-sizes-328-329/internal/host/units/carlos-instance-sidecar@.service:33)); its termination is checked only during replacement ([activator.go:2928](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/instance-egress-sizes-328-329/internal/activator/activator.go:2928)). A sidecar using more than the new cap can therefore trigger OOM killing when the parent shrinks. [Kernel memory-limit documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html). **Smallest fix:** retain the overlap allowance until the old sidecar and its descendants are proven stopped; include that proof in restore retries.

+2. **P2 — The inventory freshness and capacity requirements depend on optional region HA.** §4.2 requires edges files younger than 15 minutes, and §5.5 requires their memory totals. However, `headroomFor` returns nil without region HA ([headroom.go:10](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/instance-egress-sizes-328-329/internal/host/headroom.go:10)), and timed publication skips unchanged files unless headroom exists ([edgesfile.go:665](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/instance-egress-sizes-328-329/internal/host/edgesfile.go:665)). Adding an unchanged `addrs` field does not fix this: healthy single-box deployments eventually fail the freshness check and lack capacity evidence. **Smallest fix:** require periodic address-inventory publication and memory totals independently of region HA, with explicit handling of unreadable totals.

+3. **P2 — Older-console deletion lets a replacement instance inherit the previous instance’s override.** §3 requires deleting `sizes[host]` during instance deletion, while §7 permits older consoles that never write the policy object. Their deletion path retires the instance record without touching that object ([apiinstances.go:2245](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/instance-egress-sizes-328-329/internal/console/apiinstances.go:2245)). Recreating the hostname then matches the retained override under §5.2, violating the explicit no-inheritance requirement. **Smallest fix:** bind overrides to the instance’s creation identity and ignore mismatches; the record already carries `CreatedAt` ([provision.go:53](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/instance-egress-sizes-328-329/internal/provision/provision.go:53)).

+REJECT

\ No newline at end of file