amadan

carlos / platform Public

spec: instance sizes and egress allows (#328, #329), rev 1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude 56bd97e54841920aa7a34ad644ec1b166b9fc34c parent 3d444a1
2 files changed, +362 −0
  • docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md +332 −0
  • docs/superpowers/specs/CODEX-REVIEW-BRIEF-instance-sizes-egress.md +30 −0

CI cancelled — superseded by a newer push — run details

diff --git a/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md b/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md
new file mode 100644
index 000000000..c5568b85c
--- /dev/null
+++ b/docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md
@@ -0,0 +1,332 @@
+# Instance sizes and egress allows
+
+Amadan #328 (an isolated instance reaching a private service its operator
+runs) and #329 (operator-set CPU and memory per instance). Revision 1,
+2026-10-21.
+
+Both are the same mechanism: **an operator-set property of an app or an
+instance, delivered to the box the way update settings already are, and
+written by the control service onto the instance's own slice**
+(`carlos-instances-<host>.slice`). #329 writes memory and CPU there, as
+`sliceTier` already does. #328 writes `IPAddressAllow=` there.
+
+## 0. Rulings (Paul, 2026-10-20/21; do not reopen)
+
+1. **#328 is an operator-granted, per-app egress allow.** Not a platform
+ relay, not a doc-engine route (that is #335, separate).
+2. **Per app, not per host.** A deliberate exception to the capability
+ rule that grants are `--host` only (README "App capabilities",
+ `cli.go:695`). An app's instances are created continually, and a
+ per-host grant would need a step per new instance. The risk is one
+ named private address, not a port or a cloud role.
+3. **Address only, never a port.** `IPAddressAllow=` cannot name a port
+ and opens the address both ways. The CLI does not accept `:port`.
+4. **Named sizes, not free numbers.** A small fixed set, as tiers are
+ today.
+5. **A size has an app default and a per-instance override**, resolved the
+ way `EffectiveUpdates` resolves update settings.
+6. **The egress allow is operator-set only.** Never from a manifest,
+ never from an app's own credentials. (Sizes are operator-only too, but
+ that is my call, §6.)
+7. **Provider-neutral.** Nothing here names a cloud; a self-hosted box
+ with MinIO and plain hosts gets both features.
+
+## 1. Why
+
+- An isolated instance's unit denies every private range
+ (`IPAddressDeny=` in every `carlos-instance*` template, isolated-exec
+ spec §11.1). Only the box's DNS resolvers are reopened. An operator
+ whose app needs a private service (a document renderer in the same
+ network, a database on another host) has no way to reach it except
+ putting it on the internet. That is the wrong trade, and isolation is
+ blocked on it.
+- An isolated instance's size is one of two tiers chosen on the box from
+ the build's `children` bit (`internal/host/memtier.go:18`). An operator
+ cannot give one instance more room than another of the same app: a
+ busy shop more headroom during a sale, a dormant one less.
+
+**Success:** an operator types one command per app to open one private
+address for every instance of it, present and future, and one command to
+size an app or one instance; both take effect without a redeploy; and
+both can be seen in force on the running instance, read back from
+systemd rather than from intent.
+
+## 2. Goals, non-goals, anti-goals
+
+**Goals**
+- `carlos apps set <app> --egress-allow <addr>[,<addr>...]` and
+ `--egress-allow ""` to clear; `carlos apps show` lists them.
+- `carlos apps set <app> --size <name>` (the app default) and
+ `carlos instances set <host> --size <name>|default` (the override).
+- Every unit an isolated instance runs as (run, generation, seed,
+ sidecar, cmd, and the claim launch path) gets both, because all of
+ them sit in the per-host slice.
+- Granting, revoking and resizing reach running instances without a
+ restart (§4.4 for the one exception), and survive a restart.
+- Visible: what systemd actually holds for each instance's slice, in
+ `carlos status --layout` and `carlos instances list`.
+
+**Non-goals**
+- Port scoping (ruling 3). The operator's own firewall decides ports.
+- CIDR ranges. Single addresses only (§3.2).
+- `carlos-app@` unit-backed routes. They have their own grant model
+ (`internal/caps`), and nothing has asked.
+- Reservations or bin-packing. A size is a cap, not a reservation; see
+ §4.5 for what the console does check.
+- Self-service sizes for app owners. That belongs with Carloku's plans
+ and caps, which this must not undercut (§6).
+- A platform relay to private services, or making doc engines
+ fleet-internal: #335.
+
+**Anti-goals (failure even if it works)**
+- An app, a build or an instance able to grant itself an allow or a
+ size. Every byte that reaches `systemctl` comes from operator-written
+ state, never from a request or a manifest.
+- An allow that reaches another app's instances, or the probe, or any
+ unit outside the app's own per-host slices.
+- An allow that can open loopback, link-local, the metadata service, or
+ any of the box's own addresses.
+- A revoke that leaves the address open on running instances.
+- A size change that OOM-kills a running instance.
+- Reporting an allow or size as in force because it was requested.
+
+## 3. Egress allow (#328)
+
+### 3.1 Where it lives
+
+On the app's instance manifest, `apps/<sqid>/<app>/instances.json`
+(`provision.Manifest`), as a new field beside `Updates` and
+`PrivateStorage`:
+
+ "egress_allow": ["10.0.5.20", "fd00:5::20"]
+
+**Precondition the builder must verify and record in the plan:** no
+credential an app, a build or an instance holds can write that key.
+Today the writers are the console, the boxes, and operators holding
+bucket credentials in bucket mode. If any app-held credential can write
+`apps/<sqid>/<app>/instances.json`, stop: the allow moves to a
+console-only key under `control/`, and the plan says so.
+
+### 3.2 What is accepted
+
+Validated at the console door and again on the box (the box never trusts
+the bucket's copy more than the door's):
+
+- A single IPv4 or IPv6 address. No CIDR, no zone, no port, no hostname.
+ IPv4-mapped IPv6 is unmapped first.
+- **Must be inside the instance fence's deny list** (RFC 1918, CGNAT,
+ ULA, and the other private ranges of `IPAddressDeny=`). A public
+ address is already reachable; it is refused as "already reachable, no
+ allow needed" so the list only ever holds real exceptions.
+- **Refused anywhere:** loopback, unspecified, all link-local (v4 and
+ v6), both metadata addresses (169.254.169.254, fd00:ec2::254),
+ multicast, 0.0.0.0/8, 192.0.0.0/24, 198.18.0.0/15, 240.0.0.0/4, the
+ NAT64 prefix 64:ff9b::/96 (it maps to any IPv4 address), fec0::/10.
+- **Refused at the console:** any address a fleet box publishes as its
+ own private address (the edges file's `private_ip`, region HA 2a).
+ Opening a box opens every listener on it.
+- **Refused on the box:** any of the box's own addresses (`isLocal`,
+ `internal/instancectl/localaddrs.go`). The box drops that entry, keeps
+ the rest, and reports it (§3.5).
+- At most 8 addresses per app. Duplicates collapse; the list is stored
+ sorted.
+
+Reuse `ResolverAllowed` and `localPrefixes` where they fit; the deny-list
+ranges should come from one table shared with the template test
+(`internal/host/units_hardening_test.go`), so a range added to the fence
+is automatically one an allow must fall inside.
+
+### 3.3 Who can set it
+
+System operators only: the `apiSystemRoller` gate (owner of the ops
+account, or listed in `CARLOS_SYSTEM_OPERATORS`), with an empty service
+role set so no service credential reaches it. A new door,
+`PUT /api/cli/apps/{account}/{app}/egress`, compare-and-swap on the
+manifest's ETag.
+
+Every change is recorded with `s.audit` (account, app, old list, new
+list, operator). For this door the record is not best-effort: if the
+event cannot be written, the change is refused.
+
+### 3.4 How it reaches the instance
+
+1. The box's reconcile pass already reads each app's manifest. It
+ converges `EgressAllow` into a new registry column on each of the
+ app's route rows (`egress_allow`, the validated, sorted list), the way
+ `SetUpdateSettings` converges update settings.
+2. The control service writes it. It never takes the list from a
+ request: it looks the host's app up (`HostApp`) and reads the column
+ from the registry itself, keeping "nothing from a request reaches
+ systemctl as a string" (`server.go:33`).
+3. **When:** on every start path, unconditionally (not gated on a cold
+ slice the way `sliceTier` is for generations and cmd units), and on a
+ new `POST /egress?host=` call the agent makes after reconcile changes
+ the column for a host with a populated slice.
+4. **What:** `systemctl set-property --runtime <slice> IPAddressAllow=`
+ (reset) then `IPAddressAllow=<a>/32 <b>/128 ...`, the same reset-then-
+ write shape as `allowResolvers`. An empty list writes only the reset.
+5. **Live:** the builder must prove on a real kernel (a cgroup test, the
+ exception to "CI only") that a slice-level `set-property --runtime`
+ change both opens and closes the address for a process already running
+ in a child unit, and that the per-host slice's allow combines with the
+ resolver allow on `carlos-instances.slice` rather than replacing it.
+ If systemd does not apply a revoke live, a revoke restarts the host's
+ units, and the plan says so.
+
+A move carries the allow for free: the destination's reconcile reads the
+same manifest. Re-isolation picks it up at the next start. A revert to
+flat leaves an unused slice; flat instances have no private-range fence,
+so there is nothing to carry. The column is cleared when the route row
+goes.
+
+### 3.5 Seeing it in force
+
+The control service reads the slice back (`systemctl show -p
+IPAddressAllow <slice>`) after every write and reports per host:
+`allowed` (what systemd holds, minus the resolver entries inherited from
+the parent) and `refused` (entries the box dropped, with the reason).
+That travels in the box's status snapshot, and:
+
+- `carlos status --layout` gains an `EGRESS` column: the read-back list,
+ `-` for none, `!` plus the reason when the read-back disagrees with
+ the column.
+- `carlos instances list` gains the same, terse.
+- `carlos apps show` shows the configured list.
+
+## 4. Sizes (#329)
+
+### 4.1 The vocabulary
+
+One table in `internal/cgroup`, replacing the two `Tier*` variables as
+the source of every number:
+
+| size | memory max | high (90%) | CPU quota | CPU weight | IO weight |
+|---|---|---|---|---|---|
+| `small` | 256M | 230M | 50% | 50 | 50 |
+| `standard` | 512M | 460M | 100% | 100 | 100 |
+| `large` | 1G | 920M | 200% | 100 | 100 |
+| `xlarge` | 2G | 1840M | 400% | 200 | 200 |
+
+`standard` and `large` are exactly today's `TierDefault` and
+`TierChildren`, so an instance nobody sizes is unchanged byte for byte.
+(My call: the four names and their numbers. `small` halves the weights so
+a dormant instance yields under contention; `xlarge` doubles them so a
+busy one wins.)
+
+### 4.2 Resolution
+
+ effective = instance override, else app default, else the build's tier
+
+where the build's tier is today's rule (`children` → `large`, else
+`standard`). A size below the build's tier is allowed and the CLI warns
+("this build declares children; small may be OOM-killed"). The operator
+owns the call; the platform never silently raises it. (My call.)
+
+Stored as `Size` on `provision.Record` (override) and `DefaultSize` on
+`provision.Manifest`, resolved by an `EffectiveSize(rec, manifest,
+buildTier)` beside `EffectiveUpdates`, converged into a registry `size`
+column by reconcile.
+
+### 4.3 How it reaches the instance
+
+- The box's tier source (`ExecBackend.Tier`/`TierOf`, `memtier.go`)
+ becomes the effective size for the host, so every caller of the
+ `memory_max` parameter (run, sidecar, seed, cmd, parent) and the claim
+ snapshot (`claim_snapshot.go:229`) carry it. The claim bound-unit path's
+ hard-coded `TierDefault` (`claimunit_inputs.go:74`) goes.
+- The control service's `parseRequest` accepts exactly the four sizes'
+ `memory_max` values; `SharesOf` maps the value back to its row.
+ `ValidParentLimit` accepts one size or the sum of any two (a swap
+ overlap, including one across a resize).
+- Flat exec children get the memory half only, as today (`applyLimit`
+ writes no CPU). `carlos-app@` is untouched.
+
+### 4.4 Live changes
+
+When reconcile changes a host's effective size and the slice is
+populated, the agent asks the control service to apply it
+(`POST /size?host=`), which calls `sliceTier` with the new size:
+
+- **Growing** applies at once.
+- **Shrinking** applies at once only if the slice's `memory.current` is
+ below the new size's `MemoryHigh`. Otherwise it is held, retried on
+ every reconcile, and reported as `pending` with the current usage. It
+ is never forced: a forced shrink is an OOM kill (anti-goal).
+- A start always applies the effective size (cold slices already do; the
+ populated-slice skips for generations and cmd units now apply the size
+ when it differs from what the slice holds, under the same
+ shrink rule).
+
+### 4.5 Capacity
+
+Sizes are caps, not reservations; instances hibernate, and the sum of
+caps on a box is routinely above its memory. So the console does not
+refuse on a sum. It does:
+
+- **Refuse** a size whose memory max exceeds a quarter of the smallest
+ total memory among the boxes the app's instances can be placed on
+ (`Headroom.MemTotal` from the edges file), saying the box and its
+ total. (My call: one instance should never be able to take a box's
+ whole headroom.)
+- **Report**, on every size change, the box's committed figure: the sum
+ of the effective sizes of its awake instances against its total and
+ free memory, so the operator sees how much is left. Information, not a
+ gate.
+
+Placement (`boxchoice`) is unchanged.
+
+### 4.6 Who can set it, and how it shows
+
+- System operators only, the same gate as §3.3, recorded with `s.audit`.
+ Doors: `PUT /api/cli/apps/{account}/{app}/size` and
+ `PUT /api/cli/instances/{host}/size` (`default` clears the override).
+- `carlos instances list` and `carlos status --layout` gain `SIZE`: the
+ size the slice actually holds (read back from systemd's
+ `MemoryMax`), with `>name` when a change is pending.
+
+## 5. Rollout and compatibility
+
+- An older box ignores the new manifest and record fields: no allow
+ (fail closed) and today's tiers. An older console never writes them.
+- Boxes before consoles, dev first (standing roll order).
+- The control service is root code: a census overlay and a #280 ruling
+ before merge.
+- Nothing is on by default. With no field set, every unit, slice and
+ probe line is byte-identical to today.
+
+## 6. Calls I made (check these)
+
+- **Sizes are operator-only**, not app-owner. #329 allowed either. On a
+ multi-tenant fleet an app owner who could pick `xlarge` would bypass
+ Carloku's plan caps. Self-service can come with plans and caps.
+- The four sizes and their numbers (§4.1).
+- Shrinking below the build's tier is allowed with a warning (§4.2).
+- The quarter-of-a-box refusal (§4.5).
+- Public addresses are refused as pointless; fleet boxes' private
+ addresses are refused as dangerous (§3.2).
+- The audit write is mandatory for both doors.
+
+## 7. Left to the builder
+
+- Column names, the agent-to-control-service calls' exact shape, and the
+ status snapshot fields.
+- Whether `EGRESS`/`SIZE` fit `instances list` or only `status --layout`
+ and `instances show`, if the table gets too wide.
+- The order of the plan's tasks, provided the live-revoke kernel test
+ (§3.4 step 5) comes before anything is built on it.
+
+## 8. Testing
+
+- Validation tables for every refused address class, both doors.
+- The kernel test of §3.4 step 5, with the resolver allow present.
+- A production-wired test per start path (run, gen, seed, sidecar, cmd,
+ claim) that the slice holds the allow and the size after start, through
+ the real control service and units, not fixtures.
+- Revoke while running: the address is unreachable from the running
+ process afterwards.
+- Shrink held while `memory.current` is above the new high; applied once
+ it falls.
+- Another app's instance on the same box cannot reach the allowed
+ address; the probe cannot either.
+- An instance with nothing set: units and slice properties identical to
+ main.
diff --git a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-instance-sizes-egress.md b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-instance-sizes-egress.md
new file mode 100644
index 000000000..126b8ab21
--- /dev/null
+++ b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-instance-sizes-egress.md
@@ -0,0 +1,30 @@
+# Review brief: instance sizes and egress allows (rev 1)
+
+Review `docs/superpowers/specs/2026-10-21-instance-sizes-and-egress-design.md`
+at the commit you are run on, against the code in this repository.
+
+You are an independent second reader. The spec's §0 rulings are Paul's and
+fixed: do not argue them. Report only mechanisms that fail to honour them,
+or that fail the spec's own anti-goals (§2), against the real code.
+
+Look hardest at:
+- whether any path lets an app, a build, an instance or a non-operator reach
+ an allow or a size (bucket writers of `apps/<sqid>/<app>/instances.json`
+ and instance records, console doors, the agent-to-control-service calls);
+- whether a per-host slice `IPAddressAllow=` really is confined to that
+ host's units and combines with the resolver allow on
+ `carlos-instances.slice` (internal/instancectl/hardening.go, the
+ carlos-instance* templates, the claim unit's effective check in
+ internal/host/claimunit_effective.go);
+- start paths that would miss the allow or the size (run, gen, seed,
+ sidecar, cmd, parent, claim launch, moves, layout migrate/revert);
+- the address validation in §3.2 (anything that can still open loopback,
+ link-local, metadata, the box's own or another fleet box's addresses);
+- the size changes against internal/cgroup, parseRequest, ValidParentLimit,
+ swap overlaps and the shrink rule in §4.4;
+- anything the spec claims about the code that is false. Cite file:line.
+
+Output: a numbered list of findings, each with severity (P1 blocks
+building, P2 must fix in the spec, P3 worth fixing), the evidence with
+file:line, and the smallest fix. Then one line: ACCEPT or REJECT
+(reject if any P1). No preamble.