amadan

carlos / platform Public

Clone 
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# Plans 2: Boxes Read Account Limits Implementation Plan
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** Boxes read each account's published limits and apply the plan's idle window to instances and sidecar workers. Self-hosted deployments without a plan catalog see no change.
6
7**Architecture:** The console publishes a deployment marker `control/plans.json` (enabled plus the catalog id) once at start. On every box the host agent's existing full and five-minute passes run a `plansPass` beside `telemetryPass`. It reads the marker, and only when plans are enabled does it read `accounts/<sqid>/limits.json` for the accounts this box serves, into a last-known-good in-memory cache. The activator gets an `IdleFor` hook that can only shorten an entry's idle window. A box-level switch `CARLOS_PLAN_LIMITS=off` turns all of it off with zero bucket reads.
8
9**Tech Stack:** Go; `internal/plans`, `internal/console`, `internal/host`, `internal/activator`, `internal/hostops`, `internal/edge`.
10
11**Spec:** `docs/superpowers/specs/2026-10-20-plans-caps-packs-design.md` §5.2 (idle window), §7 (how limits reach the boxes; fail open), §10 (provider boundary). Builds on Plan 1 (`docs/superpowers/plans/2026-10-20-plans-1-catalog-and-limits.md`, merged f550afa).
12
13## Self-hosters (Paul, 2026-10-21: "make sure this isn't going to interrupt them")
14
15Three guards, each pinned by a test:
16
171. **No catalog, no change.** Without `CARLOS_PLAN_CATALOG` the console never writes `control/plans.json` as enabled. A box that finds the marker absent or disabled clears its cache and reads no limits record: one small read per pass (the marker), nothing else, and every idle window is exactly today's.
182. **No record, no limits.** A box with no limits record for an account runs that account exactly as today (fail open, spec §7).
193. **A box can opt out.** `CARLOS_PLAN_LIMITS=off` (also `false`, `0`, `no`) makes the box ignore plans entirely and read nothing.
20
21A deployment that turns its catalog off again: the console's next start writes the marker disabled, and boxes revert within one pass.
22
23## Scope changes from the spec's plan list (rulings, author's)
24
25- The **small memory tier** moves to Plan 2b. It touches nine allowlists across `internal/cgroup`, `internal/instancectl` and the root claim launch code, plus unit-backed apps' rendered drop-ins, and most of that is root-path code with its own census ruling.
26- **Throttling and the continuous-awake ceiling** move to Plan 4, where the grace engine that triggers them lives. Plan 2 carries no throttle state.
27- Plan 1's two deferred minors are fixed here (Task 1, Task 2).
28
29## Global Constraints
30
31- No new module dependencies.
32- Plans can only **shorten** an idle window: an entry's window is `min(its current window, plan idle)` (spec §5.2 "the smaller of its own and the plan's"). A plan never lengthens a box's `CARLOS_HIBERNATE_IDLE` and never touches an always-on worker.
33- Fail open everywhere on the box: a read error keeps the last good value, an absent record means no limits.
34- New box env vars go in `hostenv.Optional` (`internal/hostenv/hostenv.go:71`).
35- New bucket keys need a hostops classification and, if fleet boxes read them, the storedoor allowlist (`internal/edge/storedoor.go:436`).
36- New console workers: `claim_workers.go`, `cmd/console/main.go` RunClaimWorker, `testdata/claim-routes.txt`, a claim fixture and probe.
37- CI shards tests: no test depends on another top-level test having run first. New `internal/host` tests call `t.Parallel()` unless they set env, cwd, umask, log output or package vars; the cache is a package var, so tests that touch it stay serial. New `internal/activator` tests call `t.Parallel()` too unless they write package vars, env, log output or the umask, or listen on TCP (the rule above activator's TestMain, ci-speed-activator 4f55bfb). The Task 5 overlay loads after `plan14-root-reads-ci-speed-activator`.
38- Every change to covered packages re-pins the #280 root-read overlay; the record-write census layer may need entries too (Task 5).
39- Copy: this plan adds no human-facing text except log lines (not reviewed).
40- Local testing: touched packages only, with `-run`; CI runs the full suite.
41
42## Review Focus
43
441. **Plans off.** With the marker absent, a box must make exactly one bucket read per pass (the marker) and apply no limits. Pinned in Task 3 (`TestPlansPassOffReadsOnlyTheMarker`).
452. **Box switch off.** `CARLOS_PLAN_LIMITS=off` must make zero bucket reads and clear any cache. Pinned in Task 3 (`TestPlansPassBoxSwitchOffReadsNothing`).
463. **Bucket blips.** A marker read error keeps the whole last-good cache; a per-account read error keeps that account's last value. Pinned in Task 3 (`TestPlansPassKeepsLastGoodOnErrors`).
474. **Plans turned off again.** Marker enabled, then disabled: the next pass clears the cache and idle windows revert. Pinned in Task 3 (`TestPlansPassClearsWhenDisabled`) and Task 4 (`TestIdleForNilOrZeroChangesNothing`).
485. **A plan idle longer than the box's default.** The box default stands. Pinned in Task 4 (`TestPlanIdleOnlyShortens`).
49
50---
51
52### Task 1: The deployment marker, and checked pack arithmetic
53
54**Files:**
55- Create: `internal/plans/marker.go`, `internal/plans/marker_test.go`
56- Modify: `internal/plans/catalog.go` (`Effective`), `internal/plans/catalog_test.go`
57
58**Interfaces:**
59- Produces:
60 ```go
61 const MarkerSchema = 1
62 type Marker struct {
63 Schema int `json:"schema"`
64 Enabled bool `json:"enabled"`
65 CatalogID string `json:"catalog_id,omitempty"`
66 UpdatedAt time.Time `json:"updated_at"`
67 }
68 func MarkerKey() string // "control/plans.json"
69 func GetMarker(ctx context.Context, s release.Store) (*Marker, string, error) // nil, "", nil when absent
70 func PutMarker(ctx context.Context, s release.Store, m *Marker, ifMatch string) (string, error)
71 ```
72 `GetMarker` refuses an unknown schema (error, so the box keeps its last good state).
73
74- [ ] **Step 1: Write the failing tests**
75
76```go
77func TestMarkerKey(t *testing.T) // MarkerKey() == "control/plans.json"
78func TestMarkerRoundTrip(t *testing.T) // PutMarker create-only then GetMarker equal; absent -> nil, "", nil
79func TestMarkerRefusesUnknownSchema(t *testing.T) // a stored {"schema": 2} -> GetMarker error
80func TestPutMarkerIsCAS(t *testing.T) // second create-only put -> release.ErrCASConflict
81func TestEffectiveRefusesOverflow(t *testing.T) // pack storage adds 1<<62, quantity 2, ceiling 100 -> error wrapping ErrPackQuantity,
82 // never a negative or wrapped limit (Astra plans-1 finding 5)
83```
84
85- [ ] **Step 2: Run them to verify they fail** — `go test ./internal/plans/` → FAIL (undefined `MarkerKey`).
86
87- [ ] **Step 3: Implement** `marker.go` on the pattern of `record.go`. In `Effective`, compute each pack's contribution with checked multiplication and addition (`math/bits.Mul64`/overflow checks, or compare against `math.MaxInt64 / qty`) and return `fmt.Errorf("%w: %q overflows", ErrPackQuantity, name)` on overflow.
88
89- [ ] **Step 4: Run** `go test ./internal/plans/` → PASS.
90
91- [ ] **Step 5: Commit** `plans: the deployment marker, and checked pack arithmetic`
92
93### Task 2: The console publishes the marker; plan writes need a live account
94
95**Files:**
96- Create: `internal/console/plansmarker.go`, `internal/console/plansmarker_test.go`
97- Modify: `internal/console/accountplans.go` (`SetAccountPlan`), `internal/console/accountplans_test.go`, `internal/console/apiaccountplan.go`, `internal/console/apiaccountplan_test.go`
98- Modify: `internal/console/claim_workers.go`, `cmd/console/main.go`, `internal/console/testdata/claim-routes.txt`, `internal/console/claim_invariant_fixtures_test.go`, `internal/console/claim_invariant_probes_worker_test.go`
99- Modify: `internal/hostops/mutation_classifier.go`, `internal/hostops/limits_classifier_test.go`
100- Modify: `internal/edge/storedoor.go` (`aclRead`), its test file
101
102**Interfaces:**
103- Consumes: Task 1.
104- Produces:
105 ```go
106 func (s *Server) PublishPlansMarker(ctx context.Context) // claim worker "PublishPlansMarker"; runs once
107 var ErrAccountGone = errors.New("account deleted") // console package
108 ```
109 `SetAccountPlan` returns `ErrAccountGone` and writes nothing when the account's `deleted_at` is set (Astra plans-1 finding 3); the plan door answers 404 for it.
110
111- [ ] **Step 1: Write the failing tests**
112
113```go
114// plansmarker_test.go
115func TestPublishPlansMarkerOffAndAbsentWritesNothing(t *testing.T) // no catalog, no marker: bucket unchanged
116func TestPublishPlansMarkerEnables(t *testing.T) // catalog set: marker Enabled, CatalogID == catalog.ID()
117func TestPublishPlansMarkerIsIdempotent(t *testing.T) // second run with the same catalog: etag unchanged
118func TestPublishPlansMarkerDisablesWhenCatalogRemoved(t *testing.T) // marker enabled, then a run with no catalog: Enabled false
119// accountplans_test.go
120func TestSetAccountPlanRefusesADeletedAccount(t *testing.T) // deleted_at set: ErrAccountGone, no row
121// apiaccountplan_test.go
122func TestAccountPlanSetOnADeletedAccountIs404(t *testing.T) // deleted between lookup and write: 404, no record published
123// hostops limits_classifier_test.go
124func TestClassifyPlansMarkerIsExact(t *testing.T) // Put on control/plans.json: non-app; Delete and lookalikes: ErrUnclassified
125// edge storedoor test
126func TestFleetBoxCanReadThePlansMarker(t *testing.T) // aclRead(sqid, fleet, "control/plans.json") == true
127```
128
129- [ ] **Step 2: Run** the touched packages' tests → FAIL.
130
131- [ ] **Step 3: Implement.** `PublishPlansMarker`: with a catalog, `GetMarker`; if absent or different (`!Enabled || CatalogID != cat.ID()`), `PutMarker` with the read etag. Without a catalog, `GetMarker`; only if present and `Enabled`, write it `Enabled: false`. Log, never fail. `SetAccountPlan`: guard the insert and the update with `EXISTS (SELECT 1 FROM accounts WHERE id = ? AND deleted_at = 0)`, returning `ErrAccountGone` when no row changes. hostops: add `{exact: "control/plans.json", reason: "plans deployment marker published by the console; no app state"}` to `nonAppFamilies` (deletes false). storedoor: add `plans.MarkerKey()` to the exact case in `aclRead`.
132
133- [ ] **Step 4: Register and start the worker** (`claim_workers.go`, `cmd/console/main.go` beside `StartPlanLimitsPass`), add `worker PublishPlansMarker` to `claim-routes.txt`, a `claimNonApp` fixture and a probe asserting the marker is written when a catalog is set.
134
135- [ ] **Step 5: Run** `go test ./internal/plans/ ./internal/hostops/ ./internal/edge/ -run 'Marker|Classify|Fleet'`, `go test ./internal/console/ -run 'PlansMarker|AccountPlan|ClaimConsoleRouteCensus'`, `go test ./cmd/console/ -run TestEveryRegisteredClaimWorkerIsStarted` → PASS.
136
137- [ ] **Step 6: Commit** `console: publish the plans marker; plan writes need a live account`
138
139### Task 3: The box's plans pass and cache
140
141**Files:**
142- Create: `internal/host/plans_pass.go`, `internal/host/plans_pass_test.go`
143- Modify: `internal/host/agent.go` (call `plansPass` beside both `telemetryPass` calls, `agent.go:2809` and `:3395`), `internal/hostenv/hostenv.go` (`CARLOS_PLAN_LIMITS`)
144
145**Interfaces:**
146- Consumes: `plans.GetMarker`, `plans.GetLimits` (Task 1, Plan 1).
147- Produces:
148 ```go
149 func plansPass(ctx context.Context, s release.Store, reg *registry.Registry, getenv func(string) string)
150 func planLimitsFor(sqid string) (plans.Limits, bool) // false when plans are off or the account has no record
151 func planLimitsSwitchOn(getenv func(string) string) bool // false for off/false/0/no (case-insensitive)
152 ```
153 The cache is a package-level `atomic.Pointer` to an immutable snapshot `{byAccount map[string]plans.Limits}`; `nil` means off.
154
155- [ ] **Step 1: Write the failing tests** (serial: they touch the package cache), using `release.NewDirStore(t.TempDir())` wrapped in a counting store that records every `Get` key and can fail a key once, and a registry with routes for two accounts:
156
157```go
158func TestPlanLimitsSwitch(t *testing.T) // "", "on": true; "off", "OFF", "false", "0", "no": false
159func TestPlansPassOffReadsOnlyTheMarker(t *testing.T) // no marker: Gets == ["control/plans.json"]; planLimitsFor(any) false
160func TestPlansPassBoxSwitchOffReadsNothing(t *testing.T) // CARLOS_PLAN_LIMITS=off with an enabled marker and records: zero Gets; cache cleared
161func TestPlansPassLoadsServedAccounts(t *testing.T) // enabled marker, records for A and B: planLimitsFor(A).IdleSeconds == 90; an account with no routes on this box is not read
162func TestPlansPassAbsentRecordMeansNoLimits(t *testing.T) // enabled, no record for B: planLimitsFor(B) false
163func TestPlansPassKeepsLastGoodOnErrors(t *testing.T) // after a good pass: marker Get fails -> A and B unchanged; A's record Get fails -> A unchanged, B refreshed
164func TestPlansPassClearsWhenDisabled(t *testing.T) // enabled then marker Enabled false: planLimitsFor(A) false
165```
166
167- [ ] **Step 2: Run** `go test ./internal/host/ -run 'PlansPass|PlanLimitsSwitch'` → FAIL.
168
169- [ ] **Step 3: Implement** `plans_pass.go` after `telemetry_pass.go`'s shape: switch off → store nil, return. `GetMarker` error → log, return (keep). Absent or disabled → store nil, return. Otherwise collect distinct `release.RouteAccount(rt)` from `reg.List()` (skip status routes), `GetLimits` each: error → copy the previous value for that account if any; nil → omit; else set. Store the new snapshot. Log first failure and every tenth, as `homemap.go`'s loop does.
170
171- [ ] **Step 4: Wire** `plansPass(ctx, store, reg, os.Getenv)` after both `telemetryPass` calls; add `"CARLOS_PLAN_LIMITS"` to `hostenv.Optional` in sorted position.
172
173- [ ] **Step 5: Run** `go test ./internal/host/ -run 'PlansPass|PlanLimitsSwitch' ./internal/hostenv/` → PASS.
174
175- [ ] **Step 6: Commit** `host: read account limits each pass, marker-gated, with a box off switch`
176
177### Task 4: The activator's idle hook
178
179**Files:**
180- Modify: `internal/activator/activator.go` (`Service` gains `IdleFor`; the sweep at `:4646`)
181- Modify: `internal/activator/activator_test.go`
182- Modify: `internal/host/agent.go` (set `svc.IdleFor` after `activator.New`, `agent.go:490`)
183- Create: `internal/host/plans_idle_test.go`
184
185**Interfaces:**
186- Consumes: `planLimitsFor` (Task 3).
187- Produces:
188 ```go
189 // Service field, set before the service is published:
190 IdleFor func(registry.Route) time.Duration // 0 = no opinion; nil = today's behaviour
191 func planIdleFor(rt registry.Route) time.Duration // internal/host: Limits.IdleSeconds as a duration, 0 when none
192 ```
193
194- [ ] **Step 1: Write the failing tests**
195
196```go
197// activator_test.go, beside TestActivatorIdleSleep and TestWorkerRouteIdleWindowOverridesPlatformDefault
198func TestIdleForShortensAnInstance(t *testing.T) // cfg.Idle 15m, IdleFor returns 90s: the entry sleeps after 90s idle, not before
199func TestPlanIdleOnlyShortens(t *testing.T) // cfg.Idle 5m, IdleFor 15m: sleeps at 5m
200func TestIdleForShortensAWorkerToTheSmaller(t *testing.T) // worker IdleSeconds 600, IdleFor 90s: 90s; worker 60, IdleFor 90s: 60s
201func TestIdleForNilOrZeroChangesNothing(t *testing.T) // nil hook, and a hook returning 0: today's windows
202func TestIdleForLeavesAlwaysOnWorkersAlone(t *testing.T) // Hibernate=false worker with IdleFor 90s: never stopped for idleness
203// plans_idle_test.go (serial)
204func TestPlanIdleFor(t *testing.T) // cache off: 0; account with IdleSeconds 90: 90s; account absent: 0
205```
206
207- [ ] **Step 2: Run** `go test ./internal/activator/ -run IdleFor` and `go test ./internal/host/ -run TestPlanIdleFor` → FAIL.
208
209- [ ] **Step 3: Implement.** In the sweep, after the current `idleWindow` is chosen: `if f := a.IdleFor; f != nil { if d := f(e.route); d > 0 && d < idleWindow { idleWindow = d } }`. The "held for tick" check already uses `idleWindow`. `planIdleFor` reads `planLimitsFor(release.RouteAccount(rt))`. In `agent.go`, set `svc.IdleFor = planIdleFor` right after `activator.New`.
210
211- [ ] **Step 4: Run** the two commands above and `go test ./internal/activator/ -run 'Idle|Worker|AlwaysOn'` → PASS.
212
213- [ ] **Step 5: Commit** `activator: a plan can shorten an entry's idle window`
214
215### Task 5: Census, then the branch gates
216
217**Files:** `internal/provision/testdata/plan14-root-reads-plans-2-box-limits.tsv` (new), `internal/provision/plan14_root_reads_test.go`, possibly a record-write census layer beside `internal/provision/plans_caps_census_test.go`.
218
219- [ ] **Step 1:** Run `GOFLAGS=-buildvcs=false go test ./internal/provision/ -run 'TestPlan14RootReadCensus|TestPlan9B8'`. For each discrepancy, regenerate the overlay with the scratch-generator recipe used for Plan 1 (load every overlay, print `@legacy`/rows/`@coverage` for changed and new units, compute the seal), with basis text naming the executing identity: the box agent's plans pass reads two bucket objects and makes no root decision; the activator change narrows an idle window. Load it last; mark `rev3=PROPOSED-plans-2-box-limits`.
220- [ ] **Step 2:** Run the full claim invariant once: `nice -n 19 go test ./internal/console/ -run TestClaimConsoleMutationInvariant -timeout 30m` → ok.
221- [ ] **Step 3:** Commit `plans 2: census overlay`, merge current main, push, wait for CI green, ask for the census ruling on #280.
222
223## Done when
224
225- The commands in each task pass locally; CI is green on the tip after merging current main.
226- Review Focus 1–5 each have their named test passing.
227- The branch's final review (Astra, whole branch) has no open Critical or Important finding.
228