| index 28b191008..661b8a833 100644 |
| --- a/docs/superpowers/plans/2026-10-21-instance-policy-1-foundation.md |
| +++ b/docs/superpowers/plans/2026-10-21-instance-policy-1-foundation.md |
| @@ -17,9 +17,9 @@ |
| - New tests in `internal/host`, `internal/activator`, `internal/console`, `moveprobe` call `t.Parallel()` unless they set env, cwd, umask, log output or package vars, reach `edge.SecureUnixListen`, or listen on TCP (activator). No test depends on another having run. A new console test slower than a few seconds refreshes `.amadan/test-weights.tsv` with `.amadan/test-weights.sh`. |
| - Cross-compile: CI builds darwin and windows. Nothing in this plan may need `//go:build linux`; the new packages are pure Go. |
| - Root code: this plan changes packages in root's build graph (`internal/registry`, `internal/provision`, `internal/release`, `internal/cgroup`, and the new `internal/apppolicy`), so it ends with a census task (Task 14) in the #369 approval flow (AGENTS.md "The census: an approval, not a decision"): `make census-card`, a CLAIMS file, `make census-propose NAME=<branch> CLAIMS=<file>`, the card shown to Paul. The overlay is appended to `testdata/plan14-root-reads-chain.tsv`, so it loads after `ci-speed-activator`, `securelisten-337` and every overlay already chained. |
| -- Request budget (spec §3a, ruling 11): no reader polls the policy on its own timer. Steady state with nothing written, added per box per day: at most 48 LISTs (the backstop's `ListETags` of `control/app-policy/`) and 0 GETs; per console per day: 2 GETs per live policy (the re-sign) and no LIST of its own (plan 2's report owns the console's 48). Tasks 7 and 10 pin these with counting-store tests that fail on any increase. The writer lease's own requests go through `internal/lease`'s S3 client, not `release.Store`, and are stated in Task 8. |
| -- One policy writer per bucket (ruling 10): nothing signs a policy (door write or re-sign) without the writer lease held (Task 8). A second console refuses and names the holder. |
| -- Production-wired witnesses: the fleet-door read is tested through a real `edge.Edge` store door and the real `fleetapi` client; the door through the real console mux; reconcile through a real `DirStore` and registry. |
| +- Request budget (spec §3a, ruling 11): no reader polls the policy on its own timer. Steady state with nothing written, added per box per day: at most 48 LISTs (the backstop's `ListETags` of `control/app-policy/`) and 0 GETs; per console per day: 2 GETs per live policy (the re-sign) and no LIST of its own (plan 2's report owns the console's 48). Tasks 7 and 10 pin these with counting-store tests that fail on any increase; Task 7 also runs the box's day through the real store door (a fleet box lists the same way, Task 5). The writer lease's own requests go through `internal/lease`'s S3 client, not `release.Store`, and are stated in Task 8. |
| +- One policy writer per bucket (ruling 10): nothing signs a policy (door write or re-sign) without the writer lease held (Task 8). A second console refuses and names the holder. Every signing and every publication attempt, retries included, first rechecks that the same holding is live and inside its safe deadline (`policyAttempt`, Task 8); a lost holding aborts the write and never retries over another writer's content. |
| +- Production-wired witnesses: the fleet-door read and its daily request count are tested through a real `edge.Edge` store door and the real `fleetapi` client; the door through the real console mux; reconcile through a real `DirStore` and registry. |
| - User-facing strings (CLI output, door error messages a person reads, audit details shown in the activity tab, README prose) are collected in Task 15 for copy review before merge. Plain English, no em dashes. Internal error fragments, log lines and JSON keys are not copy. |
| - No tenant names in new code, tests or comments: `shop`, `shop-a.example.com`, `10.0.5.20`, `fd00:5::20`. (Existing fixtures that carry other names are reused as they are.) |
| - Commit per task, message ending `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. |
| @@ -39,7 +39,7 @@ |
| What was checked on main at `b06ce419`, and what this plan does about it: |
| |
| - **Bucket policy and IAM (`infra/`).** `infra/modules/deployment-buckets/main.tf` carries no Deny on `control/` beyond box evidence and fences. Writes come from IAM grants: the box role (`region-host/main.tf:166`, `region-remote-host/main.tf:85`, `s3:PutObject`/`DeleteObject` on `deployment_bucket/*`) and the console, which runs on a box. Object-store credentials an app receives are scoped to `store_bucket_prefix*` buckets (`region-host/main.tf:606-622`), never the deployment bucket. App tokens, instance tokens, deploy credentials (#229) and member sessions are console bearer tokens with no bucket credential. **So no app, build, instance or member credential can write `control/app-policy/` directly. Holds; no infra change.** |
| -- **Store door (`internal/edge/storedoor.go:529`).** `aclWrite` grants only `fleets/<fleet>/**` and `apps/<sqid>/<app>/{logs,errors,analytics}/**`. It never answers true for `control/`. **Holds;** Task 5 pins it with matrix rows. |
| +- **Store door (`internal/edge/storedoor.go:529`).** `aclWrite` grants only `fleets/<fleet>/**` and `apps/<sqid>/<app>/{logs,errors,analytics}/**`. It never answers true for `control/`. **Holds;** Task 5 pins it with matrix rows. Task 5 adds one read-only, account-scoped ETag listing of `control/app-policy/<sqid>/` (Astra plan R3-4); nothing it adds writes. |
| - **Console mutation boundary (`internal/hostops/mutation_boundary.go`, `mutation_classifier.go:100-160`).** With host claims on, `s.cfg.Releases` is the boundary, and an unreviewed key is refused (`ErrUnclassified`). `control/app-policy/` is unreviewed today, so **the console's own policy door would be refused under claims.** Task 6 classifies it as app-scoped (the `control/door-keys/` shape) for `Put` and `Delete` only. |
| - **The writer lease (Task 8).** `control/app-policy-lease/lock.json` in the deployment bucket, written by `internal/lease` through its own S3 client (the housekeeper's `control/housekeeper-lease/lock.json` precedent), so it never passes the console's mutation boundary and needs no classifier row, as the housekeeper's lock does not. The console's role already has `s3:GetObject`/`PutObject`/`DeleteObject` on `deployment_bucket/*` (`region-host/main.tf:166`). **One gap, stated:** where `store_bucket_prefix` is set (flagship), the store policy's explicit Deny of `s3:DeleteObject` on the deployment bucket (`region-host/main.tf:669`, the 2026-09-09 lease outage; hand-fixed on the live role, still on main) refuses the lease's release, so a restarted console waits out `lease.LeaseTTL` (15 minutes) before it may write policies again. No infra change in this plan; it is raised with the spec author. |
| - **Console doors.** After this plan, only four call sites touch the key: the operator door and the re-sign pass (writes, both in `apipolicy.go`), the purge sweep (delete) and app create (delete). Task 6 adds a source test that keeps it so. |
| @@ -54,10 +54,10 @@ Everything in the brief's "Cross-plan interfaces", plus: |
| - The signed record (Task 3; the brief pins `Sign`, `Verify`, `VerifyRevision`, `Lifetime`, `Revision`, `SignedAt`): `apppolicy.ErrInvalid`, `ErrUnverifiable`, `ErrExpired`; `MaxClockSkew = 5 * time.Minute`; `func VerifyKey(pub ed25519.PublicKey, wire []byte, now time.Time, sqid, app string) (Policy, error)` (`Verify` plus "the record names this key's account and app"; the agent's reads go through it via `Get`; root calls nothing here in this build, ruling 9); `func (p Policy) Expired(now time.Time) bool`; `func GetForUpdate(ctx, s, pub, sqid, app) (*Policy, []byte, string, error)` (console only: signed records whether expired or not, with their bytes); `func OpenKey(pub ed25519.PublicKey, wire []byte, sqid, app string) (Policy, error)` (console only: `VerifyKey` without the expiry, for bytes the console recorded); `func PutWire(ctx context.Context, s release.Store, sqid, app string, wire []byte, etag string) (string, error)` (console only: writes bytes already signed and recorded); `apppolicy.PutIfMatchETag(...) (string, error)`; `apppolicy.Present(ctx, s) (map[string]bool, error)`; `apppolicy.ListETags(ctx, s) (map[string]string, error)` (key `<sqid>/<app>` to the object's ETag, one listing; `errors.ErrUnsupported` from a store that is not a `release.ETagLister`); `(*Policy).SizeFor(host string, createdAt time.Time) string`; `(*Policy).EgressColumn() string`. |
| - `release.ChangeScope("control/app-policy/<sqid>/<app>.json") == "apps/<sqid>/<app>", true` (Task 3, spec §3a): every policy write and delete bumps the app's generation through the concrete stores' existing hook. |
| - `provision.PolicyCache` (Task 7): the agent's long-lived policy reader, `NewPolicyCache(watch *release.ChangeWatch) *PolicyCache`, set as `ReconcileConfig.Policies`. Plans 2 and 3 never read the store for policy; they read the columns. |
| -- The writer lease (Task 8): `console.PolicyLeaser` and `Config.PolicyLease`; `const appPolicyLeasePrefix = "control/app-policy-lease"`; `func (s *Server) policyWriter(ctx context.Context) (policyHolding, *doorRefusal)`; `func (s *Server) ReleasePolicyWriter(ctx context.Context)`. Plan 2's report runs only while `s.policyHeld()` is true. |
| +- The writer lease (Task 8): `console.PolicyLeaser` and `Config.PolicyLease`; `const appPolicyLeasePrefix = "control/app-policy-lease"`; `func (s *Server) policyWriter(ctx context.Context) (policyHolding, *doorRefusal)`; `func (s *Server) policyAttempt(ctx context.Context, h policyHolding) (context.Context, context.CancelFunc, *doorRefusal)`; `func (s *Server) ReleasePolicyWriter(ctx context.Context)`. Plan 2's report runs only while `s.policyHeld()` is true. |
| - `func (s *Store) AppPolicyRowsContext(ctx context.Context) ([]AppPolicyRow, error)` (Task 9): every recorded policy with its `synced_etag`; plan 2's report reads the live allows from it, not from the bucket. |
| - `egressaddr.MaxAllow = 8`; `cgroup.SizeNames() []string`. |
| -- Store door: a 404 for a granted, absent policy key carries `Carlos-Store-Absent: 1`. |
| +- Store door: a 404 for a granted, absent policy key carries `Carlos-Store-Absent: 1`. `GET /store/etags?prefix=` (Task 5) answers the box's own `control/app-policy/<sqid>/` keys with their ETags, and `fleetapi`'s store is a `release.ETagLister` over it, so a fleet box's backstop is one LIST and GETs only changed policies. |
| - Console: `func (s *Server) checkEgressGrant(ctx context.Context, acct *Account, app string, old, next []string) (notChecked []string, status int, msg string)` in `internal/console/apipolicy.go`. `old` is the stored list, `next` the normalized new one. **Plan 1's body is a stub returning `[]string{}, 0, ""`. Plan 2 fills it** (the fleet-address check, §4.2, on `next` minus `old`). It is called whenever the request carries `egress_allow`; a non-zero `status` refuses with `msg` before the audit. The reply's `not_checked` is its first return. |
| - Plan 3 inserts its capacity check (§5.5) in `handleAPIPolicyPut` after size validation and before `checkEgressGrant`. There is no below-tier CLI warning (spec §5.2 rev 8 drops it; the console cannot see a build's `children` bit), and no plan implements one. |
| - `provision.Reconcile` writes `routes.size` = the configured size only (`''` = nothing configured = the build's tier). Resolving `''` against the build is plan 3's. |
| @@ -413,18 +413,21 @@ func TestRemoteSetPolicy(t *testing.T) { |
| |
| --- |
| |
| -### Task 5: The store door's read grant, and refusal is not absence |
| +### Task 5: The store door's read grant and ETag listing, and refusal is not absence |
| |
| **Files:** |
| -- Modify: `internal/edge/storedoor.go`: `aclRead` (431-445) gains `appPolicyRead(sqid, key)`; `serveObject`'s GET not-found branch (157-162) adds the header; doc comment of `aclRead` gains the grant's paragraph |
| -- Modify: `internal/fleetapi/fleetapi.go`: `Get` (128-152) and `Open` (157-177) |
| +- Modify: `internal/edge/storedoor.go`: `aclRead` (432-446) gains `appPolicyRead(sqid, key)`; `serveObject`'s GET not-found branch (159-164) adds the header; doc comment of `aclRead` gains the grant's paragraph; `ServeHTTP`'s path switch (130-139) gains `storeETagsPath`; new `serveETagListing` and `aclETagListScope` beside `serveListing` (236) and `aclListScope` (585) |
| +- Modify: `internal/fleetapi/fleetapi.go`: `Get` (133-158) and `Open` (160-195); new `ListETags` beside `List` (254) |
| - Test: `internal/edge/storedoor_test.go` (rows in `TestStoreDoorACLMatrix`, one new test); `internal/fleetapi/fleetapi_test.go` (new tests) |
| |
| **Interfaces:** |
| - Consumes: `release.AppPolicyPrefix`, `release.AppPolicyKey`, `release.ErrRefused`, `release.ValidApp`. |
| - Produces: header `Carlos-Store-Absent: 1` on a 404 for a key `appPolicyRead` granted and the store reports absent. `fleetapi` (`Get` and `Open`), for a key under `release.AppPolicyPrefix`: 200 as today; 404 **with** the header → `os.ErrNotExist`; 404 **without** it → an error wrapping `release.ErrRefused` (a refusal, an older edge, a bad token). Every other key is unchanged. |
| +- Produces (Astra plan R3-4, spec §3a): `const storeETagsPath = "/store/etags"`; `func (d *storeDoor) serveETagListing(w http.ResponseWriter, r *http.Request, sqid, fleet string)`; `func aclETagListScope(sqid, prefix string) (string, bool)`; `func (s *store) ListETags(ctx context.Context, prefix string) ([]release.KeyETag, error)` in `fleetapi`, with `var _ release.ETagLister = (*store)(nil)`. Wire: `GET /store/etags?prefix=<p>` → 200 `[{"key":"…","etag":"…"}]` sorted by key, `[]` when empty, never null. |
| |
| -`appPolicyRead(sqid, key)`: exactly `release.AppPolicyKey(sqid, app)` for some `app` with `release.ValidApp(app)`, i.e. three segments under `control/app-policy/`, the box's own sqid. Not listable (`aclDir`, `aclListScope` unchanged). `aclWrite` unchanged. |
| +`appPolicyRead(sqid, key)`: exactly `release.AppPolicyKey(sqid, app)` for some `app` with `release.ValidApp(app)`, i.e. three segments under `control/app-policy/`, the box's own sqid. `aclDir` and `aclListScope` are unchanged, so `/store/list` and `/store/dirs` still refuse every `control/` prefix. `aclWrite` unchanged. |
| + |
| +The ETag listing is the one new listing, and it is read-only and policy-only: `aclETagListScope` maps exactly `control/app-policy/` and `control/app-policy/<sqid>/` (the box's own sqid) to `control/app-policy/<sqid>/` and refuses every other prefix (another account, a sqid that only prefix-matches, `control/`, `apps/`, empty), so `apppolicy.ListETags`' unmodified `Prefix` works through the door the way `apps/` maps for `List`. `serveETagListing` answers only `GET` (anything else is the bare 404), calls `d.store.(release.ETagLister).ListETags(ctx, scope)` once (a backing store without it: the bare 404, as an older door answers), keeps only entries `appPolicyRead(sqid, key)` grants, and sends a store failure through `storeFailed` (502, never an empty listing). ETags pass through as the backing store produced them, as `Get`'s do. `fleetapi.ListETags`: 200 decodes the list; 404 → an error wrapping `release.ErrRefused` (a listing has no "absent"; an older edge has no such path); any other status → a plain error. |
| |
| - [ ] **Step 1: Write the failing tests** |
| |
| @@ -464,13 +467,35 @@ func TestPolicyRefusalFromAnOldDoorIsNotAbsence(t *testing.T) { |
| // Get("control/app-policy/ab12/web.json") -> errors.Is(err, release.ErrRefused), not os.ErrNotExist |
| // Get("apps/ab12/web/x.json") -> errors.Is(err, os.ErrNotExist) // unchanged for other keys |
| // Open of the policy key behaves as Get |
| + // ListETags(ctx, "control/app-policy/") -> errors.Is(err, release.ErrRefused) |
| +} |
| +func TestPolicyETagsThroughTheDoor(t *testing.T) { |
| + // policies seeded for testSqid/web, testSqid/api and zz99/web: |
| + // apppolicy.ListETags(ctx, d.box(t)) -> exactly {testSqid+"/web": <etag Get returns>, testSqid+"/api": <etag>} |
| + // d.box(t) satisfies release.ETagLister (no fallback is taken) |
| +} |
| +``` |
| + |
| +`storedoor_test.go`: |
| + |
| +```go |
| +func TestStoreDoorListsOnlyItsOwnPolicyETags(t *testing.T) { |
| + // seeded: control/app-policy/<testSqid>/web.json, .../<testSqid>/api.json, control/app-policy/zz99/web.json, |
| + // control/app-policy/<testSqid>/x.txt |
| + // GET /store/etags?prefix=control/app-policy/ and ?prefix=control/app-policy/<testSqid>/ -> exactly the two |
| + // own .json keys, each with the ETag a GET of it returns; the backing store saw exactly one ListETags each |
| + // bare 404, byte-identical to TestStoreDoorRefusalsAreByteIdentical's: prefixes control/app-policy/zz99/, |
| + // control/app-policy/<testSqid>x/, control/, apps/<testSqid>/, ""; PUT and DELETE of /store/etags; |
| + // an unauthenticated GET |
| + // /store/list?prefix=control/app-policy/ -> still the bare 404 (plain listing unchanged) |
| + // a backing store failing ListETags -> 502, not [] |
| } |
| ``` |
| |
| -- [ ] **Step 2: Run, fail:** `go test ./internal/edge -run 'TestStoreDoorACLMatrix|TestStoreDoorMarksOnlyAGrantedAbsentPolicy|TestStoreDoorRefusalsAreByteIdentical' && go test ./internal/fleetapi -run 'TestPolicy'`. |
| -- [ ] **Step 3: Implement.** In `fleetapi`, a helper `notFound(key string, h http.Header) error` used by both `Get` and `Open`; its comment says why this one prefix differs from the uniform 404 mapping (Astra R3-3: a refusal read as absence is "no policy"). Note: `fleetapi` imports only `release`, not `apppolicy`. |
| +- [ ] **Step 2: Run, fail:** `go test ./internal/edge -run 'TestStoreDoorACLMatrix|TestStoreDoorMarksOnlyAGrantedAbsentPolicy|TestStoreDoorRefusalsAreByteIdentical|TestStoreDoorListsOnlyItsOwnPolicyETags' && go test ./internal/fleetapi -run 'TestPolicy'`. |
| +- [ ] **Step 3: Implement.** In `fleetapi`, a helper `notFound(key string, h http.Header) error` used by both `Get` and `Open`; its comment says why this one prefix differs from the uniform 404 mapping (Astra R3-3: a refusal read as absence is "no policy"). Note: `fleetapi` imports only `release`, not `apppolicy`. Comment on `serveETagListing`: spec §3a (a fleet box's backstop is one LIST, GETs only for changed ETags, Astra plan R3-4), why it is policy-only and read-only, and that entries are re-checked per key as `serveListing` does. |
| - [ ] **Step 4: Run, pass.** Same commands. |
| -- [ ] **Step 5: Commit** `storedoor, fleetapi: own-account read of the app policy; refusal is not absence (#328)`. |
| +- [ ] **Step 5: Commit** `storedoor, fleetapi: own-account read and ETag listing of the app policy; refusal is not absence (#328)`. |
| |
| --- |
| |
| @@ -525,10 +550,10 @@ func TestPolicyWritersAreTheDoorThePurgeAndCreate(t *testing.T) { |
| - Modify: `internal/provision/seedswap.go:150-161` (`swapImport` carries the columns on the import's row) |
| - Modify: `internal/host/agent.go` (the callers of `provisionTickConfig`, line 3270, which builds `provision.ReconcileConfig`: `cfg.Policies` set to one long-lived cache built after the change index is armed, over `armedChangeWatch.Load()`; `provisionTickConfig` itself keeps its `getenv`-only signature) |
| - Create: `internal/provision/apppolicy_cache.go` (`PolicyCache`: the gated reads and the backstop), `internal/provision/apppolicy_reconcile.go` (the verification, the column derivation and `PolicyReadStates`) |
| -- Test: `internal/provision/apppolicy_reconcile_test.go`, `internal/provision/apppolicy_cache_test.go` |
| +- Test: `internal/provision/apppolicy_reconcile_test.go`, `internal/provision/apppolicy_cache_test.go`, `internal/edge/fleetapi_e2e_test.go` (the fleet-door budget witness, beside `TestFleetBoxConvergesWithoutBucketCredentials`) |
| |
| **Interfaces:** |
| -- Consumes: Task 3 (`apppolicy.ListETags`, `Get`, `ErrUnverifiable`, `ErrExpired`, `SizeFor`, `EgressColumn`, `release.ErrRefused`, the `ChangeScope` row), Task 4 (`SetPolicy`, the three `Route` fields), `apppolicy.Lifetime`, `cfg.ReleasePub` (the box's `CARLOS_RELEASE_PUBKEY`, already in `ReconcileConfig`), `release.ChangeWatch.Gen`, `release.ChangeIndexBackstop`. |
| +- Consumes: Task 5 (`fleetapi`'s `ListETags`, the door's `/store/etags`), Task 3 (`apppolicy.ListETags`, `Get`, `ErrUnverifiable`, `ErrExpired`, `SizeFor`, `EgressColumn`, `release.ErrRefused`, the `ChangeScope` row), Task 4 (`SetPolicy`, the three `Route` fields), `apppolicy.Lifetime`, `cfg.ReleasePub` (the box's `CARLOS_RELEASE_PUBKEY`, already in `ReconcileConfig`), `release.ChangeWatch.Gen`, `release.ChangeWatch.Observed`, `release.ChangeIndexBackstop`. |
| - Produces: |
| - `ReconcileConfig.PolicyChanged func(host string)` (cross-plan: plan 3 sets it to the activator's `NoteSize`) |
| - `ReconcileConfig.Policies *PolicyCache`; `func NewPolicyCache(watch *release.ChangeWatch) *PolicyCache` (nil watch: the index is disarmed or this is a fleet box); `func (c *PolicyCache) read(ctx context.Context, s release.Store, pub ed25519.PublicKey, apps [][2]string, now time.Time) map[[2]string]policyRead`. A nil `Policies` (tests, `carlos` tools) builds a throwaway cache per pass, which therefore runs the backstop every pass: correct, never cheap; production always sets it. |
| @@ -540,11 +565,12 @@ func TestPolicyWritersAreTheDoorThePurgeAndCreate(t *testing.T) { |
| |
| Rules (decisions; spec §4.3 "before claims" and §3a): |
| - **Which apps.** Only apps with at least one in-scope instance record this pass (the ones this box mints or keeps rows for). Other apps' policies are never read. |
| -- **No per-pass read** (ruling 11). The cache keeps, per app, the ETag, the `policyRead` and the generation of `apps/<sqid>/<app>` taken before that read. A pass reads the store for an app only when: |
| - 1. **the backstop is due**: at the cache's first pass, then once `release.ChangeIndexBackstop + rand.N(release.ChangeIndexBackstop/10)` has passed since the last successful backstop (30 to 33 minutes, jittered upward only, so a day holds at most 48), or the last backstop failed. The backstop is one `apppolicy.ListETags`, then a `Get` only for each of the pass's apps whose listed ETag differs from the cached one or which has no cache entry; an app the listing does not name is known absent with no read. When `ListETags` fails or is unsupported (a fleet-door box: the door neither lists `control/` nor carries ETags), the backstop is one `Get` per pass app instead. Fleet boxes have no change index (#336 spec §3.4), so this is their only read: one GET per hosted app per backstop, through the console's door; |
| +- **No per-pass read** (ruling 11). The cache keeps, per app, the ETag, the `policyRead` and the generation of `apps/<sqid>/<app>` taken before that read, plus the last backstop's listing and `listSnap`, the `watch.Observed()` map taken immediately before that listing (nil when the watch is nil or unusable). A pass reads the store for an app only when: |
| + 1. **the backstop is due**: at the cache's first pass, then once `release.ChangeIndexBackstop + rand.N(release.ChangeIndexBackstop/10)` has passed since the last completed backstop (30 to 33 minutes, jittered upward only, so a day holds at most 48), or the last backstop failed. The backstop is `listSnap = watch.Observed()`, then one `apppolicy.ListETags`, then a `Get` only for each of the pass's apps whose listed ETag differs from the cached one or which has no cache entry; a pass app the listing does not name is known absent with no read, its entry's generation the `listSnap` value. Every box lists this way: a fleet-door box through the door's ETag listing (Task 5), so its day is the same 48 LISTs and GETs only for changed policies (Astra plan R3-4). A listing that fails with `release.ErrRefused` (an edge older than Task 5) makes every pass app `policyRefused` and **completes** the backstop (the next is 30 to 33 minutes on), so an old edge costs one LIST per backstop, never one per pass; any other listing error is a failed backstop. Only a store with no `ETagLister` at all (`errors.ErrUnsupported`: test fakes; no production store, since `DirStore`, `S3Store`, the prefix, boundary and record-view wrappers and now `fleetapi` all list) falls back to one `Get` per pass app per backstop. Fleet boxes have no change index (#336 spec §3.4), so the backstop is their only periodic read; |
| 2. **the app's generation moved**: the watch is usable, the cache has an entry, and `watch.Gen("apps/<sqid>/<app>")` differs from the entry's: one `Get` of that key. A watch that is nil or unusable reads nothing between backstops: ruling 11 has no timer, so a disarmed box converges at the backstop; |
| 3. **the app's last read failed** (`policyTransient`): one `Get` on the next pass, until it succeeds. |
| - An app new to the cache between backstops is known absent when the last backstop listing succeeded and did not name it (a later creation moves its generation); otherwise it gets one `Get`. Reads run in the existing prefetch `forEach`, eight at a time. The cached verified policy's expiry is re-checked against `now` on every pass with no read (`policyExpired` once it passes). |
| + 4. **an app new to the cache between backstops** (Astra plan R3-2): it reuses the last listing's absence only when that listing succeeded and did not name the app, `listSnap` is not nil, the watch is usable now and `watch.Gen("apps/<sqid>/<app>")` equals `listSnap["apps/<sqid>/<app>"]` (a missing entry is 0), i.e. nothing in the app's scope, its policy included, was written after the snapshot. Otherwise (a policy or a record written after the listing, a nil or unusable watch, a fleet box) it gets one `Get` on first encounter, its entry's generation the one taken before that `Get`. An app the listing named gets that one `Get` too. |
| + Reads run in the existing prefetch `forEach`, eight at a time. The cached verified policy's expiry is re-checked against `now` on every pass with no read (`policyExpired` once it passes). |
| - **Verify before converging** (spec §3): every read is `apppolicy.Get(ctx, s, cfg.ReleasePub, sqid, app, cfg.Now())`, so a column is only ever written from a record the release key signed for this account and app and that has not expired. A nil `ReleasePub` makes every present policy `policyUnverifiable`. |
| - **States to columns:** `policyKnown` → `{p.EgressColumn(), p.SizeFor(rec.Host, rec.CreatedAt), p.SignedAt.Add(apppolicy.Lifetime).Unix(), true}` (nil `p` gives `{"", "", 0, true}`). `policyRefused`, `policyUnverifiable` and `policyExpired` → `{"", "", 0, true}`: fail closed (no allow; sizes at the build's tier from the next start, which plan 3 never applies as a live memory shrink), log `provision: %s/%s: policy <word>: %v` once per app per pass with `<word>` `unreadable`, `unverifiable` or `expired`, and record the word in `PolicyReadStates`. `policyTransient` → `Known: false`: an existing row keeps its columns **while its `PolicyExpiresAt` is in the future**; a row whose recorded expiry has passed (or is `0` with a non-empty column) is cleared as `policyExpired` is (log and state `expired`), so a store outage longer than the lifetime closes the allow even if root never reads the store (Astra plan R1-3; root enforces the same expiry itself, plan 2 Task 5). A new row is minted with `''`/`0` (no allow, the build's tier) and converges on the next readable pass. A mint never waits for the policy (unlike the update policy). |
| - **Scope:** instance rows from `Reconcile` only. Sidecar members (`Kind "sidecar"` rows from `ReconcileSidecars`, `*.sidecar.internal`) get no policy: the columns stay `''` and `0` (spec §4.3 rev 8; plan 2 counts any such row as making the box not wholly nested, so no allow is written on that box at all). `app_id` is not checked on this path (no store-visible incarnation before claims; spec §3). |
| @@ -603,15 +629,31 @@ func TestPolicyReadsKeepTheRequestBudget(t *testing.T) { |
| // ListETags <= 48, Get == 0, List == 0 |
| // write app 1's policy (PutIfMatch, which bumps apps/<sqid>/app1); one pass: |
| // exactly 1 Get, of app 1's key; 0 Gets of app 2's; the row converged |
| - // a fourth app with records and no policy appears between backstops: 0 Gets for it |
| + // a fourth app whose records predate the last listing enters scope between backstops |
| + // (its generation unchanged since the snapshot): 0 Gets for it |
| + // a fifth app whose records are written after the listing: exactly 1 Get on its first pass, then 0 |
| +} |
| +// Astra plan R3-2: the exact ordering. Fails if a new app reuses the listing's absence after its scope moved. |
| +func TestAPolicyWrittenBeforeItsAppEntersScopeIsReadOnFirstEncounter(t *testing.T) { |
| + // watch armed; pass 1 (the backstop) hosts app 1 only and lists control/app-policy/ empty |
| + // then, before the next backstop: PutIfMatch app 5's policy {DefaultSize:"large", EgressAllow:["10.0.5.20"]} |
| + // and CreateRecord an in-scope instance of app 5; watch polled |
| + // the next pass (no backstop due): exactly 1 Get, of app 5's key; 0 ListETags; app 5's row |
| + // Size "large", EgressAllow "10.0.5.20" (the old rule left both "" until the backstop) |
| + // the same ordering with NewPolicyCache(nil) (a fleet or disarmed box): the same 1 Get and the same row |
| } |
| func TestPolicyReadsWithTheIndexDisarmedWaitForTheBackstop(t *testing.T) { |
| // NewPolicyCache(nil): a policy written after pass 1 is not read by passes inside the next |
| // 30 minutes (0 Gets); the pass after the backstop is due reads it (1 ListETags, 1 Get) and converges |
| } |
| func TestPolicyReadsFallBackToOneGetPerAppWithoutAListing(t *testing.T) { |
| - // a store hiding ETagLister (the fleet door's shape): a backstop pass makes one Get per pass app |
| - // and no List; between backstops, with no watch, 0 requests |
| + // a store hiding ETagLister (a test fake's shape; no production store): a backstop pass makes one |
| + // Get per pass app and no List; between backstops, with no watch and no new app, 0 requests |
| +} |
| +func TestARefusedListingIsUnreadableAndWaitsForTheNextBackstop(t *testing.T) { |
| + // ListETags returns fmt.Errorf("x: %w", release.ErrRefused) (an edge older than Task 5): |
| + // the backstop pass: 1 ListETags, 0 Gets, every pass app's row "" "" 0, state "policy unreadable"; |
| + // the passes inside the next 30 minutes: 0 requests |
| } |
| func TestPolicyCacheExpiresWithoutAReading(t *testing.T) { |
| // a cached verified record; the clock past SignedAt+Lifetime with no backstop due and no |
| @@ -619,9 +661,25 @@ func TestPolicyCacheExpiresWithoutAReading(t *testing.T) { |
| } |
| ``` |
| |
| -- [ ] **Step 2: Run, fail:** `go test ./internal/provision -run 'TestReconcileCarriesThePolicy|TestReconcileVerifiesBeforeConverging|TestReconcileGivesSidecarMembersNoPolicy|TestReconcileIgnoresAnOverride|TestReconcilePolicyReadFailures|TestReconcileDryRunWritesNoPolicy|TestPolicyReads|TestPolicyCache'`. |
| -- [ ] **Step 3: Implement** as the rules above. The `RouteFor` signature does not change: the mint sites set `nr.EgressAllow, nr.Size` from `policyColsFor` after calling it, so the adoption comparisons (`shapeMatches`) are untouched. In the agent, build the cache once, after `armChangeIndex` has run (so it sees the armed watch, or nil), and hand the same pointer to every pass's config. Comment on `PolicyCache`: ruling 11 and §3a, why a disarmed box waits for the backstop instead of polling, and that the generation is taken before the read (a write landing during it stays due). |
| -- [ ] **Step 4: Run, pass**, then the neighbours: `go test ./internal/provision -run 'TestReconcile|TestUpdates|TestSeed|TestHold|TestEverywhere'` and `go test ./internal/host -run 'TestProvisionTickConfig|TestChangeIndex'`. |
| +`internal/edge/fleetapi_e2e_test.go` (the real door and the real `fleetapi` client, Astra plan R3-4): |
| + |
| +```go |
| +// The §3a box budget through the store door, counted where the door reaches the backing store. |
| +func TestFleetBoxPolicyReadsKeepTheRequestBudget(t *testing.T) { |
| + // newFleetDoorWorld with EnableStatic handed a counting wrapper over w.store (forwarding |
| + // release.ETagLister) that counts List/ListETags under control/app-policy/ and Gets of keys under it. |
| + // box A hosts three apps with records; policies for two, signed by the world's release key. |
| + // NewPolicyCache(nil) (fleet boxes have no change index); cfg.Now a fake clock; ReleasePub the world's pub. |
| + // pass 1: 1 ListETags, 2 Gets (the two policies), 0 for the third; both rows converged |
| + // reset; 288 passes 5 minutes apart, nothing written: ListETags <= 48, Get == 0, List == 0 |
| + // reset; another 288 passes with both policies re-signed (same revision, later SignedAt) at hours 6 and 18: |
| + // ListETags <= 48, Get == 4 (2 per live policy), 0 Gets of the third app's key, List == 0 |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run, fail:** `go test ./internal/provision -run 'TestReconcileCarriesThePolicy|TestReconcileVerifiesBeforeConverging|TestReconcileGivesSidecarMembersNoPolicy|TestReconcileIgnoresAnOverride|TestReconcilePolicyReadFailures|TestReconcileDryRunWritesNoPolicy|TestPolicyReads|TestPolicyCache|TestAPolicyWrittenBeforeItsAppEntersScope|TestARefusedListing' && go test ./internal/edge -run TestFleetBoxPolicyReadsKeepTheRequestBudget`. |
| +- [ ] **Step 3: Implement** as the rules above. The `RouteFor` signature does not change: the mint sites set `nr.EgressAllow, nr.Size` from `policyColsFor` after calling it, so the adoption comparisons (`shapeMatches`) are untouched. In the agent, build the cache once, after `armChangeIndex` has run (so it sees the armed watch, or nil), and hand the same pointer to every pass's config. Comment on `PolicyCache`: ruling 11 and §3a, why a disarmed box waits for the backstop instead of polling, that the generation is taken before the read (a write landing during it stays due), and why a new app reuses the listing's absence only while its scope is unmoved since the snapshot taken before that listing (Astra plan R3-2). |
| +- [ ] **Step 4: Run, pass** (the Step 2 commands), then the neighbours: `go test ./internal/provision -run 'TestReconcile|TestUpdates|TestSeed|TestHold|TestEverywhere'`, `go test ./internal/host -run 'TestProvisionTickConfig|TestChangeIndex'` and `go test ./internal/edge -run TestFleetBox`. |
| - [ ] **Step 5: Commit** `provision: converge the app policy into the registry (#328, #329)`. |
| |
| --- |
| @@ -634,21 +692,26 @@ func TestPolicyCacheExpiresWithoutAReading(t *testing.T) { |
| - Modify: `cmd/console/main.go` (the `console.Config` literal at ~398: `PolicyLease: policyLeaseFor(os.Getenv)`; a signal handler started after the server is built), `cmd/console/main_test.go` |
| |
| **Interfaces:** |
| -- Consumes: `lease.AcquireTTL(ctx, bucket, prefix, region string, ttl time.Duration) (*lease.Held, error)`, `*lease.HeldError{Owner string; ExpiresAt time.Time}`, `lease.LeaseTTL`, `(*lease.Held).Err`, `(*lease.Held).Release`; `s.cfg.Bucket` (the deployment bucket's name, `CARLOS_DEPLOYMENT_BUCKET`). |
| +- Consumes: `lease.AcquireTTL(ctx, bucket, prefix, region string, ttl time.Duration) (*lease.Held, error)`, `*lease.HeldError{Owner string; ExpiresAt time.Time}`, `lease.LeaseTTL`, `(*lease.Held).Err`, `(*lease.Held).Release`, `(*lease.Held).Lost() <-chan struct{}` and `(*lease.Held).SafeUntil() (time.Time, error)` (`lease.go:234-251`: the expiry less `lease.RenewMargin`, 5 s; "a worker must recheck Err/SafeUntil"); `s.cfg.Bucket` (the deployment bucket's name, `CARLOS_DEPLOYMENT_BUCKET`). |
| - Produces: |
| |
| ```go |
| const appPolicyLeasePrefix = "control/app-policy-lease" // the lock is <prefix>/lock.json |
| -type PolicyLease interface { |
| +type PolicyLease interface { // *lease.Held satisfies it |
| Err() error |
| Release(ctx context.Context) error |
| + Lost() <-chan struct{} |
| + SafeUntil() (time.Time, error) |
| } |
| type PolicyLeaser func(ctx context.Context, bucket, prefix string, ttl time.Duration) (PolicyLease, error) |
| // lease.AcquireTTL(ctx, bucket, prefix, "", ttl), returning a nil interface on error |
| // (the typed-nil guard acquireDeploymentLease has, internal/host/housekeeper.go:121). |
| func ProductionPolicyLeaser(ctx context.Context, bucket, prefix string, ttl time.Duration) (PolicyLease, error) |
| -type policyHolding struct { n uint64; sole bool } |
| +type policyHolding struct { n uint64; sole bool; lease PolicyLease } // lease nil when sole |
| func (s *Server) policyWriter(ctx context.Context) (policyHolding, *doorRefusal) |
| +// Astra plan R3-1: called before every signing and every publication attempt. |
| +const policyAttemptWindow = 10 * time.Second |
| +func (s *Server) policyAttempt(ctx context.Context, h policyHolding) (context.Context, context.CancelFunc, *doorRefusal) |
| func (s *Server) policyHeld() bool // non-blocking: a holding exists and its Err() is nil (sole: always) |
| func (s *Server) ReleasePolicyWriter(ctx context.Context) |
| // cmd/console |
| @@ -660,14 +723,15 @@ Why this mechanism: it is the platform's one election, `internal/lease` (litestr |
| Rules: |
| - **Modes.** `cfg.Bucket == ""` (a `CARLOS_DEPLOYMENT_DIR` deployment) is **sole**, the `housekeeperSole` precedent: a directory store is one console by construction, so `policyWriter` returns holding `{1, true}` and never refuses. `cfg.Bucket != ""` with `cfg.PolicyLease == nil` refuses every call with 503 `policyLeaseMissingMsg` (fail closed: a console that lost the wiring signs nothing). Otherwise the lease. |
| - **Lazy, then long-lived.** Nothing is acquired at start. The first `policyWriter` call (a door PUT, Task 9; a re-sign pass with something to sign, Task 10) acquires `appPolicyLeasePrefix` in `cfg.Bucket` with TTL `cfg.PolicyLeaseTTL` (`0` = `lease.LeaseTTL`, 15 minutes); `internal/lease` renews it every TTL/3 for as long as it is held. Later calls return the same holding while its `Err()` is nil. Once `Err()` is non-nil (a renewal lost), the holding ends (`Release` best-effort, log `console: policy writer lease lost: %v`) and that call makes one fresh attempt. Each successful acquisition is a new holding, `n` one above the last. `s.policyLeaseMu` serializes acquisition; it is never held across a policy read. |
| +- **Every write is bound to its holding** (Astra plan R3-1). A caller keeps the `policyHolding` its `policyWriter` call returned and never asks for a new one mid-write. `policyAttempt(ctx, h)` is the check before each signing and each publication attempt: sole → `ctx` unchanged, a no-op cancel, no refusal. Otherwise it refuses with 503 `policyLeaseLostMsg` unless `h` is still the server's current holding (`h.n` equals it: a later holding does not revive an earlier one), `h.lease.Err()` is nil, `h.lease.SafeUntil()` succeeds and `safe` is at least `policyAttemptWindow` away. On success it returns a context that ends at the earlier of `now + policyAttemptWindow` and `safe`, and as soon as `h.lease.Lost()` closes. A refusal for a lost lease also ends the holding as `policyWriter` does. An attempt cut off by that context counts as the same refusal, never as a store failure. Bound, stated: `safe` is 5 s before the lease's expiry and no other console can acquire until that expiry, so an attempt either reaches the store before another console can hold the lease or is cancelled; a request the network holds back past its own cancellation is the residual, and a later holder's CAS write then sees a changed ETag (Task 9 step 11). |
| - **Refusal names the holder.** `*lease.HeldError` (via `errors.As`) → 503 `policyLeaseHeldMsg` with the lock's `Owner` (litestream's `<hostname>:<pid>`) and `ExpiresAt` as RFC 3339 UTC. Any other acquire error → 503 `policyLeaseUnavailableMsg`, logged with the error. |
| - **Release.** `ReleasePolicyWriter` releases the held lease within its own 5 s budget and ends the holding. `cmd/console` calls it from a `signal.Notify(SIGTERM, os.Interrupt)` handler, then `os.Exit(0)`, so a rolled console's successor does not wait out the TTL (the housekeeper's "graceful stand-down must be reachable"). Where the role's `DeleteObject` is denied (Boundary verification), the release fails, is logged, and the successor waits up to 15 minutes. |
| - **Cost** (outside `release.Store`; stated, not counted by the §3a tests): nothing on a console that never signs a policy. While held, one conditional PUT per renewal: 288 PUTs a day for the one writer console, independent of apps and boxes. A refused console tries again only on its next need (an operator write or its 12-hour re-sign pass), never on a timer. |
| - **What a holding means to the content.** A new holding is "another console may have written since I last did". Task 9's base rules use `n` for that: on the first touch of each app in a holding, the console builds on the bucket's verified record unless the bucket still holds exactly what this console last published. |
| |
| -Copy (Task 15): `policyLeaseHeldMsg` = `another console (%s) is the policy writer for this deployment until %s. Make the change there, or try again after that`; `policyLeaseUnavailableMsg` = `could not take the policy writer lease. Try again`; `policyLeaseMissingMsg` = `this console is not set up to write app policies (no writer lease). Deploy a console built from current main`. |
| +Copy (Task 15): `policyLeaseHeldMsg` = `another console (%s) is the policy writer for this deployment until %s. Make the change there, or try again after that`; `policyLeaseUnavailableMsg` = `could not take the policy writer lease. Try again`; `policyLeaseMissingMsg` = `this console is not set up to write app policies (no writer lease). Deploy a console built from current main`; `policyLeaseLostMsg` = `this console stopped being the policy writer during the change, so it may not have been published. Read the policy again before you retry`. |
| |
| -- [ ] **Step 1: Write the failing tests** (`policylease_test.go`, each `t.Parallel()`, a fake `PolicyLeaser` recording its calls and returning a fake `PolicyLease` whose `Err` the test sets) |
| +- [ ] **Step 1: Write the failing tests** (`policylease_test.go`, each `t.Parallel()`, a fake `PolicyLeaser` recording its calls and returning a fake `PolicyLease` whose `Err`, `SafeUntil` and `Lost` channel the test sets; `SafeUntil` defaults to now + 10 minutes) |
| |
| ```go |
| func TestPolicyWriterIsSoleWithoutABucket(t *testing.T) { |
| @@ -690,6 +754,16 @@ func TestPolicyWriterNamesTheHolder(t *testing.T) { |
| func TestReleasePolicyWriterEndsTheHolding(t *testing.T) { |
| // held: ReleasePolicyWriter -> fake Release called once, policyHeld() false; a second call is a no-op |
| } |
| +// Astra plan R3-1. Fails if any check is dropped. |
| +func TestPolicyAttemptBindsToTheSameLiveHolding(t *testing.T) { |
| + // sole: policyAttempt -> the same ctx (no deadline), nil refusal |
| + // holding 1, SafeUntil now+10m: nil refusal; ctx deadline within policyAttemptWindow of now |
| + // SafeUntil now+30m: deadline still now+policyAttemptWindow; SafeUntil now+8s: 503 policyLeaseLostMsg |
| + // SafeUntil returning an error: 503 policyLeaseLostMsg |
| + // Err set: 503 policyLeaseLostMsg; then policyWriter -> holding 2 (live): policyAttempt(holding 1) -> 503, |
| + // policyAttempt(holding 2) -> nil refusal |
| + // holding 2's Lost channel closed after policyAttempt returned: that ctx's Done() closes |
| +} |
| ``` |
| |
| `cmd/console/main_test.go`: |
| @@ -700,8 +774,8 @@ func TestPolicyLeaseForFollowsTheDeploymentBucket(t *testing.T) { |
| } |
| ``` |
| |
| -- [ ] **Step 2: Run, fail:** `go test ./internal/console -run 'TestPolicyWriter|TestReleasePolicyWriter' && go test ./cmd/console -run TestPolicyLeaseFor` → FAIL (undefined). |
| -- [ ] **Step 3: Implement.** File comment on `policylease.go`: ruling 10; why `internal/lease` and not a new record (the housekeeper's rule); why lazy (no requests on a deployment without policies); what a holding number is for (Task 9); the flagship `DeleteObject` caveat. |
| +- [ ] **Step 2: Run, fail:** `go test ./internal/console -run 'TestPolicyWriter|TestReleasePolicyWriter|TestPolicyAttempt' && go test ./cmd/console -run TestPolicyLeaseFor` → FAIL (undefined). |
| +- [ ] **Step 3: Implement.** File comment on `policylease.go`: ruling 10; why `internal/lease` and not a new record (the housekeeper's rule); why lazy (no requests on a deployment without policies); what a holding number is for (Task 9); why every signing and publication attempt goes through `policyAttempt` (a write that outlives its holding could overwrite the next writer's revoke, Astra plan R3-1) and the stated bound; the flagship `DeleteObject` caveat. |
| - [ ] **Step 4: Run, pass.** Same commands, plus `GOOS=windows go build ./cmd/console && GOOS=darwin go build ./cmd/console`. |
| - [ ] **Step 5: Commit** `console: one policy writer per bucket, on the platform's lease (#328, #329)`. |
| |
| @@ -720,7 +794,7 @@ func TestPolicyLeaseForFollowsTheDeploymentBucket(t *testing.T) { |
| - Test: `internal/console/apipolicy_test.go` |
| |
| **Interfaces:** |
| -- Consumes: Tasks 1-3, 6, 8 (`policyWriter`, `policyHolding`); `s.cfg.Key` (the release signing key, as `apidoorkeys.go:142` signs with it) and `doorKeyWritable` (`apidoorkeys.go:184`, its approved 503 messages reused). |
| +- Consumes: Tasks 1-3, 6, 8 (`policyWriter`, `policyHolding`, `policyAttempt`, `policyLeaseLostMsg`); `s.cfg.Key` (the release signing key, as `apidoorkeys.go:142` signs with it) and `doorKeyWritable` (`apidoorkeys.go:184`, its approved 503 messages reused). |
| - Produces: |
| - `CREATE TABLE IF NOT EXISTS app_policy_revisions (account_id INTEGER NOT NULL, app TEXT NOT NULL, revision INTEGER NOT NULL, wire BLOB NOT NULL, synced_etag TEXT NOT NULL DEFAULT '', PRIMARY KEY (account_id, app))`, classified in `claimSQLTables` as `{Account: "account_id", App: "app"}` (gated, app-scoped; the purge never writes it, so no `PurgeMonotone`). `synced_etag` is the bucket ETag the row last matched: the ETag this console's publication of `wire` got back, or the ETag of a bucket record it adopted; `''` = nothing published from this row (a first write that never landed, or an object found absent). |
| - `type AppPolicyRow struct { AccountID int64; Sqid, App string; Revision uint64; Wire []byte; SyncedETag string }` |
| @@ -734,19 +808,20 @@ func TestPolicyLeaseForFollowsTheDeploymentBucket(t *testing.T) { |
| - `func (s *Server) policyBase(ctx context.Context, acct *Account, app string, h policyHolding, commit bool) (base apppolicy.Policy, etag string, next uint64, row AppPolicyRow, err error)`: step 7's reads and rules below, shared by the GET (`commit` false: nothing recorded, no holding mark), the PUT and Task 10's re-sign (`commit` true). |
| - The table is the authority for content once a revision is recorded, as the door keys' database is (`doorkeys_store.go:3-11`): the door records the signed bytes **before** the bucket write, so a write whose reply is lost, or a bucket writer that puts an older record back, can never leave the console building on, or re-signing, content older than what it last signed (Astra plan R1-1). The purge leaves the row: revisions only rise, so a recreated app's first policy is above every record its predecessor had. The recorded content is used only while the bucket holds an object at the key: an absent object (a purge, a create-time clear, or a first write that never landed) starts empty, so a purged app's content is never brought back. |
| - `func (s *Server) policyLock(sqid, app string) *sync.Mutex` (the app's policy mutex, created on first use as `doorPublishLock` is; Task 10 shares it) |
| + - `Server.policyAfterBase func(sqid, app string)`: a test seam, nil in production, called right after `policyBase` returns in the PUT and in Task 10's `resignAppPolicy` (the `beforeClaimInsert` precedent, `server.go:4338`), so a test can pause a writer that has read its base (Astra plan R3-1) |
| - `GET /api/cli/apps/{account}/{app}/policy` → `handleAPIPolicyGet`; `PUT` same path → `handleAPIPolicyPut`. Both `svcrole.Set{}`. |
| - Request (strict reader, 8 KiB): `type policyPutReq struct { EgressAllow *[]string `json:"egress_allow"`; DefaultSize *string `json:"default_size"`; Size *policySizeReq `json:"size"` }`, `type policySizeReq struct { Host string `json:"host"`; Value string `json:"value"` }`. `null` or absent leaves a field; `[]` clears the allow; `""` clears the default size; `value: "default"` removes the override. |
| - Reply: `type policyReply struct { Policy apppolicy.Policy `json:"policy"`; ETag string `json:"etag"`; NotChecked []string `json:"not_checked"`; Stale []string `json:"stale_overrides,omitempty"` }`; the `ETag` header carries the same value. An absent object reads as `{V:1}` with `etag ""`. |
| - `checkEgressGrant(ctx, acct, app, old, next []string)` (stub; see "Interfaces produced"), `auditRequired(ctx context.Context, acct *Account, app, addr, detail string) error` (wraps `s.store.RecordEventContext`, action `"app-policy"`). |
| |
| PUT order (each refusal writes nothing): |
| -1. `apiSystemRoller` (bare 404). 2. `AccountBySqid(r.PathValue("account"))`, then `HasAppIn`: either missing → bare 404. 3. `allowWrite` → 429 `apiSwapLimited`. 4. `doorKeyWritable`: no signing key or no bucket → 503 with its message. 5. Body; nothing set → 400. 6. Precondition: `If-Match: <etag>` or `If-None-Match: *` (= `""`); neither → 428. 7. Take `policyLock(acct.Sqid, app)` until the reply, then `hold, ref := s.policyWriter(ctx)` (Task 8); a refusal is returned as it is (503 naming the holder), before any read, audit or write. Then `policyBase(..., hold, true)`: read the bucket with `apppolicy.GetForUpdate(ctx, s.cfg.Releases, pub, acct.Sqid, app)` (`b`, its bytes, `etag`) and the table with `AppPolicyRecordContext` (row: `h` its revision, `rw` its wire, `se` its synced ETag); a read error of either other than absence or `ErrUnverifiable` → 502. Choose the base, in order: |
| +1. `apiSystemRoller` (bare 404). 2. `AccountBySqid(r.PathValue("account"))`, then `HasAppIn`: either missing → bare 404. 3. `allowWrite` → 429 `apiSwapLimited`. 4. `doorKeyWritable`: no signing key or no bucket → 503 with its message. 5. Body; nothing set → 400. 6. Precondition: `If-Match: <etag>` or `If-None-Match: *` (= `""`); neither → 428. 7. Take `policyLock(acct.Sqid, app)` until the reply, then `hold, ref := s.policyWriter(ctx)` (Task 8); a refusal is returned as it is (503 naming the holder), before any read, audit or write. Then `policyBase(..., hold, true)` (and `s.policyAfterBase(acct.Sqid, app)` when set): read the bucket with `apppolicy.GetForUpdate(ctx, s.cfg.Releases, pub, acct.Sqid, app)` (`b`, its bytes, `etag`) and the table with `AppPolicyRecordContext` (row: `h` its revision, `rw` its wire, `se` its synced ETag); a read error of either other than absence or `ErrUnverifiable` → 502. Choose the base, in order: |
| - the bucket object is absent → `{V:1}`, `etag ""` (Task 10 never revives an absent object, so neither does the door); |
| - **a new holding** (ruling 10's takeover): `hold` is not sole, this app has not been touched in holding `hold.n` (`s.policySynced[sqid+"/"+app] != hold.n`), and the bucket's `etag != se` and its bytes are not `rw`, so another console may have written since this one last did → the bucket is the base: `b` verified → `AdoptAppPolicyContext(b.Revision, its bytes, etag)`, base `b`; unverifiable → `{V:1}` with its ETag (the row's content is not used). Logged `console: app policy %s/%s: new writer holding; building on the bucket's record (revision %d)`. When the bucket still holds what this console last published (`etag == se`, or the bytes are `rw` after a lost reply), the rules below apply as within a holding, so a publication a crash interrupted is still finished (Astra plan R1-1). A sole console never rebases: it is the only writer by construction; |
| - `b` verified at a revision above `h` (a database restored from backup, or a record this console signed and then lost the row of) → adopt it: `AdoptAppPolicyContext(b.Revision, its bytes, etag)`, base `b`; |
| - `h > 0` → base `OpenKey(pub, rw, acct.Sqid, app)` (the recorded content; an error here → 502), whatever the bucket holds; when the bucket's object is unverifiable, below `h`, or at `h` with other content (`rw`'s payload differs), log `console: app policy %s/%s: not building on the stored object: %v` (the operator's repair path); |
| - no row and `b` verified → adopt it as above; no row and `ErrUnverifiable` → `{V:1}` with the object's ETag. |
| - With `s.cfg.HostClaim`: `cur := s.store.ClaimAppID(acct.ID, app)` (error → 503); a base whose `AppID` is not `cur` (a predecessor's policy a failed create-time clear left, Task 11) is replaced by `{V:1}` **before** the change is applied, keeping the object's ETag and the revision floor, logged `console: app policy %s/%s: stored policy is for incarnation %q, not %q; starting empty` (Astra plan R1-2). With `commit`, `s.policySynced[sqid+"/"+app] = hold.n` once the base is chosen (a map under its own mutex). The precondition must equal the bucket's current ETag, else 412. `next := max(h, revision of a verified b) + 1`. `old` (for step 10 and the audit) is the base after these rules. 8. Apply: `egressaddr.Normalize` (400 with its message); `default_size` via `cgroup.SizeByName` or `""`; `size`: `ValidHostShape` (400 `invalid host`), `provision.GetRecord(ctx, s.cfg.Releases, acct.Sqid, app, strings.ToLower(host))` (absent → 404 `no instance %s in this app`), then set `{value, rec.CreatedAt}` or delete. Drop every other override whose record is absent or whose `CreatedAt` differs (the spec's "removes the entry, for tidiness", done here so no member door writes the object). 9. With `s.cfg.HostClaim`, `p.AppID = cur` (step 7); without, `AppID` stays as read (empty). 10. `checkEgressGrant(ctx, acct, app, old.EgressAllow, p.EgressAllow)` whenever `egress_allow` was sent. 11. `s.appMutation(ctx, acct, app, "app-policy", ...)`: inside, `auditRequired` first (failure → 503, no write); then set `p.Account, p.App, p.Revision, p.SignedAt = acct.Sqid, app, next, time.Now().UTC()`, `wire := apppolicy.Sign(s.cfg.Key, p)`, and **record before publishing**: `RecordAppPolicyContext(ctx, acct.ID, app, next, wire)` (failure → 503 `the change could not be recorded, so it was not made. Try again`, no bucket write). Then `apppolicy.PutWire(ctx, s.cfg.Releases, acct.Sqid, app, wire, etag)`; on success `SyncedAppPolicyContext(ctx, acct.ID, app, next, newETag)` (a failure is logged only: the next touch finds the bytes equal to `rw` and treats them as this console's). `ErrCASConflict` (the object changed after step 7, which under the lock and the writer lease only a bucket writer can do: two operators are serialised by the lock and refused at the precondition, a second console by the lease) → re-read the ETag and put the same bytes again, up to `doorKeyPublishAttempts` (the door keys' publish loop, `apidoorkeys.go:106-155`); any other failure, or the attempts spent → 502 `the change is recorded but not yet published. Run the command again to publish it now` (the door keys' 502 precedent, `apidoorkeys.go:321`). The change stands once recorded: Task 10's pass (at console start and every 12 h) publishes the recorded content over whatever an existing object holds, and the next operator write builds on it. 12. `s.bumpSerial(ctx)`. 13. 200 with the reply. |
| + With `s.cfg.HostClaim`: `cur := s.store.ClaimAppID(acct.ID, app)` (error → 503); a base whose `AppID` is not `cur` (a predecessor's policy a failed create-time clear left, Task 11) is replaced by `{V:1}` **before** the change is applied, keeping the object's ETag and the revision floor, logged `console: app policy %s/%s: stored policy is for incarnation %q, not %q; starting empty` (Astra plan R1-2). With `commit`, `s.policySynced[sqid+"/"+app] = hold.n` once the base is chosen (a map under its own mutex). The precondition must equal the bucket's current ETag, else 412. `next := max(h, revision of a verified b) + 1`. `old` (for step 10 and the audit) is the base after these rules. 8. Apply: `egressaddr.Normalize` (400 with its message); `default_size` via `cgroup.SizeByName` or `""`; `size`: `ValidHostShape` (400 `invalid host`), `provision.GetRecord(ctx, s.cfg.Releases, acct.Sqid, app, strings.ToLower(host))` (absent → 404 `no instance %s in this app`), then set `{value, rec.CreatedAt}` or delete. Drop every other override whose record is absent or whose `CreatedAt` differs (the spec's "removes the entry, for tidiness", done here so no member door writes the object). 9. With `s.cfg.HostClaim`, `p.AppID = cur` (step 7); without, `AppID` stays as read (empty). 10. `checkEgressGrant(ctx, acct, app, old.EgressAllow, p.EgressAllow)` whenever `egress_allow` was sent. 11. `s.appMutation(ctx, acct, app, "app-policy", ...)`: inside, `s.policyAttempt(ctx, hold)` first, the holding step 7 took (a refusal → 503 `policyLeaseLostMsg`: nothing audited, recorded or written); then `auditRequired` (failure → 503, no write); then set `p.Account, p.App, p.Revision, p.SignedAt = acct.Sqid, app, next, time.Now().UTC()`, `policyAttempt(ctx, hold)` again (the check before signing; a refusal → 503 `policyLeaseLostMsg`, nothing recorded), `wire := apppolicy.Sign(s.cfg.Key, p)`, and **record before publishing**: `RecordAppPolicyContext(ctx, acct.ID, app, next, wire)` (failure → 503 `the change could not be recorded, so it was not made. Try again`, no bucket write). Then publish, **each attempt bound to the holding** (Astra plan R3-1): every attempt, the first and each retry, begins `actx, cancel, ref := s.policyAttempt(ctx, hold)`; a refusal, or `actx` ending during the attempt, → 503 `policyLeaseLostMsg` and no further attempt. The attempt is `apppolicy.PutWire(actx, s.cfg.Releases, acct.Sqid, app, wire, etag)`; on success `SyncedAppPolicyContext(ctx, acct.ID, app, next, newETag)` (a failure is logged only: the next touch finds the bytes equal to `rw` and treats them as this console's). `ErrCASConflict` (the object changed after step 7, which under the lock and a live holding only a bucket writer can do: two operators are serialised by the lock and refused at the precondition, a second console by the lease) → re-read with `GetForUpdate(actx, ...)` under that attempt's check. A record that verifies for this key at a revision of `next` or above with bytes other than `wire` is another writer's (only the release key signs one) → 503 `policyLeaseLostMsg`, never overwritten. Otherwise take its ETag and start the next attempt with the same bytes, up to `doorKeyPublishAttempts` (the door keys' publish loop, `apidoorkeys.go:106-155`). Any other failure, or the attempts spent → 502 `the change is recorded but not yet published. Run the command again to publish it now` (the door keys' 502 precedent, `apidoorkeys.go:321`). The change stands once recorded: Task 10's pass (at console start and every 12 h) publishes the recorded content over whatever an existing object holds, and the next operator write builds on it. After a lost holding the recorded revision is published only if this console's next holding finds the bucket still holding its last publication (Task 9 step 7's new-holding rule); otherwise the next holding builds on the other writer's record. 12. `s.bumpSerial(ctx)`. 13. 200 with the reply. |
| |
| Audit detail (copy, Task 15): the changed parts joined with `"; "`, each `egress allow: <old> -> <new>`, `default size: <old> -> <new>`, `size of <host>: <old> -> <new>`, with `none`, `the build's size` and `the app default` for empty values, ending ` (cli)`. |
| |
| @@ -837,13 +912,23 @@ func TestPolicyRevokeSurvivesALostWriteAndAReplay(t *testing.T) { |
| // stored Revision 3, no egress_allow, default_size small |
| } |
| func TestPolicyPutStartsEmptyOverAnotherIncarnationsPolicy(t *testing.T) { |
| - // Astra plan R1-2. claim-enabled fixture (newClaimCase): PUT egress ["10.0.5.20"] -> app_id a1, rev 1 |
| - // purge and recreate the app through the doors with a Releases wrapper failing Delete of the policy key |
| - // (Task 11's create-time clear fails and the create stands); ClaimAppID is now a2 |
| + // Astra plan R1-2. claim-enabled fixture (newClaimCase): PUT egress ["10.0.5.20"] -> app_id a1, rev 1; keep its bytes w1 |
| + // purge and recreate the app through the doors (Task 11 clears the key); ClaimAppID is now a2 |
| + // put w1 back at the key directly (an older console's create, which never clears, spec §3 "Older consoles") |
| // GET -> {v:1}, no egress_allow |
| // PUT If-Match <the object's etag> {"default_size":"small"} -> stored app_id a2, Revision 2, |
| // no egress_allow, default_size small; the audit detail has no "egress allow" part |
| } |
| +// Astra plan R3-1, the retry half. Fake leaser (cfg.Bucket "deploy"); a Releases wrapper over the policy key. |
| +func TestPolicyPublishRechecksTheHoldingOnEveryAttempt(t *testing.T) { |
| + // the wrapper's first Put returns ErrCASConflict and sets the fake lease's Err in the same call: |
| + // PUT -> 503 policyLeaseLostMsg; exactly 1 Put reached the store; the row is the recorded revision |
| + // the fake's SafeUntil now+8s before the PUT: 503 policyLeaseLostMsg, 0 Puts, nothing recorded, no event |
| + // the fake's SafeUntil moved to now+8s by the wrapper's first (conflicting) Put: 503, exactly 1 Put |
| + // the wrapper's first Put writes a record the fixture's key signed at Revision next (another writer's) |
| + // and returns ErrCASConflict, the lease still live: 503 policyLeaseLostMsg, no second Put, the object is that record |
| + // control: a first Put that only returns ErrCASConflict (the object unchanged): the second attempt lands, 200 |
| +} |
| func TestOpenGatedClassifiesAppPolicyRevisions(t *testing.T) { |
| // Astra plan R1-4. openGatedStore(t) succeeds (OpenGated runs the migration and the census); |
| // claimSQLCensus(s.bootDB(), claimSQLTables) lists app_policy_revisions as gated (NonApp == ""); |
| @@ -888,18 +973,18 @@ func TestPolicyPutStampsTheAppIDUnderClaims(t *testing.T) { |
| - Test: `internal/console/apipolicy_resign_test.go`, `internal/console/policylease_twoconsoles_test.go` |
| |
| **Interfaces:** |
| -- Consumes: Task 3 (`Present`, `Key`, `GetForUpdate`, `OpenKey`, `Sign`, `PutWire`, `ErrUnverifiable`), Task 8 (`policyWriter`, `policyHeld`), Task 9 (`policyLock`, `policyBase`, `AppPolicyRowsContext`, `AppPolicyRecordContext`, `SyncedAppPolicyContext`, `ResignAppPolicyContext`), `s.appMutation`, `AccountBySqid`, `doorKeyPublishBudget`. |
| +- Consumes: Task 3 (`Present`, `Key`, `GetForUpdate`, `OpenKey`, `Sign`, `PutWire`, `ErrUnverifiable`), Task 8 (`policyWriter`, `policyHeld`, `policyAttempt`, `policyLeaseLostMsg`), Task 9 (`policyLock`, `policyBase`, `policyAfterBase`, `AppPolicyRowsContext`, `AppPolicyRecordContext`, `SyncedAppPolicyContext`, `ResignAppPolicyContext`), `s.appMutation`, `AccountBySqid`, `doorKeyPublishBudget`. |
| - Produces: `func (s *Server) resignAppPolicies(ctx context.Context)`; `func (s *Server) resignAppPolicy(ctx context.Context, acct *Account, app string, hold policyHolding) error`. |
| |
| Rules (the door keys' re-sign, `apidoorkeys.go:328-365`, with the console's table as the content's source, Astra plan R1-1, and the writer lease, ruling 10): |
| - Runs only where `StartDoorKeyResign` runs (a signing key and a bucket): at start, then every `doorKeyResignEvery` (12 h). A record lives `Lifetime` (72 h), so a console down for up to 60 h past its last pass leaves no policy expired; one down longer closes every granted address until it is back (spec §3, the door keys' trade), and its first pass re-signs them. The pass at start is also the restart recovery: it finishes any publication a crash or a lost write interrupted. |
| - **Which apps, with no listing in steady state** (spec §3a). The candidates are `AppPolicyRowsContext`'s rows (every policy this console published or adopted). The pass also lists `control/app-policy/` with `Present` (one LIST) when this process has not listed in its current holding: its first pass, the first pass of a new holding, and every pass while another console holds the lease (so it sees what the holder has before it takes over). A listed key with no row is a candidate too; each must be exactly `Key(sqid, app)` with a valid account and app, else skipped. `AccountBySqid` missing → logged and skipped (the purge sweep owns it). No candidates → the pass ends with no lease taken. |
| -- **The lease.** With candidates, `hold, ref := s.policyWriter(ctx)`; a refusal is logged once per pass (`console: app policies: not re-signing; %s`, the refusal's message naming the holder) and nothing is signed. |
| +- **The lease.** With candidates, `hold, ref := s.policyWriter(ctx)`; a refusal is logged once per pass (`console: app policies: not re-signing; %s`, the refusal's message naming the holder) and nothing is signed. The pass keeps that one holding, and every app's signing and publication go through `policyAttempt(ctx, hold)` (below), so a holding lost part way through the pass stops it (Astra plan R3-1): log `console: app policies: stopped re-signing at %s/%s; the writer lease was lost`, and no later candidate is read or written in that pass. |
| - Per app, under `policyLock` and `s.appMutation(ctx, acct, app, "app-policy-resign", ...)`, within `doorKeyPublishBudget`: `policyBase(ctx, acct, app, hold, true)`, Task 9's rules exactly, including the new-holding rebase. Then: |
| - the object is absent → `SyncedAppPolicyContext(row.Revision, "")` (the row leaves the candidates until an operator write), nothing written; an absent object is never recreated (purge, create-time clear); |
| - the base is a row's recorded content → the content to sign is `OpenKey(pub, row.Wire, ...)`, **never the bucket's** within a holding: the bucket object may be the recorded revision, an interrupted publication's predecessor, a replayed older record or unverifiable bytes. When it is anything but the recorded content, log `console: app policy %s/%s: the bucket held revision %d, not the recorded %d; publishing the recorded one`; |
| - the base is `{V:1}` from an unverifiable object with no usable row → logged `console: app policy %s/%s: not re-signing an object that does not verify; an operator write repairs it`, nothing written (the console never signs bytes it neither recorded nor verified). |
| - - Re-sign: that content at the row's **recorded revision** with `SignedAt = now`; `ResignAppPolicyContext(rev, wire')` before the write (the record never lags the bucket); then `PutWire(..., wire', etag)` (replace-only on the ETag just read, so a purge that deleted the key between the read and the write is never undone; `ErrCASConflict` and a not-found are both logged and skipped); on success `SyncedAppPolicyContext(rev, newETag)`. The same revision is safe to re-sign: every signing of it carries the same content with a different expiry, so replaying an older one gains nothing. |
| + - Re-sign: `policyAttempt(ctx, hold)` (the check before signing), then that content at the row's **recorded revision** with `SignedAt = now`; `ResignAppPolicyContext(rev, wire')` before the write (the record never lags the bucket); then `actx, cancel, ref := s.policyAttempt(ctx, hold)` and `PutWire(actx, ..., wire', etag)` (one attempt, replace-only on the ETag just read, so a purge that deleted the key between the read and the write is never undone, and a writer that took over meanwhile is never overwritten; `ErrCASConflict` and a not-found are both logged and skipped, never retried); a refusal from either check, or `actx` ending, stops the pass as above; on success `SyncedAppPolicyContext(rev, newETag)`. The same revision is safe to re-sign: every signing of it carries the same content with a different expiry, so replaying an older one gains nothing. |
| - A failure for one app is logged and does not stop the others. |
| - **Cost:** per pass, one `Get` per candidate (the base read) and one `Put` per re-signed policy; a `List` only on the passes named above. With the change index enabled each re-sign `Put` bumps the app's scope, so each armed box re-reads that one policy (Task 7) twice a day. |
| |
| @@ -933,6 +1018,11 @@ func TestStartDoorKeyResignAlsoResignsPolicies(t *testing.T) { |
| // StartDoorKeyResign with a context cancelled after its first pass -> the policy's SignedAt moved |
| // a console with no signing key -> StartDoorKeyResign returns at once; the object is untouched |
| } |
| +func TestResignStopsWhenTheHoldingIsLost(t *testing.T) { |
| + // Astra plan R3-1. cfg.Bucket "deploy", the fake leaser; three recorded policies a, b, c; |
| + // a Releases wrapper sets the fake lease's Err once the first policy Put returns |
| + // -> exactly 1 policy re-signed; the other two objects byte-identical; no further Get or Put of the prefix |
| +} |
| func TestResignWithNoPoliciesTakesNoLease(t *testing.T) { |
| // cfg.Bucket "deploy", the fake leaser: an empty bucket and table -> one Present listing, 0 leaser calls |
| } |
| @@ -965,10 +1055,25 @@ func TestPolicyWriterLeaseIsOneConsolePerBucket(t *testing.T) { |
| // default_size small; B's row (3, that wire, its etag) |
| // 7. A: PUT -> 503 naming B's owner; A.resignAppPolicies -> nothing signed |
| } |
| +// Astra plan R3-1: a writer paused inside a write must not overwrite the next writer's revoke. |
| +// Same world as above (one leasetest bucket, two console databases, TTL 3s, not parallel). |
| +func TestALapsedWriterCannotOverwriteTheNextWritersRevoke(t *testing.T) { |
| + // 1. A: PUT egress ["10.0.5.20"] -> 200, rev 1 (A holds) |
| + // 2. A.policyAfterBase blocks on a channel; A: PUT {"default_size":"small"} If-Match <rev 1's etag> in a |
| + // goroutine; wait until the seam is entered (A has read the allow as its base) |
| + // 3. FailOn refuses puts to the lock key; poll (5 s deadline) until A.policyHeld() is false and the |
| + // lock's ExpiresAt has passed; clear FailOn |
| + // 4. B: GET -> egress 10.0.5.20; B: PUT {"egress_allow":[]} If-Match <that etag> -> 200, rev 2; keep the bytes wB |
| + // 5. release A's seam: A's PUT -> 503 policyLeaseLostMsg; the object is wB byte for byte (no egress_allow) |
| + // 6. the re-sign: B.policyAfterBase blocks and B.resignAppPolicies runs in a goroutine; B's lease lapses as |
| + // in 3; A: PUT {"default_size":"large"} -> 200, rev 3, no egress_allow; keep wA |
| + // 7. release B's seam: B's pass writes nothing; the object is wA byte for byte |
| + // (with policyAttempt's checks removed, step 5 publishes A's rev over wB and the allow returns) |
| +} |
| ``` |
| |
| -- [ ] **Step 2: Run, fail:** `go test ./internal/console -run 'TestResign|TestStartDoorKeyResignAlsoResignsPolicies|TestPolicyResignKeepsTheRequestBudget|TestPolicyWriterLeaseIsOneConsolePerBucket|TestClaimConsoleMutationInvariant'`. |
| -- [ ] **Step 3: Implement.** Comment on `resignAppPolicies`: why the content and the revision both come from the table (Task 9; the door keys' database-as-authority rule, Astra plan R1-1), why only present objects are re-signed (an absent one is a purge or a clear, never revived), why adopting a higher bucket revision is safe (only a console with the release key can have signed it), why the pass lists only once per holding (spec §3a), and that nothing is signed without the writer lease (ruling 10). |
| +- [ ] **Step 2: Run, fail:** `go test ./internal/console -run 'TestResign|TestStartDoorKeyResignAlsoResignsPolicies|TestPolicyResignKeepsTheRequestBudget|TestPolicyWriterLeaseIsOneConsolePerBucket|TestALapsedWriterCannotOverwriteTheNextWritersRevoke|TestClaimConsoleMutationInvariant'`. |
| +- [ ] **Step 3: Implement.** Comment on `resignAppPolicies`: why the content and the revision both come from the table (Task 9; the door keys' database-as-authority rule, Astra plan R1-1), why only present objects are re-signed (an absent one is a purge or a clear, never revived), why adopting a higher bucket revision is safe (only a console with the release key can have signed it), why the pass lists only once per holding (spec §3a), that nothing is signed without the writer lease (ruling 10), and that the whole pass is bound to one holding through `policyAttempt` (Astra plan R3-1). |
| - [ ] **Step 4: Run, pass.** Same command, plus `go test ./internal/console -run 'TestDoorKey'` (the door keys' own pass is unchanged). If the two-console test runs longer than a few seconds, refresh `.amadan/test-weights.tsv` with `.amadan/test-weights.sh`. |
| - [ ] **Step 5: Commit** `console: re-sign every app policy on the door keys' loop (#328, #329)`. |
| |
| @@ -978,12 +1083,15 @@ func TestPolicyWriterLeaseIsOneConsolePerBucket(t *testing.T) { |
| |
| **Files:** |
| - Modify: `internal/release/trash.go:276-295` (append `AppPolicyKey(acct, app)` to `keys` before the delete loop, with a comment in the ledger paragraph's style) |
| -- Modify: `internal/console/server.go` `claimAppWith` (around `createClaimAppRow`, line 4193) |
| +- Modify: `internal/console/server.go`: `claimOutcome` (4103: `claimPolicyClearFailed`), `claimAppWith` (its `createClaimAppRow` call, 4341-4349), new `clearAppPolicy`, the browser door's outcome switch (4458); `internal/console/claim_app_enroll.go` `createClaimAppRow` (16-44); `internal/console/apicli.go` (the create door's outcome switch, 465-483); `internal/console/apitemporary.go` (its outcome switch, 205-219) |
| - Test: `internal/release/trash_test.go`; `internal/console/apipolicy_test.go` |
| |
| **Interfaces:** |
| -- Consumes: `release.AppPolicyKey`, `apppolicy.Delete`, `s.appMutation`. |
| -- Rules: purge deletes the policy before the marker (a purge that dies is retried by the next pass; covers the console's "Delete now" at `ownership.go:514` and the auto-purge at `domains/reconcile.go:271`). Create: with `HostClaim` off, `apppolicy.Delete` on `s.cfg.Releases` **before** `createClaimAppRow`; a failure returns `claimStoreFailed` and the name is not handed out. With `HostClaim` on, the delete runs **after** `createClaimAppRow` inside `s.appMutation(ctx, acct, name, "app-policy-clear", ...)` (no permit exists before the row); a failure is logged and the create stands: under claims root writes no egress allow at all in this build (ruling 9), so a lingering predecessor's policy can at most size the new app's instances (the trust sizes already have, spec §5.3) until the next operator write, which the door starts empty (Task 9 step 7, Astra plan R1-2). The claim-mode follow-up binds the policy to the app's incarnation in root; nothing here reads `app_id` on a box. Both apply to the owner's adopt-existing case too (fail closed: the operator grants again). |
| +- Consumes: `release.AppPolicyKey`, `apppolicy.Delete`, `s.appMutation`, `claimEnrollmentContext` (`claim_app_admission.go:189`) and `mutationAppID`'s enrollment branch (212-220). |
| +- Produces: `claimPolicyClearFailed` (a `claimOutcome`); `func (s *Server) clearAppPolicy(ctx context.Context, acct *Account, app string) error` = `s.appMutation(ctx, acct, app, "app-policy-clear", func(ctx) error { return apppolicy.Delete(ctx, s.cfg.Releases, acct.Sqid, app) })`, wrapped in `errAppPolicyClear` on failure; `createClaimAppRow` keeps its signature. |
| +- Rules: purge deletes the policy before the marker (a purge that dies is retried by the next pass; covers the console's "Delete now" at `ownership.go:514` and the auto-purge at `domains/reconcile.go:271`). **Create never succeeds while a policy object at the key cannot be deleted, in every mode** (spec §3: "before the name is usable"; Astra plan R3-3). `createClaimAppRow` calls `clearAppPolicy` **before the row is inserted**: with `HostClaim` off, first thing; with `HostClaim` on, after `EnrollNewApp` and `claimEnrollmentContext`, with the enrollment context, whose identity `mutationAppID` already answers before the row exists, so the boundary admits the app-scoped delete (Task 6), and before `withAutoAppSQL("sql.CreateApp")`. A failure (a store error, or the boundary refusing) inserts no row and returns `errAppPolicyClear`; `claimAppWith` maps it to `claimPolicyClearFailed`. An enrollment left with no row is the state an insert failure after `EnrollNewApp` leaves today, so a retry takes that existing path. The doors answer `claimPolicyClearFailed` as retryable: the CLI create door and `/new-temporary` 503 `appCreatePolicyClearMsg`, the browser door the flash `appCreatePolicyClearFlash`. A retry of the create re-runs the clear. Applies to the owner's adopt-existing case too. Task 9 step 7's `AppID` rule stays, for an object an older console's create left (spec §3 "Older consoles"); nothing here reads `app_id` on a box. |
| + |
| +Copy (Task 15): `appCreatePolicyClearMsg` = `an earlier app's settings under this name could not be cleared, so nothing was created. Try again`; `appCreatePolicyClearFlash` = `Could not clear an earlier app's settings under this name, so nothing was created. Please try again.` |
| |
| - [ ] **Step 1: Write the failing tests** |
| |
| @@ -1000,15 +1108,27 @@ func TestPurgeAppSweepsThePolicyBeforeTheMarker(t *testing.T) { |
| ```go |
| // apipolicy_test.go |
| func TestCreatingAnAppClearsAStalePolicy(t *testing.T) { |
| + // for each mode: newConsoleFixture (claims off) and newClaimCase (claims on) |
| // seed a policy at the key of a name no app holds; create the app through the CLI create door |
| - // -> 2xx and the object is gone |
| - // a store failing Delete of that key -> the create is refused (claimStoreFailed's answer) and no row exists |
| + // -> 2xx and the object is gone (under claims the delete passed the boundary as "app-policy-clear") |
| +} |
| +// Astra plan R3-3. Fails if any mode lets the create stand after a failed clear. |
| +func TestCreateIsRefusedWhileAPredecessorsPolicyLingers(t *testing.T) { |
| + // for each mode: newConsoleFixture and newClaimCase |
| + // seed a predecessor's policy at the key (signed by the fixture's key, DefaultSize "xlarge", |
| + // EgressAllow ["10.0.5.20"]); a Releases wrapper fails Delete of that key |
| + // CLI create -> 503 appCreatePolicyClearMsg; HasAppIn false; the object unchanged; no "create" event; |
| + // /new-temporary of a tmp- name over a seeded tmp- key and the browser form: the same refusal, no row |
| + // an instance create for the app through the instances door -> refused (no app); no record under its prefix |
| + // provision.Reconcile over fx.rel (apiinstancesenable_test.go's shape): no route row for the app |
| + // clear the fault; CLI create again -> 2xx, the object gone; create an instance record; Reconcile with |
| + // ReleasePub the fixture's pub -> its row Size "", EgressAllow "", PolicyExpiresAt 0 |
| } |
| ``` |
| |
| -- [ ] **Step 2: Run, fail:** `go test ./internal/release -run 'TestPurgeApp' && go test ./internal/console -run 'TestCreatingAnAppClearsAStalePolicy'`. |
| +- [ ] **Step 2: Run, fail:** `go test ./internal/release -run 'TestPurgeApp' && go test ./internal/console -run 'TestCreatingAnAppClearsAStalePolicy|TestCreateIsRefusedWhileAPredecessorsPolicyLingers'`. |
| - [ ] **Step 3: Implement.** |
| -- [ ] **Step 4: Run, pass**, plus `go test ./internal/console -run 'TestPurge|TestAppCreate|TestClaimApp'` and `go test ./internal/apppolicy -run TestPolicyWriters`. |
| +- [ ] **Step 4: Run, pass**, plus `go test ./internal/console -run 'TestPurge|TestAppCreate|TestClaimApp|TestClaimMutationStaleConsoleCannotMutateRecreatedApp|TestTemporary|TestPolicyPutStartsEmpty'` and `go test ./internal/apppolicy -run TestPolicyWriters`. |
| - [ ] **Step 5: Commit** `purge and create clear the app policy (#328, #329)`. |
| |
| --- |
| @@ -1136,7 +1256,7 @@ The flow is AGENTS.md "The census: an approval, not a decision" (merged #369). ` |
| - Scratch (not committed): `$TMPDIR/census-claims-instance-policy-1.tsv` |
| - Created by the tool: `internal/provision/testdata/plan14-root-reads-instance-policy-1.tsv`, its line in `plan14-root-reads-chain.tsv`, `docs/superpowers/notes/census-proposal-instance-policy-1.md` (the card) |
| |
| -- [ ] **Step 1: Read the card.** `make census-card` (writes nothing). Expected classes: `body` for the moved units (`scanRoute`, `routeArgs`, `convergeOwned`, the prefetch, `purgeApp`, `ChangeScope`, the classifier, `fleetapi.Get`/`Open`, ...), `envelope` for the re-pinned packages (`internal/apppolicy`, `internal/registry`, `internal/provision`, `internal/release`, `internal/cgroup`, `internal/fleetapi`, `internal/hostops`), and `sites` where a unit gains reads of a kind it already makes (`PolicyCache.read`'s `Get` and `ListETags` beside the walk's). Root reads neither new column in this plan. |
| +- [ ] **Step 1: Read the card.** `make census-card` (writes nothing). Expected classes: `body` for the moved units (`scanRoute`, `routeArgs`, `convergeOwned`, the prefetch, `purgeApp`, `ChangeScope`, the classifier, `fleetapi.Get`/`Open`, ...), `envelope` for the re-pinned packages (`internal/apppolicy`, `internal/registry`, `internal/provision`, `internal/release`, `internal/cgroup`, `internal/fleetapi`, `internal/hostops`), and `sites` where a unit gains reads of a kind it already makes (`PolicyCache.read`'s `Get` and `ListETags` beside the walk's). `fleetapi`'s `ListETags` (Task 5) is a new unit: if the card classes it `new`, Step 3's hand-row path applies. Root reads neither new column in this plan. |
| - [ ] **Step 2: Write the CLAIMS file**, one line per changed unit the card lists: `<file>:<unit><TAB><what changed, and that no read was added or which one was>`, for example `internal/provision/reconcile.go:convergeOwned body: converges egress_allow, size and policy_expires_at from the verified policy; no read added` and `internal/release/changeindex.go:ChangeScope body: maps control/app-policy/<a>/<b>.json to apps/<a>/<b>; no read added`. The executing identity is in each claim where it is not obvious: the agent (`carlos-edge.service`, user `carlos`) for reconcile and the policy cache, the console for the classifier and purge, boxes for the auto-purge. |
| - [ ] **Step 3: Propose.** `make census-propose NAME=instance-policy-1 CLAIMS=$TMPDIR/census-claims-instance-policy-1.tsv`. Expected: it writes the overlay (rows `rev3=census-ruling-instance-policy-1`), the `@legacy` fingerprints, the envelope re-pins, the seal, the chain line and the card, and checks the census is clean. If the card has a decision class (`reads`, `new`, `removed`, `other`), the tool refuses: write those rows by hand following the chained overlays, and the card goes to #280 for the owner's decision, not an approval. |
| - [ ] **Step 4: Ask for the approval.** Show Paul the card (`docs/superpowers/notes/census-proposal-instance-policy-1.md`) with one question: approve or hold. If he is not in the session, post the card on #280. Record the approval on #280: who, when, and the tip. |
| @@ -1151,9 +1271,9 @@ The flow is AGENTS.md "The census: an approval, not a decision" (merged #369). ` |
| - Modify: `README.md` under "Releasing the platform" (line 243): a dated entry, drafted for review, stating the rollout rule of spec §3 and §7 (every console that writes the bucket runs this version before the first policy is granted; one console per bucket writes policies, held by the writer lease, ruling 10; a console downgraded below it must delete `control/app-policy/` first; with nothing granted, the release changes nothing) |
| - After review: `approved_instance_policy_copy_test.go` in the shape of `approved_run_copy_test.go`, pinning the approved strings |
| |
| -- [ ] **Step 1: Collect every user-facing string** from: Task 1's error table; Task 9's door messages and audit detail parts (`egress allow: …`, `default size: …`, `size of …: …`, `none`, `the build's size`, `the app default`, ` (cli)`; the new 502 `the change is recorded but not yet published. Run the command again to publish it now`); Task 8's three lease refusals (`policyLeaseHeldMsg`, `policyLeaseUnavailableMsg`, `policyLeaseMissingMsg`); Task 12's table, the changed `appsSetNothing`, the `appsUsage` `show` line and `appsSetPolicyUsage`; the README entry. Count each string's occurrences at the base commit first (memory: shared strings need their own review). |
| +- [ ] **Step 1: Collect every user-facing string** from: Task 1's error table; Task 9's door messages and audit detail parts (`egress allow: …`, `default size: …`, `size of …: …`, `none`, `the build's size`, `the app default`, ` (cli)`; the new 502 `the change is recorded but not yet published. Run the command again to publish it now`); Task 8's four lease refusals (`policyLeaseHeldMsg`, `policyLeaseUnavailableMsg`, `policyLeaseMissingMsg`, `policyLeaseLostMsg`); Task 11's create refusals (`appCreatePolicyClearMsg`, `appCreatePolicyClearFlash`); Task 12's table, the changed `appsSetNothing`, the `appsUsage` `show` line and `appsSetPolicyUsage`; the README entry. Count each string's occurrences at the base commit first (memory: shared strings need their own review). |
| - [ ] **Step 2: Run the copy-review skill** with that list; wait for Paul's edits. |
| -- [ ] **Step 3: Apply the edits verbatim**, rerun `go test . -run 'TestApprovedCopy|TestAppsSet|TestAppsShow|TestChangeAppPolicy'`, `go test ./internal/egressaddr`, `go test ./internal/console -run 'TestPolicy|TestReleasePolicyWriter'`. |
| +- [ ] **Step 3: Apply the edits verbatim**, rerun `go test . -run 'TestApprovedCopy|TestAppsSet|TestAppsShow|TestChangeAppPolicy'`, `go test ./internal/egressaddr`, `go test ./internal/console -run 'TestPolicy|TestReleasePolicyWriter|TestCreateIsRefusedWhileAPredecessorsPolicyLingers'`. |
| - [ ] **Step 4: Add the approved-copy test** pinning the reviewed strings; run it. |
| - [ ] **Step 5: Commit** `instance policy: approved copy and rollout note`. |
| |
| @@ -1165,6 +1285,7 @@ The flow is AGENTS.md "The census: an approval, not a decision" (merged #369). ` |
| - **Names:** the brief's pinned names (rev 8: `Get`, `PutIfMatch` and `Policy` changed for the signature, and the brief was updated with them) are used unchanged; additions are listed under "Interfaces produced". `apppolicy.Prefix`/`Key` are defined through `release` (cycle), with the same values. |
| - **Decisions the spec did not pin:** see the report that accompanies this plan. |
| - **Spec rev 9 re-check** (run after the dispositions below). Coverage: ruling 10's writer lease → Task 8 (mechanism, refusal naming the holder), Tasks 9 and 10 (no signing without it; a new holding builds on the bucket); ruling 11 and §3a → Task 3 (`ChangeScope` row and its table test), Task 7 (gated reads, backstop, budget test), Task 10 (no steady-state listing, budget test); ruling 9 → Boundary verification and Task 11 (no box reads `app_id`; root reads no policy). The policy itself stays signed and expiring, and the agent still verifies before converging (§3, still needed pre-claim). Names: `AppPolicyRecordContext` now returns `AppPolicyRow` everywhere it is used (Tasks 9, 10); `AdoptAppPolicyContext`, `SyncedAppPolicyContext`, `AppPolicyRowsContext`, `policyBase`, `policyWriter`, `policyHeld`, `ReleasePolicyWriter`, `PolicyLeaser`, `PolicyCache`, `apppolicy.ListETags` are defined once and used with the same signatures in Tasks 3, 7, 8, 9, 10, the "Interfaces produced" list and the brief; `apppolicy.Present` is now used only by Task 10's once-per-holding listing; the per-pass `Present` read in Task 7 is gone. Proportion: the plan grew by one task and two test files; no step carries a body the signature and tests already decide. Placeholders: none. |
| +- **Astra plan R3 re-check** (run after the R3 dispositions). Spec coverage: §3 "before the name is usable" → Task 11 (create refused in every mode); §3a's box budget now holds for fleet-door boxes too → Tasks 5 and 7 (the door's ETag listing, the fleet-door witness); ruling 10's one writer now holds through a lapse mid-write → Tasks 8 to 10 (`policyAttempt`). Names: `policyAttempt`, `policyAttemptWindow`, `policyLeaseLostMsg`, `PolicyLease`'s `Lost`/`SafeUntil`, `policyHolding.lease`, `policyAfterBase`, `storeETagsPath`, `serveETagListing`, `aclETagListScope`, `fleetapi`'s `ListETags`, `listSnap`, `claimPolicyClearFailed`, `clearAppPolicy`, `errAppPolicyClear`, `appCreatePolicyClearMsg`/`Flash` are each defined once and used the same way in Tasks 5, 7, 8, 9, 10, 11, 15, "Interfaces produced" and the brief. Plans 2 and 3 consume none of the changed names (plan 2 uses `policyHeld` and Task 8's fake leaser, both unchanged in use). Copy: three new strings, listed in Task 15. Placeholders: none. Proportion: five tests and two rule paragraphs added; no bodies. |
| - **Astra plan R1 re-check:** the self-review was run again after the dispositions below. Spec coverage is unchanged (the table now records content as the door keys' database does, spec §3's precedent; the expiry column carries §3's lifetime to the box before claims). Names: `GetForUpdate` gains the bytes, `OpenKey`, `PutWire`, `AppPolicyRecordContext`, `RecordAppPolicyContext`, `ResignAppPolicyContext` and `Route.PolicyExpiresAt` are used the same way in Tasks 3, 4, 7, 9 and 10 and in the brief; `RaiseAppPolicyRevisionContext` and `AppPolicyRevisionContext` are gone everywhere. Copy: one new door message (the 502), and the `app-policy-not-applied` row is gone. |
| |
| ## Review dispositions (Astra plan R1) |
| @@ -1187,3 +1308,14 @@ The flow is AGENTS.md "The census: an approval, not a decision" (merged #369). ` |
| | Ruling 11, spec §3a | Task 3: `ChangeScope` maps the policy key to `apps/<sqid>/<app>` (table rows and a bump test). Task 7: the agent's per-pass LIST and GETs are gone; reads happen on that app's generation move, a failed read's retry, or the backstop (one `ListETags`, GET only changed ETags). Task 10: the re-sign visits the table's rows and lists only once per holding. Budget tests: `TestPolicyReadsKeepTheRequestBudget` (box: at most 48 LISTs and 0 GETs a day with nothing written), `TestPolicyResignKeepsTheRequestBudget` (console: 0 LISTs and 2 GETs per live policy a day). | |
| | Console invariant (main 521c6ee) | Task 9: the PUT's fixture in `TestClaimConsoleMutationInvariant` injects a fake leaser on its own server, so it stays parallel; the serial alternative (`claimSerialDoors` plus `claimMustRunAlone`) is spelled out. Task 10's worker fixture does the same. | |
| | Census (#369) | Task 14 is the approval flow: `make census-card`, a CLAIMS file, `make census-propose`, the card to Paul; the chain manifest loads it after `ci-speed-activator` and `securelisten-337`. | |
| + |
| +## Review dispositions (Astra plan R3) |
| + |
| +All four findings' citations were checked and hold (`lease.go:234-251`, `storedoor.go:159`, spec §3 "The policy dies with its app"; the door has `/store/list` and `/store/dirs` only, with no ETag or `ListMod` listing). |
| + |
| +| Finding | What changed | |
| +|---|---| |
| +| R3-1 (P1) an in-flight writer overwrites a revoke after losing its lease | Task 8: `PolicyLease` gains `Lost` and `SafeUntil` (`*lease.Held` has both); `policyHolding` carries its lease; `policyAttempt(ctx, h)` refuses (503 `policyLeaseLostMsg`) unless `h` is still the current holding, live, and at least `policyAttemptWindow` (10 s) inside `SafeUntil`, and returns a context that ends at that deadline or on `Lost`. Task 9 step 11: checked before the audit, before signing and before every publication attempt including CAS retries; a CAS retry that finds another writer's record at `next` or above stops instead of overwriting. Task 10: the pass is bound to one holding, checked before each signing and each single `PutWire`; a loss stops the pass. Tests: `TestPolicyAttemptBindsToTheSameLiveHolding`, `TestPolicyPublishRechecksTheHoldingOnEveryAttempt`, `TestResignStopsWhenTheHoldingIsLost`, and the two-console witness `TestALapsedWriterCannotOverwriteTheNextWritersRevoke` (A paused by `policyAfterBase` after reading the allow, its lease lapses, B revokes, A resumes: B's bytes stand; the same for a paused re-sign). | |
| +| R3-2 (P1) a newly hosted app misses an already-written policy until the backstop | Task 7: the backstop takes `listSnap = watch.Observed()` before its listing; a new app reuses the listing's absence only if its scope's generation still equals `listSnap`'s, else one `Get` on first encounter (always on a fleet or disarmed box). The budget test's "new app" lines now say which case is free. Test: `TestAPolicyWrittenBeforeItsAppEntersScopeIsReadOnFirstEncounter` (empty listing, then policy and record for an app not yet in scope, then the first pass that hosts it reads the policy). | |
| +| R3-3 (P1) failed create-time cleanup lets a recreated app inherit its predecessor's size | Task 11: `createClaimAppRow` clears the key before inserting the row in every mode; under claims it does so in the enrollment context, which `mutationAppID` already resolves before the row exists. A failure inserts nothing and answers `claimPolicyClearFailed` (503 `appCreatePolicyClearMsg`, browser flash `appCreatePolicyClearFlash`, both in Task 15); a retry re-runs the clear. Task 9's `TestPolicyPutStartsEmptyOverAnotherIncarnationsPolicy` now gets its lingering object from an older console's create, since a failed clear no longer lets the create stand. Test: `TestCreateIsRefusedWhileAPredecessorsPolicyLingers` (both modes: failed clear, refused create, no instance, no row; then a clean create and provision at the build's size with no allow). | |
| +| R3-4 (P2) fleet-door reads exceed the request budget | Task 5: the store door gains `GET /store/etags`, read-only and policy-only, mapping `control/app-policy/` to the box's own `control/app-policy/<sqid>/` and re-checking each key with `appPolicyRead`; `fleetapi` implements `release.ETagLister` over it (404 → `release.ErrRefused`). Task 7: fleet boxes list like every other box; a refused listing (an older edge) marks the apps unreadable and completes the backstop; the one-`Get`-per-app fallback is left for stores with no `ETagLister` (no production store). Tests: `TestStoreDoorListsOnlyItsOwnPolicyETags`, `TestPolicyETagsThroughTheDoor`, `TestARefusedListingIsUnreadableAndWaitsForTheNextBackstop`, and `TestFleetBoxPolicyReadsKeepTheRequestBudget`, counted at the backing store behind the real door: at most 48 LISTs and 0 GETs on a quiet day, 2 GETs per live policy on a day with two re-signs (§3a unchanged). | |
amadan