amadan
spec: releases retention — ambient prune for shipped releases (#136)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+259
−0
docs/superpowers/specs/2026-08-10-releases-retention-design.md+259 −0
diff --git a/docs/superpowers/specs/2026-08-10-releases-retention-design.md b/docs/superpowers/specs/2026-08-10-releases-retention-design.md| new file mode 100644 |
| index 000000000..3f13db1cb |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-10-releases-retention-design.md |
| @@ -0,0 +1,259 @@ |
| +# Releases retention — ambient prune for shipped releases |
| + |
| +**Date:** 2026-08-10 |
| +**Issue:** #136 (carlos releases prune: retention for shipped releases) |
| +**Status:** approved by Paul 2026-08-10 (brainstorming session) |
| + |
| +## 1. Problem |
| + |
| +Nothing has ever deleted a `releases/<version>/` object. CI-shipped apps |
| +accumulate versions forever: titogo holds 320+ (one per merge, ~2 objects |
| +each), growing daily. PR #134 (main 8f64abd) fixed the O(N)-manifest-read |
| +render cost; this design fixes the other half — the history itself must be |
| +able to shrink, without any member needing box or AWS access (the no-infra |
| +rule) and without anyone having to remember to run a cleanup. |
| + |
| +Decided scope (Paul, 2026-08-10): **ambient retention first.** An opt-in |
| +per-app policy the platform applies continuously. A manual one-shot |
| +`carlos releases prune` verb is a possible later wrapper over the same |
| +engine, not part of this branch. |
| + |
| +## 2. Shape of the feature |
| + |
| +- A member sets a per-app policy: `carlos releases retention -app X -keep 50`. |
| +- The policy is a bucket record; boxes enforce it ambiently on a slow loop. |
| +- The safety list (§5) always wins over the policy: promoted, rollback- |
| + reachable, tagged, or recently-shipped versions are never deleted. |
| +- Verification lives inside the command: the door answers with the policy |
| + as written plus how many versions are currently eligible; `carlos |
| + releases` afterward shows the shrunken history. |
| + |
| +## 3. Retention record |
| + |
| +Key: `apps/<acct>/<app>/retention.json`. Plain unsigned JSON (signing is |
| +for channel pointers; this follows `instances.json` / `restart-gen.json`). |
| +Absence means retention is off — the `instances.json` convention. Clearing |
| +the policy deletes the record. |
| + |
| +```go |
| +type Retention struct { |
| + Schema int `json:"schema"` // 1 |
| + App string `json:"app"` // must agree with the key path |
| + Keep int `json:"keep"` // >= KeepFloor |
| + SetBy string `json:"set_by,omitempty"` |
| + SetAt time.Time `json:"set_at"` |
| +} |
| +``` |
| + |
| +`GetRetention` enforces path/body agreement (the `provision.Record` |
| +precedent: a record that lies about identity is never guessed at) and |
| +pins Schema to 1 — unknown schema refuses, the wire-shape rule. |
| + |
| +`KeepFloor = 20` lives in `internal/release`, documented against |
| +`releasesCap`/`versionsCap` (both 20): the floor guarantees a pruned hole |
| +can never appear inside what the console page or CLI listing shows. |
| + |
| +Writes go through the console door with CAS (`Put` with the etag from the |
| +read), last-writer-wins on genuine races. |
| + |
| +## 4. CLI verb |
| + |
| +`carlos releases` grows sub-verb dispatch for exactly one word — |
| +`retention` — the `carlos version target` precedent (flat verb, one |
| +dispatched sub-verb, judgment call documented at the dispatch site). |
| + |
| +- `carlos releases retention -app X -keep 50` — set. Refuses `-keep` |
| + below `KeepFloor` client-side with the reason before any network call. |
| +- `carlos releases retention -app X -off` — clear. |
| +- `carlos releases retention -app X` — show policy + current eligible count. |
| +- `carlos releases retention -app X -keep 50 -dry-run` — full would-prune |
| + preview from the door; writes nothing. |
| + |
| +API-mode only, the `carlos restart` pattern: direct-bucket mode is refused |
| +with its own message (the audit trail and the eligibility preview are |
| +console-side). Flags via `accountFlags` (`-account`/`-account-id`) and |
| +`-console`, like every member verb. |
| + |
| +Output is honest about tense — the policy is *set*, pruning is *pending*: |
| + |
| + retention for titogo: keep 50 |
| + 272 of 322 versions currently eligible; boxes prune within ~10 minutes |
| + verify later with: carlos releases -app titogo |
| + |
| +## 5. Safety engine (shared selector) |
| + |
| +One shared function in `internal/release`, used by both the console door |
| +(preview / eligible count) and the box leg (enforcement): |
| + |
| +```go |
| +// PruneEligible returns versions safe to delete, oldest-first. |
| +// Fail-closed: any read/list/verify error aborts the whole app's prune. |
| +func PruneEligible(ctx context.Context, s Store, pub ed25519.PublicKey, |
| + acct, app string, keep int, now time.Time, |
| + alsoProtected map[string]bool) ([]string, error) |
| +``` |
| + |
| +A version is protected if it is any of: |
| + |
| +1. **Among the newest `keep`**, ranked by manifest.json's own mod stamp |
| + via the `ListMod` machinery from PR #134. The ranking core is shared |
| + with `VersionCandidates`, not duplicated. |
| +2. **Stamped newer than 72h** — `release.DefaultHolds`' fail-closed |
| + ceiling (`HoldFor` returns 72h for any unknown channel), so an |
| + in-flight promote can never race its own prune. Sourced from bake.go, |
| + not a fresh constant. |
| +3. **Named by any channel pointer** — `Version` plus the full `History` |
| + (≤ `historyMax` = 10 per channel), across everything under |
| + `ChannelsPrefix` including nested `canary/<branch>` names. |
| + Pointers are signature-verified with the release public key; an |
| + unverifiable pointer aborts the prune for the app (fail closed). |
| +4. **Named by any tag's `Release` field** (`ListTags`; note the mapping |
| + direction — `Tag.Version` is the semver name, `Tag.Release` is the |
| + shipped version it protects). |
| +5. **In the caller's `alsoProtected` set** — the box passes its own |
| + registry's `Version`/`PrevVersion` for the app; the console passes nil. |
| + |
| +Manifest-less version dirs (aborted ships) rank by their newest object's |
| +stamp, same as `VersionCandidates`, and are eligible once outside both |
| +keep-N and the 72h window. They are the one class that can shrink below |
| +the visible listing floor — they already surface as error rows, so |
| +removing them is cleanup, not a hole. |
| + |
| +`ValidVersion` is applied to every version name derived from listings |
| +before it is ever turned into a delete key. |
| + |
| +### Explicit rulings (Paul, 2026-08-10) |
| + |
| +- **Pointer `Audit` entries are NOT protected.** The audit trail |
| + (≤ `auditMax` = 20 promotions) is a log, not a rollback target; only |
| + `Version` + `History` are reachable by `Rollback`. |
| +- **The promote-vs-prune TOCTOU is accepted.** Promoting an ancient, |
| + untagged, un-pointed version can in principle race enforcement in the |
| + seconds between pointer read and delete. The outcome is a clean adopt |
| + failure (`ensureStaged` cannot fetch the manifest), recoverable by |
| + promoting again; preventing it would need bucket transactions we do |
| + not have. |
| +- **Enforcement scope is hosted-apps-only per box** (§6). An app with |
| + routes on no box never prunes; nothing is at stake there and |
| + trash→purge owns end of life. |
| + |
| +## 6. Box enforcement leg |
| + |
| +A new slow loop in the host agent — 5 min + jitter, first tick delayed — |
| +the `domainsLoop` cadence but **not inside it** (that loop is gated on |
| +`PLATFORM_APPS_DOMAIN`, an unrelated condition). Not the 2s serial fast |
| +pass: pruning is never urgent. |
| + |
| +Per tick, for each app the box has registry routes for (the registry is |
| +the work list): |
| + |
| +1. GET `retention.json`; 404 → skip the app. |
| +2. Run `PruneEligible` with the box's own registry `Version`/`PrevVersion` |
| + values for that app as `alsoProtected`. |
| +3. Delete each eligible version: **manifest first** (a half-deleted |
| + version then reads as an aborted ship — never a complete version with |
| + missing artifacts), then artifacts. |
| + |
| +Deletes are idempotent by store contract (`Delete` on a missing key is |
| +not an error), so boxes racing each other — or racing a re-run — is |
| +benign. A missing `CARLOS_RELEASE_PUBKEY` disables the leg with one log |
| +line, exactly like auto-purge (agent.go's domainsLoop precedent). |
| + |
| +One summary log line per app per pruning tick: |
| +`pruned 27 versions for bae/titogo (kept 50, protected 9)`. Quiet when |
| +nothing is eligible. |
| + |
| +## 7. Console door |
| + |
| +Routes (registered in `authcli.go` beside the other CLI API routes): |
| + |
| +- `GET /api/cli/apps/{account}/{app}/retention` — policy + eligible count. |
| +- `POST /api/cli/apps/{account}/{app}/retention` — body |
| + `{"keep": 50, "dry_run": false}` or `{"off": true}`. |
| + |
| +Gate order is `handleAPIRestart`'s, verbatim: |
| + |
| +1. `s.apiAccountApp(w, r)` — bearer token, membership-scoped account |
| + resolution, app existence; unknown account / foreign account / missing |
| + app are one byte-identical 404. |
| +2. `s.cli.write.allow(addr, now)` → 429 (POST only). |
| +3. `s.cfg.Releases == nil` → 503. |
| +4. Validate: `keep >= release.KeepFloor` else 400 with the floor named; |
| + `off` and `keep` are mutually exclusive. |
| +5. The bucket write (CAS Put, or Delete for `off`). |
| +6. `s.audit(acct, app, addr, "retention", ...)`. |
| +7. `s.bumpSerial(ctx)` — strictly after the successful write (the |
| + standing console-writes-bump-serial rule; nothing fast-acts on |
| + retention today, which is fine and future-proof). |
| + |
| +`dry_run: true` computes the eligible list with the shared selector |
| +(`alsoProtected` nil — the console cannot see box registries, and the |
| +box-side set only ever shrinks the delete list further) and returns it |
| +without writing, auditing, or bumping. |
| + |
| +Authorization ruling (Paul, 2026-08-10): **member-reachable**, like every |
| +existing CLI door. Retention is app config; the safety engine is the |
| +guardrail. No owner tier, no step-up. |
| + |
| +Console UI in v1: the Releases tab shows a read-only |
| +"retention: keep 50" line when the record exists. No browser write path. |
| + |
| +## 8. Invariants restated |
| + |
| +- `putImmutable`'s comment is amended in place: release objects are |
| + **immutable-while-present** — their bytes never change under a key, but |
| + retention (this design) and whole-app purge (trash.go) may delete them. |
| +- Wire-shape rule: `Retention` is a new record, additive; nothing existing |
| + parses it. Schema pinned to 1, unknown refuses. |
| +- PR #137 (route records) touches `provision.Record`; retention is a |
| + separate record at a separate key — no collision. |
| + |
| +## 9. Error handling summary |
| + |
| +- Selector: fail closed per app — any list/read/signature error skips the |
| + app this tick with a log line; never "prune what we could see". |
| +- Door: floor violation 400, identity failures byte-identical 404, rate |
| + limit 429, no store 503; dry-run can fail with 502 if the selector |
| + errors (surfaced, not swallowed). |
| +- Box leg: per-app isolation — one app's error never stops another app's |
| + enforcement; per-key delete errors log and leave the remainder for the |
| + next tick (idempotency makes retry free). |
| +- CLI: client-side validation first; API errors surfaced with the |
| + restart-style "or this console predates retention" hint on 404 of the |
| + route itself. |
| + |
| +## 10. Testing |
| + |
| +- Selector unit tests, one per protected class: keep-N ranking, 72h |
| + window, pointer Version, pointer History, tag Release, alsoProtected, |
| + manifest-less dirs, unverifiable-pointer abort, empty-app no-op. |
| +- DirStore end-to-end: seed releases + pointers + tags, set a policy, |
| + run the leg once, assert exactly the eligible set is gone and |
| + manifest-first ordering held (observable via a store that records |
| + delete order). |
| +- Door tests: 404 identity (foreign account = missing app byte-identical), |
| + floor 400, rate limit, audit row, serial bump after write, dry-run |
| + writes/audits/bumps nothing. |
| +- CLI test shells through `carlosTestEnv` with `cmd.Dir` set to a scratch |
| + dir (the 40-accounts rule). |
| +- Full gate: `go build ./...`, `GOOS=darwin go build ./...`, `go vet |
| + ./...`, `gofmt -l` clean on touched files, `go test ./... -count=1`. |
| + |
| +## 11. Rollout |
| + |
| +1. Merge + deploy consoles (door live; CLI verb usable). |
| +2. Deploy box binaries (leg live; existing update-platform-binary.sh path, |
| + no new units or arming needed — the leg is in-process in the agent). |
| +3. Set `keep 50` on titogo (account `bae`) as the proving case; verify |
| + `carlos releases -app titogo` shrinks and the box log line appears. |
| +4. File the follow-up for the manual `carlos releases prune` wrapper and |
| + any console-UI write path, if wanted after living with ambient. |
| + |
| +## 12. Non-goals |
| + |
| +- Manual one-shot prune verb (follow-up over the same engine). |
| +- Age-based or combined retention knobs (`-keep-days`) — YAGNI until asked. |
| +- Per-version trash/tombstone cooloff — the safety list plus the 72h |
| + ceiling cover the realistic risks; a pruned CI version is not an app. |
| +- Console browser write path for the policy. |
| +- Pruning apps hosted on no box. |