| new file mode 100644 |
| index 000000000..a353ff9fc |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-24-edge-asset-cache-plan-2-disk-store.md |
| @@ -0,0 +1,160 @@ |
| +# Edge asset cache, plan 2: the disk store Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** A self-contained package, `internal/assetcache`, that stores cached asset responses on a box's disk under a budget, with worst-case reservations, a free-space floor, per-box and per-app fill bounds, fair-share eviction, expiry and a sweep, and that nothing calls yet (plan 3 does). |
| + |
| +**Architecture:** One `Store` per process over one directory. Each entry is three files named by the hex sha256 of its key: `<id>.body` (identity), optional `<id>.gz`, and `<id>.meta` (JSON, written last, which is what makes an entry exist). An in-memory index is rebuilt from the `.meta` files at `Open`. A fill is a `Fill` holding a reservation; it writes temporaries in the same directory and renames them on `Commit`. Free space comes from an `FS` interface whose Linux implementation is `statfs`. The package knows nothing about HTTP semantics beyond storing headers it is given: admission, freshness rules and serving belong to plan 3. The design is spec §7, written so the media roadmap's stage 1 can reuse it with its own directory and budget (spec §11). |
| + |
| +**Tech Stack:** Go 1.26 standard library (`os`, `compress/gzip`, `crypto/sha256`, `encoding/json`), `golang.org/x/sys/unix` for `Statfs` on Linux. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-24-edge-asset-cache-design.md` (rev 8). Read §5.3 ("Coalescing", "Every fill has a deadline"), §5.4 and §7 before starting. |
| + |
| +**Branch:** `edge-asset-cache-2`, cut from amadan `main` after the spec branch has merged. Independent of plan 1. |
| + |
| +## Global Constraints |
| + |
| +Every task's requirements include this section. |
| + |
| +- **Real files.** Every test runs on a real directory from `t.TempDir()`. A fake `FS` (free-space source) is allowed and is the only stand-in; tests using it say "not acceptance" in the doc comment. Task 2's `TestStatfsReportsTheTempDir` is the acceptance test for the real `FS` and runs on Linux only. |
| +- **Targeted local runs; CI runs the full suite.** `go test ./internal/assetcache/ -count=1` is allowed in full (new, small package). `go build ./...`, `go vet ./...`, `gofmt -l .` run locally. |
| +- **Cross-compile gate.** `internal/edge` is linked into the CLI, and plan 3 imports this package from it. `statfs_linux.go` carries `//go:build linux`; `statfs_other.go` carries `//go:build !linux` and returns `ErrUnsupported`. Before every push: `GOOS=darwin GOARCH=arm64 go build ./...` and `GOOS=windows GOARCH=amd64 go build ./...`. |
| +- **No user-facing strings.** Every error in this package is an internal fragment, so no copy rows. |
| +- **No tenant names.** Keys in tests use account `abc`, apps `shop` and `blog`, host `shop.example.com`. |
| +- **Go cache.** Leave `GOCACHE`, `GOMODCACHE` and `GOTMPDIR` at their defaults. |
| +- **Census overlay per branch, loaded LAST.** `plan14-root-reads-edge-asset-cache-2.tsv`, `rev3=PROPOSED-edge-asset-cache-2`, after whatever loads last at the merge (plan 1's overlay if it has merged), previous seal checked first. |
| +- **Merge main first; amadan tasks; scratch under `t.TempDir()`, removed by name.** As in plan 1. |
| +- **Spec numbers, verbatim.** Default budget: the smaller of 4 GiB and 10% of the filesystem's size (`CARLOS_ASSET_CACHE_MAX` overrides, read by plan 3). Free-space floor: no fill admitted unless free space minus outstanding reservations minus this one stays at or above 15% of the filesystem's size. Reservation: `Content-Length` if known, else 32 MiB, doubled for the gzip copy, plus 64 KiB of metadata. Body cap: 32 MiB. Fill bounds: 16 per box, 4 per app. Fair share: budget divided by the number of apps with entries, never more than 25% of the budget for one app. |
| + |
| +## Review Focus |
| + |
| +1. **A crash between writing `.body` and `.meta`.** A person expects a restart to leave no half entry and no leaked disk. Test: Task 1, `TestOpenRemovesOrphanBodiesAndTemporaries`. |
| +2. **A `.meta` file from a newer binary** with fields this one does not know. A person expects it to load (unknown JSON keys ignored), not to be deleted. Test: Task 1, `TestOpenKeepsMetaWithUnknownFields`; a `.meta` that does not parse is deleted with its files. |
| +3. **A filesystem that fills from outside the cache** (logs, a database) while fills are in flight. A person expects in-flight writes to stop at their reservation and new fills to be refused, never the cache pushing past the floor. Test: Task 3, `TestReserveRefusesWhenSomethingElseFillsTheDisk`. |
| +4. **One app over its share when a second app arrives.** A person expects the second app's fill to evict the first app's oldest entries, not its own and not to be refused. Test: Task 5, `TestNewAppEvictsTheAppOverItsShare`. |
| +5. **Clock moving backwards.** An entry stored "in the future" after a clock step. A person expects it to stay valid until its stored expiry and never to report a negative age. Test: Task 4, `TestAgeNeverNegative`. |
| + |
| +## Census impact |
| + |
| +- New package `internal/assetcache`: its file-reading declarations (`Open`'s scan, `Store.Get`'s open of `.body`/`.gz`) are new `@legacy` rows of class `cache`, task `accepted-legacy`, with their envelope pinned. If the census has no `cache` class, use the class the static site cache's units carry (`internal/edge/static.go` `*`) and say so in the notes file. |
| +- Notes: `docs/superpowers/notes/census-proposal-edge-asset-cache-2.md`, ruling asked on #280. |
| + |
| +## File structure |
| + |
| +| File | Responsibility | |
| +|---|---| |
| +| `internal/assetcache/key.go` | `Key`, its id. | |
| +| `internal/assetcache/meta.go` | `Meta`, its file encoding. | |
| +| `internal/assetcache/store.go` | `Config`, `Open`, the index, `Get`, `Retain`, `Stats`. | |
| +| `internal/assetcache/fill.go` | `Reserve`, `Fill`, `Commit`, `Abort`. | |
| +| `internal/assetcache/evict.go` | Fair-share eviction. | |
| +| `internal/assetcache/fs.go`, `statfs_linux.go`, `statfs_other.go` | Free space. | |
| +| `internal/assetcache/*_test.go` | Tests. | |
| + |
| +--- |
| + |
| +### Task 1: Keys, metadata and `Open` |
| + |
| +**Files:** |
| +- Create: `internal/assetcache/key.go`, `meta.go`, `store.go`, `store_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `type Key struct { Account, App, Gen, Host, Path, RawQuery string }`; `func (k Key) ID() string` (hex sha256 over the six fields, each length-prefixed so no two keys collide); `func (k Key) AppScope() string` (`Account + "/" + App`) |
| + - `type Meta struct { Key Key; Header http.Header; Lifetime time.Duration; Expires time.Time; StoredAt time.Time; Size int64; GzipSize int64 }` (`GzipSize == 0` means no gzip copy) |
| + - `type Config struct { Dir string; Budget int64; FS FS; Now func() time.Time }` |
| + - `func Open(cfg Config) (*Store, error)`: creates `Dir` (0700), deletes every `*.tmp-*` file, deletes `.body`/`.gz` with no `.meta`, deletes a `.meta` that does not parse together with its files, and indexes the rest. |
| + - `type Stats struct { Entries int; Bytes int64; Reserved int64; Apps int }`; `func (s *Store) Stats() Stats` |
| + |
| +- [ ] **Step 1: Write the failing tests.** `TestKeyIDIsStableAndDistinct` (same key same id; moving one character between `Path` and `RawQuery` changes it); `TestOpenCreatesTheDirectory`; `TestOpenRemovesOrphanBodiesAndTemporaries`; `TestOpenKeepsMetaWithUnknownFields`; `TestOpenDropsUnparseableMeta`; `TestOpenIndexesExistingEntries` (write three entries' files by hand in the documented layout, `Open`, `Stats` counts them and their bytes). |
| +- [ ] **Step 2:** Run `go test ./internal/assetcache/ -count=1`. Expected: FAIL (package missing). |
| +- [ ] **Step 3:** Implement. |
| +- [ ] **Step 4:** Run the same command. Expected: PASS. |
| +- [ ] **Step 5:** Commit. |
| + |
| +### Task 2: Free space |
| + |
| +**Files:** |
| +- Create: `internal/assetcache/fs.go`, `statfs_linux.go`, `statfs_other.go`, `fs_test.go`, `fs_linux_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `type FS interface { Space(dir string) (total, free uint64, err error) }` |
| + - `var Statfs FS` (Linux: `unix.Statfs`, `free` from `Bavail * Bsize`; other systems: returns `ErrUnsupported`) |
| + - `var ErrUnsupported = errors.New("assetcache: free space unknown on this system")` |
| + - `func DefaultBudget(total uint64) int64` (the smaller of 4 GiB and `total/10`) |
| + |
| +- [ ] **Step 1: Write the failing tests.** `TestDefaultBudget` (20 GiB gives 2 GiB, 60 GiB gives 4 GiB, 0 gives 0). `TestStatfsReportsTheTempDir` (Linux only; `total > 0`, `free <= total`). |
| +- [ ] **Step 2:** Run `go test ./internal/assetcache/ -run 'TestDefaultBudget|TestStatfs' -count=1`. Expected: FAIL. |
| +- [ ] **Step 3:** Implement. |
| +- [ ] **Step 4:** Run the same command, then both cross-compiles. Expected: PASS, clean builds. |
| +- [ ] **Step 5:** Commit. |
| + |
| +### Task 3: Reservations, the floor and fill bounds |
| + |
| +**Files:** |
| +- Create: `internal/assetcache/fill.go`, `fill_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: `FS`, `Store` (Tasks 1 and 2). |
| +- Produces: |
| + - `const MaxBody = 32 << 20`, `const metaReserve = 64 << 10`, `const maxFillsPerBox = 16`, `const maxFillsPerApp = 4` |
| + - `var ErrNoRoom`, `ErrBelowFloor`, `ErrBusy`, `ErrTooLarge` (sentinels) |
| + - `func (s *Store) Reserve(k Key, contentLength int64) (*Fill, error)`: `contentLength < 0` means unknown; reserves `2*n + metaReserve` where `n` is `contentLength` or `MaxBody`; `ErrTooLarge` when `contentLength > MaxBody`; `ErrBusy` over either fill bound; evicts (Task 5's `makeRoom`, a no-op until then) then `ErrNoRoom` if `used + reserved + r > Budget`; `ErrBelowFloor` if `free - reserved - r < total*15/100`; an `FS` error refuses with that error. |
| + - `type Fill struct` implementing `io.Writer` for the identity body; a write past the reservation's identity half or past `MaxBody` returns `ErrTooLarge` and marks the fill failed. |
| + - `func (f *Fill) Abort()`: deletes temporaries, releases the reservation and the fill slot; safe to call twice and after `Commit`. |
| + |
| +- [ ] **Step 1: Write the failing tests** (a fake `FS` with settable total and free; "not acceptance"). |
| + - `TestReserveWorstCaseWithUnknownLength` (`Stats().Reserved == 2*MaxBody + 64<<10`), `TestReserveUsesContentLength`. |
| + - `TestReserveRefusesBelowTheFloor`: total 8 GiB, free 15% plus one reservation fits once, the second concurrent reservation is `ErrBelowFloor`. This is spec §7's sixteen-fill case reduced to its rule. |
| + - `TestReserveRefusesWhenSomethingElseFillsTheDisk` (free drops between two calls). |
| + - `TestReserveBoundsFillsPerBoxAndPerApp` (17th fill and 5th per app are `ErrBusy`). |
| + - `TestWritePastTheReservationFails`, `TestAbortReleasesEverything` (reserved bytes and slots back to zero, no files left), `TestAbortTwiceIsSafe`. |
| +- [ ] **Step 2:** Run `go test ./internal/assetcache/ -run 'TestReserve|TestWritePast|TestAbort' -count=1`. Expected: FAIL. |
| +- [ ] **Step 3:** Implement. The index lock is held only to update counters, never across file I/O. |
| +- [ ] **Step 4:** Run the same command. Expected: PASS. |
| +- [ ] **Step 5:** Commit. |
| + |
| +### Task 4: Commit, lookup and expiry |
| + |
| +**Files:** |
| +- Modify: `internal/assetcache/fill.go`, `store.go` |
| +- Test: `internal/assetcache/entry_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `func (f *Fill) Commit(m Meta, gzipCopy bool) error`: gzips the identity body into a temporary when `gzipCopy`, abandoning the copy (`GzipSize = 0`) if it would exceed the identity size or the reservation's gzip half; fsyncs and renames `.body`, then `.gz`, then writes `.meta` last; sets `Size`, `GzipSize`, `StoredAt = Now()`; releases the reservation; on any error behaves as `Abort`. A `Commit` for a key that already has an entry replaces it. |
| + - `type Entry struct { Meta Meta }`; `func (e *Entry) Open(gzip bool) (*os.File, error)` |
| + - `func (s *Store) Get(k Key) (*Entry, bool)`: false and the entry deleted when `Now()` is at or past `Meta.Expires`; marks the entry recently used. |
| + - `func (m Meta) Age(now time.Time) time.Duration` (`Lifetime - Expires.Sub(now)`, never below zero) |
| + |
| +- [ ] **Step 1: Write the failing tests.** `TestCommitThenGetRoundTrips` (header and body bytes equal); `TestCommitWritesMetaLast` (a crash simulated by stopping before the meta rename leaves `Open` with no entry); `TestGzipCopyIsAbandonedWhenLarger` (random bytes); `TestGzipCopyIsKept` (CSS text; reading `.gz` through `gzip.NewReader` gives the identity bytes); `TestGetExpires` (fake `Now` past `Expires`: false, files gone); `TestCommitReplacesAnEntry`; `TestAgeNeverNegative`. |
| +- [ ] **Step 2:** Run `go test ./internal/assetcache/ -run 'TestCommit|TestGzipCopy|TestGetExpires|TestAge' -count=1`. Expected: FAIL. |
| +- [ ] **Step 3:** Implement. |
| +- [ ] **Step 4:** Run the same command. Expected: PASS. |
| +- [ ] **Step 5:** Commit. |
| + |
| +### Task 5: Fair-share eviction and the sweep |
| + |
| +**Files:** |
| +- Create: `internal/assetcache/evict.go`, `evict_test.go` |
| +- Modify: `fill.go` (`Reserve` calls `makeRoom`), `store.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + - `func (s *Store) makeRoom(need int64)` (internal): while `used + reserved + need > Budget`, evict the least recently used entry of the app with the largest `bytes - share`, `share = min(Budget/apps, Budget/4)`; stops when nothing is left to evict. |
| + - `func (s *Store) Retain(keep func(Key) bool) int`: deletes every entry whose key `keep` rejects, returns how many. Plan 3 calls it with "this key's generation and owner are the host's current ones". |
| + |
| +- [ ] **Step 1: Write the failing tests.** `TestEvictionTakesFromTheAppFurthestOverItsShare` (app `shop` holds 80% of the budget, `blog` 10%; a `blog` fill evicts `shop`'s oldest); `TestOneAppWalkingUniqueQueriesEvictsItself` (only `shop` has entries, it fills past the budget, `blog`'s single entry survives); `TestNewAppEvictsTheAppOverItsShare`; `TestShareNeverExceedsAQuarter` (one app alone is still evicted first once it passes 25%); `TestRetainDeletesRejectedKeys`. |
| +- [ ] **Step 2:** Run `go test ./internal/assetcache/ -run 'TestEviction|TestOneApp|TestNewApp|TestShare|TestRetain' -count=1`. Expected: FAIL. |
| +- [ ] **Step 3:** Implement; per-app LRU lists inside the index. |
| +- [ ] **Step 4:** Run `go test ./internal/assetcache/ -count=1 -race`. Expected: PASS. |
| +- [ ] **Step 5:** Commit. |
| + |
| +### Task 6: Census, merge and CI |
| + |
| +- [ ] **Step 1:** `git fetch origin && git merge origin/main`; re-read the last overlay. |
| +- [ ] **Step 2:** Run `go test ./internal/provision/ -run TestPlan14RootReadCensus -count=1`. Expected: FAIL naming the new package's units and envelope. |
| +- [ ] **Step 3:** Write `plan14-root-reads-edge-asset-cache-2.tsv` and the notes file; post on #280. |
| +- [ ] **Step 4:** Census test PASS; cross-compiles and `gofmt -l .` clean. |
| +- [ ] **Step 5:** Commit, push, CI green on the tip, merge per plan 1's Global Constraints with this branch's name. |