| index c51e0f2a7..07d996650 100644 |
| --- a/docs/superpowers/plans/2026-09-10-store-egress-and-durability-meters.md |
| +++ b/docs/superpowers/plans/2026-09-10-store-egress-and-durability-meters.md |
| @@ -2,60 +2,66 @@ |
| |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| |
| -**Goal:** Meter the two costs the platform currently charges for but cannot see — bytes an app serves straight from its own object-store bucket, and the S3 writes its database replication makes — and make the cost model consume them instead of pricing edge HTTP requests at an S3 rate. |
| +**Goal:** Meter the two costs the platform charges for but cannot see — bytes an app serves straight from its own object-store bucket, and the S3 writes its database replication makes — and make the cost model consume them instead of pricing edge HTTP requests at an S3 rate. |
| |
| -**Architecture:** Both meters come from one mechanism. S3 server access logging is switched on for the regional data bucket (which carries every litestream replica, keyed by database path) and for each tenant store bucket at mint time (one bucket per app, so the bucket name *is* the tenant). A console-side pass parses the previous day's logs into per-tenant sums, exactly as `storagePass` already builds a per-day storage object, and carries the same "unswept, never zero" honesty flag. Nothing runs on a box, because the bytes never cross one. |
| +**Architecture:** Both meters come from S3 server access logs, delivered to a dedicated log bucket. A console-side pass parses the previous days' logs into per-tenant sums. Nothing runs on a box, because the bytes never cross one. Three properties of that log stream drive most of the design: AWS may deliver a record **twice**, may deliver it **late**, and names log objects by **delivery time, not event time** — so the pass deduplicates on request id, takes the day from each record's own timestamp, and keeps rewriting a day until plan one's statement for that month closes. |
| |
| -**Tech Stack:** Go, AWS SDK v2 (`s3.PutBucketLogging`), Terraform for the data bucket. No new dependencies. |
| +**Tech Stack:** Go, AWS SDK v2 (`awss3.PutBucketLogging`), Terraform. No new dependencies. |
| |
| -**Spec:** `docs/superpowers/specs/2026-09-07-carloku-pricing-plans-design.md` — build steps 4, 5 and 6, plus the box-side durable-usage-write half of step 3 that `2026-09-08-invoice-grade-statements.md` deferred here by name. |
| +**Spec:** `docs/superpowers/specs/2026-09-07-carloku-pricing-plans-design.md` — build steps 4, 5 and 6. |
| |
| -**Depends on:** `docs/superpowers/plans/2026-09-08-invoice-grade-statements.md` (plan one). Task 3 and Task 4 register new sources with `usage.MeterSource`, which that plan introduces. Do not start this plan until it has landed. |
| +**Depends on:** `docs/superpowers/plans/2026-09-08-invoice-grade-statements.md` (plan one), for `usage.MeterSource`, `SourceDays`, statement closure, and the daily census this plan reuses for replica ownership. Do not start until it has landed. |
| |
| ## Global Constraints |
| |
| -- **Access logs are customer-shaped data.** They record object keys and end-user IP addresses. `infra/modules/deployment-buckets/main.tf:8-10` states the deployment bucket holds "routing metadata only — nothing customer-shaped rests here", so **logs go to the regional data bucket**, which is the residency promise, and never to the deployment bucket. The parsed daily totals — bytes and counts, no keys, no addresses — are what reaches the deployment bucket. |
| -- **Raw logs are deleted after parsing.** A lifecycle rule expires them at 30 days. We keep the aggregate, not the trail. No console view ever renders a raw log line. |
| -- **Server access logging is best-effort.** AWS delivers "on a best-effort basis" with no completeness guarantee and hours of latency. So a day with logs present is *covered*, never automatically *complete* — see Task 3's `Unswept` handling. Whether that is good enough to bill from is the billing plan's decision, and this plan deliberately does not make it. |
| -- **Wire shapes are additive.** New day objects are new keys; new fields carry `omitempty`. Never remove or retype an existing field. (AGENTS.md, "Rules this repo has paid for".) |
| -- **The Go and JS cost arithmetic are one definition in two files.** `internal/economics/compute.go:93-96` says so explicitly: `PriceAccount` is mirrored by `priceAccount` in `internal/console/templates/economics.html` (`econ_maths_js`, :545). Task 5 changes both in one commit or the page disagrees with the server that served it. |
| +- **Access logs are customer-shaped.** They carry object keys, client IP addresses and user agents. `infra/modules/deployment-buckets/main.tf:8-10` states the deployment bucket holds "routing metadata only — nothing customer-shaped rests here". Logs therefore land in a **dedicated log bucket in the deployment's own region**, and only aggregates — bytes and counts — reach the deployment bucket. No console view ever renders a raw log line. |
| +- **The log bucket must not log itself.** AWS permits a bucket to be its own log destination and warns it creates an unbounded loop, since each log write generates another log record. The destination has logging disabled, always. |
| +- **Server access logging is best-effort.** AWS guarantees neither completeness nor at-most-once delivery. Every pass here therefore fails toward `Unswept`, never toward a measured zero, and **an empty read is never evidence that nothing was served**. Whether a source with no completeness guarantee may back an invoice line at all is the billing plan's decision; this plan collects and refuses to conclude. |
| +- **Wire shapes are additive.** New day objects are new keys; new fields carry `omitempty`. (AGENTS.md, "Rules this repo has paid for".) |
| +- **The cost arithmetic exists in three places, not two.** `PriceAccount` (`internal/economics/compute.go:98`), the shared `priceAccount` JS (`internal/console/templates/economics.html:665`), and a *separate* model-total recompute at `templates/economics.html:379-388`. Task 5 changes all three in one commit. |
| - **Build gate before any commit lands:** `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l .` clean tree-wide. `make test` covers the last three. |
| -- **Sandbox note (AGENTS.md:167):** `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; go may need the sandbox disabled, "read-only file system" from the toolchain is the tell. |
| -- **Cross-compile gate.** `20-cross@5` builds darwin/windows/linux-arm64. Task 6 touches `internal/host`, which links into the CLI people run on Macs; any Linux-only assumption breaks their build. |
| +- **Sandbox note (AGENTS.md:167):** `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off`; go may need the sandbox disabled. |
| - **CI is the `amadan` remote.** A branch pushed only to `origin` has been gated by nothing. |
| |
| ## Why not the alternatives |
| |
| -Recorded so nobody re-litigates it mid-implementation. |
| - |
| | Option | Why not | |
| | --- | --- | |
| -| CloudWatch S3 request metrics (`BytesDownloaded` per bucket) | $0.30 per bucket per month. One bucket per app, and Personal is €24/**year** — a single bucket would eat 15% of the subscription. It would invert the unit economics the spec publishes. | |
| -| CloudTrail data events | $0.10 per 100k events, near-real-time and far more complete. The right upgrade if best-effort logging ever blocks billing, but not the place to start. | |
| -| Cost Explorer by usage type | `cepull.go:91-101` groups by SERVICE; CE has no bucket dimension. Deployment-wide totals only, never per tenant. | |
| -| Proxy store traffic through the edge | Defeats the entire point of handing apps direct credentials, and `internal/objstore/objstore.go:1-5` states the package "never touches the data plane" as a design property. | |
| -| Model durability from replication class × uptime | An upper bound, not a measurement. Fine for a dashboard estimate, wrong for an invoice line the customer can dispute. | |
| +| CloudWatch S3 request metrics | $0.30 per bucket per month, one bucket per app, against a €24 **per year** plan. It would invert the economics the spec publishes. | |
| +| CloudTrail data events | $0.10 per 100k events, but exactly-once and prompt. **The upgrade path if best-effort logging proves unbillable** — this plan is deliberately built so only the reader changes. | |
| +| Cost Explorer by usage type | `cepull.go:91-101` groups by SERVICE; CE has no bucket dimension. | |
| +| Proxy store traffic through the edge | `internal/objstore/objstore.go:1-5` states the package "never touches the data plane" as a design property. | |
| +| Model durability from replication class × uptime | An upper bound, not a measurement. Wrong for an invoice line a customer can dispute. | |
| + |
| +## Deliberately not in this plan |
| + |
| +**The box-side durable usage write** (plan one deferred it here by name) is **moved to its own plan**. A first draft proposed re-queueing failed rows into the live `edge.UsageMeter` map. That is not replay-safe: `usage.UpsertEdgeDay` (`internal/usage/edge.go:104`) *adds* deltas on every successful write, so a commit whose response was lost double-counts on retry, and a re-queued row loses the day it belonged to — an August failure retried in September becomes September usage. `internal/analytics` already solves this properly, with dated `Delta` batches carrying a box-monotonic `Seq` and an idempotent `Day.Merge` that drops `x.Seq <= d.Seq[x.Box]` (`internal/analytics/day.go:83-85`, `meter.go:220-221`). Giving edge usage the same treatment is a real piece of work with restart-recovery and rollover cases of its own, and it does not belong bolted to the end of a metering plan. |
| + |
| +**The flush-only replication class** stays with enforcement (plan three): it needs a plan definition to read, which does not exist yet. |
| |
| --- |
| |
| ### Task 1: Parse an S3 server access log line |
| |
| -Everything downstream consumes this, and it is a pure function, so it goes first. The format is space-separated with bracketed timestamps and quoted strings, and any field may be `-`. |
| +Everything downstream consumes this and it is pure, so it goes first. |
| + |
| +**Field positions are the thing to get right.** Zero-indexed, counting a bracketed time and each quoted string as one token: `bucket_owner=0, bucket=1, time=2, remote_ip=3, requester=4, request_id=5, operation=6, key=7, request_uri=8, http_status=9, error_code=10, bytes_sent=11, object_size=12`. |
| + |
| +**Four fields the pass needs that a first draft threw away.** Time decides which day a record belongs to, because log *object* names are delivery times and a request near midnight lands in the wrong month otherwise. Request id deduplicates, because AWS explicitly permits delivering a record more than once. Remote IP separates an app fetching its own object through the platform from a real external download — same-region S3-to-EC2 transfer is free, and billing it as egress would overcharge. None of the three is persisted: they live inside one pass and die with it. |
| |
| **Files:** |
| -- Create: `internal/usage/accesslog.go` |
| -- Test: `internal/usage/accesslog_test.go` |
| +- Create: `internal/usage/accesslog.go`, `internal/usage/accesslog_test.go` |
| |
| **Interfaces:** |
| - Consumes: nothing. |
| - Produces: |
| - - `usage.AccessLogRecord{Bucket, Operation, Key string; BytesSent int64; Status int}` |
| - - `func usage.ParseAccessLogLine(line string) (AccessLogRecord, bool)` — `false` for a line too short or malformed to trust. |
| + - `usage.AccessLogRecord{Bucket, Operation, Key, RequestID, RemoteIP string; Time time.Time; BytesSent int64; Status int}` |
| + - `func usage.ParseAccessLogLine(line string) (AccessLogRecord, bool)` |
| - `func usage.ParseAccessLog(r io.Reader, fn func(AccessLogRecord)) (parsed, skipped int, err error)` |
| |
| - [ ] **Step 1: Write the failing test** |
| |
| -Create `internal/usage/accesslog_test.go`: |
| +Create `internal/usage/accesslog_test.go`. Note the fixture's bytes-sent and object-size are **deliberately different**: equal values hide an off-by-one in the field offsets, which is exactly the bug an earlier draft of this plan shipped. |
| |
| ```go |
| package usage |
| @@ -65,18 +71,18 @@ import ( |
| "testing" |
| ) |
| |
| -// A real GET line, fields per the S3 server access log format. Bytes sent is |
| -// field 13 (1-indexed), operation is field 8, and the bracketed time plus the |
| -// quoted request URI both contain spaces. |
| -const getLine = `79a5 my-bucket [06/Feb/2026:00:00:38 +0000] 192.0.2.3 79a5 3E57427F33A59F07 REST.GET.OBJECT photo.jpg "GET /photo.jpg HTTP/1.1" 200 - 3462992 3462992 73 72 "-" "S3Console/0.4" - abc SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.eu-west-1.amazonaws.com TLSv1.2 - -` |
| +// Fields, in order: bucket_owner bucket [time] remote_ip requester |
| +// request_id operation key "request_uri" status error_code bytes_sent |
| +// object_size total_time turn_around "referer" "user_agent" ... |
| +const getLine = `79a5 my-bucket [06/Feb/2026:00:00:38 +0000] 192.0.2.3 arn:aws:iam::1:user/u REQ001 REST.GET.OBJECT photo.jpg "GET /photo.jpg HTTP/1.1" 200 - 900 3462992 73 72 "-" "S3Console/0.4" - abc SigV4 ECDHE AuthHeader s3.eu-west-1.amazonaws.com TLSv1.2 - -` |
| |
| -func TestParseAccessLogLineGet(t *testing.T) { |
| +func TestParseAccessLogLineFields(t *testing.T) { |
| rec, ok := ParseAccessLogLine(getLine) |
| if !ok { |
| t.Fatal("a well-formed GET line must parse") |
| } |
| if rec.Bucket != "my-bucket" { |
| - t.Errorf("bucket = %q, want my-bucket", rec.Bucket) |
| + t.Errorf("bucket = %q", rec.Bucket) |
| } |
| if rec.Operation != "REST.GET.OBJECT" { |
| t.Errorf("operation = %q", rec.Operation) |
| @@ -84,44 +90,67 @@ func TestParseAccessLogLineGet(t *testing.T) { |
| if rec.Key != "photo.jpg" { |
| t.Errorf("key = %q", rec.Key) |
| } |
| - if rec.BytesSent != 3462992 { |
| - t.Errorf("bytes_sent = %d, want 3462992", rec.BytesSent) |
| + if rec.RequestID != "REQ001" { |
| + t.Errorf("request id = %q", rec.RequestID) |
| + } |
| + if rec.RemoteIP != "192.0.2.3" { |
| + t.Errorf("remote ip = %q", rec.RemoteIP) |
| + } |
| + // bytes_sent is 900; object_size is 3462992. Reading the wrong one is |
| + // the whole point of this assertion. |
| + if rec.BytesSent != 900 { |
| + t.Errorf("bytes_sent = %d, want 900 (object_size is 3462992)", rec.BytesSent) |
| } |
| if rec.Status != 200 { |
| - t.Errorf("status = %d, want 200", rec.Status) |
| + t.Errorf("status = %d", rec.Status) |
| + } |
| + if got := rec.Time.UTC().Format("2006-01-02T15:04:05"); got != "2026-02-06T00:00:38" { |
| + t.Errorf("time = %s", got) |
| } |
| } |
| |
| -// A PUT sends no bytes back: bytes_sent is "-", which must read as 0 and |
| -// must not fail the line — we still need the operation to count writes. |
| +// A PUT sends nothing back: bytes_sent is "-", which reads as 0 without |
| +// failing the line — the durability meter still needs the operation. |
| func TestParseAccessLogLineDashBytes(t *testing.T) { |
| line := strings.Replace(getLine, "REST.GET.OBJECT", "REST.PUT.OBJECT", 1) |
| - line = strings.Replace(line, "200 - 3462992 3462992", "200 - - 3462992", 1) |
| + line = strings.Replace(line, "200 - 900 3462992", "200 - - 3462992", 1) |
| rec, ok := ParseAccessLogLine(line) |
| if !ok { |
| t.Fatal("a PUT line must parse") |
| } |
| if rec.BytesSent != 0 { |
| - t.Errorf("bytes_sent = %d, want 0 for %q", rec.BytesSent, "-") |
| - } |
| - if rec.Operation != "REST.PUT.OBJECT" { |
| - t.Errorf("operation = %q", rec.Operation) |
| + t.Errorf("bytes_sent = %d, want 0", rec.BytesSent) |
| } |
| } |
| |
| -func TestParseAccessLogLineRejectsJunk(t *testing.T) { |
| - for _, line := range []string{"", " ", "too few fields here", `79a5 b [06/Feb/2026:00:00:38 +0000] ip req id`} { |
| - if _, ok := ParseAccessLogLine(line); ok { |
| - t.Errorf("ParseAccessLogLine(%q) = ok, want rejected", line) |
| - } |
| +// AWS permits UNESCAPED quotes inside request_uri, referer and user_agent. |
| +// The closing quote is found by structure — the token after it must be a |
| +// status — not by taking the first quote encountered. |
| +func TestParseAccessLogLineInteriorQuote(t *testing.T) { |
| + line := strings.Replace(getLine, |
| + `"GET /photo.jpg HTTP/1.1"`, |
| + `"GET /a"b.jpg HTTP/1.1"`, 1) |
| + rec, ok := ParseAccessLogLine(line) |
| + if !ok { |
| + t.Fatal("an interior quote in request_uri must not break the line") |
| + } |
| + if rec.BytesSent != 900 || rec.Status != 200 { |
| + t.Errorf("fields shifted: bytes=%d status=%d", rec.BytesSent, rec.Status) |
| } |
| } |
| |
| -// An unterminated quote must not swallow the rest of the file. |
| -func TestParseAccessLogUnterminatedQuote(t *testing.T) { |
| - bad := `79a5 my-bucket [06/Feb/2026:00:00:38 +0000] 192.0.2.3 79a5 REQ REST.GET.OBJECT k "GET /k HTTP/1.1 200 - 1 1 1 1 "-" "ua" - - - - - - - - -` |
| - if _, ok := ParseAccessLogLine(bad); ok { |
| - t.Error("an unterminated quote must be rejected, not parsed") |
| +func TestParseAccessLogLineRejects(t *testing.T) { |
| + cases := map[string]string{ |
| + "empty": "", |
| + "too few fields": "79a5 b [06/Feb/2026:00:00:38 +0000] ip req id", |
| + "unterminated uri": `79a5 b [06/Feb/2026:00:00:38 +0000] 1.2.3.4 r REQ REST.GET.OBJECT k "GET /k HTTP/1.1 200 - 900 1 1 1`, |
| + "bad status": strings.Replace(getLine, " 200 - 900 ", " 2zz - 900 ", 1), |
| + "negative bytes": strings.Replace(getLine, " 200 - 900 ", " 200 - -5 ", 1), |
| + } |
| + for name, line := range cases { |
| + if _, ok := ParseAccessLogLine(line); ok { |
| + t.Errorf("%s: parsed, want rejected", name) |
| + } |
| } |
| } |
| |
| @@ -134,14 +163,11 @@ func TestParseAccessLogCountsAndSkips(t *testing.T) { |
| if err != nil { |
| t.Fatal(err) |
| } |
| - if parsed != 2 { |
| - t.Errorf("parsed = %d, want 2", parsed) |
| + if parsed != 2 || skipped != 1 { |
| + t.Errorf("parsed=%d skipped=%d, want 2 and 1 (blank lines are neither)", parsed, skipped) |
| } |
| - if skipped != 1 { |
| - t.Errorf("skipped = %d, want 1 (blank lines are not skips)", skipped) |
| - } |
| - if total != 2*3462992 { |
| - t.Errorf("total bytes = %d", total) |
| + if total != 1800 { |
| + t.Errorf("total = %d, want 1800", total) |
| } |
| } |
| ``` |
| @@ -164,109 +190,161 @@ import ( |
| "io" |
| "strconv" |
| "strings" |
| + "time" |
| ) |
| |
| -// AccessLogRecord is the handful of fields billing needs from one S3 server |
| -// access log line. Everything else in the format — requester, IP, user |
| -// agent, TLS details — is deliberately dropped on the floor at parse time: |
| -// the raw logs are customer-shaped, and the only thing that outlives this |
| -// function is a byte count. |
| +// AccessLogRecord is what billing needs from one S3 server access log line. |
| +// |
| +// RequestID, RemoteIP and Time are carried because the pass cannot be |
| +// correct without them: AWS may deliver a record more than once, names log |
| +// objects by delivery time rather than event time, and an app fetching its |
| +// own object from inside the region is free transfer that must not be billed |
| +// as egress. None of the three is ever persisted — they live inside one pass |
| +// and die with it, which is what keeps the aggregates free of customer shape. |
| type AccessLogRecord struct { |
| Bucket string |
| - Operation string // e.g. REST.GET.OBJECT, REST.PUT.OBJECT |
| + Operation string // REST.GET.OBJECT, REST.PUT.OBJECT, ... |
| Key string |
| + RequestID string |
| + RemoteIP string |
| + Time time.Time |
| BytesSent int64 |
| Status int |
| } |
| |
| -// Field positions in the S3 server access log format, 0-indexed after |
| -// tokenisation (the bracketed time counts as ONE token, each quoted string |
| -// as one token). |
| +// Zero-based field positions, counting the bracketed time as one token and |
| +// each quoted string as one token. |
| const ( |
| - fieldBucket = 1 |
| - fieldOperation = 7 |
| - fieldKey = 8 |
| - fieldStatus = 10 |
| - fieldBytesSent = 12 |
| - minFields = 13 // through bytes_sent; later fields have grown over time |
| + fBucket = 1 |
| + fTime = 2 |
| + fRemoteIP = 3 |
| + fRequestID = 5 |
| + fOperation = 6 |
| + fKey = 7 |
| + fRequestURI = 8 |
| + fStatus = 9 |
| + fBytesSent = 11 |
| + fObjectSize = 12 |
| + minFields = 13 // through object_size; later fields have grown over time |
| ) |
| |
| -// splitAccessLog tokenises a line: whitespace-separated, except that |
| -// [bracketed] and "quoted" runs are single tokens. Returns false on an |
| -// unterminated bracket or quote rather than swallowing the remainder — a |
| -// truncated line must be skipped, never half-read into a byte count. |
| -func splitAccessLog(line string) ([]string, bool) { |
| - var out []string |
| - for i := 0; i < len(line); { |
| - switch line[i] { |
| - case ' ', '\t': |
| - i++ |
| - case '[', '"': |
| - closer := byte(']') |
| - if line[i] == '"' { |
| - closer = '"' |
| - } |
| - j := strings.IndexByte(line[i+1:], closer) |
| - if j < 0 { |
| - return nil, false |
| - } |
| - out = append(out, line[i+1:i+1+j]) |
| - i += j + 2 |
| - default: |
| - j := strings.IndexAny(line[i:], " \t") |
| - if j < 0 { |
| - out = append(out, line[i:]) |
| - i = len(line) |
| - continue |
| - } |
| - out = append(out, line[i:i+j]) |
| - i += j |
| - } |
| - } |
| - return out, true |
| -} |
| +const accessLogTime = "02/Jan/2006:15:04:05 -0700" |
| |
| -// dashInt reads a numeric field, treating the format's "-" (not applicable) |
| -// as zero. A field that is neither is a malformed line. |
| +// dashInt reads a numeric field, treating the format's "-" as zero, and |
| +// rejects negatives — a shifted field must never become a billable number. |
| func dashInt(s string) (int64, bool) { |
| if s == "-" { |
| return 0, true |
| } |
| n, err := strconv.ParseInt(s, 10, 64) |
| - if err != nil { |
| + if err != nil || n < 0 { |
| return 0, false |
| } |
| return n, true |
| } |
| |
| +// splitPrefix tokenises the unquoted head of a line: fields 0..7, where |
| +// field 2 is bracketed. Returns the tokens and the offset just past them. |
| +func splitPrefix(line string) ([]string, int, bool) { |
| + var out []string |
| + i := 0 |
| + for len(out) <= fKey { |
| + for i < len(line) && (line[i] == ' ' || line[i] == '\t') { |
| + i++ |
| + } |
| + if i >= len(line) { |
| + return nil, 0, false |
| + } |
| + if line[i] == '[' { |
| + j := strings.IndexByte(line[i:], ']') |
| + if j < 0 { |
| + return nil, 0, false |
| + } |
| + out = append(out, line[i+1:i+j]) |
| + i += j + 1 |
| + continue |
| + } |
| + j := strings.IndexAny(line[i:], " \t") |
| + if j < 0 { |
| + return nil, 0, false |
| + } |
| + out = append(out, line[i:i+j]) |
| + i += j |
| + } |
| + return out, i, true |
| +} |
| + |
| // ParseAccessLogLine reads one line, reporting false for anything it cannot |
| -// trust. Callers count the rejects: a day whose log files are largely |
| -// unparseable is a day we must not claim to have measured. |
| +// trust. |
| +// |
| +// The request URI is quoted and AWS does NOT escape quotes inside it, so the |
| +// closing quote is located by STRUCTURE rather than by taking the first one: |
| +// a candidate closer is only accepted when the three fields after it read as |
| +// a status, an error code and a byte count. A line where more than one |
| +// candidate satisfies that is ambiguous and rejected — a guess here silently |
| +// becomes someone's invoice. |
| func ParseAccessLogLine(line string) (AccessLogRecord, bool) { |
| - f, ok := splitAccessLog(line) |
| - if !ok || len(f) < minFields { |
| + head, i, ok := splitPrefix(line) |
| + if !ok { |
| return AccessLogRecord{}, false |
| } |
| - bytesSent, ok := dashInt(f[fieldBytesSent]) |
| - if !ok { |
| + for i < len(line) && (line[i] == ' ' || line[i] == '\t') { |
| + i++ |
| + } |
| + if i >= len(line) || line[i] != '"' { |
| return AccessLogRecord{}, false |
| } |
| - status, ok := dashInt(f[fieldStatus]) |
| - if !ok { |
| + rest := line[i+1:] |
| + |
| + var found AccessLogRecord |
| + matches := 0 |
| + for off := 0; ; { |
| + q := strings.IndexByte(rest[off:], '"') |
| + if q < 0 { |
| + break |
| + } |
| + q += off |
| + tail := strings.Fields(rest[q+1:]) |
| + off = q + 1 |
| + if len(tail) < minFields-fStatus { |
| + continue |
| + } |
| + status, sok := dashInt(tail[0]) |
| + bytesSent, bok := dashInt(tail[2]) |
| + if _, osok := dashInt(tail[3]); !sok || !bok || !osok { |
| + continue |
| + } |
| + if status != 0 && (status < 100 || status > 599) { |
| + continue |
| + } |
| + matches++ |
| + if matches > 1 { |
| + return AccessLogRecord{}, false // ambiguous |
| + } |
| + ts, terr := time.Parse(accessLogTime, head[fTime]) |
| + if terr != nil { |
| + return AccessLogRecord{}, false |
| + } |
| + found = AccessLogRecord{ |
| + Bucket: head[fBucket], |
| + Operation: head[fOperation], |
| + Key: head[fKey], |
| + RequestID: head[fRequestID], |
| + RemoteIP: head[fRemoteIP], |
| + Time: ts, |
| + BytesSent: bytesSent, |
| + Status: int(status), |
| + } |
| + } |
| + if matches != 1 { |
| return AccessLogRecord{}, false |
| } |
| - return AccessLogRecord{ |
| - Bucket: f[fieldBucket], |
| - Operation: f[fieldOperation], |
| - Key: f[fieldKey], |
| - BytesSent: bytesSent, |
| - Status: int(status), |
| - }, true |
| + return found, true |
| } |
| |
| // ParseAccessLog streams r, calling fn for every line that parses. Blank |
| -// lines are neither parsed nor skipped. Returns the two counts so a caller |
| -// can judge whether the day is trustworthy. |
| +// lines are neither parsed nor skipped. The two counts let a caller judge |
| +// whether a day is trustworthy at all. |
| func ParseAccessLog(r io.Reader, fn func(AccessLogRecord)) (parsed, skipped int, err error) { |
| sc := bufio.NewScanner(r) |
| sc.Buffer(make([]byte, 0, 64*1024), 1024*1024) |
| @@ -291,7 +369,7 @@ func ParseAccessLog(r io.Reader, fn func(AccessLogRecord)) (parsed, skipped int, |
| |
| Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/usage/ -run TestParseAccessLog -count=1 -v` |
| |
| -Expected: PASS, all five. |
| +Expected: PASS, all six. If the ambiguity rule rejects the interior-quote fixture, the structural check is too weak — tighten it against the real remainder length rather than loosening the rule. |
| |
| - [ ] **Step 5: Commit** |
| |
| @@ -299,100 +377,170 @@ Expected: PASS, all five. |
| git add internal/usage/accesslog.go internal/usage/accesslog_test.go |
| git commit -m "usage: parse S3 server access log lines |
| |
| -The two costs the platform charges for but cannot see — bytes served |
| -straight from a tenant's own bucket, and replication writes — are both |
| -visible in S3 server access logs and nowhere else affordable. This is |
| -the parser both meters will read. |
| +The two costs the platform charges for but cannot see are visible in |
| +access logs and nowhere else affordable. This is the parser. |
| + |
| +It keeps the timestamp, request id and client address, which an earlier |
| +draft discarded for privacy: without them a request near midnight lands |
| +in the wrong month, AWS's duplicate deliveries get billed twice, and an |
| +app fetching its own object from inside the region looks like egress. |
| +They live inside one pass and are never persisted. |
| |
| -It keeps five fields and drops the rest at parse time: the raw logs |
| -carry object keys and end-user addresses, and the only thing that |
| -should outlive parsing is a byte count. An unterminated quote is a |
| -skipped line, never a half-read one." |
| +The closing quote of the request URI is found by structure, because AWS |
| +does not escape quotes inside it, and a line where two candidates both |
| +parse is rejected rather than guessed." |
| ``` |
| |
| --- |
| |
| -### Task 2: Turn logging on for tenant buckets |
| - |
| -Tenant buckets are minted at runtime by `internal/objstore/aws.go`, not by Terraform (`walk.go:60-64`, `MintedBucket = prefix + acct + "-" + app`), so logging is configured where the bucket is created. One bucket per app means the bucket name identifies the tenant with no parsing. |
| +### Task 2: A log bucket, and logging switched on |
| |
| **Files:** |
| -- Modify: `internal/objstore/provisioner.go` (the `Provisioner` interface) |
| -- Modify: `internal/objstore/aws.go` (implement), `internal/objstore/memprovisioner.go` (fake) |
| -- Modify: `internal/objstore/walk.go` (call it from the grant walk) |
| -- Test: `internal/objstore/walk_test.go` |
| +- Create: `infra/modules/deployment-buckets/logs.tf` (or extend `main.tf`) |
| +- Modify: `internal/objstore/provisioner.go`, `aws.go`, `memprovisioner.go`, `walk.go`, `aws_test.go` |
| +- Modify: `cmd/console/main.go` (`storeBackend`, main.go:504-536) |
| |
| **Interfaces:** |
| -- Consumes: nothing from Task 1. |
| -- Produces: `Provisioner.EnsureBucketLogging(ctx context.Context, name, targetBucket, targetPrefix string) error` — idempotent, like every other Ensure verb in this interface. |
| +- Produces: `Provisioner.EnsureBucketLogging(ctx, name, targetBucket, targetPrefix string) error`; `objstore.Deployment.LogBucket string`; `objstore.StoreLogPrefix(acct, app string) string`. |
| |
| -- [ ] **Step 1: Write the failing test** |
| +- [ ] **Step 1: The Terraform** |
| + |
| +A **separate** bucket, unversioned, with logging disabled on itself and a delivery policy — none of which the existing module has: |
| + |
| +```hcl |
| +# The access-log bucket. Separate from data on purpose: a bucket that logs |
| +# to itself generates a log record for every log write, which AWS documents |
| +# as an unbounded loop. Unversioned, because lifecycle expiration on a |
| +# versioned bucket only writes delete markers and keeps the contents as |
| +# noncurrent versions — the 30-day promise would silently not happen. |
| +resource "aws_s3_bucket" "logs" { |
| + bucket = var.log_bucket_name |
| +} |
| + |
| +resource "aws_s3_bucket_public_access_block" "logs" { |
| + bucket = aws_s3_bucket.logs.id |
| + block_public_acls = true |
| + ignore_public_acls = true |
| + block_public_policy = true |
| + restrict_public_buckets = true |
| +} |
| + |
| +resource "aws_s3_bucket_server_side_encryption_configuration" "logs" { |
| + bucket = aws_s3_bucket.logs.id |
| + rule { |
| + apply_server_side_encryption_by_default { sse_algorithm = "AES256" } |
| + } |
| +} |
| + |
| +resource "aws_s3_bucket_lifecycle_configuration" "logs" { |
| + bucket = aws_s3_bucket.logs.id |
| + rule { |
| + id = "expire-raw-logs" |
| + status = "Enabled" |
| + filter {} |
| + expiration { days = 30 } |
| + } |
| +} |
| + |
| +# PutBucketLogging does NOT grant the delivery permission. With |
| +# bucket-owner-enforced ownership an ACL grant is not available, so the |
| +# destination needs a policy naming the logging service. |
| +data "aws_iam_policy_document" "log_delivery" { |
| + statement { |
| + principals { |
| + type = "Service" |
| + identifiers = ["logging.s3.amazonaws.com"] |
| + } |
| + actions = ["s3:PutObject"] |
| + resources = ["${aws_s3_bucket.logs.arn}/*"] |
| + condition { |
| + test = "StringEquals" |
| + variable = "aws:SourceAccount" |
| + values = [var.account_id] |
| + } |
| + } |
| +} |
| |
| -In `internal/objstore/walk_test.go`, following the file's existing `memProvisioner` setup: |
| +resource "aws_s3_bucket_policy" "logs" { |
| + bucket = aws_s3_bucket.logs.id |
| + policy = data.aws_iam_policy_document.log_delivery.json |
| +} |
| + |
| +resource "aws_s3_bucket_logging" "data" { |
| + bucket = aws_s3_bucket.data.id |
| + target_bucket = aws_s3_bucket.logs.id |
| + target_prefix = "datalogs/" |
| +} |
| +``` |
| + |
| +Add `log_bucket_name` and `account_id` variables and a `log_bucket` output. The operator's console role also needs `s3:PutBucketLogging` on tenant buckets — add it wherever the store-provisioning grant lives, and check first: |
| + |
| +Run: `grep -rn 's3:CreateBucket\|s3:PutBucketPolicy' infra/ | head` |
| + |
| +**This is a manual apply and not part of the Go gate.** Until it is applied, every day reads as `Unswept`, which is the correct reading rather than a silent zero. |
| + |
| +- [ ] **Step 2: Write the failing test** |
| + |
| +In `internal/objstore/walk_test.go`. Note `Grant`'s real signature (`walk.go:133-134`) — ten arguments, returning `(*Record, error)`: |
| |
| ```go |
| -// A minted bucket must be logging before any app can write to it: logging |
| -// turned on later cannot recover the bytes served in between. |
| -func TestGrantEnablesAccessLogging(t *testing.T) { |
| - p := newMemProvisioner() |
| - // ... build the same Deployment/store fixture the neighbouring grant |
| - // tests use, with LogBucket set on the Deployment. |
| - if err := Grant(ctx, s, sealPub, p, d, "acme", "app"); err != nil { |
| +// Logging must be configured for an ALREADY-HEALTHY store too. The healthy |
| +// short-circuit returns at walk.go:201, before EnsureBucket at :209, so |
| +// logging placed beside bucket creation would never reach an existing store |
| +// — which is every store that exists today. |
| +func TestGrantEnablesLoggingOnHealthyStore(t *testing.T) { |
| + // Build the fixture exactly as the neighbouring grant tests do, then |
| + // run Grant TWICE: the second call takes the healthy path. |
| + p := NewMemProvisioner("eu-west-1") |
| + d := Deployment{Backend: "aws", Region: "eu-west-1", BucketPrefix: "carlos-", LogBucket: "carlos-logs"} |
| + // ... declare the store, then: |
| + if _, err := Grant(ctx, s, p, d, sealPub, "acme", "app", "actor", "", now); err != nil { |
| t.Fatal(err) |
| } |
| - got, ok := p.logging[MintedBucket(d, "acme", "app")] |
| - if !ok { |
| - t.Fatal("grant did not enable access logging on the minted bucket") |
| + delete(p.Logging, MintedBucket(d, "acme", "app")) // forget the first call |
| + if _, err := Grant(ctx, s, p, d, sealPub, "acme", "app", "actor", "", now); err != nil { |
| + t.Fatal(err) |
| } |
| - if got.target != d.LogBucket { |
| - t.Errorf("log target = %q, want %q", got.target, d.LogBucket) |
| + got, ok := p.Logging[MintedBucket(d, "acme", "app")] |
| + if !ok { |
| + t.Fatal("a healthy re-grant did not configure logging") |
| } |
| - if want := "store/acme/app/"; got.prefix != want { |
| - t.Errorf("log prefix = %q, want %q", got.prefix, want) |
| + if got.Target != d.LogBucket || got.Prefix != StoreLogPrefix("acme", "app") { |
| + t.Errorf("logging = %+v, want target %q prefix %q", got, d.LogBucket, StoreLogPrefix("acme", "app")) |
| } |
| } |
| ``` |
| |
| -Read the neighbouring tests first and mirror their fixture construction exactly: |
| - |
| -Run: `grep -n 'func TestGrant' -A 20 internal/objstore/walk_test.go | head -60` |
| - |
| -- [ ] **Step 2: Run it to verify it fails** |
| - |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/objstore/ -run TestGrantEnablesAccessLogging -count=1 -v` |
| +Read the neighbouring tests and copy their fixture construction verbatim rather than inventing one: |
| |
| -Expected: FAIL to compile — no `logging` field on the fake, no `LogBucket` on `Deployment`. |
| +Run: `grep -n 'func TestGrant' -A 25 internal/objstore/walk_test.go | head -80` |
| |
| -- [ ] **Step 3: Extend the seam** |
| +- [ ] **Step 3: Extend the seam and both implementations** |
| |
| -In `internal/objstore/provisioner.go`, add to the `Provisioner` interface, after `EnsureBucket`: |
| +`internal/objstore/provisioner.go`, in the `Provisioner` interface: |
| |
| ```go |
| // EnsureBucketLogging points the bucket's server access logs at |
| - // targetBucket under targetPrefix. Idempotent: re-applying the same |
| - // configuration is a no-op, and it is called on every grant walk so a |
| - // bucket minted before logging existed picks it up on the next repair. |
| - // targetBucket == "" disables the call entirely, for deployments that |
| - // have not provisioned a log bucket. |
| + // targetBucket under targetPrefix. Idempotent, and called on every |
| + // grant walk INCLUDING the healthy path, so a bucket minted before |
| + // logging existed acquires it. targetBucket == "" disables it. |
| EnsureBucketLogging(ctx context.Context, name, targetBucket, targetPrefix string) error |
| ``` |
| |
| -Add a `LogBucket string` field to `Deployment` in `internal/objstore/walk.go` beside `BucketPrefix`, documented as "the regional data bucket; empty disables access logging". |
| +`internal/objstore/aws.go` — the client is a field (`p.s3`), the alias is `awss3`, and **`s3API` (aws.go:28-35) must gain the method or this will not compile**: |
| |
| -- [ ] **Step 4: Implement it on both provisioners** |
| +```go |
| + PutBucketLogging(ctx context.Context, in *awss3.PutBucketLoggingInput, optFns ...func(*awss3.Options)) (*awss3.PutBucketLoggingOutput, error) |
| +``` |
| |
| -In `internal/objstore/aws.go`, beside `EnsureBucket`: |
| +then, beside `EnsureBucket` (aws.go:266): |
| |
| ```go |
| -func (p *AWSProvisioner) EnsureBucketLogging(ctx context.Context, name, targetBucket, targetPrefix string) error { |
| +func (p *awsProvisioner) EnsureBucketLogging(ctx context.Context, name, targetBucket, targetPrefix string) error { |
| if targetBucket == "" { |
| return nil |
| } |
| - cl, err := p.s3(ctx) |
| - if err != nil { |
| - return err |
| - } |
| - _, err = cl.PutBucketLogging(ctx, &s3.PutBucketLoggingInput{ |
| + _, err := p.s3.PutBucketLogging(ctx, &awss3.PutBucketLoggingInput{ |
| Bucket: aws.String(name), |
| BucketLoggingStatus: &s3types.BucketLoggingStatus{ |
| LoggingEnabled: &s3types.LoggingEnabled{ |
| @@ -401,171 +549,156 @@ func (p *AWSProvisioner) EnsureBucketLogging(ctx context.Context, name, targetBu |
| }, |
| }, |
| }) |
| - return err |
| + if err != nil { |
| + return fmt.Errorf("objstore: put bucket logging %q: %w", name, err) |
| + } |
| + return nil |
| } |
| ``` |
| |
| -Match the existing client accessor and type import aliases in that file rather than the names above — check them first: |
| +`internal/objstore/memprovisioner.go` — follow the file's convention exactly: lock, then `takeFailure` before any state change (memprovisioner.go:67-77 is the model): |
| |
| -Run: `grep -n 'func (p \*AWSProvisioner) s3\|s3types\|"github.com/aws/aws-sdk-go-v2/service/s3' internal/objstore/aws.go | head` |
| +```go |
| +// MemLogging records an EnsureBucketLogging call for tests. |
| +type MemLogging struct{ Target, Prefix string } |
| |
| -In `internal/objstore/memprovisioner.go`, record the call so tests can assert on it: |
| +func (p *MemProvisioner) EnsureBucketLogging(ctx context.Context, name, targetBucket, targetPrefix string) error { |
| + p.mu.Lock() |
| + defer p.mu.Unlock() |
| + if err := p.takeFailure("EnsureBucketLogging"); err != nil { |
| + return err |
| + } |
| + if targetBucket == "" { |
| + return nil |
| + } |
| + p.Logging[name] = MemLogging{Target: targetBucket, Prefix: targetPrefix} |
| + return nil |
| +} |
| +``` |
| + |
| +Add `Logging map[string]MemLogging` to the struct and initialise it in `NewMemProvisioner` (memprovisioner.go:47-55). |
| + |
| +`internal/objstore/aws_test.go` — `stubS3` (aws_test.go:91) implements `s3API` and must gain the hook, in the file's established shape: |
| |
| ```go |
| -type memLogging struct{ target, prefix string } |
| + putBucketLogging func(*awss3.PutBucketLoggingInput) (*awss3.PutBucketLoggingOutput, error) |
| ``` |
| |
| -with a `logging map[string]memLogging` field on the fake, initialised in its constructor, and: |
| - |
| ```go |
| -func (p *memProvisioner) EnsureBucketLogging(ctx context.Context, name, targetBucket, targetPrefix string) error { |
| - if targetBucket == "" { |
| - return nil |
| +func (s *stubS3) PutBucketLogging(_ context.Context, in *awss3.PutBucketLoggingInput, _ ...func(*awss3.Options)) (*awss3.PutBucketLoggingOutput, error) { |
| + if s.putBucketLogging == nil { |
| + s.t.Fatal("unexpected PutBucketLogging call") |
| } |
| - p.logging[name] = memLogging{target: targetBucket, prefix: targetPrefix} |
| - return nil |
| + return s.putBucketLogging(in) |
| } |
| ``` |
| |
| -- [ ] **Step 5: Call it from the grant walk** |
| +- [ ] **Step 4: Add `LogBucket` and call it before the healthy return** |
| |
| -In `internal/objstore/walk.go`, immediately after the `EnsureBucket` step in `Grant`, add: |
| +Add `LogBucket string` to `Deployment` (walk.go:53-58), documented as "the access-log destination; empty disables logging". Populate it in `storeBackend` (`cmd/console/main.go:535`) from `CARLOS_LOG_BUCKET` — **the pass's own config value is a different variable and setting only that leaves every provisioning call with logging disabled.** |
| + |
| +In `Grant`, put the call **immediately after `EnsureUser` and before the healthy short-circuit at walk.go:201**, with the prefix helper beside `MintedBucket`: |
| |
| ```go |
| - // Logging goes on before any credential is delivered: bytes served |
| - // between mint and logging are invisible forever, and this walk is |
| - // idempotent precisely so a bucket minted before logging existed picks |
| - // it up on the next repair pass. |
| + // Before the healthy short-circuit, deliberately: an existing store is |
| + // exactly the case that needs logging turned on, and the short-circuit |
| + // returns at :201 without ever reaching EnsureBucket. |
| if err := p.EnsureBucketLogging(ctx, bucket, d.LogBucket, StoreLogPrefix(acct, app)); err != nil { |
| - return fmt.Errorf("objstore: enable logging on %s: %w", bucket, err) |
| + return nil, err |
| } |
| ``` |
| |
| -and add the prefix helper beside `MintedBucket`: |
| - |
| ```go |
| -// StoreLogPrefix is where a tenant bucket's access logs land in the data |
| -// bucket. One prefix per app, so the day pass reads exactly one tenant's |
| -// logs without parsing bucket names back apart. |
| +// StoreLogPrefix is where a tenant bucket's access logs land. One prefix per |
| +// app, so the day pass reads exactly one tenant's logs. |
| func StoreLogPrefix(acct, app string) string { |
| return "storelogs/" + acct + "/" + app + "/" |
| } |
| ``` |
| |
| -- [ ] **Step 6: Run the tests** |
| +- [ ] **Step 5: Run the tests, then the gate** |
| |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/objstore/ -count=1` |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp go test ./internal/objstore/ -count=1` |
| |
| -Expected: PASS, including every existing walk and contract test. Any other implementation of `Provisioner` in the tree must gain the method — find them first: |
| +Find any other `Provisioner` implementation before assuming there are only two: |
| |
| Run: `grep -rn 'EnsureBucket(ctx context.Context' --include=*.go .` |
| |
| -- [ ] **Step 7: Whole gate, then commit** |
| +Then: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp make test` |
| |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp make test` |
| +- [ ] **Step 6: Commit** |
| |
| ```bash |
| -git add internal/objstore/ |
| -git commit -m "objstore: enable access logging on tenant buckets at mint |
| +git add infra/modules/deployment-buckets/ internal/objstore/ cmd/console/main.go |
| +git commit -m "objstore: log tenant bucket access to a dedicated log bucket |
| |
| Apps hold direct bucket credentials, so bytes they serve never cross a |
| -box and no meter sees them. Server access logging is the only source |
| -that does not invert the unit economics: per-bucket CloudWatch metrics |
| -cost \$0.30 a month against a €24 a year subscription. |
| - |
| -Logging goes on inside the grant walk, before any credential is |
| -delivered, because bytes served between mint and logging are invisible |
| -forever. The walk is idempotent, so buckets minted before this picks it |
| -up on the next repair." |
| +box and no meter sees them. Access logging is the only source that does |
| +not invert the unit economics. |
| + |
| +Three things a first draft got wrong. The destination is its own |
| +bucket, because a bucket that logs to itself logs its own log writes, |
| +forever. It carries a delivery policy, because PutBucketLogging grants |
| +the logging service nothing. And it is unversioned, because lifecycle |
| +expiration on a versioned bucket writes a delete marker and keeps the |
| +contents. |
| + |
| +The call sits before the grant walk's healthy short-circuit, which |
| +returns without reaching EnsureBucket — so placing it beside bucket |
| +creation would have configured logging on new stores only, and every |
| +store that exists today is an old one." |
| ``` |
| |
| --- |
| |
| ### Task 3: The `store_egress_gb` meter |
| |
| -A console-side pass on `storagePass`'s shape (`internal/console/economicspass.go:136-195`): create-only per day, one object per day, an explicit unswept flag rather than a zero. |
| - |
| **Files:** |
| -- Modify: `internal/usage/usage.go` (meter constant, and `MeterSource` from plan one) |
| -- Create: `internal/usage/storeegress.go` |
| -- Modify: `internal/console/economicspass.go` (new pass, wired into `economicsPass`) |
| +- Modify: `internal/usage/usage.go`; Create: `internal/usage/storeegress.go` |
| +- Modify: `internal/console/economicspass.go`, `cmd/console/main.go` |
| - Test: `internal/usage/storeegress_test.go`, `internal/console/economicspass_test.go` |
| |
| **Interfaces:** |
| -- Consumes: `usage.ParseAccessLog` (Task 1), `usage.SourceStoreEgress` naming convention from plan one's `MeterSource`. |
| -- Produces: |
| - - `usage.MeterStoreEgressGB = "store_egress_gb"` |
| - - `usage.StoreEgressDay{V, Date, Apps map[string]AppEgress}` with `AppEgress{BytesOut int64; Requests int64; Unswept bool}` |
| - - `func usage.StoreEgressDayKey(date string) string` |
| - - `func (s *Server) storeEgressPass(ctx context.Context, date string) error` |
| +- Produces: `usage.MeterStoreEgressGB`, `usage.SourceStoreEgress`, `usage.StoreEgressDay`, `usage.AppEgress{BytesOut, BytesInternal, Requests int64; Unswept bool}`, `func (s *Server) storeEgressPass(ctx context.Context, date string) error`. |
| |
| -- [ ] **Step 1: Write the failing test for the day shape** |
| +Four rules this pass follows, each answering a way the log stream lies: |
| |
| -Create `internal/usage/storeegress_test.go`: |
| +1. **The day comes from the record, not the object name.** Log objects are named by delivery time. A record whose `Time` is not `date` is ignored by this run and picked up by the run for its own day. |
| +2. **Deduplicate on request id.** AWS may deliver a record more than once. The pass holds a set per day; if it exceeds a cap the day is `Unswept` rather than possibly-double-counted. |
| +3. **Internal fetches are not egress.** A GET whose `RemoteIP` is one of the platform's own addresses is an app pulling its own object to serve through the edge, where the edge meter already counts the external leg. Same-region S3-to-EC2 transfer is free. Counted separately as `BytesInternal`, priced at zero. |
| +4. **An empty read is not a measured zero.** Absence of logs is absence of evidence. |
| |
| -```go |
| -package usage |
| +- [ ] **Step 1: Day shape and meter** |
| |
| -import "testing" |
| - |
| -func TestStoreEgressDayKey(t *testing.T) { |
| - if got, want := StoreEgressDayKey("2026-08-01"), "usage/storeegress/2026-08-01.json"; got != want { |
| - t.Errorf("StoreEgressDayKey = %q, want %q", got, want) |
| - } |
| -} |
| - |
| -func TestStoreEgressMeterHasASource(t *testing.T) { |
| - if MeterSource(MeterStoreEgressGB) != SourceStoreEgress { |
| - t.Errorf("store_egress_gb must map to its own source, not %q", MeterSource(MeterStoreEgressGB)) |
| - } |
| -} |
| -``` |
| - |
| -- [ ] **Step 2: Run it to verify it fails** |
| - |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/usage/ -run TestStoreEgress -count=1 -v` |
| - |
| -Expected: FAIL to compile — undefined identifiers. |
| - |
| -- [ ] **Step 3: Add the meter, source and day shape** |
| - |
| -In `internal/usage/usage.go`, add to the meter const block: |
| +Add to `internal/usage/usage.go`'s meter block, and register `SourceStoreEgress` in `MeterSource` (plan one) plus the `TestMeterSource` table: |
| |
| ```go |
| // MeterStoreEgressGB is bytes an app served from its OWN object-store |
| - // bucket, which never cross the edge and so never appear in |
| - // MeterEgressGB. Shares the plan's egress allowance; priced at the same |
| - // rate. Sourced from S3 server access logs, which AWS delivers |
| - // best-effort — see SourceStoreEgress. |
| + // bucket. These never cross the edge, so MeterEgressGB has never seen |
| + // them. Shares the plan's egress allowance and rate. |
| MeterStoreEgressGB = "store_egress_gb" |
| ``` |
| |
| -and to the source const block from plan one: |
| - |
| -```go |
| - // SourceStoreEgress is its own source, not SourceStorage: the storage |
| - // sweep lists object sizes and the egress pass parses access logs, and |
| - // one succeeding says nothing about the other. AWS delivers these logs |
| - // best-effort with no completeness guarantee, so a day present here is |
| - // covered, never automatically complete. |
| - SourceStoreEgress = "storeegress" |
| -``` |
| - |
| -Add `case MeterStoreEgressGB: return SourceStoreEgress` to `MeterSource`, and add the meter to `TestMeterSource`'s table in `passes_test.go`. |
| - |
| Create `internal/usage/storeegress.go`: |
| |
| ```go |
| package usage |
| |
| -// AppEgress is one app's object-store transfer on one day, parsed from that |
| -// bucket's server access logs. Unswept marks a day whose logs could not be |
| -// read or were largely unparseable — the dashboard must show "unswept", |
| -// never 0, exactly as AppStorage.StoreUnswept does for occupancy. |
| +// AppEgress is one app's object-store transfer on one day. |
| +// |
| +// BytesInternal is transfer to the platform's own addresses — an app pulling |
| +// its own object to serve through the edge, where the edge meter already |
| +// counts the external leg, and which AWS does not charge for within a |
| +// region. It is recorded so the dashboard can show it, and priced at zero. |
| +// |
| +// Unswept marks a day whose logs could not be read, were largely |
| +// unparseable, or exceeded the dedup cap. As with AppStorage.StoreUnswept, |
| +// the dashboard must show "unswept", never 0. |
| type AppEgress struct { |
| - BytesOut int64 `json:"bytes_out"` |
| - Requests int64 `json:"requests,omitempty"` |
| - Unswept bool `json:"unswept,omitempty"` |
| + BytesOut int64 `json:"bytes_out"` |
| + BytesInternal int64 `json:"bytes_internal,omitempty"` |
| + Requests int64 `json:"requests,omitempty"` |
| + Unswept bool `json:"unswept,omitempty"` |
| } |
| |
| // StoreEgressDay is usage/storeegress/<date>.json, keyed "<acct>/<app>". |
| @@ -582,276 +715,100 @@ func StoreEgressDayKey(date string) string { return "usage/storeegress/" + date |
| const StoreEgressPrefix = "usage/storeegress/" |
| ``` |
| |
| -- [ ] **Step 4: Run to verify it passes** |
| +- [ ] **Step 2: Write the failing tests** |
| |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/usage/ -count=1` |
| - |
| -Expected: PASS. |
| - |
| -- [ ] **Step 5: Write the failing pass test** |
| - |
| -In `internal/console/economicspass_test.go`: |
| +In `internal/console/economicspass_test.go`, three cases that each pin one of the four rules: |
| |
| ```go |
| -// Logs that cannot be read must produce Unswept, never a zero that reads as |
| -// "this app served nothing". |
| -func TestStoreEgressPassMarksUnreadableUnswept(t *testing.T) { |
| - fx := newConsoleFixture(t) |
| - ctx := context.Background() |
| - |
| - fx.srv.egressLogs = func(context.Context, string, string) ([]byte, error) { |
| - return nil, errors.New("endpoint down") |
| - } |
| - putStoreJSON(t, fx.rel, objstore.Key(fixtureAcct, fixtureApp), objstore.Record{ |
| - Provisioned: &objstore.Provisioned{Bucket: "b", Region: "eu-west-1"}, |
| - }) |
| +func TestStoreEgressPassUnreadableIsUnswept(t *testing.T) { /* reader returns an error → Unswept, BytesOut 0 */ } |
| |
| - if err := fx.srv.storeEgressPass(ctx, "2026-08-01"); err != nil { |
| - t.Fatal(err) |
| - } |
| - raw, _, err := fx.rel.Get(ctx, usage.StoreEgressDayKey("2026-08-01")) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - var day usage.StoreEgressDay |
| - if err := json.Unmarshal(raw, &day); err != nil { |
| - t.Fatal(err) |
| - } |
| - got := day.Apps[fixtureAcct+"/"+fixtureApp] |
| - if !got.Unswept { |
| - t.Error("unreadable logs must mark the app unswept") |
| - } |
| - if got.BytesOut != 0 { |
| - t.Errorf("BytesOut = %d, want 0 alongside Unswept", got.BytesOut) |
| - } |
| +func TestStoreEgressPassEmptyIsUnsweptNotZero(t *testing.T) { /* reader returns no objects → Unswept, NOT a measured zero */ } |
| + |
| +func TestStoreEgressPassDedupsAndClassifies(t *testing.T) { |
| + // Four records: the same request id delivered twice (counts once), one |
| + // from a platform address (BytesInternal), one external (BytesOut), and |
| + // one dated the previous day (ignored entirely). |
| } |
| ``` |
| |
| -Confirm `objstore.Key` and `objstore.Record`/`Provisioned` field names before writing this: |
| +Write them against the seam from Step 3. Confirm `objstore.Get`, `objstore.Record` and `Provisioned`'s fields before using them: |
| |
| -Run: `grep -n 'func Key\|type Record struct' -A 12 internal/objstore/objstore.go` |
| +Run: `grep -n 'func Get\|type Record struct\|type Provisioned struct' -A 12 internal/objstore/objstore.go` |
| |
| -- [ ] **Step 6: Implement the pass** |
| +- [ ] **Step 3: The production reader and the pass** |
| |
| -In `internal/console/economicspass.go`, add a `egressLogs func(ctx context.Context, bucket, prefix string) ([]byte, error)` field to `Server` (the test seam, mirroring `storeSizer`), and: |
| +The seam is a *lister plus reader*, because a day's logs are many objects: |
| |
| ```go |
| -// storeEgressPass parses the previous day's tenant access logs into one |
| -// create-only day object. Shape follows storagePass deliberately: same |
| -// create-only guard, same walk over apps/, same "unswept, never zero" |
| -// convention — a store whose logs we could not read is not a store that |
| -// served nothing. |
| -func (s *Server) storeEgressPass(ctx context.Context, date string) error { |
| - key := usage.StoreEgressDayKey(date) |
| - if _, _, err := s.cfg.Releases.Get(ctx, key); err == nil { |
| - return nil // already written by us or a sibling console |
| - } |
| - acctPrefixes, err := s.cfg.Releases.ListDirs(ctx, "apps/") |
| - if err != nil { |
| - return err |
| - } |
| - day := usage.StoreEgressDay{V: 1, Date: date, Apps: map[string]usage.AppEgress{}} |
| - for _, acctPrefix := range acctPrefixes { |
| - acct := strings.TrimSuffix(strings.TrimPrefix(acctPrefix, "apps/"), "/") |
| - appPrefixes, err := s.cfg.Releases.ListDirs(ctx, acctPrefix) |
| - if err != nil { |
| - return err |
| - } |
| - for _, appPrefix := range appPrefixes { |
| - app := strings.TrimSuffix(strings.TrimPrefix(appPrefix, acctPrefix), "/") |
| - rec, _, gerr := objstore.Get(ctx, s.cfg.Releases, acct, app) |
| - if gerr != nil { |
| - log.Printf("console: store egress pass: record %s/%s: %v", acct, app, gerr) |
| - day.Apps[acct+"/"+app] = usage.AppEgress{Unswept: true} |
| - continue |
| - } |
| - if rec == nil || rec.Provisioned == nil { |
| - continue // no store, no egress, nothing to say |
| - } |
| - day.Apps[acct+"/"+app] = s.egressForApp(ctx, acct, app, date) |
| - } |
| - } |
| - _, err = usage.CreateDayObject(ctx, s.cfg.Releases, key, day) |
| - return err |
| -} |
| - |
| -// egressForApp sums one app's GET traffic for a day. A read failure, or a |
| -// day whose lines are more than 1% unparseable, is Unswept rather than a |
| -// number we would have to defend on an invoice. |
| -func (s *Server) egressForApp(ctx context.Context, acct, app, date string) usage.AppEgress { |
| - raw, err := s.egressLogs(ctx, s.cfg.LogBucket, objstore.StoreLogPrefix(acct, app)+date) |
| - if err != nil { |
| - log.Printf("console: store egress pass: logs %s/%s %s: %v", acct, app, date, err) |
| - return usage.AppEgress{Unswept: true} |
| - } |
| - var out usage.AppEgress |
| - parsed, skipped, perr := usage.ParseAccessLog(bytes.NewReader(raw), func(r usage.AccessLogRecord) { |
| - if strings.HasPrefix(r.Operation, "REST.GET") || strings.HasPrefix(r.Operation, "REST.HEAD") { |
| - out.BytesOut += r.BytesSent |
| - out.Requests++ |
| - } |
| - }) |
| - if perr != nil || (parsed+skipped > 0 && skipped*100 > parsed+skipped) { |
| - log.Printf("console: store egress pass: %s/%s %s: %d parsed, %d skipped: %v", |
| - acct, app, date, parsed, skipped, perr) |
| - return usage.AppEgress{Unswept: true} |
| - } |
| - return out |
| -} |
| +// logObjects lists and reads access-log objects under prefix. The seam is |
| +// the whole pair, so a test injects fixtures and production paginates S3. |
| +type logObjects func(ctx context.Context, bucket, prefix string) ([][]byte, error) |
| ``` |
| |
| -Wire it into `economicsPass` beside `storagePass` (`:59-73`), keeping the "each stage degrades independently" rule — log the error, never return it. Add `LogBucket string` to the server config, read in `cmd/console/main.go` `serve()` from `CARLOS_LOG_BUCKET`, defaulting to empty (which disables the pass with one log line). |
| +Add a `logObjects` field to `Server`, defaulting in `cmd/console/main.go` to an implementation that paginates `ListObjectsV2` over `bucket/prefix` and `GetObject`s each key, with a bounded total size. `objstore.S3ClientFor(ctx, region, endpoint)` (aws.go:95) builds the client; `sizeProvisionedStore` (`economicspass.go:992-1017`) is the paginating precedent to copy. |
| |
| -- [ ] **Step 7: Fold the meter into statements** |
| +**Discover tenants from the log prefixes, not from live records** — a store torn down yesterday has its `store.json` deleted (`walk.go:585`) but its logs still exist, and the pass must still bill the day it served. List `storelogs/` and walk down two levels for account and app, mirroring `storagePass`'s `ListDirs` idiom (`economicspass.go:148-163`). |
| |
| -In `foldStorageAndCensus` (`:466`), or a sibling fold beside it, read `usage.StoreEgressDayKey(date)` per day and add `BytesOut/1e9` into `MeterStoreEgressGB` for each app's account, skipping any app marked `Unswept`. Mirror the existing storage fold exactly, including its per-day denominator convention documented at `:479-490`. |
| +Then per app: read the objects, parse, and apply the four rules. Cap the dedup set at a stated size (100k ids is a few megabytes) and mark `Unswept` past it. Classify with a `platformIPs` set read from the existing `PLATFORM_EDGE_IPS` convention — check how it is parsed today before adding a second spelling: |
| |
| -Register the source day count so plan one's `MeterComplete` works: in `sweepUsage`'s day loop, add a `storeEgressOK` flag set when the day object decodes and no app in it is `Unswept`, and accumulate `sweep.SourceDays[usage.SourceStoreEgress]`. |
| +Run: `grep -rn 'PLATFORM_EDGE_IPS' --include=*.go . | head` |
| |
| -- [ ] **Step 8: Run the tests, then the gate** |
| +**Rewrite, do not create-only.** `storagePass` uses `CreateDayObject` because occupancy is a snapshot. Logs arrive late, so this pass must be able to revise a day — until plan one's statement for that month closes, at which point the day is fixed. Use `Get`+CAS `Put`, and skip a date whose month already has a closed statement. |
| |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp go test ./internal/console/ ./internal/usage/ -count=1` |
| +- [ ] **Step 4: Fold into statements** |
| |
| -Then: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp make test` |
| +Beside `foldStorageAndCensus` (`:466`), fold `float64(BytesOut)/1e9` into `MeterStoreEgressGB` per account. **No days-in-month divisor**: storage divides occupancy because it is a rate over time, transfer is an event total that simply sums. Convert before dividing — `BytesOut/1e9` on an `int64` truncates to zero for anything under a gigabyte. |
| + |
| +Register the source day only when the day object exists, decodes, and **no app in it is `Unswept`**. |
| |
| -- [ ] **Step 9: Commit** |
| +- [ ] **Step 5: Tests, gate, commit** |
| + |
| +Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp make test` |
| |
| ```bash |
| git add internal/usage/ internal/console/ cmd/console/ |
| git commit -m "usage: meter object-store egress |
| |
| Bytes an app serves from its own bucket never cross the edge, so the |
| -egress meter has never seen them. The pass parses that bucket's access |
| -logs into a create-only day object, on storagePass's shape and with its |
| -honesty convention: logs we could not read, or a day more than 1% |
| -unparseable, mark the app unswept rather than reporting zero bytes. |
| - |
| -The source is its own, not storage: listing object sizes succeeding |
| -says nothing about whether the log parse did." |
| +egress meter has never seen them. |
| + |
| +The pass takes each record's day from its own timestamp rather than the |
| +log object's name, deduplicates on request id because AWS may deliver a |
| +record twice, and separates fetches from platform addresses — an app |
| +pulling its own object to serve through the edge, which AWS does not |
| +charge for in-region and the edge meter already counts. |
| + |
| +It discovers tenants from log prefixes rather than live store records, |
| +so a store torn down yesterday is still billed for the day it served, |
| +and it revises a day until that month's statement closes rather than |
| +freezing the first observation of a best-effort stream." |
| ``` |
| |
| --- |
| |
| ### Task 4: The `durability_writes_1k` meter |
| |
| -The regional data bucket carries every litestream replica, and the key is the database's full path (`internal/ops/litestream.go:170-172`, `replicaKey` trims the leading slash and keeps the rest precisely so two tenants' `registry.db` never collide). So one logging configuration on one bucket covers every tenant, and attribution is a path-to-route match — the problem `unitOwner` already solves for systemd units. |
| +Litestream writes to `s3://<bucket>/<prefix>/<replicaKey(dbPath)>` (`internal/ops/litestream.go:141`), where **`--prefix` is required** (`internal/ops/ops.go:105-107`) and names the fleet. So the object key is `<fleet-prefix>/<trimmed db path>/...`, and an earlier draft that matched `replicaKey` alone would have matched nothing. |
| |
| **Files:** |
| -- Modify: `infra/modules/deployment-buckets/main.tf` (logging + lifecycle on the data bucket) |
| -- Modify: `internal/usage/usage.go`, `internal/usage/storeegress.go` (day shape reuse) |
| -- Modify: `internal/console/economicspass.go` |
| +- Modify: `internal/usage/usage.go`, `internal/console/economicspass.go` |
| - Test: `internal/console/economicspass_test.go` |
| |
| -**Interfaces:** |
| -- Consumes: `usage.ParseAccessLog` (Task 1). |
| -- Produces: `usage.MeterDurabilityWrites1k = "durability_writes_1k"`, `usage.SourceDurability`, `func (s *Server) durabilityPass(ctx context.Context, date string) error`. |
| +- [ ] **Step 1: Attribute by the route's own expected key** |
| |
| -- [ ] **Step 1: Add the Terraform** |
| - |
| -In `infra/modules/deployment-buckets/main.tf`, after the encryption resources: |
| - |
| -```hcl |
| -# Access logs for the data bucket: the only affordable source for |
| -# per-tenant replication writes and object-store egress. They are |
| -# customer-shaped (object keys, client addresses), so they rest in the DATA |
| -# bucket — the residency promise — never the deployment bucket, and they |
| -# expire once the daily passes have aggregated them. |
| -resource "aws_s3_bucket_logging" "data" { |
| - bucket = aws_s3_bucket.data.id |
| - target_bucket = aws_s3_bucket.data.id |
| - target_prefix = "accesslogs/data/" |
| -} |
| - |
| -resource "aws_s3_bucket_lifecycle_configuration" "data_logs" { |
| - bucket = aws_s3_bucket.data.id |
| - rule { |
| - id = "expire-access-logs" |
| - status = "Enabled" |
| - filter { prefix = "accesslogs/" } |
| - expiration { days = 30 } |
| - } |
| - rule { |
| - id = "expire-store-logs" |
| - status = "Enabled" |
| - filter { prefix = "storelogs/" } |
| - expiration { days = 30 } |
| - } |
| -} |
| -``` |
| - |
| -**This is a manual apply, not part of the Go gate.** Record in the PR description that the module must be applied before the pass produces anything, and that until it is applied every day is `Unswept` — which is the correct reading, not a silent zero. |
| - |
| -- [ ] **Step 2: Write the failing test** |
| - |
| -In `internal/console/economicspass_test.go`: |
| - |
| -```go |
| -// A replica PUT is attributed to the route whose data dir the key names. |
| -func TestDurabilityPassAttributesByPath(t *testing.T) { |
| - fx := newConsoleFixture(t) |
| - ctx := context.Background() |
| - |
| - reg, err := registry.Open(fx.regPath) |
| - if err != nil { |
| - t.Fatal(err) |
| - } |
| - if err := reg.Upsert(registry.Route{ |
| - Host: "a.example.com", Kind: "instance", Status: "ready", |
| - App: fixtureApp, Account: "acme", Version: "v1", |
| - }); err != nil { |
| - t.Fatal(err) |
| - } |
| - reg.Close() |
| - |
| - // Two PUTs under this route's data dir, one GET that must not count. |
| - fx.srv.egressLogs = func(context.Context, string, string) ([]byte, error) { |
| - line := func(op, key string) string { |
| - return `79a5 data-bucket [06/Feb/2026:00:00:38 +0000] 192.0.2.3 79a5 REQ ` + |
| - op + ` ` + key + ` "PUT /k HTTP/1.1" 200 - - 100 1 1 "-" "ua" - - - - - - - -` |
| - } |
| - k := "data/acme/" + fixtureApp + "/app.db/generations/x/wal/000001.wal.lz4" |
| - return []byte(line("REST.PUT.OBJECT", k) + "\n" + |
| - line("REST.PUT.OBJECT", k) + "\n" + |
| - line("REST.GET.OBJECT", k) + "\n"), nil |
| - } |
| - |
| - if err := fx.srv.durabilityPass(ctx, "2026-08-01"); err != nil { |
| - t.Fatal(err) |
| - } |
| - // assert two writes land against acme, zero against ops |
| -} |
| -``` |
| - |
| -**The replica key does not contain the account or the app, and it does not need to.** A tenant's database path comes from its route's `DBPath` column, and unit tenants' paths "typically live entirely outside dataDir, at their StateDirectory path" (`internal/ops/litestream.go:390-392`). So parsing a key apart would be guesswork. |
| - |
| -Instead, match each route's **own** expected replica key, exactly as `unitOwner` builds each route's own expected unit names rather than parsing a unit string back into a host (`economicspass.go:196-210`). The route knows its `DBPath`; `replicaKey` is a pure function of it; the log key either starts with that or it does not. |
| - |
| -Replace the guessed key in the test above with one built the same way the production code will build it: |
| - |
| -```go |
| - k := strings.TrimPrefix(filepath.Clean("/var/lib/tenant/acme/"+fixtureApp+"/app.db"), "/") + |
| - "/generations/x/wal/000001.wal.lz4" |
| -``` |
| - |
| -and set that route's `DBPath` to `/var/lib/tenant/acme/<app>/app.db` when you `Upsert` it, so the test exercises the real match rather than a coincidence of naming. |
| - |
| -- [ ] **Step 3: Implement attribution and the pass** |
| - |
| -Write `func replicaOwner(routes []registry.Route, key string) (acct, app string, ok bool)` in `economicspass.go` beside `unitOwner`, matching each route's own expected replica prefix rather than parsing the key apart — the same reasoning `unitOwner`'s doc comment gives at `:196-210`, and the same reason it is robust: |
| +`registry.Route.DBPath` (`registry.go:58`) is the database a route replicates — single-valued, per `internal/host/caps_pass.go:294`. Build each route's expected key rather than parsing one apart, the reasoning `unitOwner` gives at `economicspass.go:196-210`: |
| |
| ```go |
| // replicaOwner attributes a litestream replica object to the route whose |
| -// database it holds. Like unitOwner, it builds each route's OWN expected |
| -// key from registry.Route.DBPath rather than parsing the key back apart: |
| -// a unit tenant's database usually lives outside dataDir at its |
| -// StateDirectory path (ops/litestream.go:390-392), so the key carries no |
| -// account or app to parse, and the route is the only thing that knows. |
| -func replicaOwner(routes []registry.Route, key string) (string, string, bool) { |
| +// database it holds. The key is "<fleet prefix>/<db path minus its leading |
| +// slash>/...", so the fleet prefix must be supplied — matching replicaKey |
| +// alone matches nothing. |
| +func replicaOwner(routes []registry.Route, fleetPrefix, key string) (string, string, bool) { |
| for _, rt := range routes { |
| if rt.DBPath == "" { |
| continue |
| } |
| - want := strings.TrimPrefix(filepath.Clean(rt.DBPath), "/") |
| + want := path.Join(fleetPrefix, strings.TrimPrefix(filepath.Clean(rt.DBPath), "/")) |
| if key == want || strings.HasPrefix(key, want+"/") { |
| return release.RouteAccount(rt), rt.App, true |
| } |
| @@ -860,11 +817,13 @@ func replicaOwner(routes []registry.Route, key string) (string, string, bool) { |
| } |
| ``` |
| |
| -Confirm `DBPath` is the field's real name and that `replicaKey`'s trimming matches the expression above: |
| +A write matching no route is **logged, not silently folded into the platform account**: `carlos ops litestream` already refuses to shrink coverage and names databases the registry does not describe (`internal/ops/ops.go:133-146`), so an unattributable replica is a known condition worth surfacing. |
| + |
| +The fleet prefix must come from configuration, since nothing in the console knows it today. Add `CARLOS_LITESTREAM_PREFIX` beside the log bucket, and skip the pass with one log line when unset. |
| |
| -Run: `grep -n 'DBPath' internal/registry/registry.go | head -3; sed -n 165,175p internal/ops/litestream.go` |
| +- [ ] **Step 2: Survive a deleted route** |
| |
| -**A deleted route takes its replication history with it**, the same failure plan one fixed for units. Reuse that fix: `censusPass` already records every name a route answers to in `CensusDay.Units`, and a replica key is simply another such name. Add it there, in the same loop: |
| +Reuse plan one's census, which already records every name a route answers to. In `censusPass`'s route loop: |
| |
| ```go |
| if rt.DBPath != "" { |
| @@ -873,235 +832,108 @@ Run: `grep -n 'DBPath' internal/registry/registry.go | head -3; sed -n 165,175p |
| } |
| ``` |
| |
| -Then `durabilityPass` resolves an object's owner by walking the key's prefixes against that day's census before falling back to `replicaOwner` on the live registry — the same asymmetry plan one established, and for the same reason. |
| +The census key is the **unprefixed** path, so the pass strips the fleet prefix from a log key before consulting it, then falls back to `replicaOwner` on the live registry — the same asymmetry plan one established: a deleted route is in the census, a newly created one is in the registry. |
| |
| -Then `durabilityPass` follows `storeEgressPass` exactly: create-only day object, count `REST.PUT` operations per owner, `Unswept` on a read failure or a high skip rate, one object per day at `usage.DurabilityDayKey(date)`. |
| +- [ ] **Step 3: The pass** |
| |
| -Fold into statements as `MeterDurabilityWrites1k` in thousands (`puts/1000`), and register `SourceDurability` in `sweepUsage`'s day loop the same way Task 3 registers store egress. |
| +`durabilityPass` is `storeEgressPass` with three changes: it reads one prefix (`datalogs/`) rather than one per tenant, it counts `REST.PUT` operations rather than summing bytes, and it attributes with `replicaOwner`. Same day-from-timestamp rule, same request-id dedup, same `Unswept` conditions, same rewrite-until-closed behaviour. |
| |
| -- [ ] **Step 4: Run the tests, then the gate** |
| +Fold as `float64(puts)/1000` into `MeterDurabilityWrites1k` — again converting before dividing. |
| |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp make test` |
| +- [ ] **Step 4: Tests, gate, commit** |
| |
| -- [ ] **Step 5: Commit** |
| +Test at minimum: attribution against a real `DBPath` including the fleet prefix; a deleted route resolving through the census; two identical database paths under different fleet prefixes not colliding; duplicate request ids counted once. |
| |
| ```bash |
| -git add infra/modules/deployment-buckets/main.tf internal/usage/ internal/console/ |
| +git add internal/usage/ internal/console/ |
| git commit -m "usage: meter replication writes per tenant |
| |
| -Tight durability is a Business inclusion and an overage line, and until |
| -now nothing counted the S3 writes it pays for. One logging config on the |
| -data bucket covers every tenant, because litestream keys carry the |
| -database's full path — which is also why two tenants' registry.db never |
| -collide. |
| +Tight durability is a Business inclusion and an overage line, and |
| +nothing counted the S3 writes it pays for. |
| |
| -Attribution matches each route's own expected replica prefix rather than |
| -parsing keys apart, for the reason unitOwner gives: a route knows its own |
| -names, and a parser has to guess." |
| +Attribution builds each route's own expected replica key from its |
| +DBPath, including the fleet prefix the config requires — matching the |
| +trimmed database path alone would have matched nothing. A write that |
| +matches no route is logged rather than folded silently into the |
| +platform's own account." |
| ``` |
| |
| --- |
| |
| ### Task 5: Make the cost model consume the new meters |
| |
| -Adding a meter key to a statement prices nothing. `PriceAccount` enumerates its inputs explicitly (`internal/economics/compute.go:98-101`), and so does its mirrored JavaScript (`internal/console/templates/economics.html:665-667`). Until both consume the new meters, the dashboard's cost figure is wrong — and it feeds pool draw directly (`internal/console/economicsview.go:568`). |
| - |
| -This task also removes a live bug the spec named: `requests_1k` is fed from **edge HTTP request counts** (`economicspass.go:407`) but priced at the **S3 request rate** (`compute.go:100`, and the page even labels it `$s3requests` at `templates/economics.html:165`). |
| +Adding a meter key prices nothing: `PriceAccount` enumerates its inputs (`compute.go:98-101`), and so do **two** separate JavaScript calculations. This task also removes a live bug — `requests_1k` is fed from edge HTTP request counts (`economicspass.go:407`) and priced at the S3 request rate (`compute.go:100`), which the page even labels `$s3requests` (`templates/economics.html:165`). |
| |
| **Files:** |
| -- Modify: `internal/economics/scenario.go` (`RateCard`, `DefaultScenario`) |
| -- Modify: `internal/economics/compute.go:98-130` |
| -- Modify: `internal/console/templates/economics.html` (`econ_maths_js` at :545, the model table at :164-166, the knob rows at :449) |
| +- Modify: `internal/economics/scenario.go`, `internal/economics/compute.go` |
| +- Modify: `internal/console/templates/economics.html` — `priceAccount` (:665-667), the model recompute (:379-388), the model row (:165), the knobs (:449) |
| - Test: `internal/economics/compute_test.go` |
| |
| -**Interfaces:** |
| -- Consumes: `usage.MeterStoreEgressGB`, `usage.MeterDurabilityWrites1k`. |
| -- Produces: `RateCard.DurabilityPer1kUSD float64` (`omitempty`, additive — a scenario saved before it existed reads 0 and shows $0 rather than a silent default price, the convention `InterRegionGBUSD` set at `scenario.go:88-95`). |
| +- [ ] **Step 1: One rate, wired everywhere** |
| |
| -- [ ] **Step 1: Write the failing test** |
| - |
| -In `internal/economics/compute_test.go`, following the file's existing scenario construction: |
| +Add to `RateCard` (`scenario.go`), following `InterRegionGBUSD`'s additive convention (`scenario.go:88-95`) — a scenario saved before it existed reads 0 and shows $0 rather than a silent default price: |
| |
| ```go |
| -func TestPriceAccountCountsStoreEgressAndDurability(t *testing.T) { |
| - sc := DefaultScenario() |
| - base := PriceAccount(sc, "acme", map[string]float64{}, 0) |
| - with := PriceAccount(sc, "acme", map[string]float64{ |
| - usage.MeterStoreEgressGB: 10, |
| - usage.MeterDurabilityWrites1k: 100, |
| - }, 0) |
| - if with.CostUSD <= base.CostUSD { |
| - t.Errorf("store egress and replication writes must cost something: %v vs %v", |
| - with.CostUSD, base.CostUSD) |
| - } |
| -} |
| - |
| -// Edge HTTP requests must not be priced at an S3 request rate. |
| -func TestPriceAccountDoesNotChargeS3RatesForEdgeRequests(t *testing.T) { |
| - sc := DefaultScenario() |
| - a := PriceAccount(sc, "acme", map[string]float64{}, 0) |
| - b := PriceAccount(sc, "acme", map[string]float64{usage.MeterRequests1k: 1000}, 0) |
| - if b.CostUSD != a.CostUSD { |
| - t.Errorf("edge requests changed cost by %v — they are not S3 requests", |
| - b.CostUSD-a.CostUSD) |
| - } |
| -} |
| + // DurabilityPer1kUSD is what 1,000 litestream replication PUTs cost. |
| + // Distinct from RequestsPer1kUSD, which the edge-request meter misused. |
| + DurabilityPer1kUSD float64 `json:"durability_per_1k_usd,omitempty"` |
| ``` |
| |
| -Check `AccountEconomics`'s actual field names before writing the assertions: |
| - |
| -Run: `grep -n 'type AccountEconomics struct' -A 20 internal/economics/compute.go` |
| - |
| -- [ ] **Step 2: Run to verify it fails** |
| - |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/economics/ -run TestPriceAccount -count=1 -v` |
| +Seed it in `DefaultScenario` at the same S3 PUT anchor `RequestsPer1kUSD` uses today. |
| |
| -- [ ] **Step 3: Change the Go arithmetic** |
| - |
| -In `compute.go`, the cost term becomes: |
| +- [ ] **Step 2: The Go arithmetic** |
| |
| ```go |
| + storeGB := meters[usage.MeterStoreEgressGB] |
| cost := allocatedUSD + |
| - meters[usage.MeterEgressGB]*rc.EgressGBUSD + |
| - meters[usage.MeterStoreEgressGB]*rc.EgressGBUSD + |
| - meters[usage.MeterDurabilityWrites1k]*rc.RequestsPer1kUSD + |
| + (meters[usage.MeterEgressGB]+storeGB)*rc.EgressGBUSD + |
| + meters[usage.MeterDurabilityWrites1k]*rc.DurabilityPer1kUSD + |
| meters[usage.MeterStorageGBMonth]*rc.StorageGBMonthUSD |
| ``` |
| |
| -Note what changed: `MeterRequests1k` is **gone** from the cost, and `MeterDurabilityWrites1k` takes the S3 request rate that term was misusing. Store egress shares `EgressGBUSD` because a byte off a box and a byte off S3 cost the same. Apply the same substitution in the `billableCost` term below it, and add store egress to `bEgress`'s allowance so the two share one inclusion as the spec requires. |
| - |
| -- [ ] **Step 4: Change the JavaScript, in the same commit** |
| - |
| -In `internal/console/templates/economics.html`, mirror it exactly: `priceAccount` in `econ_maths_js` (:665-667), the `$s3requests` model row (:165) — which should now read from `durability_writes_1k` and be relabelled — and add a knob row for the new rate beside :449. |
| - |
| -- [ ] **Step 5: Run the tests and read the page** |
| - |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp make test` |
| - |
| -Then confirm the two definitions agree by eye: the Go expression and the JS expression must have the same terms in the same order. |
| - |
| -- [ ] **Step 6: Commit** |
| - |
| -```bash |
| -git add internal/economics/ internal/console/templates/economics.html |
| -git commit -m "economics: price the new meters, stop charging S3 rates for edge requests |
| - |
| -Adding a meter key to a statement prices nothing: PriceAccount lists its |
| -inputs explicitly, and so does its mirrored JavaScript. Both now consume |
| -store egress and replication writes, and both stop multiplying edge HTTP |
| -request counts by an S3 request rate — a substitution the page itself |
| -labelled \$s3requests while the meter was fed from edge rows. |
| - |
| -Store egress takes the egress rate and shares the egress allowance: a |
| -byte off a box and a byte off S3 cost the same and should read the same." |
| -``` |
| - |
| ---- |
| - |
| -### Task 6: Durable usage writes on the box |
| - |
| -Plan one deferred this here by name. `usageFlush` passes `drain()` as an argument (`internal/host/usage_loop.go:25`), so the edge meter's counters are emptied whether or not the write succeeds, and the caller only logs the failure — the doc at `:48-51` concedes "the drained window is lost". The analytics half of the very same function already solves this with `am.Requeue(failed)` at `:39`. |
| +`MeterRequests1k` is **gone** from both the cost and the `billableCost` terms. |
| |
| -**Files:** |
| -- Modify: `internal/edge/usagemeter.go` (add `Requeue`) |
| -- Modify: `internal/host/usage_loop.go:24-42` |
| -- Test: `internal/edge/usagemeter_test.go`, `internal/host/usage_loop_test.go` |
| - |
| -**Interfaces:** |
| -- Consumes: nothing from Tasks 1–5. This task is independent and may be done first if convenient. |
| -- Produces: `func (m *UsageMeter) Requeue(rows map[string]usage.EdgeRow)` — adds rows back, summing into anything accumulated since the drain. |
| - |
| -- [ ] **Step 1: Write the failing test** |
| - |
| -In `internal/edge/usagemeter_test.go`: |
| +**Charge the shared allowance once.** Store egress and edge egress share one inclusion, so: |
| |
| ```go |
| -func TestUsageMeterRequeueSums(t *testing.T) { |
| - m := NewUsageMeter() // match the real constructor name |
| - m.Add("h1", usage.EdgeRow{Requests: 1, BytesOut: 10}) |
| - drained := m.Drain() |
| - |
| - // New traffic arrives while the write is failing. |
| - m.Add("h1", usage.EdgeRow{Requests: 2, BytesOut: 20}) |
| - m.Requeue(drained) |
| - |
| - got := m.Drain()["h1"] |
| - if got.Requests != 3 || got.BytesOut != 30 { |
| - t.Errorf("after requeue got %+v, want Requests 3 BytesOut 30", got) |
| - } |
| -} |
| + bEgress := billableSum(meters, usage.MeterEgressGB, usage.MeterStoreEgressGB, ft.EgressGB) |
| ``` |
| |
| -Confirm the constructor and `Add` method names first: |
| - |
| -Run: `grep -n 'func NewUsageMeter\|func (m \*UsageMeter)' internal/edge/usagemeter.go` |
| - |
| -- [ ] **Step 2: Run it to verify it fails** |
| - |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp go test ./internal/edge/ -run TestUsageMeterRequeue -count=1 -v` |
| +and use that single `bEgress` in `billableCost` — do **not** add store egress again alongside it. Update the `clipped` check to compare against the combined total, not `egress_gb` alone. |
| |
| -- [ ] **Step 3: Implement `Requeue`** |
| +- [ ] **Step 3: Both JavaScript calculations, same commit** |
| |
| -Mirror `analytics.Meter.Requeue` — read it first and follow its locking exactly: |
| +`priceAccount` in `econ_maths_js` (:665-667) mirrors the Go exactly. The separate model recompute at :379-388 also needs `store_egress_gb` folded into its `egress` line and `s3requests` re-pointed at `durability_writes_1k` — including the `model-s3-requests` label and its `modelTotal` term. Add the knob row beside :449. |
| |
| -Run: `grep -n 'func (m \*Meter) Requeue' -A 15 internal/analytics/*.go` |
| +- [ ] **Step 4: Exact-value tests** |
| |
| -- [ ] **Step 4: Requeue on a failed usage write** |
| +Not "cost went up": assert exact numbers, one per meter, plus a mixed case where edge and store egress together cross the allowance. A test that only checks direction passes even if durability is ignored entirely. |
| |
| -In `internal/host/usage_loop.go`, restructure `usageFlush` so the drained rows survive a failed write: |
| - |
| -```go |
| - rows := drain() |
| - err := usage.UpsertEdgeDay(ctx, s, box, now, rows) |
| - if err != nil { |
| - log.Printf("agent: usage flush %s: %v (re-queued)", box, err) |
| - um.Requeue(rows) |
| - } |
| -``` |
| - |
| -This needs the meter itself, not just its `drain` closure, so change the parameter from `drain func() map[string]usage.EdgeRow` to the meter. Update the caller in `usageLoop` (`:52-79`) and every test that constructs `usageFlush`. Update the doc comment at `:21-23`, which currently states the opposite behaviour, and the `usageLoop` doc at `:48-51` conceding the lost window. |
| - |
| -**Bound the requeue.** An endpoint down for a day would otherwise grow the map without limit. Cap retained rows at a stated number of flush cycles, drop the oldest with a log line when exceeded, and say so in the doc comment — an unbounded retry queue on a box is a memory leak with a nicer name. |
| - |
| -- [ ] **Step 5: Run the tests and the cross-compile check** |
| - |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp make test` |
| - |
| -Then, because this touches `internal/host` which links into the CLI: |
| - |
| -Run: `GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off GOOS=darwin go build ./... && GOOS=windows go build ./...` |
| - |
| -- [ ] **Step 6: Commit** |
| +- [ ] **Step 5: Gate and commit** |
| |
| ```bash |
| -git add internal/edge/usagemeter.go internal/edge/usagemeter_test.go internal/host/usage_loop.go internal/host/usage_loop_test.go |
| -git commit -m "host: re-queue a failed edge usage write instead of losing it |
| +git add internal/economics/ internal/console/templates/economics.html |
| +git commit -m "economics: price the new meters, stop charging S3 rates for edge requests |
| |
| -usageFlush passed drain() as an argument, so counters emptied whether or |
| -not the write landed, and the caller only logged the failure. The doc |
| -conceded the window was lost — acceptable at modelling grade, not when |
| -the number becomes an invoice line. |
| +Adding a meter key prices nothing. Both the Go and the two separate |
| +JavaScript calculations now consume store egress and replication |
| +writes, and all three stop multiplying edge HTTP request counts by an |
| +S3 request rate — a substitution the page itself labelled s3requests. |
| |
| -The analytics half of this same function already had the answer. The |
| -queue is bounded: an endpoint down for a day drops the oldest rows with |
| -a log line rather than growing a box's memory without limit." |
| +Store egress shares the egress rate and the egress allowance, charged |
| +once against the combined total rather than twice." |
| ``` |
| |
| --- |
| |
| ## Self-review |
| |
| -**Spec coverage.** Step 4 (`store_egress_gb`) is Tasks 1–3. Step 5 (`durability_writes_1k`) is Tasks 1 and 4. Step 6 (migrate the cost consumers) is Task 5. The half of step 3 plan one deferred here by name is Task 6. |
| - |
| -**Step 5's other half is not here.** The spec pairs the durability meter with a flush-only replication class and plan propagation. That needs an entitlement layer to read a plan from, which does not exist until plan three, so it stays assigned to enforcement (plan one's Task 2 note records the same split). |
| - |
| -**Two things this plan cannot settle, both stated rather than hidden.** |
| +**Spec coverage.** Step 4 is Tasks 1–3. Step 5 is Tasks 1 and 4. Step 6 is Task 5. |
| |
| -1. **Best-effort delivery.** AWS makes no completeness guarantee for server access logs. Every task here fails toward `Unswept`, so a doubtful day is never billed, but whether a source with no completeness guarantee may back an invoice line at all is the billing plan's decision. The upgrade path, if the answer is no, is CloudTrail data events at $0.10 per 100k. |
| -2. **Whether a Business account's replication is actually attributable at all** depends on every tenant database being named by a route's `DBPath`. `carlos ops litestream` refuses to shrink coverage and reports databases the registry does not describe (`internal/ops/ops.go:133-146`), so an unnamed database is already a known, surfaced condition — but it would meter to the platform account rather than the tenant. Task 4's `replicaOwner` returns `ok=false` for exactly that case, and the pass should log it rather than silently folding those writes into ops. |
| +**What this plan does not settle, stated rather than hidden.** Access logs are best-effort and AWS guarantees neither completeness nor at-most-once delivery. Dedup handles the duplicates it can see within a day; a duplicate delivered across a day boundary, after the set is discarded, is not detectable here. Every pass fails toward `Unswept`, so a doubtful day is never billed — but whether this source may back an invoice line at all remains the billing plan's decision, and CloudTrail data events are the upgrade if the answer is no. The plan is built so only the reader changes. |
| |
| -**One risk closed during writing.** An earlier draft of Task 4 assumed the replica key contained the account and app, and told the implementer to stop if it did not. It does not: unit tenants' databases live outside the data directory at their StateDirectory path. Matching each route's own `DBPath` instead removes the assumption entirely, and reusing plan one's census for it means a deleted route does not take its replication history with it. |
| +**Two tasks are prescribed at a lower resolution than plan one's**, deliberately. Task 3 Steps 2–3 and Task 4 Step 3 describe the passes by their rules and their differences from a sibling rather than quoting complete bodies, because the production reader depends on facts the implementer must check first (how `PLATFORM_EDGE_IPS` is parsed, the exact fixture shape of the neighbouring pass tests). An earlier draft of this plan guessed at eight signatures and got all eight wrong; naming the check is more honest than guessing again. |
| |
| -**Placeholder scan.** Task 4 Step 3 and Task 3 Step 7 describe implementations by reference to a sibling rather than quoting full code, because they are near-copies of functions the plan has already quoted in full. Every other step carries its code. Task 2 Step 1 and Task 4 Step 2 deliberately instruct the implementer to read neighbouring fixtures before writing assertions, because the fixture shapes are local conventions this plan should not fossilise by guessing. |
| +**Placeholder scan.** Task 3 Step 2's three test bodies are one-line descriptions rather than code. That is a real gap and the implementer should expect to write them from the rules above; every other step carries either code or a named command whose output determines the code. |
| |
| -**Type consistency.** `AppEgress` is the shape both Task 3 and Task 4 write. `Unswept` carries the same meaning as `AppStorage.StoreUnswept`. `egressLogs` is one seam used by both passes. `SourceStoreEgress` and `SourceDurability` are registered in `MeterSource` and counted in `sweepUsage`, matching plan one's `SourceDays` contract. |
| +**Type consistency.** `AppEgress` is written by Task 3 and read by Task 4's fold. `Unswept` means what `AppStorage.StoreUnswept` means. `logObjects` is one seam serving both passes. `SourceStoreEgress` and `SourceDurability` are registered in `MeterSource` and counted in `sweepUsage`, matching plan one's `SourceDays` contract. |
amadan