amadan
Prevent failure drain from hiding billing incompleteness in snapshots
3 files changed,
+111
−8
implementation/carloku-paid-launch/AGENT_HANDOFF.md+3 −3internal/usage/durabilityfailures_test.go+57 −0internal/usage/durabilityledger.go+51 −5
CI passed — run details
diff --git a/implementation/carloku-paid-launch/AGENT_HANDOFF.md b/implementation/carloku-paid-launch/AGENT_HANDOFF.md| index e1847d090..b9b786568 100644 |
| --- a/implementation/carloku-paid-launch/AGENT_HANDOFF.md |
| +++ b/implementation/carloku-paid-launch/AGENT_HANDOFF.md |
| @@ -1,6 +1,6 @@ |
| # Agent pickup instructions — Carloku paid launch |
| -CURRENT GATE: RED. Fix `TestDurabilityLateIncompleteVoidsClosedDayPersistently` before any merge or deployment. The handoff is a checkpoint of unfinished work. |
| +CURRENT GATE: local build, vet and `make test` GREEN after fixing the intermittent durability snapshot failure. Amadan CI must run on the new commit. This remains a checkpoint of unfinished work. |
| This file is context and instructions for the next coding agent. The branch is a work-in-progress checkpoint, not a release candidate. Do not merge, deploy, enable billing, or infer that the overnight launch goal was achieved. |
| @@ -40,7 +40,7 @@ Read `AGENTS.md` and `docs/superpowers/specs/2026-09-07-carloku-pricing-plans-de |
| - Quantity/checkout/membership audit tests passed `-race` (49.015s), before the final invoice lock changes. Log: `quantity-complete-root-race.txt`. |
| - New host ownership, locked-ledger timeout, shutdown/publication and actual fake-HTTP SDK List/Copy/Delete-to-ledger tests passed `-race` (1.551s). Log: `host-durability-sdk-root-race.txt`. |
| - New activator meter selection plus existing staging-restore tests passed `-race` (1.032s). Log: `activator-metering-root-race.txt`. |
| -- Current complete-tree handoff gate FAILED on `internal/usage/durabilityfailures_test.go:76`, `TestDurabilityLateIncompleteVoidsClosedDayPersistently`: immediately after Incomplete, Snapshot returned RecordingFailures=0 for an already-closed day. All other package tests, build, vet, formatting and nested modules passed. Investigate the read-transaction/latch-drain race: a snapshot can retain the pre-drain DB view while the latch is cleared after a newer commit. This is a hypothesis to verify, not a completed fix. Do not weaken the test or enable billing. Log: `handoff-full-gate.txt`. |
| +- The prior intermittent failure was real: a snapshot could read SQLite before a background failure drain, then read the latch after that drain cleared it. The fix retries when the latch changes during the read and checks the final latch state under the writer mutex, which spans commit and subtraction. A forced-interleaving regression and the original late-failure test passed `-race -count=20`; the original passed `-count=30` without race instrumentation. The complete local gate (`go build ./...`, `go vet ./...`, `make test` with JS parity) passed on 2026-09-18. Amadan CI at the previous handoff commit `099586ea` also passed, demonstrating why a single green run did not settle this intermittent bug. New-commit CI is pending. |
| - CURRENT tree passed all five cross-builds (Darwin arm64/amd64, Linux arm64, Windows amd64/arm64). Log: `handoff-cross.txt`. |
| - Final whole-branch Fable review and production/authority Opus review remain outstanding. All three Claude agents stopped at the shared usage limit, which reports reset at05:50 Europe/Dublin on2026-09-17 (04:50UTC). No agents remain running. That reset time has now passed; availability has not been rechecked for this handoff. Relevant task briefs/reports are in scratch, especially `durability-host-integration-brief.txt`, `opus-durability-integration-design.txt`, `quantity-history-hardening-brief.txt` and the invoice reports. The older design's arbitrary five-minute late-callback slack is NOT accepted proof. |
| @@ -77,7 +77,7 @@ A scratch SQLite experiment was also found replacing local root main.go. It was |
| ## Next agent actions |
| -1. Fix the closed-day late-failure snapshot race first, with a deterministic regression and repeated/race tests. Then read branch CI and rerun the full gate. Inspect the complete diff as WIP; do not infer launch readiness from prior checkpoint test counts. |
| +1. Read Amadan CI for the new durability-fix commit and inspect the complete branch diff as WIP. The local gate is green, but that does not make the launch ready. |
| 2. Finish/review newest host wiring: full per-route move/checkpoint tests and client inventory, source completeness folding, scoped external-client exclusion, crash/late-callback/restart proof, bounded ledger retention and real-disk latency measurement. Keep unattested usage unbillable until proved. |
| 3. Implement P5 collector with durable deduplication, immutable historical bucket ownership and a settled transfer contract. Prepare exact scoped logging activation before seeking production approval. |
| 4. Complete partial-month billing, quantity/proration collection, operator review/house doors, Free enrollment/caps/idle/flush-only, dormancy warnings/archive and revenue-pool accounting required by the spec. Use the requirements audit rather than treating these as optional follow-ups. |
diff --git a/internal/usage/durabilityfailures_test.go b/internal/usage/durabilityfailures_test.go| index 8425803ba..b2b63445b 100644 |
| --- a/internal/usage/durabilityfailures_test.go |
| +++ b/internal/usage/durabilityfailures_test.go |
| @@ -5,12 +5,69 @@ import ( |
| "os" |
| "path/filepath" |
| "strings" |
| + "sync" |
| "testing" |
| "time" |
| "github.com/carlosframework/platform/internal/release" |
| ) |
| +// A snapshot can hold an older WAL view while the failure drainer commits. |
| +// It must retry that view instead of combining old rows with a cleared latch. |
| +func TestDurabilitySnapshotConcurrentFailureDrain(t *testing.T) { |
| + dir := t.TempDir() |
| + clock := newLedgerClock(ledgerDay0) |
| + l := openTestLedger(t, dir, "box1", clock, true) |
| + defer l.Close(context.Background(), nil) |
| + date := DayOf(ledgerDay0) |
| + paused := make(chan struct{}) |
| + resume := make(chan struct{}) |
| + var once sync.Once |
| + l.snapshotPause = func(string) { |
| + once.Do(func() { |
| + close(paused) |
| + <-resume |
| + }) |
| + } |
| + // Keep the background drainer from committing until the snapshot has |
| + // opened its read transaction and reached the pause. |
| + l.mu.Lock() |
| + l.latchFailure(ledgerDay0) |
| + type result struct { |
| + content DurabilityContent |
| + err error |
| + } |
| + done := make(chan result, 1) |
| + go func() { |
| + content, _, err := l.snapshot(context.Background(), date) |
| + done <- result{content, err} |
| + }() |
| + select { |
| + case <-paused: |
| + case <-time.After(5 * time.Second): |
| + l.mu.Unlock() |
| + t.Fatal("snapshot did not reach the read transaction") |
| + } |
| + l.mu.Unlock() |
| + deadline := time.Now().Add(5 * time.Second) |
| + for !l.latch.empty() && time.Now().Before(deadline) { |
| + time.Sleep(time.Millisecond) |
| + } |
| + if !l.latch.empty() { |
| + close(resume) |
| + t.Fatal("failure was not drained") |
| + } |
| + close(resume) |
| + select { |
| + case got := <-done: |
| + if got.err != nil || got.content.RecordingFailures != 1 { |
| + t.Fatalf("snapshot after concurrent drain: failures=%d err=%v", got.content.RecordingFailures, got.err) |
| + } |
| + case <-time.After(5 * time.Second): |
| + t.Fatal("snapshot did not finish") |
| + } |
| +} |
| + |
| // ---- Recording failures: the non-blocking latch and its proofs ------------ |
| // |
| // Incomplete is called from the request path with a 25 ms budget around |
diff --git a/internal/usage/durabilityledger.go b/internal/usage/durabilityledger.go| index f824798c1..96c3e07f4 100644 |
| --- a/internal/usage/durabilityledger.go |
| +++ b/internal/usage/durabilityledger.go |
| @@ -129,8 +129,11 @@ type DurabilityLedger struct { |
| // ledgerLatchMaxDays, and beyond that a single range that conservatively |
| // covers every day a further failure touched. |
| type failureLatch struct { |
| - mu sync.Mutex |
| - days map[string]int64 // date -> failures not yet in the database |
| + mu sync.Mutex |
| + // generation changes whenever a failure arrives or a committed drain |
| + // removes one. A snapshot retries if either happens during its read. |
| + generation uint64 |
| + days map[string]int64 // date -> failures not yet in the database |
| // overflow holds failures past the per-date bound: n of them, spread |
| // over an unknown subset of the days from..to. Drained as n on every |
| // day of the range — over-counted, never dropped. |
| @@ -235,6 +238,16 @@ func (la *failureLatch) pending(date string) int64 { |
| return n |
| } |
| +func (la *failureLatch) pendingAt(date string) (int64, uint64) { |
| + la.mu.Lock() |
| + defer la.mu.Unlock() |
| + n := la.days[date] |
| + if la.overflow.n > 0 && DayOf(la.overflow.from) <= date && date <= DayOf(la.overflow.to) { |
| + n += la.overflow.n |
| + } |
| + return n, la.generation |
| +} |
| + |
| // dates lists every date the latch holds a failure for, the overflow |
| // range expanded day by day (bounded like a gap), so Publish can send a |
| // day whose only news is a failure not yet on disk. |
| @@ -276,6 +289,9 @@ func (la *failureLatch) take() (map[string]int64, ledgerFailureRange) { |
| func (la *failureLatch) subtract(days map[string]int64, overflow ledgerFailureRange) { |
| la.mu.Lock() |
| defer la.mu.Unlock() |
| + if len(days) > 0 || overflow.n > 0 { |
| + la.generation++ |
| + } |
| for d, n := range days { |
| if left := la.days[d] - n; left > 0 { |
| la.days[d] = left |
| @@ -1220,6 +1236,7 @@ func (l *DurabilityLedger) latchFailure(at time.Time) { |
| } |
| la := &l.latch |
| la.mu.Lock() |
| + la.generation++ |
| if la.closed { |
| la.late++ |
| if la.lateFrom.IsZero() || at.Before(la.lateFrom) { |
| @@ -1616,14 +1633,34 @@ func (l *DurabilityLedger) Snapshot(date string) (DurabilityContent, error) { |
| return c, err |
| } |
| -// snapshot builds the content and reads the day's publication lineage in |
| +// snapshot retries when a failure enters or leaves the latch during its |
| +// database read. Otherwise a read may precede a drain's commit while the |
| +// final latch check follows its subtraction, making a failed day look clean. |
| +func (l *DurabilityLedger) snapshot(ctx context.Context, date string) (DurabilityContent, dayLineage, error) { |
| + for range 8 { |
| + _, before := l.latch.pendingAt(date) |
| + content, lineage, err := l.snapshotOnce(ctx, date, before) |
| + if errors.Is(err, errSnapshotLatchChanged) { |
| + continue |
| + } |
| + if err != nil { |
| + return DurabilityContent{}, dayLineage{}, err |
| + } |
| + return content, lineage, nil |
| + } |
| + return DurabilityContent{}, dayLineage{}, errors.New("usage ledger: recording failures changed during snapshot") |
| +} |
| + |
| +var errSnapshotLatchChanged = errors.New("usage ledger: recording failures changed during snapshot") |
| + |
| +// snapshotOnce builds the content and reads the day's publication lineage in |
| // ONE read transaction on the reader connection: the watermark is the |
| // ledger's next_seq as that transaction sees it, so every row with a |
| // lower sequence is in the snapshot and none with a higher one. It holds |
| // no ledger mutex: a Start or Finish commits beside it, and is simply |
| // above the watermark. Latched failures are added after the read; they |
| // only ever make a snapshot more incomplete. |
| -func (l *DurabilityLedger) snapshot(ctx context.Context, date string) (DurabilityContent, dayLineage, error) { |
| +func (l *DurabilityLedger) snapshotOnce(ctx context.Context, date string, before uint64) (DurabilityContent, dayLineage, error) { |
| start, end, err := DayBounds(date) |
| if err != nil { |
| return DurabilityContent{}, dayLineage{}, fmt.Errorf("usage ledger: %q is not a date", date) |
| @@ -1781,7 +1818,16 @@ func (l *DurabilityLedger) snapshot(ctx context.Context, date string) (Durabilit |
| // Failures latched and not yet drained count now, not after the next |
| // seal: a snapshot must never read complete over a failure the |
| // process already knows about. |
| - c.RecordingFailures += l.latch.pending(date) |
| + // The writer holds l.mu through both commit and latch subtraction. |
| + // Checking under it cannot observe a committed row alongside its |
| + // not-yet-subtracted latch entry. |
| + l.mu.Lock() |
| + pending, after := l.latch.pendingAt(date) |
| + l.mu.Unlock() |
| + if after != before { |
| + return DurabilityContent{}, dayLineage{}, errSnapshotLatchChanged |
| + } |
| + c.RecordingFailures += pending |
| return c, lineage, nil |
| } |