amadan

carlos / platform Public

Prove a box-day by what its ledger covered; expire the absent-month memory

Durability fold (Astra F5). The registration tolerance is gone. It let a
ledger whose coverage began minutes late pass for the minutes it never
saw, and dropped an exclusion that fell inside them. Each box must now
prove the union of its durability, metrics and edge windows, for
coverage and exclusions alike, and the proof is what its ledger covered.
A ledger that was open before the box did anything makes a registration
race harmless on its own. One that opened late leaves a gap, and the day
holds. Opus asked for the tolerance in round four; holding is the rule
until a policy for a box's first minutes is approved.

Statement catch-up (Astra F8, Opus N6). The memory of a month with no
statement expires after six hours: another console's pass that began
before a month boundary can still write that month afterwards. An
account's statement that cannot be read three passes running is
reported once and the month is classified without it, so it no longer
keeps every other account's month from being swept. The platform's
statement stays unknown, and that rule is now pinned.

The partial-month comment says what the rule does at its boundary: a
partial boundary day is left out, a midnight start counts, and the
rounding is bounded per continuous paid span, not per month (Astra F11).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
claude 6d2ef5085d1fa0baaefac43ee0471e23ac82332b parent 8936244
5 files changed, +215 −94
  • internal/console/billingcalculation.go +7 −4
  • internal/console/durabilityfold.go +45 −36
  • internal/console/economicspass.go +37 −9
  • internal/console/meterfolds_test.go +124 −45
  • internal/console/server.go +2 −0
diff --git a/internal/console/billingcalculation.go b/internal/console/billingcalculation.go
index d19df5356..aaf1a5602 100644
--- a/internal/console/billingcalculation.go
+++ b/internal/console/billingcalculation.go
@@ -140,10 +140,13 @@ var errBillingDaysMismatch = errors.New("billing_statement_days_mismatch")
// billingUsageOnPaidDays returns each billed meter summed over the UTC days
// that lie wholly inside one paid span, and those days in order.
//
-// A day counts only when all 24 hours are inside a single covered interval:
-// the day a plan starts and the day it ends are not counted, because the
-// statement cannot say which of that day's usage fell on which side. That
-// rounds toward the customer by at most the boundary days.
+// A day counts only when all 24 hours are inside a single covered interval.
+// A day the plan covered only part of (it started, ended or lapsed inside
+// the day) is not counted, because the statement cannot say which of that
+// day's usage fell on which side. A plan that starts exactly at midnight
+// covers that whole day, and it counts. The rounding is toward the
+// customer, by at most two partial days for each continuous paid span, so
+// a month with several spans (lapses) can leave out more than two.
//
// Before anything is counted the days must account for the statement: every
// day inside the month, and each billed meter's days summing to its monthly
diff --git a/internal/console/durabilityfold.go b/internal/console/durabilityfold.go
index 528439465..f8b6ccc51 100644
--- a/internal/console/durabilityfold.go
+++ b/internal/console/durabilityfold.go
@@ -46,29 +46,24 @@ import (
// legacy number is metrics, edge and storage only.
//
// active is every box another per-box source knows (metrics, edge), keyed
-// by that source. A box that was alive on the day, by its own metrics or
-// edge expectation, must be expected by the durability source for the same
-// span. A box with no ledger directory, or whose ledger registered later
-// than the box began serving, ran its replication unmetered for the
-// difference and must not be left out of the question. It holds the day.
+// by that source. A box that was alive on the day, by its own durability
+// registration or by its metrics or edge expectation, must PROVE the whole
+// of that span: its published day is asked (usage.DurabilityDay.Scope)
+// about the union of those windows, for coverage and for exclusions alike.
+// A box with no ledger directory proves nothing and holds the day.
+//
+// When each source was first registered does not enter into it. Sources
+// register on their own first ticks, seconds apart, and first-seen is
+// write-once; comparing those instants either held a new box's first day
+// for good over a race, or, with a tolerance, let a ledger whose coverage
+// began minutes late pass for the minutes it never saw, and dropped an
+// exclusion that fell inside them. What the ledger covered is the proof,
+// and it is asked for all of the span. A ledger that was open before the
+// box did anything shows a registration race to be harmless on its own;
+// one that opened late leaves a real gap, and the day holds. Whether a
+// box's first minutes, before its ledger existed, may ever be waved
+// through is a policy nobody has approved, so they hold.
//
-// The spans are compared with a tolerance (durabilityRegistrationSlack). A
-// box registers each source on that source's first tick, each with its own
-// first-seen instant, and they can differ by seconds on a healthy box;
-// first-seen is write-once, so an exact comparison would hold a new box's
-// first day for good over a registration race. A tolerance, not a
-// comparison of clock hours: two seconds either side of the top of an hour
-// are still two seconds. In practice the host backdates a late ledger's
-// first-seen to the box's earliest evidence (host.boxEarliestEvidence), so
-// a ledger that opened late is mostly caught further down, by its missing
-// day files. This check is the backstop for a registration that was not
-// backdated.
-// durabilityRegistrationSlack is how much later than the box's other
-// sources its durability source may have been first seen and still be the
-// same boot. Sources register seconds apart; this is minutes, and far
-// short of the hour or more a ledger that opened late would show.
-const durabilityRegistrationSlack = 5 * time.Minute
-
// excluded is the union, over every box, of the accounts that box's
// attestation left out for the day. It is meaningful only when complete is
// true.
@@ -77,13 +72,30 @@ func (s *Server) foldDurabilityDay(ctx context.Context, cands []sourceCandidate,
// ledger has no evidence that nothing replicated.
complete = lifecycleOK && len(cands) > 0
out := map[string]bool{}
- // Every box another per-box source expected that day must be expected
- // by the durability source too, for at least the same span. A box with
- // no ledger directory, or one whose ledger was first registered after
- // the box was already serving, replicated unmetered for the difference.
- ledgers := map[string]sourceCandidate{}
+ // The span each box must prove for the day: the union of what every
+ // per-box source expected of it.
+ type span struct {
+ from, to time.Time
+ expected bool
+ }
+ need := map[string]span{}
+ widen := func(box string, from, to time.Time) {
+ w := need[box]
+ if !w.expected || from.Before(w.from) {
+ w.from = from
+ }
+ if !w.expected || to.After(w.to) {
+ w.to = to
+ }
+ w.expected = true
+ need[box] = w
+ }
+ ledgers := map[string]bool{}
for _, cand := range cands {
- ledgers[cand.Box] = cand
+ ledgers[cand.Box] = true
+ if from, to, expected := cand.window(usage.SourceDurability, d); expected {
+ widen(cand.Box, from, to)
+ }
}
for source, others := range active {
for _, other := range others {
@@ -91,20 +103,17 @@ func (s *Server) foldDurabilityDay(ctx context.Context, cands []sourceCandidate,
if !expected {
continue
}
- cand, ok := ledgers[other.Box]
- if !ok {
- complete = false
- continue
- }
- dFrom, dTo, dExpected := cand.window(usage.SourceDurability, d)
- if !dExpected || dFrom.Sub(from) > durabilityRegistrationSlack || to.Sub(dTo) > durabilityRegistrationSlack {
+ widen(other.Box, from, to)
+ if !ledgers[other.Box] {
+ // Alive, and no ledger at all.
complete = false
}
}
}
for _, cand := range cands {
box := cand.Box
- wFrom, wTo, expected := cand.window(usage.SourceDurability, d)
+ want := need[box]
+ expected, wFrom, wTo := want.expected, want.from, want.to
key := usage.DurabilityDayKey(box, date)
raw, _, gerr := s.cfg.Releases.Get(ctx, key)
if gerr != nil {
diff --git a/internal/console/economicspass.go b/internal/console/economicspass.go
index 32a7c57d3..10e5ff9f4 100644
--- a/internal/console/economicspass.go
+++ b/internal/console/economicspass.go
@@ -1345,6 +1345,14 @@ const (
statementCatchUpWarnMonths = 6
)
+// statementAbsentTTL is how long a month seen with no statement is taken to
+// still have none. statementUnreadPasses is how many passes running an
+// account's statement may fail to be read before the month is swept
+// without it.
+var statementAbsentTTL = 6 * time.Hour
+
+const statementUnreadPasses = 3
+
type statementMonth int
const (
@@ -1383,15 +1391,17 @@ const (
// for: its open month is not retried and cannot be billed, which errs in
// the customer's favour.
//
-// Two things are remembered for the life of the process, so that a steady
-// state costs no reads. A statement seen closed is not read again: closure
-// is irreversible (usage.UpsertStatement). A month seen with no statement
-// at all is not read again until this process sweeps it: no pass writes a
-// month two or more back except the catch-up, which only sweeps months
-// inside the metered range. Anything that ever reopens or deletes a closed
-// statement must run in a fresh process or clear s.statementClosed.
+// Two things are remembered so that a steady state costs few reads. A
+// statement seen closed is not read again for the life of the process:
+// closure is irreversible (usage.UpsertStatement), and anything that ever
+// reopens or deletes a closed statement must run in a fresh process or
+// clear s.statementClosed. A month seen with no statement at all is not
+// read again for statementAbsentTTL, and no longer: another console's pass
+// that began before a month boundary can still write that month as its
+// "previous month" after this process looked, and a memory that never
+// expired would hide the statement it wrote for good.
func (s *Server) statementMonthState(ctx context.Context, month string, monthEnd time.Time, accounts []Account) statementMonth {
- if _, ok := s.statementAbsent.Load(month); ok {
+ if seen, ok := s.statementAbsent.Load(month); ok && time.Since(seen.(time.Time)) < statementAbsentTTL {
return statementMonthAbsent
}
const (
@@ -1411,9 +1421,27 @@ func (s *Server) statementMonthState(ctx context.Context, month string, monthEnd
return missing
}
if err != nil {
+ // A store error. For the platform's statement it concludes
+ // nothing, pass after pass. For an account's, the first few
+ // conclude nothing either; after that the object is treated
+ // like one that will not parse (reported once, counted as
+ // present), so that one statement nobody can read does not
+ // keep every other account's month from ever being swept.
+ n := 1
+ if prev, ok := s.statementUnread.Load(key); ok {
+ n = prev.(int) + 1
+ }
+ s.statementUnread.Store(key, n)
+ if acct != release.OpsAccount && n >= statementUnreadPasses {
+ if _, logged := s.statementCorrupt.LoadOrStore(key, struct{}{}); !logged {
+ log.Printf("console: statements pass: catch-up: %s could not be read on %d passes running; its month is swept without it and someone must repair it: %v", key, n, err)
+ }
+ return corrupt
+ }
log.Printf("console: statements pass: catch-up: read %s: %v", key, err)
return unread
}
+ s.statementUnread.Delete(key)
var st usage.Statement
if err := json.Unmarshal(raw, &st); err != nil {
if _, logged := s.statementCorrupt.LoadOrStore(key, struct{}{}); !logged {
@@ -1455,7 +1483,7 @@ func (s *Server) statementMonthState(ctx context.Context, month string, monthEnd
}
switch {
case !any:
- s.statementAbsent.Store(month, struct{}{})
+ s.statementAbsent.Store(month, time.Now())
return statementMonthAbsent
case owed:
return statementMonthOpen
diff --git a/internal/console/meterfolds_test.go b/internal/console/meterfolds_test.go
index c0f2851c6..510e0351f 100644
--- a/internal/console/meterfolds_test.go
+++ b/internal/console/meterfolds_test.go
@@ -289,16 +289,23 @@ func publishDurabilityDay(t *testing.T, fx *consoleFixture, box string, attested
// exclusion set: the accounts whose databases an uninstrumented process on
// the box replicates.
func publishScopedDurabilityDay(t *testing.T, fx *consoleFixture, box string, attested bool, unmetered []string, record func(bind func(usage.ReplicationAttribution) *usage.DurabilityClient, at time.Time)) {
+ t.Helper()
+ publishDurabilityDayOpenedAt(t, fx, box, attested, unmetered, time.Date(2026, 9, 16, 23, 0, 0, 0, time.UTC), record)
+}
+
+// publishDurabilityDayOpenedAt publishes the 17th from a ledger first
+// opened at openAt: its coverage begins there.
+func publishDurabilityDayOpenedAt(t *testing.T, fx *consoleFixture, box string, attested bool, unmetered []string, openAt time.Time, record func(bind func(usage.ReplicationAttribution) *usage.DurabilityClient, at time.Time)) {
t.Helper()
ctx := context.Background()
day := time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC)
- clock := &foldClock{t: day.Add(-time.Hour)}
+ clock := &foldClock{t: openAt}
l, err := usage.OpenDurabilityLedger(t.TempDir(), box, clock.now, usage.DurabilityLedgerOptions{AllClientsInstrumented: attested, Unmetered: unmetered})
if err != nil {
t.Fatal(err)
}
defer l.Close(ctx, nil) //nolint:errcheck
- clock.t = day.Add(10 * time.Hour)
+ clock.t = day.Add(12 * time.Hour)
record(func(attr usage.ReplicationAttribution) *usage.DurabilityClient {
c, err := l.Bind(attr)
if err != nil {
@@ -542,62 +549,77 @@ func TestStoreEgressHeldMonthIsRevisitedAfterTheNextMonthEnds(t *testing.T) {
}
}
-// A box whose ledger registered later than the box began serving replicated
-// unmetered for the difference. Its ledger directory exists, and its own
-// durability window says "not expected yet", but another source saw it
-// alive: the day holds.
-func TestDurabilityFoldHoldsBeforeABoxRegisteredItsLedger(t *testing.T) {
+// What a box must prove for a day is the union of what its sources
+// expected of it, and the proof is what its ledger covered, not when each
+// source happened to be registered.
+//
+// - A ledger that was open before the box did anything proves the day,
+// however its registration instant compares with the edge's: seconds
+// apart, across the top of an hour, or hours late.
+// - A ledger that opened after the box began serving leaves a real gap,
+// minutes or hours, and the day holds. A box's first minutes before
+// its ledger existed are not waved through.
+func TestDurabilityFoldProvesTheUnionOfABoxsWindows(t *testing.T) {
ctx := context.Background()
day := time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC)
- edgeFrom := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)
- run := func(durabilityFrom time.Time) int {
+ evening := day.Add(-time.Hour)
+ run := func(edgeFrom, durabilityRegistered, ledgerOpened time.Time) int {
fx := newConsoleFixture(t)
- // The box has served edge traffic since edgeFrom.
if _, err := usage.RegisterBoxSource(ctx, fx.rel, "box1", usage.SourceEdge, edgeFrom); err != nil {
t.Fatal(err)
}
- if _, err := usage.RegisterBoxSource(ctx, fx.rel, "box1", usage.SourceDurability, durabilityFrom); err != nil {
+ if _, err := usage.RegisterBoxSource(ctx, fx.rel, "box1", usage.SourceDurability, durabilityRegistered); err != nil {
t.Fatal(err)
}
putStoreJSON(t, fx.rel, usage.EdgeDayKey("box1", "2026-09-17"), watchedEdgeDay("box1", "2026-09-17"))
- publishDurabilityDay(t, fx, "box1", true, tenantWrites(t))
+ publishDurabilityDayOpenedAt(t, fx, "box1", true, nil, ledgerOpened, tenantWrites(t))
if err := fx.srv.statementsPass(ctx, time.Date(2026, 9, 20, 6, 0, 0, 0, time.UTC)); err != nil {
t.Fatal(err)
}
return septemberStatement(t, fx, foldAcctA).SourceDays[usage.SourceDurability]
}
- if got := run(time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)); got != 1 {
- t.Fatalf("control: ledger registered with the box: durability source days = %d, want 1", got)
- }
- // Registered at 10:00 on the day itself: the first ten hours of the
- // 17th were served with no ledger.
- if got := run(day.Add(10 * time.Hour)); got != 0 {
- t.Errorf("durability source days = %d, want 0: the ledger registered after the box began serving", got)
- }
- // A new box, on its first day: each source registers on its own first
- // tick, seconds apart. First-seen is write-once, so holding on that
- // difference would hold the box's first day for good.
- // The days before the box existed owe nothing, so the count is compared
- // with a ledger registered at the very instant the edge was.
- edgeFrom = day.Add(8*time.Hour + 5*time.Second)
- same := run(edgeFrom)
- if got := run(day.Add(8*time.Hour + 45*time.Second)); got != same {
- t.Errorf("durability source days = %d, want %d: a registration race of seconds held a new box's first day", got, same)
- }
- // An hour or more is not a race.
- if got := run(day.Add(9*time.Hour + 5*time.Second)); got != same-1 {
- t.Errorf("durability source days = %d, want %d: the ledger registered an hour after the box began serving", got, same-1)
+ month := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)
+ base := run(month, month, evening)
+ if base < 1 {
+ t.Fatalf("control: durability source days = %d, want the 17th counted", base)
+ }
+ // Registration instants, with the ledger open all along: none of these
+ // changes the proof.
+ for name, registered := range map[string]time.Time{
+ "registered ten hours into the day": day.Add(10 * time.Hour),
+ "registered an hour late": day.Add(time.Hour),
+ } {
+ if got := run(month, registered, evening); got != base {
+ t.Errorf("%s, ledger open all along: durability source days = %d, want %d", name, got, base)
+ }
}
- // Nor is ten minutes.
- if got := run(edgeFrom.Add(10 * time.Minute)); got != same-1 {
- t.Errorf("durability source days = %d, want %d: the ledger registered ten minutes after the box began serving", got, same-1)
+ // A box born on the day, its sources registering seconds apart, the
+ // ledger open before either: complete, including across an hour.
+ for name, c := range map[string][2]time.Time{
+ "seconds apart": {day.Add(8*time.Hour + 5*time.Second), day.Add(8*time.Hour + 45*time.Second)},
+ "across the top of an hour": {day.Add(8*time.Hour + 59*time.Minute + 59*time.Second), day.Add(9*time.Hour + time.Second)},
+ } {
+ born := run(c[0], c[0], evening)
+ if got := run(c[0], c[1], evening); got != born {
+ t.Errorf("%s: durability source days = %d, want %d", name, got, born)
+ }
}
- // The same race of seconds, across the top of an hour: the edge at
- // 08:59:59, the ledger at 09:00:01. Still one boot.
- edgeFrom = day.Add(8*time.Hour + 59*time.Minute + 59*time.Second)
- same = run(edgeFrom)
- if got := run(day.Add(9*time.Hour + time.Second)); got != same {
- t.Errorf("durability source days = %d, want %d: two seconds across an hour boundary held a new box's first day", got, same)
+ // The ledger itself opened late: the box served edge traffic from
+ // midnight and the ledger's coverage begins at 09:04, or at 00:04.
+ // Those minutes are not proven, whatever the registration says.
+ for name, opened := range map[string]time.Time{
+ "nine hours late": day.Add(9*time.Hour + 4*time.Minute),
+ "four minutes late": day.Add(4 * time.Minute),
+ } {
+ if got := run(month, month, opened); got != base-1 {
+ t.Errorf("ledger opened %s: durability source days = %d, want %d (the day holds)", name, got, base-1)
+ }
+ // Registering the durability source at the same late instant does
+ // not shrink what must be proven: the edge still expected the box
+ // from midnight.
+ if got := run(month, opened, opened); got != base-1 {
+ t.Errorf("ledger opened and registered %s: durability source days = %d, want %d", name, got, base-1)
+ }
}
}
@@ -861,15 +883,72 @@ func TestStatementMonthStateEdges(t *testing.T) {
if _, ok := fresh.statementCorrupt.Load(key); !ok {
t.Fatal("the unparseable statement was not recorded as reported")
}
- // A store error concludes nothing: not absent, not open, not settled.
+ // A store error on the platform's statement concludes nothing, however
+ // many passes it lasts: not absent, not open, not settled.
broken := &Server{cfg: fx.srv.cfg, store: fx.srv.store}
broken.cfg.Releases = failingGets{Store: fx.rel, fail: usage.StatementKey(fixtureAcct, "2026-09")}
- if got := broken.statementMonthState(ctx, "2026-09", start.AddDate(0, 1, 0), accounts); got != statementMonthUnknown {
- t.Fatalf("a month whose platform statement could not be fetched = %v, want unknown", got)
+ for pass := 0; pass < statementUnreadPasses+2; pass++ {
+ if got := broken.statementMonthState(ctx, "2026-09", start.AddDate(0, 1, 0), accounts); got != statementMonthUnknown {
+ t.Fatalf("pass %d: a month whose platform statement could not be fetched = %v, want unknown", pass, got)
+ }
}
if _, ok := broken.statementAbsent.Load("2026-09"); ok {
t.Fatal("a store error was remembered as an absent month")
}
+ // A store error on one account's statement concludes nothing at first.
+ // When it persists, the month is classified without that statement, so
+ // the others in it are not held hostage. (The fixture's other
+ // statements are closed, so the month is settled.)
+ stuck := &Server{cfg: fx.srv.cfg, store: fx.srv.store}
+ stuck.cfg.Releases = failingGets{Store: fx.rel, fail: key}
+ for pass := 1; pass < statementUnreadPasses; pass++ {
+ if got := stuck.statementMonthState(ctx, "2026-09", start.AddDate(0, 1, 0), accounts); got != statementMonthUnknown {
+ t.Fatalf("pass %d: an account statement that could not be fetched = %v, want unknown", pass, got)
+ }
+ }
+ if got := stuck.statementMonthState(ctx, "2026-09", start.AddDate(0, 1, 0), accounts); got != statementMonthSettled {
+ t.Fatalf("after %d failed passes = %v, want the month classified without the unreadable statement", statementUnreadPasses, got)
+ }
+ // One good read starts the count again.
+ stuck.cfg.Releases = fx.rel
+ stuck.statementCorrupt.Delete(key)
+ _ = stuck.statementMonthState(ctx, "2026-09", start.AddDate(0, 1, 0), accounts)
+ if _, ok := stuck.statementUnread.Load(key); ok {
+ t.Fatal("a successful read did not reset the failure count")
+ }
+}
+
+// Around a month boundary. Console A's catch-up looks at September and
+// finds no statement. Console B, whose pass began before the boundary,
+// then writes September as its previous month, open, and exits. A is still
+// running. A must find that statement: its memory of September being
+// absent expires.
+func TestStatementCatchUpRevalidatesAnAbsentMonth(t *testing.T) {
+ fx := newConsoleFixture(t)
+ ctx := context.Background()
+ fx.srv.cfg.StoreEgressReportPrefix = foldReportPrefix
+ accounts, _ := fx.st.Accounts()
+ end := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
+ if got := fx.srv.statementMonthState(ctx, "2026-09", end, accounts); got != statementMonthAbsent {
+ t.Fatalf("fixture: September should be absent, got %v", got)
+ }
+ // Console B: another process, the same bucket.
+ consoleB := &Server{cfg: fx.srv.cfg, store: fx.srv.store}
+ if err := consoleB.statementsPass(ctx, time.Date(2026, 9, 30, 23, 59, 0, 0, time.UTC)); err != nil {
+ t.Fatal(err)
+ }
+ if !statementExists(t, fx, fixtureAcct, "2026-09") {
+ t.Fatal("fixture: console B should have written September")
+ }
+ // Within the memory's life A still answers from memory.
+ if got := fx.srv.statementMonthState(ctx, "2026-09", end, accounts); got != statementMonthAbsent {
+ t.Fatalf("within the memory's life: %v", got)
+ }
+ // Once it has expired, A reads again and finds the open statement.
+ fx.srv.statementAbsent.Store("2026-09", time.Now().Add(-statementAbsentTTL-time.Minute))
+ if got := fx.srv.statementMonthState(ctx, "2026-09", end, accounts); got != statementMonthOpen {
+ t.Fatalf("after the memory expired: %v, want open", got)
+ }
}
// failingGets fails every Get of one key with a store error.
diff --git a/internal/console/server.go b/internal/console/server.go
index b1646f944..cded07cf9 100644
--- a/internal/console/server.go
+++ b/internal/console/server.go
@@ -701,6 +701,8 @@ type Server struct {
// statementCorrupt the unparseable statement keys already logged.
statementAbsent sync.Map
statementCorrupt sync.Map
+ // statementUnread counts consecutive failed reads of a statement key.
+ statementUnread sync.Map
// claims folds the deployment's domain claims for the read paths — the
// Settings render and the domains list door — reading only the claims
// whose bucket stamp moved since the last fold (#173). Process-lifetime