carlos / platform Public

console: a Database stats tile on the app Overview

Human

Prompt

can you create a new branch for a feature to be added to the carloku console to show some statistics in a box for the database. Size etc on the overview of an instance... (reference: a Supabase 'Primary Database' card showing region/instance type and CPU/Disk/RAM/connections)

From this commit's own message. The prompt ledger has no entries for it — which is what a squashed, rebased or amended commit looks like, and what every commit looks like in a repo without a ledger.

Summary

Adds a Database tile to the per-app Overview page (internal/console/templates/app_overview.html), alongside Running/Resources/Web addresses/Release history. Shows total SQLite size across the account's own routes (reused from the existing status.BoxMetrics.RouteDBBytes join that already backs the instances/routes APIs, via the new boxJoin param on assembleOverview) and the fullest disk-used fraction among the boxes backing the app's reporting instances (new InstanceMetrics.DiskFrac, sourced from the metrics sweep's existing per-box BoxMetricsView.Disk, same pipeline Resources' CPU/Mem already use). DiskFracMax is deliberately the worst box, not an average: a shared volume filling up is what took down all three bdn/ips instances together on 2026-10-01 (SQLITE_FULL then SQLITE_IOERR_SHMSIZE crash-looping for 4h18m), and an average would have hidden that.
Agent notes
Backend plumbing for per-route DB size already existed end-to-end (internal/host/probes.go dbFileBytes -> internal/status RouteDBBytes -> console apiroutes.go/apiinstances.go) from an earlier 'show database and bucket sizes' commit (11709577) — this branch only had to wire it into the Overview tile via the existing loadBoxJoin helper, not invent a new pipeline. Disk headroom reuses the existing internal/metrics sweep (BoxMetricsView.Disk) the same way. Two pre-existing, unrelated test failures observed in this sandbox: internal/tunnelclient TestDialAndServeInvertsConnection hangs past 600s, and internal/edge's fleet/tunnel e2e tests fail with 'tunnel lost — re-dialing' — both are tunnel/relay-subsystem tests with zero import relation to internal/console, reproduce on a clean checkout, and are almost certainly this sandbox's networking limits rather than a real regression. Filing a discussion about it separately rather than debugging it here.
claude agent, operated by tonywood@keymail.dev pushed by tonywood@keymail.dev d9bf4606dc2ef6f0696876424a86dbc6afd3a4a0 parent e8406ce
6 files changed, +142 −20
  • .claude/settings.json +24 −0
  • internal/console/metricsview.go +8 −0
  • internal/console/overview.go +26 −1
  • internal/console/overview_test.go +77 −18
  • internal/console/server.go +2 −1
  • internal/console/templates/app_overview.html +5 −0

CI passed — run details

diff --git a/.claude/settings.json b/.claude/settings.json
new file mode 100644
index 000000000..7a1468fb5
--- /dev/null
+++ b/.claude/settings.json
@@ -0,0 +1,24 @@
+{
+ "hooks": {
+ "Stop": [
+ {
+ "hooks": [
+ {
+ "command": "amadan ledger sync-transcript",
+ "type": "command"
+ }
+ ]
+ }
+ ],
+ "UserPromptSubmit": [
+ {
+ "hooks": [
+ {
+ "command": "amadan ledger add-prompt",
+ "type": "command"
+ }
+ ]
+ }
+ ]
+ }
+}
diff --git a/internal/console/metricsview.go b/internal/console/metricsview.go
index 7f1c70307..52a0857ea 100644
--- a/internal/console/metricsview.go
+++ b/internal/console/metricsview.go
@@ -674,6 +674,13 @@ type InstanceMetrics struct {
Box string
CPUFrac *float64
MemBytes *int64
+ // DiskFrac is the used-space fraction of the BOX this instance currently
+ // sits on — a host-level reading (every instance sharing the box reports
+ // the same value), carried per-instance only so the Overview tile can
+ // read it off InstanceMetrics like every other number there. It says
+ // nothing about which box, so it is safe to render even though Box
+ // itself is not (see Box's own doc comment).
+ DiskFrac *float64
SparkCPU, SparkMem string
@@ -773,6 +780,7 @@ func (s *Server) OwnerAppMetrics(ctx context.Context, hosts []string, unitByHost
Box: box.Box,
CPUFrac: u.CPUFrac,
MemBytes: u.MemBytes,
+ DiskFrac: box.Disk,
SparkCPU: u.SparkCPU,
SparkMem: u.SparkMem,
}
diff --git a/internal/console/overview.go b/internal/console/overview.go
index 104e4cb2c..f193ec403 100644
--- a/internal/console/overview.go
+++ b/internal/console/overview.go
@@ -72,6 +72,22 @@ type OverviewStats struct {
CPUSum *float64
MemSum *int64
+ // DiskFracMax is the fullest disk-used fraction among the boxes backing
+ // this app's reporting instances — the worst case, not an average,
+ // because a shared volume filling up is what crashes every tenant on it
+ // at once (the 2026-10-01 bdn/ips outage: SQLITE_FULL on one instance's
+ // writes, then every instance on the box failing to reopen its database
+ // at all). Nil when nothing reports, same em-dash contract as CPUSum.
+ DiskFracMax *float64
+
+ // DBSizeSum is this app's total on-disk SQLite footprint (each route's
+ // file plus its -wal, status.BoxMetrics.RouteDBBytes via dbJoin), summed
+ // across every one of the account's own routes with a sized database —
+ // not gated on the metrics sweep reporting, since a hibernating
+ // instance's database is still on disk and still worth showing. Nil
+ // when no route could be sized.
+ DBSizeSum *int64
+
// Domains, and how many need someone's attention: a custom domain not
// pointing here yet, or a cert that is not ready.
DomainTotal int
@@ -93,7 +109,7 @@ type OverviewStats struct {
// instance count and the domains count must skip rows stamped with another
// account — a same-named app elsewhere must not inflate this dashboard or
// wave its failed cert on it.
-func assembleOverview(view *AppView, acct string, metrics []InstanceMetrics, versions []VersionView) OverviewStats {
+func assembleOverview(view *AppView, acct string, metrics []InstanceMetrics, versions []VersionView, dbJoin boxJoin) OverviewStats {
st := OverviewStats{MetricsUnavailable: metrics == nil}
st.Deployment = observedDeployment(view.Routes, acct)
reporting := make(map[string]bool, len(metrics))
@@ -114,11 +130,20 @@ func assembleOverview(view *AppView, acct string, metrics []InstanceMetrics, ver
}
*st.MemSum += *m.MemBytes
}
+ if m.DiskFrac != nil && (st.DiskFracMax == nil || *m.DiskFrac > *st.DiskFracMax) {
+ st.DiskFracMax = m.DiskFrac
+ }
}
for _, rt := range view.Routes {
if rt.Account != acct || rt.Platform {
continue
}
+ if n := dbJoin.dbSize(rt.Host); n != nil {
+ if st.DBSizeSum == nil {
+ st.DBSizeSum = new(int64)
+ }
+ *st.DBSizeSum += *n
+ }
if rt.Kind == "sidecar" {
st.SidecarTotal++
continue
diff --git a/internal/console/overview_test.go b/internal/console/overview_test.go
index 380b3ad0b..96dcbbe3e 100644
--- a/internal/console/overview_test.go
+++ b/internal/console/overview_test.go
@@ -30,7 +30,7 @@ func TestOverviewLiveVersionFromOwnedRoutes(t *testing.T) {
{Name: "beta", Version: "v9"},
{Name: "stable", Version: "v7", Err: "obsolete unreadable pointer"},
}, Routes: []RouteView{{Host: "kass.example", Account: "bab", Channel: "edge", Version: "v9"}}}
- st := assembleOverview(view, "bab", nil, nil)
+ st := assembleOverview(view, "bab", nil, nil, boxJoin{})
if !st.Deployment.HasReportedVersion() {
t.Fatal("an app with a reported route version is not reported deployed")
}
@@ -39,7 +39,7 @@ func TestOverviewLiveVersionFromOwnedRoutes(t *testing.T) {
}
// A pointer with no owned route is a target, not a deployment.
- st = assembleOverview(&AppView{App: "kass", Channels: []ChannelView{{Name: "stable", Version: "v9"}}}, "bab", nil, nil)
+ st = assembleOverview(&AppView{App: "kass", Channels: []ChannelView{{Name: "stable", Version: "v9"}}}, "bab", nil, nil, boxJoin{})
if st.Deployment.HasReportedVersion() || st.Deployment.UniformVersion() != "" {
t.Fatalf("a never-promoted app reads as live: %+v", st)
}
@@ -52,7 +52,7 @@ func TestOverviewUnknownRouteVersionIsUnknownNotOffline(t *testing.T) {
view := &AppView{App: "kass", Channels: []ChannelView{
{Name: "stable", Err: "signature does not verify"},
}, Routes: []RouteView{{Host: "kass.example", Account: "bab", Channel: "stable"}}}
- st := assembleOverview(view, "bab", nil, nil)
+ st := assembleOverview(view, "bab", nil, nil, boxJoin{})
if st.Deployment.HasReportedVersion() {
t.Fatal("an unreadable pointer was reported as a known live version")
}
@@ -88,7 +88,7 @@ func TestOverviewSumsOnlyReportingInstances(t *testing.T) {
{Host: "d", CPUFrac: f64(99), MemBytes: i64(99), Hibernate: true},
{Host: "e"},
}
- st := assembleOverview(view, "bab", metrics, nil)
+ st := assembleOverview(view, "bab", metrics, nil, boxJoin{})
if st.InstanceTotal != 5 {
t.Fatalf("InstanceTotal = %d, want 5 (the routes, not the sweep, and never another account's)", st.InstanceTotal)
}
@@ -111,7 +111,7 @@ func TestOverviewSumsOnlyReportingInstances(t *testing.T) {
// Nobody reported: nil, not zero. A zero here would render as "0%" and
// claim an idle app where there is simply no reading.
quietView := &AppView{App: "kass", Routes: []RouteView{{Host: "a", Account: "bab", Kind: "instance"}, {Host: "b", Account: "bab", Kind: "instance"}}}
- quiet := assembleOverview(quietView, "bab", []InstanceMetrics{{Host: "a"}, {Host: "b"}}, nil)
+ quiet := assembleOverview(quietView, "bab", []InstanceMetrics{{Host: "a"}, {Host: "b"}}, nil, boxJoin{})
if quiet.CPUSum != nil || quiet.MemSum != nil {
t.Fatalf("sums = %v %v with nothing reporting, want nil so the tiles show the em-dash",
quiet.CPUSum, quiet.MemSum)
@@ -122,7 +122,7 @@ func TestOverviewSumsOnlyReportingInstances(t *testing.T) {
// The sweep FAILED (nil metrics): the count must survive — "readings
// unavailable" is not "no hosts yet".
- failed := assembleOverview(quietView, "bab", nil, nil)
+ failed := assembleOverview(quietView, "bab", nil, nil, boxJoin{})
if failed.InstanceTotal != 2 {
t.Fatalf("InstanceTotal = %d after a failed sweep, want the routes' 2", failed.InstanceTotal)
}
@@ -143,7 +143,7 @@ func TestOverviewSeparatesNoReadings(t *testing.T) {
{Host: "d.x", Kind: "instance", Account: "acct1", Hibernate: true}, // asleep
}, Domains: []DomainView{{}, {}, {}, {}}}
metrics := []InstanceMetrics{{Host: "a.x", Box: "box1"}}
- st := assembleOverview(view, "acct1", metrics, nil)
+ st := assembleOverview(view, "acct1", metrics, nil, boxJoin{})
if st.InstanceExpected != 4 || st.InstanceReporting != 1 || st.InstanceAsleep != 1 ||
st.InstanceNoReadings != 1 || st.InstanceCounted != 3 {
t.Fatalf("counts: %+v", st)
@@ -188,7 +188,7 @@ func TestOverviewDomainAttentionCountsGuidanceAndFailedTLS(t *testing.T) {
{Domain: "e.example.com", TLS: "ready", Guidance: "Point an A record for e.example.com at ..."},
{Domain: "f.example.com", TLS: "failed: dns-01 timeout"},
}}
- st := assembleOverview(view, "bab", nil, nil)
+ st := assembleOverview(view, "bab", nil, nil, boxJoin{})
if st.DomainTotal != 5 {
t.Fatalf("DomainTotal = %d, want 5 — the other account's domain is not this tenant's", st.DomainTotal)
}
@@ -220,7 +220,7 @@ func TestOverviewReleasesCapped(t *testing.T) {
for i := range full {
full[i] = VersionView{Version: "v" + string(rune('a'+i))}
}
- st := assembleOverview(&AppView{App: "kass"}, "bab", nil, full)
+ st := assembleOverview(&AppView{App: "kass"}, "bab", nil, full, boxJoin{})
if st.ReleaseCount != versionsCap || !st.ReleasesCapped {
t.Fatalf("count/capped = %d/%v, want %d/true", st.ReleaseCount, st.ReleasesCapped, versionsCap)
}
@@ -228,11 +228,11 @@ func TestOverviewReleasesCapped(t *testing.T) {
t.Fatalf("LatestVersion = %q, want the head of the newest-first list (%q)", st.LatestVersion, full[0].Version)
}
- short := assembleOverview(&AppView{App: "kass"}, "bab", nil, full[:versionsCap-1])
+ short := assembleOverview(&AppView{App: "kass"}, "bab", nil, full[:versionsCap-1], boxJoin{})
if short.ReleasesCapped {
t.Fatalf("a list one short of the cap was reported capped (%d)", short.ReleaseCount)
}
- none := assembleOverview(&AppView{App: "kass"}, "bab", nil, nil)
+ none := assembleOverview(&AppView{App: "kass"}, "bab", nil, nil, boxJoin{})
if none.ReleaseCount != 0 || none.ReleasesCapped || none.LatestVersion != "" {
t.Fatalf("empty release list = %+v, want zeros and no version", none)
}
@@ -572,7 +572,7 @@ func TestOverviewAndAppListAgreeWhenFailedRouteHasNoVersion(t *testing.T) {
Host: "kass.example", Account: "bab", Kind: "instance", Status: "failed", Provisioned: true,
}}}
row := summarize(view, "bab")
- overview := assembleOverview(&view, "bab", []InstanceMetrics{}, nil)
+ overview := assembleOverview(&view, "bab", []InstanceMetrics{}, nil, boxJoin{})
if row.Verdict != "attention" || overview.Verdict != "attention" {
t.Fatalf("list=%q overview=%q, want attention on both pages", row.Verdict, overview.Verdict)
}
@@ -587,7 +587,7 @@ func TestOverviewIgnoresPlatformAliasWithoutInventingStaticVersion(t *testing.T)
},
Channels: []ChannelView{{Name: "edge", Version: "site-v3"}},
}
- st := assembleOverview(&view, "bab", []InstanceMetrics{}, nil)
+ st := assembleOverview(&view, "bab", []InstanceMetrics{}, nil, boxJoin{})
if st.Verdict != "ok" || st.Deployment.RouteCount != 1 || st.Deployment.ReportableCount != 0 || st.Deployment.HasReportedVersion() {
t.Fatalf("overview = %+v, want a healthy static deployment with no invented observation", st)
}
@@ -596,7 +596,7 @@ func TestOverviewIgnoresPlatformAliasWithoutInventingStaticVersion(t *testing.T)
func TestServiceOnlyAppIsPresentAndHealthyOnBothPages(t *testing.T) {
view := AppView{App: "proxy", Routes: []RouteView{{Host: "proxy.example", Account: "bab", Kind: "service", Status: "ready"}}}
row := summarize(view, "bab")
- overview := assembleOverview(&view, "bab", []InstanceMetrics{}, nil)
+ overview := assembleOverview(&view, "bab", []InstanceMetrics{}, nil, boxJoin{})
if row.Verdict != "ok" || overview.Verdict != "ok" || !overview.Deployment.HasRoutes() || overview.Deployment.ReportableCount != 0 {
t.Fatalf("list=%+v overview=%+v, want a present healthy service with no version report", row, overview)
}
@@ -608,7 +608,7 @@ func TestStaticPlusSilentInstanceIsUnknownOnBothPages(t *testing.T) {
{Host: "app.example", Account: "bab", Kind: "instance"},
}}
row := summarize(view, "bab")
- overview := assembleOverview(&view, "bab", []InstanceMetrics{}, nil)
+ overview := assembleOverview(&view, "bab", []InstanceMetrics{}, nil, boxJoin{})
if row.Verdict != "unknown" || overview.Verdict != "unknown" {
t.Fatalf("list=%q overview=%q, want unknown on both pages", row.Verdict, overview.Verdict)
}
@@ -622,7 +622,7 @@ func TestPlatformOnlyFailureDoesNotCreateCrossPageAttention(t *testing.T) {
Host: "alias.bab.oncarlos.com", Account: "bab", Platform: true, Status: "failed",
}}, Domains: []DomainView{{Guidance: "ignored platform guidance"}}}
row := summarize(view, "bab")
- overview := assembleOverview(&view, "bab", []InstanceMetrics{}, nil)
+ overview := assembleOverview(&view, "bab", []InstanceMetrics{}, nil, boxJoin{})
if row.Verdict != "notdeployed" || overview.Verdict != "notdeployed" {
t.Fatalf("list=%q overview=%q, want notdeployed on both pages", row.Verdict, overview.Verdict)
}
@@ -634,7 +634,7 @@ func TestOverviewExpectedCountsInstanceKindOnly(t *testing.T) {
{Host: "b", Account: "bab", Kind: "static"},
{Host: "c", Account: "other", Kind: "instance"},
}}
- st := assembleOverview(view, "bab", []InstanceMetrics{}, nil)
+ st := assembleOverview(view, "bab", []InstanceMetrics{}, nil, boxJoin{})
if st.InstanceExpected != 1 {
t.Errorf("InstanceExpected = %d, want 1 (instance-kind, this account only)", st.InstanceExpected)
}
@@ -656,7 +656,7 @@ func TestOverviewCountsSidecarsSeparately(t *testing.T) {
// Another account's sidecar must not count here either.
{Host: "bot.eleven.other.sidecar.internal", Account: "other", Kind: "sidecar"},
}}
- st := assembleOverview(view, "bab", []InstanceMetrics{}, nil)
+ st := assembleOverview(view, "bab", []InstanceMetrics{}, nil, boxJoin{})
if st.InstanceTotal != 1 {
t.Errorf("InstanceTotal = %d, want 1 — sidecars must not inflate it", st.InstanceTotal)
}
@@ -665,6 +665,65 @@ func TestOverviewCountsSidecarsSeparately(t *testing.T) {
}
}
+// TestOverviewDiskFracMaxIsWorstBoxNotAverage: the Database tile's disk
+// figure must name the box closest to the SQLITE_FULL cliff, not blend it
+// away into an average — the 2026-10-01 bdn/ips outage was one box's volume
+// filling up and every instance on it failing together, a fact an average
+// across boxes would have hidden behind a calm-looking number.
+func TestOverviewDiskFracMaxIsWorstBoxNotAverage(t *testing.T) {
+ view := &AppView{App: "ips", Routes: []RouteView{
+ {Host: "a", Account: "bab", Kind: "instance"},
+ {Host: "b", Account: "bab", Kind: "instance"},
+ {Host: "c", Account: "bab", Kind: "instance"},
+ }}
+ metrics := []InstanceMetrics{
+ {Host: "a", Box: "box-1", DiskFrac: f64(0.40)},
+ {Host: "b", Box: "box-2", DiskFrac: f64(0.97)},
+ // Reporting, but this box's disk reading was unreadable: must not
+ // win by default or be treated as the floor.
+ {Host: "c", Box: "box-3"},
+ }
+ st := assembleOverview(view, "bab", metrics, nil, boxJoin{})
+ if st.DiskFracMax == nil || *st.DiskFracMax != 0.97 {
+ t.Fatalf("DiskFracMax = %v, want 0.97 (the fullest box, not an average)", st.DiskFracMax)
+ }
+
+ quiet := assembleOverview(view, "bab", []InstanceMetrics{{Host: "a"}, {Host: "b"}, {Host: "c"}}, nil, boxJoin{})
+ if quiet.DiskFracMax != nil {
+ t.Fatalf("DiskFracMax = %v with no disk reading anywhere, want nil so the tile shows the em-dash", quiet.DiskFracMax)
+ }
+}
+
+// TestOverviewDBSizeSumsEveryRouteRegardlessOfReporting: a hibernating
+// instance's database is still sitting on disk even though the metrics
+// sweep hears nothing from it, so the Database tile's size must come from
+// the account's ROUTES (via dbJoin, the same status.BoxMetrics.RouteDBBytes
+// join the instances/routes APIs already use), not be gated on the sweep
+// the way CPU/Mem are. It must also stay tenant-scoped like every other
+// number assembleOverview produces.
+func TestOverviewDBSizeSumsEveryRouteRegardlessOfReporting(t *testing.T) {
+ view := &AppView{App: "ips", Routes: []RouteView{
+ {Host: "a", Account: "bab", Kind: "instance"},
+ {Host: "b", Account: "bab", Kind: "instance", Hibernate: true},
+ {Host: "c", Account: "bab", Kind: "instance"}, // no sized database
+ {Host: "theirs", Account: "other", Kind: "instance"},
+ }}
+ join := boxJoin{dbBytes: map[string]int64{
+ "a": 1 << 20,
+ "b": 2 << 20, // asleep, unreported by the sweep, still on disk
+ "theirs": 99 << 20,
+ }}
+ st := assembleOverview(view, "bab", nil, nil, join)
+ if st.DBSizeSum == nil || *st.DBSizeSum != 3<<20 {
+ t.Fatalf("DBSizeSum = %v, want 3MiB (a+b only — hibernating counts, another account never does)", st.DBSizeSum)
+ }
+
+ none := assembleOverview(&AppView{App: "kass", Routes: []RouteView{{Host: "x", Account: "bab", Kind: "instance"}}}, "bab", nil, nil, boxJoin{})
+ if none.DBSizeSum != nil {
+ t.Fatalf("DBSizeSum = %v with no route sized, want nil so the tile shows the em-dash", none.DBSizeSum)
+ }
+}
+
// TestAppPageShowsPartialMismatchNotFullAlarm guards the boundary
// AllMismatched exists to draw: when only SOME of an app's routes disagree
// with the viewed account, the page must name the disagreeing route(s), not
diff --git a/internal/console/server.go b/internal/console/server.go
index db0eea533..b579f4c8d 100644
--- a/internal/console/server.go
+++ b/internal/console/server.go
@@ -2815,7 +2815,8 @@ func (s *Server) handleApp(w http.ResponseWriter, r *http.Request) {
}
p.Versions = versions
metrics := s.appResourceMetrics(r.Context(), p.Detail, acct)
- p.Overview = assembleOverview(p.Detail, acct.Sqid, metrics, versions)
+ dbJoin := loadBoxJoin(r.Context(), s.cfg.Releases, time.Now())
+ p.Overview = assembleOverview(p.Detail, acct.Sqid, metrics, versions, dbJoin)
// The diagram is drawn from the SAME tenant-filtered rows the Running tab
// renders, so the two can never disagree about which addresses exist or
// how they are doing — see flow.go's package comment for why it is not
diff --git a/internal/console/templates/app_overview.html b/internal/console/templates/app_overview.html
index 1ed17eaab..72e576d4f 100644
--- a/internal/console/templates/app_overview.html
+++ b/internal/console/templates/app_overview.html
@@ -37,6 +37,11 @@
<p class="tile-value">{{pct .CPUSum}} · {{bytesOrDash .MemSum}}</p>
<p class="tile-detail">{{if .CPUSum}}CPU · memory, from the copies responding{{else if .MetricsUnavailable}}no data right now{{else}}no recent data{{end}}</p>
</a>
+ <a class="tile" href="{{$.Nav}}/apps/{{$.AppName}}/instances">
+ <h2>Database</h2>
+ <p class="tile-value">{{bytesOrDash .DBSizeSum}} · {{pct .DiskFracMax}}</p>
+ <p class="tile-detail">{{if .DBSizeSum}}size · disk used, from the copies responding{{else if .MetricsUnavailable}}no data right now{{else}}no recent data{{end}}</p>
+ </a>
<a class="tile" href="{{$.Nav}}/apps/{{$.AppName}}/settings/domains">
<h2>Web addresses</h2>
<p class="tile-value">{{.DomainTotal}}</p>