amadan
Merge origin/main into sec-deps-dos-bumps
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
7 files changed,
+1101
−30
AGENTS.md+44 −0Makefile+22 −1internal/provision/census_propose_internal_test.go+373 −0internal/provision/census_propose_test.go+571 −0internal/provision/plan14_root_reads_test.go+88 −26internal/provision/testdata/plan14-root-reads-chain.tsv+3 −0internal/provision/testdata/plan14-root-reads-sec-deps-dos-bumps.tsv+0 −3
diff --git a/AGENTS.md b/AGENTS.md| index f3637c3d5..59be1f922 100644 |
| --- a/AGENTS.md |
| +++ b/AGENTS.md |
| @@ -222,6 +222,50 @@ anything. |
| GOPROXY=off`; go commands may need the sandbox disabled ("read-only file |
| system" from the toolchain is the tell). |
| +## The census: an approval, not a decision (Paul, 2026-10-24, #369) |
| + |
| +`TestPlan14RootReadCensus` (`internal/provision`) fails when a branch |
| +changes a unit that reads root-owned state, or a package inside the |
| +census envelope. Every such change needs the census owner's approval on |
| +#280 before the merge. Most are approvals: the same reads in an edited |
| +body, or an envelope fingerprint that moved. Get them approved like this: |
| + |
| +1. `make census-card` prints what changed and writes nothing. Each change |
| + has a class: |
| + - `body`: the digest moved and the reads did not; |
| + - `sites`: more or fewer reads of kinds the unit already makes; |
| + - `envelope`: a package was re-pinned; |
| + - `reads` (a kind of read the unit did not make before), `new`, |
| + `removed` or `other` (any census discrepancy none of these explains, |
| + shown with the census's own message): a decision. |
| +2. Write a CLAIMS file with one line per changed unit: |
| + `<file>:<unit><TAB><what changed, and that no read was added or which one was>`. |
| +3. `make census-propose NAME=<branch> CLAIMS=<file>` writes the rest: |
| + - the overlay (`testdata/plan14-root-reads-<NAME>.tsv`), each unit |
| + re-frozen at its existing disposition with the claim appended; |
| + - the `@legacy` fingerprints, the envelope re-pins and the new seal; |
| + - its line in `testdata/plan14-root-reads-chain.tsv`; |
| + - the approval card, `docs/superpowers/notes/census-proposal-<NAME>.md`. |
| + |
| + It checks that the overlay leaves the census clean, and writes nothing |
| + if any step fails. If the code changes again, rerun it with the same |
| + NAME: it replaces its own overlay (only one it wrote itself), which must |
| + still be the last line of the chain. |
| +4. Show the card to the census owner and ask for an approval. Ask in the |
| + session when they are there (one question: approve or hold); otherwise |
| + post the card on #280. Record the approval on #280: who, when, and the |
| + tip. Rows carry `rev3=census-ruling-<NAME>` from the start, so there is |
| + nothing to flip afterwards and no second CI run. |
| + |
| +The tool refuses decisions (`reads`, `new`, `removed`, `other`): a new |
| +kind of read can make a unit's disposition wrong, so it is the owner's |
| +call, not an approval. Write those rows by hand, following the overlays in the |
| +chain, and ask on #280 with the card attached; the census owner decides |
| +those. Nothing about the census's strength changes: every changed read is |
| +still named, digested and seen by a person before merge. The record-write |
| +census (9B8, the `*_census_test.go` inventories) still takes hand-written |
| +entries. |
| + |
| ## Rules this repo has paid for |
| - **A rule stated as an absence can only be checked by looking for the absent |
diff --git a/Makefile b/Makefile| index fd9a7f63f..efae74f45 100644 |
| --- a/Makefile |
| +++ b/Makefile |
| @@ -3,7 +3,7 @@ LDFLAGS := -s -w -X github.com/carlosframework/platform/internal/version.Build=$ |
| GOOS ?= |
| GOARCH ?= |
| -.PHONY: test build fmt |
| +.PHONY: test build fmt census-card census-propose |
| # The build gate: vet, format, tests. `.amadan/ci.d/10-gate` runs this same |
| # target, so what CI checks and what you check are one definition. |
| @@ -23,3 +23,24 @@ build: |
| fmt: |
| gofmt -w . |
| + |
| +# The root-read census as an approval (#369, AGENTS.md "The census"): |
| +# census-card prints what changed and writes nothing; census-propose writes |
| +# the overlay, its chain manifest line and the approval card. CLAIMS has |
| +# one line per changed unit, "<file>:<unit><TAB><what changed>". Kept here |
| +# rather than in a script: every .sh in the tree is itself a census unit. |
| +CENSUS_RUN = out=$$(mktemp "$${TMPDIR:-/tmp}/census.XXXXXX") || exit 1; \ |
| + trap 'rm -f "$$out"' EXIT; \ |
| + GOFLAGS="$${GOFLAGS:--buildvcs=false}" go test ./internal/provision/ -run '^TestPlan14RootReadCensusPropose$$' -count=1 -v >"$$out" 2>&1; rc=$$?; \ |
| + if [ $$rc -eq 0 ]; then sed -n '/^--- census card ---$$/,/^--- end census card ---$$/p' "$$out" | sed '1d;$$d'; \ |
| + else grep -v '^20[0-9][0-9]/' "$$out" | tail -20 >&2; fi; exit $$rc |
| +# CLAIMS is kept as one value, spaces and all, and made absolute for go test. |
| +CENSUS_CLAIMS = c='$(CLAIMS)'; if [ -n "$$c" ]; then case "$$c" in /*) ;; *) c="$$PWD/$$c" ;; esac; \ |
| + CARLOS_CENSUS_CLAIMS="$$c"; export CARLOS_CENSUS_CLAIMS; fi |
| + |
| +census-card: |
| + @CARLOS_CENSUS_CARD=1; export CARLOS_CENSUS_CARD; $(CENSUS_CLAIMS); $(CENSUS_RUN) |
| + |
| +census-propose: |
| + @if [ -z '$(NAME)' ] || [ -z '$(CLAIMS)' ]; then echo "usage: make census-propose NAME=<branch> CLAIMS=<file>" >&2; exit 2; fi; \ |
| + CARLOS_CENSUS_PROPOSE='$(NAME)'; export CARLOS_CENSUS_PROPOSE; $(CENSUS_CLAIMS); $(CENSUS_RUN) |
diff --git a/internal/provision/census_propose_internal_test.go b/internal/provision/census_propose_internal_test.go| new file mode 100644 |
| index 000000000..19a267956 |
| --- /dev/null |
| +++ b/internal/provision/census_propose_internal_test.go |
| @@ -0,0 +1,373 @@ |
| +package provision |
| + |
| +import ( |
| + "os" |
| + "path/filepath" |
| + "strings" |
| + "testing" |
| +) |
| + |
| +// proposeFixture is a small census: one legacy unit, one authority unit, one |
| +// envelope, sealed, and a scan that matches it. |
| +func proposeFixture(t *testing.T) (rootReadScan, rootReadInventory, string) { |
| + t.Helper() |
| + legacy := classifiedRootRead{rootReadEntry: rootReadEntry{File: "internal/a/a.go", Unit: "Wake", Kinds: "registry:Host@10,registry:DBPath@20", Digest: strings.Repeat("1", 64)}, Class: "legacy", Task: "accepted-legacy", Basis: "accepted-legacy;rev3=J1;basis=root-route"} |
| + auth := classifiedRootRead{rootReadEntry: rootReadEntry{File: "internal/a/b.go", Unit: "Route", Kinds: "registry:Host@5", Digest: strings.Repeat("2", 64)}, Class: "authority", Task: "E4", Basis: "root-route"} |
| + fp := rootReadLegacyFingerprint(legacy) |
| + inv := rootReadInventory{ |
| + Entries: map[string]classifiedRootRead{"internal/a/a.go:Wake": legacy, "internal/a/b.go:Route": auth}, |
| + Frozen: map[string]classifiedRootRead{"internal/a/a.go:Wake": legacy}, |
| + Legacy: map[string]bool{fp: true}, |
| + Coverage: map[string]string{"internal/a": strings.Repeat("c", 64)}, |
| + } |
| + got := rootReadScan{ |
| + Entries: []rootReadEntry{legacy.rootReadEntry, auth.rootReadEntry}, |
| + Declarations: map[string]string{"internal/a/a.go:Wake": legacy.Digest, "internal/a/b.go:Route": auth.Digest}, |
| + Coverage: map[string]string{"internal/a": strings.Repeat("c", 64)}, |
| + } |
| + return got, inv, rootReadLegacySeal(inv) |
| +} |
| + |
| +// change edits the scan as a code change would: a new digest, and the |
| +// unit's read sites (kinds) as given. |
| +func change(got *rootReadScan, key, kinds, digest string) { |
| + for i, e := range got.Entries { |
| + if e.File+":"+e.Unit == key { |
| + got.Entries[i].Kinds, got.Entries[i].Digest = kinds, digest |
| + } |
| + } |
| + got.Declarations[key] = digest |
| +} |
| + |
| +func TestCensusClassifiesEachKindOfChange(t *testing.T) { |
| + got, inv, seal := proposeFixture(t) |
| + // body: the same reads at moved offsets. |
| + change(&got, "internal/a/a.go:Wake", "registry:Host@14,registry:DBPath@24", strings.Repeat("3", 64)) |
| + // reads: a kind of read the unit did not make. |
| + change(&got, "internal/a/b.go:Route", "registry:Host@5,registry:Socket@9", strings.Repeat("4", 64)) |
| + got.Coverage["internal/a"] = strings.Repeat("d", 64) |
| + got.Entries = append(got.Entries, rootReadEntry{File: "internal/a/c.go", Unit: "New", Kinds: "file:Open@1", Digest: strings.Repeat("5", 64)}) |
| + got.Declarations["internal/a/c.go:New"] = strings.Repeat("5", 64) |
| + |
| + classes := map[string]string{} |
| + for _, c := range classifyRootReads(got, inv, seal) { |
| + classes[c.Key] = c.Class |
| + } |
| + want := map[string]string{ |
| + "internal/a/a.go:Wake": censusBody, |
| + "internal/a/b.go:Route": censusReads, |
| + "internal/a": censusEnvelope, |
| + "internal/a/c.go:New": censusNew, |
| + } |
| + for k, v := range want { |
| + if classes[k] != v { |
| + t.Errorf("class of %s = %q, want %q (all: %v)", k, classes[k], v, classes) |
| + } |
| + } |
| + if len(classes) != len(want) { |
| + t.Errorf("classes = %v, want exactly %v", classes, want) |
| + } |
| +} |
| + |
| +// sites: another read of a kind the unit already makes. |
| +func TestCensusClassifiesMoreReadsOfAKnownKindAsSites(t *testing.T) { |
| + got, inv, seal := proposeFixture(t) |
| + change(&got, "internal/a/b.go:Route", "registry:Host@5,registry:Host@9", strings.Repeat("4", 64)) |
| + cs := classifyRootReads(got, inv, seal) |
| + if len(cs) != 1 || cs[0].Class != censusSites { |
| + t.Fatalf("changes = %+v, want one sites", cs) |
| + } |
| +} |
| + |
| +// The card never shows less than the census: a discrepancy no unit or |
| +// envelope change explains is an "other" decision with the census's own |
| +// message (Astra review, census-approvals P2). |
| +func TestCensusClassifiesEveryOtherDiscrepancy(t *testing.T) { |
| + got, inv, seal := proposeFixture(t) |
| + for name, tc := range map[string]struct { |
| + edit func(*rootReadScan, *rootReadInventory) string |
| + want string |
| + }{ |
| + "a wrong seal": { |
| + edit: func(*rootReadScan, *rootReadInventory) string { return strings.Repeat("e", 64) }, |
| + want: "changed legacy freeze", |
| + }, |
| + "a removed declaration back in the tree": { |
| + edit: func(g *rootReadScan, i *rootReadInventory) string { |
| + i.Removed = map[string]string{"internal/a/z.go:Gone": "E4;census-ruling-x"} |
| + g.Declarations["internal/a/z.go:Gone"] = strings.Repeat("6", 64) |
| + return seal |
| + }, |
| + want: "internal/a/z.go:Gone", |
| + }, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + g, i := got, cloneRootReadInventory(inv) |
| + g.Declarations = cloneMap(got.Declarations) |
| + s := tc.edit(&g, &i) |
| + cs := classifyRootReads(g, i, s) |
| + if len(cs) == 0 { |
| + t.Fatal("classified nothing") |
| + } |
| + for _, c := range cs { |
| + if c.Class != censusOther || !strings.Contains(c.Key, tc.want) { |
| + t.Fatalf("changes = %+v, want an other carrying %q", cs, tc.want) |
| + } |
| + } |
| + if card := rootReadCard("x", cs, nil, s, ""); strings.Contains(card, "Nothing to approve") || !strings.Contains(card, tc.want) { |
| + t.Errorf("card:\n%s", card) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestCensusClassifiesARemovedUnit(t *testing.T) { |
| + got, inv, seal := proposeFixture(t) |
| + got.Entries = got.Entries[:1] |
| + delete(got.Declarations, "internal/a/b.go:Route") |
| + cs := classifyRootReads(got, inv, seal) |
| + if len(cs) != 1 || cs[0].Class != censusRemoved || cs[0].Key != "internal/a/b.go:Route" { |
| + t.Fatalf("changes = %+v, want one removed", cs) |
| + } |
| +} |
| + |
| +// The proposal is the whole job: loaded onto the inventory it leaves no |
| +// discrepancy, under the seal it reports, and every row names the ruling. |
| +func TestCensusProposalClosesEveryApprovableChange(t *testing.T) { |
| + got, inv, seal := proposeFixture(t) |
| + change(&got, "internal/a/a.go:Wake", "registry:Host@14,registry:DBPath@24", strings.Repeat("3", 64)) |
| + change(&got, "internal/a/b.go:Route", "registry:Host@5,registry:Host@9", strings.Repeat("4", 64)) |
| + got.Coverage["internal/a"] = strings.Repeat("d", 64) |
| + if len(censusDiscrepancies(got, inv, seal)) == 0 { |
| + t.Fatal("fixture change made no discrepancy") |
| + } |
| + claims := map[string]string{ |
| + "internal/a/a.go:Wake": "logs the wake's timing; no read added", |
| + "internal/a/b.go:Route": "reads the route's host again to log it", |
| + } |
| + p, err := proposeRootReads(got, inv, seal, "wake-timing", claims) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + path := filepath.Join(t.TempDir(), "overlay.tsv") |
| + if err := os.WriteFile(path, []byte(p.Overlay), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + after := cloneRootReadInventory(inv) |
| + if err := loadRootReadInventory(path, &after, true); err != nil { |
| + t.Fatal(err) |
| + } |
| + if errs := censusDiscrepancies(got, after, p.Seal); len(errs) != 0 { |
| + t.Fatalf("after the proposal: %v\n%s", errs, p.Overlay) |
| + } |
| + if rootReadLegacySeal(inv) != seal { |
| + t.Error("proposing changed the inventory it was given") |
| + } |
| + for _, line := range strings.Split(p.Overlay, "\n") { |
| + if line != "" && !strings.HasPrefix(line, "#") && !strings.HasPrefix(line, "@coverage") && !strings.Contains(line, "rev3=census-ruling-wake-timing;basis=") { |
| + t.Errorf("row does not name the ruling: %s", line) |
| + } |
| + } |
| + for _, want := range []string{"census-ruling-wake-timing", "internal/a/a.go", "body", "sites", "+registry:Host", "internal/a", seal, p.Seal, "logs the wake's timing"} { |
| + if !strings.Contains(p.Card, want) { |
| + t.Errorf("card lacks %q:\n%s", want, p.Card) |
| + } |
| + } |
| +} |
| + |
| +func TestCensusProposalRefuses(t *testing.T) { |
| + for name, tc := range map[string]struct { |
| + edit func(*rootReadScan) |
| + claims map[string]string |
| + want string |
| + }{ |
| + "a change without a claim": { |
| + edit: func(g *rootReadScan) { |
| + change(g, "internal/a/a.go:Wake", "registry:Host@14,registry:DBPath@24", strings.Repeat("3", 64)) |
| + }, |
| + claims: map[string]string{}, |
| + want: "no claim", |
| + }, |
| + "a claim for no change": { |
| + edit: func(g *rootReadScan) { |
| + change(g, "internal/a/a.go:Wake", "registry:Host@14,registry:DBPath@24", strings.Repeat("3", 64)) |
| + }, |
| + claims: map[string]string{"internal/a/a.go:Wake": "x", "internal/a/z.go:Typo": "y"}, |
| + want: "not a change", |
| + }, |
| + "a new kind of read": { |
| + edit: func(g *rootReadScan) { |
| + change(g, "internal/a/b.go:Route", "registry:Host@5,file:Open@9", strings.Repeat("4", 64)) |
| + }, |
| + claims: map[string]string{"internal/a/b.go:Route": "opens a file"}, |
| + want: "decision", |
| + }, |
| + "a new unit": { |
| + edit: func(g *rootReadScan) { |
| + g.Entries = append(g.Entries, rootReadEntry{File: "internal/a/c.go", Unit: "New", Kinds: "file:Open@1", Digest: strings.Repeat("5", 64)}) |
| + g.Declarations["internal/a/c.go:New"] = strings.Repeat("5", 64) |
| + }, |
| + claims: map[string]string{"internal/a/c.go:New": "opens a file"}, |
| + want: "decision", |
| + }, |
| + "a claim with a tab": { |
| + edit: func(g *rootReadScan) { |
| + change(g, "internal/a/a.go:Wake", "registry:Host@14,registry:DBPath@24", strings.Repeat("3", 64)) |
| + }, |
| + claims: map[string]string{"internal/a/a.go:Wake": "a\tb"}, |
| + want: "tab", |
| + }, |
| + "nothing to propose": { |
| + edit: func(*rootReadScan) {}, |
| + claims: map[string]string{}, |
| + want: "nothing", |
| + }, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + got, inv, seal := proposeFixture(t) |
| + tc.edit(&got) |
| + _, err := proposeRootReads(got, inv, seal, "x", tc.claims) |
| + if err == nil || !strings.Contains(err.Error(), tc.want) { |
| + t.Fatalf("err = %v, want one mentioning %q", err, tc.want) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +func TestCensusClaimsFile(t *testing.T) { |
| + path := filepath.Join(t.TempDir(), "claims") |
| + if err := os.WriteFile(path, []byte("# what changed\n\ninternal/a/a.go:Wake\tlogs the timing; no read added\n"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + claims, err := readCensusClaims(path) |
| + if err != nil || claims["internal/a/a.go:Wake"] != "logs the timing; no read added" || len(claims) != 1 { |
| + t.Fatalf("claims = %v, %v", claims, err) |
| + } |
| + if err := os.WriteFile(path, []byte("internal/a/a.go:Wake no tab\n"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := readCensusClaims(path); err == nil { |
| + t.Fatal("a line without a tab was accepted") |
| + } |
| +} |
| + |
| +// writeFixtureRoot is a tree with an empty chain manifest and the notes dir. |
| +func writeFixtureRoot(t *testing.T) string { |
| + t.Helper() |
| + root := t.TempDir() |
| + for _, d := range []string{"internal/provision/testdata", "docs/superpowers/notes"} { |
| + if err := os.MkdirAll(filepath.Join(root, d), 0o755); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if err := os.WriteFile(filepath.Join(root, rootReadChainManifest), []byte("# manifest\n"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + return root |
| +} |
| + |
| +func readFile(t *testing.T, path string) string { |
| + t.Helper() |
| + raw, err := os.ReadFile(path) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return string(raw) |
| +} |
| + |
| +func TestCensusProposalWritesAndReplacesItsOwnOverlay(t *testing.T) { |
| + root := writeFixtureRoot(t) |
| + first := rootReadProposal{Overlay: "# Census overlay x, " + censusProposeMark + ".\nrow1\n", Seal: strings.Repeat("a", 64), Card: "card 1"} |
| + if err := writeRootReadProposal(root, "x", first); err != nil { |
| + t.Fatal(err) |
| + } |
| + second := rootReadProposal{Overlay: "# Census overlay x, " + censusProposeMark + ".\nrow2\n", Seal: strings.Repeat("b", 64), Card: "card 2"} |
| + if err := writeRootReadProposal(root, "x", second); err != nil { |
| + t.Fatal(err) |
| + } |
| + if got, want := readFile(t, filepath.Join(root, rootReadChainManifest)), "# manifest\nplan14-root-reads-x.tsv\t"+strings.Repeat("b", 64)+"\n"; got != want { |
| + t.Errorf("manifest = %q, want %q", got, want) |
| + } |
| + if got := readFile(t, filepath.Join(root, "internal/provision/testdata/plan14-root-reads-x.tsv")); got != second.Overlay { |
| + t.Errorf("overlay = %q", got) |
| + } |
| + if got := readFile(t, filepath.Join(root, "docs/superpowers/notes/census-proposal-x.md")); got != "card 2" { |
| + t.Errorf("card = %q", got) |
| + } |
| +} |
| + |
| +func TestCensusProposalRefusesWhatItDidNotWrite(t *testing.T) { |
| + for name, setup := range map[string]func(t *testing.T, root string){ |
| + "a hand-written overlay of that name": func(t *testing.T, root string) { |
| + if err := os.WriteFile(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-x.tsv"), []byte("# Census overlay x (by hand)\n"), 0o644); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, |
| + "its overlay followed by another": func(t *testing.T, root string) { |
| + mine := rootReadProposal{Overlay: "# " + censusProposeMark + "\n", Seal: strings.Repeat("a", 64)} |
| + if err := writeRootReadProposal(root, "x", mine); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := writeRootReadProposal(root, "y", mine); err != nil { |
| + t.Fatal(err) |
| + } |
| + }, |
| + } { |
| + t.Run(name, func(t *testing.T) { |
| + root := writeFixtureRoot(t) |
| + setup(t, root) |
| + manifest := readFile(t, filepath.Join(root, rootReadChainManifest)) |
| + overlay, _ := os.ReadFile(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-x.tsv")) |
| + card, _ := os.ReadFile(filepath.Join(root, "docs/superpowers/notes/census-proposal-x.md")) |
| + p := rootReadProposal{Overlay: "# " + censusProposeMark + "\nnew\n", Seal: strings.Repeat("c", 64), Card: "card"} |
| + if err := writeRootReadProposal(root, "x", p); err == nil { |
| + t.Fatal("replaced an overlay it should not have") |
| + } |
| + if got := readFile(t, filepath.Join(root, rootReadChainManifest)); got != manifest { |
| + t.Errorf("manifest changed: %q", got) |
| + } |
| + if got, _ := os.ReadFile(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-x.tsv")); string(got) != string(overlay) { |
| + t.Errorf("overlay changed: %q", got) |
| + } |
| + if got, _ := os.ReadFile(filepath.Join(root, "docs/superpowers/notes/census-proposal-x.md")); string(got) != string(card) { |
| + t.Errorf("card changed: %q", got) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// Keys may contain spaces, and a directory is a prefix of its units' keys. |
| +func TestCensusNamesWholeKeys(t *testing.T) { |
| + for _, tc := range []struct { |
| + msg, key string |
| + want bool |
| + }{ |
| + {"changed root read internal/h/k.go:type gen; review class", "internal/h/k.go:type gen", true}, |
| + {"changed legacy declaration internal/h/k.go:type gen", "internal/h/k.go:type gen", true}, |
| + {"changed root read internal/h/k.go:type general; review", "internal/h/k.go:type gen", false}, |
| + {"changed root read internal/a/x.go:Foo; review", "internal/a", false}, |
| + {"changed root source envelope internal/a", "internal/a", true}, |
| + {"stale root read internal/a/x.go:Foo", "internal/a/x.go:Foo", true}, |
| + } { |
| + if got := censusNames(tc.msg, tc.key); got != tc.want { |
| + t.Errorf("censusNames(%q, %q) = %v, want %v", tc.msg, tc.key, got, tc.want) |
| + } |
| + } |
| +} |
| + |
| +// A body change to a unit whose key has a space is still an approval. |
| +func TestCensusClassifiesAKeyWithASpace(t *testing.T) { |
| + got, inv, seal := proposeFixture(t) |
| + key := "internal/a/t.go:type gen" |
| + e := classifiedRootRead{rootReadEntry: rootReadEntry{File: "internal/a/t.go", Unit: "type gen", Kinds: "registry:Host@3", Digest: strings.Repeat("7", 64)}, Class: "legacy", Task: "accepted-legacy", Basis: "accepted-legacy;rev3=J1;basis=root-route"} |
| + inv.Entries[key], inv.Frozen[key] = e, e |
| + inv.Legacy[rootReadLegacyFingerprint(e)] = true |
| + seal = rootReadLegacySeal(inv) |
| + got.Entries = append(got.Entries, e.rootReadEntry) |
| + got.Declarations[key] = e.Digest |
| + change(&got, key, "registry:Host@4", strings.Repeat("8", 64)) |
| + cs := classifyRootReads(got, inv, seal) |
| + if len(cs) != 1 || cs[0].Class != censusBody || cs[0].Key != key { |
| + t.Fatalf("changes = %+v, want one body for %q", cs, key) |
| + } |
| +} |
diff --git a/internal/provision/census_propose_test.go b/internal/provision/census_propose_test.go| new file mode 100644 |
| index 000000000..7dd3fa823 |
| --- /dev/null |
| +++ b/internal/provision/census_propose_test.go |
| @@ -0,0 +1,571 @@ |
| +package provision |
| + |
| +// census-propose (#369): census rulings as approvals. The root-read census |
| +// (TestPlan14RootReadCensus) fails on any change to a unit it tracks or to |
| +// a package envelope. Most of those changes are approvals, not decisions: |
| +// the same reads in an edited body, or an envelope fingerprint that moved. |
| +// This file classifies each discrepancy, writes the overlay that closes the |
| +// approvable ones (rows, @legacy fingerprints, envelope re-pins, the seal, |
| +// the chain manifest line), and the approval card a person reads to |
| +// approve it. Changes that need a disposition (a new unit, a removed one, |
| +// the scanner's field list) are decisions: it names them and writes |
| +// nothing. |
| +// |
| +// make census-card print the card; writes nothing |
| +// make census-propose NAME=<branch> CLAIMS=<file> write the overlay and the card |
| +// |
| +// CLAIMS has one line per changed unit, "<file>:<unit>\t<what changed>". |
| +// Rows carry rev3=census-ruling-<NAME> from the start: approval happens |
| +// before the merge, so there is no PROPOSED row to flip afterwards. |
| + |
| +import ( |
| + "fmt" |
| + "os" |
| + "path/filepath" |
| + "regexp" |
| + "sort" |
| + "strings" |
| + "testing" |
| +) |
| + |
| +// Change classes. The first three are approvals; the rest are decisions. |
| +const ( |
| + censusBody = "body" // digest moved, the same reads (offsets aside) |
| + censusSites = "sites" // more or fewer reads of kinds the unit already makes |
| + censusEnvelope = "envelope" // a package's source fingerprint moved |
| + censusReads = "reads" // a kind of read the unit did not make before |
| + censusNew = "new" // an unclassified unit: needs a disposition |
| + censusRemoved = "removed" // a unit or envelope gone: needs a removal row |
| + censusOther = "other" // a census discrepancy no change above explains |
| +) |
| + |
| +// censusDecision: a new kind of read can make a unit's disposition wrong |
| +// (a registry reader that starts opening files is not the unit that was |
| +// classified), so it is the owner's call, as a new or removed unit is |
| +// (Astra review, census-approvals P1). |
| +func censusDecision(class string) bool { |
| + return class == censusReads || class == censusNew || class == censusRemoved || class == censusOther |
| +} |
| + |
| +type censusChange struct { |
| + Key, Class string |
| + Before classifiedRootRead // the inventory's row; zero for a new unit |
| + After rootReadEntry // the scan's; zero for a removed unit |
| + // An envelope's fingerprints. |
| + OldHash, NewHash string |
| +} |
| + |
| +// kindCounts is a unit's read sites without their offsets. |
| +func kindCounts(kinds string) map[string]int { |
| + m := map[string]int{} |
| + if kinds == "" { |
| + return m |
| + } |
| + for _, k := range strings.Split(kinds, ",") { |
| + name, _, _ := strings.Cut(k, "@") |
| + m[name]++ |
| + } |
| + return m |
| +} |
| + |
| +// kindDiff says how a unit's read sites changed: "+registry:Socket", |
| +// "-file:Open ×2", or "unchanged (N)". |
| +func kindDiff(before, after string) string { |
| + b, a := kindCounts(before), kindCounts(after) |
| + names := map[string]bool{} |
| + for k := range b { |
| + names[k] = true |
| + } |
| + for k := range a { |
| + names[k] = true |
| + } |
| + var sorted []string |
| + for k := range names { |
| + sorted = append(sorted, k) |
| + } |
| + sort.Strings(sorted) |
| + var parts []string |
| + for _, k := range sorted { |
| + d := a[k] - b[k] |
| + sign := "+" |
| + if d < 0 { |
| + sign, d = "-", -d |
| + } |
| + switch { |
| + case d == 1: |
| + parts = append(parts, sign+k) |
| + case d > 1: |
| + parts = append(parts, fmt.Sprintf("%s%s ×%d", sign, k, d)) |
| + } |
| + } |
| + if len(parts) == 0 { |
| + n := 0 |
| + for _, c := range a { |
| + n += c |
| + } |
| + return fmt.Sprintf("unchanged (%d)", n) |
| + } |
| + return strings.Join(parts, ", ") |
| +} |
| + |
| +func sameKinds(before, after string) bool { |
| + return kindDiff(before, after) == kindDiff(after, after) |
| +} |
| + |
| +// newKinds reports whether after reads a kind before did not. |
| +func newKinds(before, after string) bool { |
| + b := kindCounts(before) |
| + for k := range kindCounts(after) { |
| + if b[k] == 0 { |
| + return true |
| + } |
| + } |
| + return false |
| +} |
| + |
| +// classifyRootReads names every discrepancy the census would report under |
| +// seal, one change per unit or envelope. A discrepancy no unit or envelope |
| +// change accounts for (a removed declaration back in the tree, the seal |
| +// itself, the scanner's field list) is an "other" decision carrying the |
| +// census's own message, so the card never shows less than the census does |
| +// (Astra review, census-approvals P2). |
| +func classifyRootReads(got rootReadScan, inv rootReadInventory, seal string) []censusChange { |
| + var cs []censusChange |
| + seen := map[string]bool{} |
| + for _, g := range got.Entries { |
| + key := g.File + ":" + g.Unit |
| + seen[key] = true |
| + w, ok := inv.Entries[key] |
| + switch { |
| + case !ok: |
| + cs = append(cs, censusChange{Key: key, Class: censusNew, After: g}) |
| + case w.Kinds == g.Kinds && w.Digest == g.Digest && declarationHolds(got, inv, key): |
| + case sameKinds(w.Kinds, g.Kinds): |
| + cs = append(cs, censusChange{Key: key, Class: censusBody, Before: w, After: g}) |
| + case newKinds(w.Kinds, g.Kinds): |
| + cs = append(cs, censusChange{Key: key, Class: censusReads, Before: w, After: g}) |
| + default: |
| + cs = append(cs, censusChange{Key: key, Class: censusSites, Before: w, After: g}) |
| + } |
| + } |
| + for key, w := range inv.Entries { |
| + if seen[key] { |
| + continue |
| + } |
| + decl, declared := got.Declarations[key] |
| + switch { |
| + case w.Class != "legacy" && w.Kinds == "" && declared && decl == w.Digest: |
| + // Declared, reads nothing, unchanged: not a discrepancy. |
| + case w.Class != "legacy" && w.Kinds == "" && declared: |
| + cs = append(cs, censusChange{Key: key, Class: censusBody, Before: w, After: rootReadEntry{File: w.File, Unit: w.Unit, Digest: decl}}) |
| + default: |
| + cs = append(cs, censusChange{Key: key, Class: censusRemoved, Before: w}) |
| + } |
| + } |
| + for dir, hash := range got.Coverage { |
| + if inv.Coverage[dir] != hash { |
| + cs = append(cs, censusChange{Key: dir, Class: censusEnvelope, OldHash: inv.Coverage[dir], NewHash: hash}) |
| + } |
| + } |
| + for dir, hash := range inv.Coverage { |
| + if _, ok := got.Coverage[dir]; !ok { |
| + cs = append(cs, censusChange{Key: dir, Class: censusRemoved, OldHash: hash}) |
| + } |
| + } |
| + keys := map[string]bool{} |
| + for _, c := range cs { |
| + keys[c.Key] = true |
| + } |
| + for _, msg := range censusDiscrepancies(got, inv, seal) { |
| + explained := false |
| + for key := range keys { |
| + if censusNames(msg, key) { |
| + explained = true |
| + break |
| + } |
| + } |
| + if !explained { |
| + cs = append(cs, censusChange{Key: msg, Class: censusOther}) |
| + } |
| + } |
| + sort.Slice(cs, func(i, j int) bool { |
| + if cs[i].Class != cs[j].Class { |
| + return cs[i].Class < cs[j].Class |
| + } |
| + return cs[i].Key < cs[j].Key |
| + }) |
| + return cs |
| +} |
| + |
| +// censusNames reports whether a census message names key whole: keys may |
| +// contain spaces ("file.go:type T"), and an envelope's directory is a |
| +// prefix of its units' keys, so the key must be bounded by a space before |
| +// it and a space, a semicolon or the end after it (Astra round 2). |
| +func censusNames(msg, key string) bool { |
| + for i := 0; ; { |
| + j := strings.Index(msg[i:], " "+key) |
| + if j < 0 { |
| + return false |
| + } |
| + end := i + j + 1 + len(key) |
| + if end == len(msg) || msg[end] == ' ' || msg[end] == ';' { |
| + return true |
| + } |
| + i += j + 1 |
| + } |
| +} |
| + |
| +// declarationHolds is checkRootReadDeclarations's test for one unit. |
| +func declarationHolds(got rootReadScan, inv rootReadInventory, key string) bool { |
| + frozen, ok := inv.Frozen[key] |
| + if !ok { |
| + return true |
| + } |
| + cur := inv.Entries[key] |
| + if cur.Class == "legacy" { |
| + return got.Declarations[key] == frozen.Digest && rootReadLegacyFingerprint(cur) == rootReadLegacyFingerprint(frozen) |
| + } |
| + return got.Declarations[key] == cur.Digest |
| +} |
| + |
| +type rootReadProposal struct { |
| + Overlay, Seal, Card string |
| +} |
| + |
| +var censusName = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`) |
| + |
| +// proposeRootReads writes the overlay that closes every change in the scan, |
| +// re-freezing each changed unit at the disposition it already has with its |
| +// claim appended, under census-ruling-<name>. It refuses decisions, a |
| +// change with no claim and a claim for no change, and checks its own work: |
| +// loaded onto a copy of inv, the overlay must leave the census clean. |
| +func proposeRootReads(got rootReadScan, inv rootReadInventory, seal, name string, claims map[string]string) (rootReadProposal, error) { |
| + if !censusName.MatchString(name) { |
| + return rootReadProposal{}, fmt.Errorf("census name %q: want lower-case letters, digits and dashes", name) |
| + } |
| + cs := classifyRootReads(got, inv, seal) |
| + if len(cs) == 0 { |
| + return rootReadProposal{}, fmt.Errorf("nothing to propose: the census is clean") |
| + } |
| + var decisions []string |
| + units := map[string]bool{} |
| + for _, c := range cs { |
| + if censusDecision(c.Class) { |
| + decisions = append(decisions, c.Class+" "+c.Key) |
| + } |
| + if c.Class == censusBody || c.Class == censusSites { |
| + units[c.Key] = true |
| + } |
| + } |
| + if len(decisions) > 0 { |
| + return rootReadProposal{}, fmt.Errorf("a decision is needed, not an approval: %s; write those rows by hand and ask on #280", strings.Join(decisions, "; ")) |
| + } |
| + for key := range units { |
| + if strings.TrimSpace(claims[key]) == "" { |
| + return rootReadProposal{}, fmt.Errorf("no claim for %s: say what changed in the CLAIMS file", key) |
| + } |
| + } |
| + for key, claim := range claims { |
| + if !units[key] { |
| + return rootReadProposal{}, fmt.Errorf("claim for %s: not a change the census sees", key) |
| + } |
| + if strings.ContainsAny(claim, "\t\n\r") { |
| + return rootReadProposal{}, fmt.Errorf("claim for %s: a tab or newline would break the row", key) |
| + } |
| + } |
| + |
| + ruling := "census-ruling-" + name |
| + var frozen, plain, cover []string |
| + counts := map[string]int{} |
| + for _, c := range cs { |
| + counts[c.Class]++ |
| + if c.Class == censusEnvelope { |
| + cover = append(cover, "@coverage\t"+c.Key+"\t"+c.NewHash) |
| + continue |
| + } |
| + e := classifiedRootRead{rootReadEntry: c.After, Class: c.Before.Class, Task: c.Before.Task, Basis: c.Before.Basis + ";rev3=" + ruling + ";basis=" + strings.TrimSpace(claims[c.Key])} |
| + row := strings.Join([]string{e.File, e.Unit, e.Class, e.Task, e.Basis, e.Kinds, e.Digest}, "\t") |
| + if e.Class == "legacy" { |
| + frozen = append(frozen, "@legacy\t"+rootReadLegacyFingerprint(e)+"\t"+row) |
| + } |
| + plain = append(plain, row) |
| + } |
| + header := []string{ |
| + fmt.Sprintf("# Census overlay %s, "+censusProposeMark+". %s, approved by the census owner before the merge; the approval card is docs/superpowers/notes/census-proposal-%s.md.", name, ruling, name), |
| + fmt.Sprintf("# %d body, %d sites, %d envelope: each changed unit re-frozen at its existing disposition with its claim appended.", counts[censusBody], counts[censusSites], counts[censusEnvelope]), |
| + } |
| + overlay := strings.Join(append(append(append(header, frozen...), plain...), cover...), "\n") + "\n" |
| + |
| + after := cloneRootReadInventory(inv) |
| + f, err := os.CreateTemp("", "census-overlay-*.tsv") |
| + if err != nil { |
| + return rootReadProposal{}, err |
| + } |
| + defer os.Remove(f.Name()) |
| + if _, err := f.WriteString(overlay); err != nil { |
| + f.Close() |
| + return rootReadProposal{}, err |
| + } |
| + if err := f.Close(); err != nil { |
| + return rootReadProposal{}, err |
| + } |
| + if err := loadRootReadInventory(f.Name(), &after, true); err != nil { |
| + return rootReadProposal{}, fmt.Errorf("the overlay does not load: %w", err) |
| + } |
| + newSeal := rootReadLegacySeal(after) |
| + if errs := censusDiscrepancies(got, after, newSeal); len(errs) != 0 { |
| + return rootReadProposal{}, fmt.Errorf("the overlay leaves discrepancies: %s", strings.Join(errs, "; ")) |
| + } |
| + return rootReadProposal{Overlay: overlay, Seal: newSeal, Card: rootReadCard(name, cs, claims, seal, newSeal)}, nil |
| +} |
| + |
| +func censusDiscrepancies(got rootReadScan, inv rootReadInventory, seal string) []string { |
| + errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) |
| + return append(errs, checkRootReadDeclarations(got, inv)...) |
| +} |
| + |
| +// rootReadCard is what a person reads to approve the change. newSeal is |
| +// empty for a dry run (make census-card). |
| +func rootReadCard(name string, cs []censusChange, claims map[string]string, seal, newSeal string) string { |
| + var b strings.Builder |
| + fmt.Fprintf(&b, "# Census approval: %s\n\n", name) |
| + var decisions []censusChange |
| + var envelopes []string |
| + var units []censusChange |
| + for _, c := range cs { |
| + switch { |
| + case censusDecision(c.Class): |
| + decisions = append(decisions, c) |
| + case c.Class == censusEnvelope: |
| + envelopes = append(envelopes, "`"+c.Key+"`") |
| + default: |
| + units = append(units, c) |
| + } |
| + } |
| + switch { |
| + case len(cs) == 0: |
| + fmt.Fprintf(&b, "Nothing to approve: the census is clean (seal `%s`).\n", seal) |
| + return b.String() |
| + case len(decisions) == 0: |
| + fmt.Fprintf(&b, "Ruling `census-ruling-%s`. Every change below keeps the disposition it already has, so it needs an approval, not a decision.\n\n", name) |
| + default: |
| + fmt.Fprintf(&b, "Ruling `census-ruling-%s`. %d change(s) need a decision (below), which census-propose cannot write.\n\n", name, len(decisions)) |
| + } |
| + if len(units) > 0 { |
| + b.WriteString("| unit | change | read sites | what changed |\n|---|---|---|---|\n") |
| + for _, c := range units { |
| + claim := strings.TrimSpace(claims[c.Key]) |
| + if claim == "" { |
| + claim = "(no claim yet)" |
| + } |
| + fmt.Fprintf(&b, "| `%s` `%s` | %s | %s | %s |\n", c.After.File, c.After.Unit, c.Class, kindDiff(c.Before.Kinds, c.After.Kinds), claim) |
| + } |
| + b.WriteString("\n") |
| + } |
| + if len(envelopes) > 0 { |
| + fmt.Fprintf(&b, "Envelopes re-pinned: %s.\n\n", strings.Join(envelopes, ", ")) |
| + } |
| + if len(decisions) > 0 { |
| + b.WriteString("Decisions (write their rows by hand and ask on #280):\n\n") |
| + for _, c := range decisions { |
| + detail := "" |
| + switch c.Class { |
| + case censusNew: |
| + detail = ": reads " + kindDiff("", c.After.Kinds) |
| + case censusReads: |
| + detail = ": " + kindDiff(c.Before.Kinds, c.After.Kinds) |
| + } |
| + fmt.Fprintf(&b, "- %s `%s`%s\n", c.Class, c.Key, detail) |
| + } |
| + b.WriteString("\n") |
| + } |
| + if newSeal != "" { |
| + fmt.Fprintf(&b, "Seal: `%s` → `%s`.\n\nOverlay: `internal/provision/testdata/%s`, appended to `%s`.\n", seal, newSeal, rootReadOverlayFile(name), rootReadChainManifest) |
| + } else { |
| + fmt.Fprintf(&b, "Seal now: `%s`. Dry run: nothing written.\n", seal) |
| + } |
| + return b.String() |
| +} |
| + |
| +// censusProposeMark is in the first line of every overlay census-propose |
| +// writes: a re-run replaces only a file that carries it. |
| +const censusProposeMark = "written by census-propose (#369)" |
| + |
| +func rootReadOverlayFile(name string) string { return "plan14-root-reads-" + name + ".tsv" } |
| + |
| +func cloneRootReadInventory(inv rootReadInventory) rootReadInventory { |
| + out := rootReadInventory{Fields: inv.Fields} |
| + out.Entries = cloneMap(inv.Entries) |
| + out.Frozen = cloneMap(inv.Frozen) |
| + out.Removed = cloneMap(inv.Removed) |
| + out.Coverage = cloneMap(inv.Coverage) |
| + out.Legacy = cloneMap(inv.Legacy) |
| + return out |
| +} |
| + |
| +func cloneMap[V any](m map[string]V) map[string]V { |
| + if m == nil { |
| + return nil |
| + } |
| + out := make(map[string]V, len(m)) |
| + for k, v := range m { |
| + out[k] = v |
| + } |
| + return out |
| +} |
| + |
| +// readCensusClaims reads a CLAIMS file: "<file>:<unit>\t<what changed>" |
| +// per line, blank lines and # comments ignored. |
| +func readCensusClaims(path string) (map[string]string, error) { |
| + raw, err := os.ReadFile(path) |
| + if err != nil { |
| + return nil, err |
| + } |
| + claims := map[string]string{} |
| + for i, line := range strings.Split(string(raw), "\n") { |
| + line = strings.TrimRight(line, "\r") |
| + if strings.TrimSpace(line) == "" || strings.HasPrefix(line, "#") { |
| + continue |
| + } |
| + key, claim, ok := strings.Cut(line, "\t") |
| + if !ok || !strings.Contains(key, ":") || strings.TrimSpace(claim) == "" { |
| + return nil, fmt.Errorf("%s:%d: want <file>:<unit><TAB><what changed>", path, i+1) |
| + } |
| + if _, dup := claims[key]; dup { |
| + return nil, fmt.Errorf("%s:%d: a second claim for %s", path, i+1, key) |
| + } |
| + claims[key] = claim |
| + } |
| + return claims, nil |
| +} |
| + |
| +const ( |
| + censusCardStart = "--- census card ---" |
| + censusCardEnd = "--- end census card ---" |
| +) |
| + |
| +// TestPlan14RootReadCensusPropose is census-propose and census-card (see |
| +// the Makefile). It does nothing unless one of them asked. |
| +func TestPlan14RootReadCensusPropose(t *testing.T) { |
| + name, dry := os.Getenv("CARLOS_CENSUS_PROPOSE"), os.Getenv("CARLOS_CENSUS_CARD") != "" |
| + if name == "" && !dry { |
| + t.Skip("run by make census-card and make census-propose") |
| + } |
| + root := censusRoot(t) |
| + // A re-run (the code changed again) proposes against the chain before |
| + // its own overlay, which must still be the manifest's last; nothing is |
| + // changed on disk until the new proposal is built and checked. |
| + before := "" |
| + if !dry { |
| + if !censusName.MatchString(name) { |
| + t.Fatalf("census name %q: want lower-case letters, digits and dashes", name) |
| + } |
| + before = rootReadOverlayFile(name) |
| + } |
| + got, err := rootReadProduction(root) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + inv, seal := rootReadChainBefore(t, root, before) |
| + claims := map[string]string{} |
| + if path := os.Getenv("CARLOS_CENSUS_CLAIMS"); path != "" { |
| + if claims, err = readCensusClaims(path); err != nil { |
| + t.Fatal(err) |
| + } |
| + } |
| + if dry { |
| + if name == "" { |
| + name = "<name>" |
| + } |
| + fmt.Printf("%s\n%s%s\n", censusCardStart, rootReadCard(name, classifyRootReads(got, inv, seal), claims, seal, ""), censusCardEnd) |
| + return |
| + } |
| + p, err := proposeRootReads(got, inv, seal, name, claims) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := writeRootReadProposal(root, name, p); err != nil { |
| + t.Fatal(err) |
| + } |
| + fmt.Printf("%s\n%s%s\n", censusCardStart, p.Card, censusCardEnd) |
| +} |
| + |
| +// writeRootReadProposal writes a checked proposal: the overlay, its line |
| +// at the end of the chain manifest (replacing its own line on a re-run) |
| +// and the approval card. It refuses, writing nothing, to replace an |
| +// overlay census-propose did not write, or one that is not the manifest's |
| +// last. |
| +func writeRootReadProposal(root, name string, p rootReadProposal) error { |
| + file := rootReadOverlayFile(name) |
| + overlay := filepath.Join(root, "internal/provision/testdata", file) |
| + if raw, err := os.ReadFile(overlay); err == nil { |
| + first, _, _ := strings.Cut(string(raw), "\n") |
| + if !strings.Contains(first, censusProposeMark) { |
| + return fmt.Errorf("%s exists and census-propose did not write it: propose under a new name", file) |
| + } |
| + } else if !os.IsNotExist(err) { |
| + return err |
| + } |
| + manifest := filepath.Join(root, rootReadChainManifest) |
| + raw, err := os.ReadFile(manifest) |
| + if err != nil { |
| + return err |
| + } |
| + lines := strings.Split(strings.TrimRight(string(raw), "\n"), "\n") |
| + links, err := readRootReadChainManifest(manifest) |
| + if err != nil { |
| + return err |
| + } |
| + for i, l := range links { |
| + if l.File != file { |
| + continue |
| + } |
| + if i != len(links)-1 || lines[len(lines)-1] != l.File+"\t"+l.Seal { |
| + return fmt.Errorf("%s is not the last overlay in %s: propose under a new name", file, rootReadChainManifest) |
| + } |
| + lines = lines[:len(lines)-1] |
| + } |
| + lines = append(lines, file+"\t"+p.Seal) |
| + note := filepath.Join(root, "docs/superpowers/notes", "census-proposal-"+name+".md") |
| + // Every file is written beside its target first and renamed only once |
| + // all three are on disk, the manifest last, so a failed write leaves |
| + // the previous proposal whole (Astra round 2). A rename failing between |
| + // the others is not rolled back (Astra round 3, declined): the files |
| + // are a working tree's, the census fails on any mismatch, and a re-run |
| + // recovers, since the manifest still ends in this name's line. |
| + writes := []struct{ path, text string }{ |
| + {overlay, p.Overlay}, |
| + {note, p.Card}, |
| + {manifest, strings.Join(lines, "\n") + "\n"}, |
| + } |
| + var temps []string |
| + defer func() { |
| + for _, tmp := range temps { |
| + os.Remove(tmp) |
| + } |
| + }() |
| + for _, w := range writes { |
| + f, err := os.CreateTemp(filepath.Dir(w.path), "."+filepath.Base(w.path)+".*") |
| + if err != nil { |
| + return err |
| + } |
| + temps = append(temps, f.Name()) |
| + if _, err := f.WriteString(w.text); err != nil { |
| + f.Close() |
| + return err |
| + } |
| + if err := f.Chmod(0o644); err != nil { |
| + f.Close() |
| + return err |
| + } |
| + if err := f.Close(); err != nil { |
| + return err |
| + } |
| + } |
| + for i, w := range writes { |
| + if err := os.Rename(temps[i], w.path); err != nil { |
| + return err |
| + } |
| + } |
| + return nil |
| +} |
diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go| index 41573409f..72cc46997 100644 |
| --- a/internal/provision/plan14_root_reads_test.go |
| +++ b/internal/provision/plan14_root_reads_test.go |
| @@ -593,6 +593,36 @@ func TestPlan14RootReadCensus(t *testing.T) { |
| if err != nil { |
| t.Fatal(err) |
| } |
| + inv, seal := rootReadChain(t, root) |
| + errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) |
| + errs = append(errs, checkRootReadDeclarations(got, inv)...) |
| + if len(errs) > 0 { |
| + t.Fatalf("root read census: %d discrepancies\n%s\n\nmake census-card shows what changed; make census-propose NAME=<branch> CLAIMS=<file> writes the overlay and its approval card (AGENTS.md, The census)", len(errs), strings.Join(errs, "\n")) |
| + } |
| + counts := map[string]int{} |
| + sites := 0 |
| + for _, e := range inv.Entries { |
| + counts[e.Class]++ |
| + sites += len(strings.Split(e.Kinds, ",")) |
| + } |
| + t.Logf("%d units, %d sites, %d source envelopes; classes %v", len(inv.Entries), sites, len(inv.Coverage), counts) |
| +} |
| + |
| +// rootReadChain loads the census inventory: the base file and every overlay |
| +// in order, each loaded only after its predecessor's seal is checked, then |
| +// the overlays the chain manifest lists (rootReadManifest). It returns the |
| +// inventory and the seal the last overlay leaves. |
| +func rootReadChain(t testing.TB, root string) (rootReadInventory, string) { |
| + t.Helper() |
| + return rootReadChainBefore(t, root, "") |
| +} |
| + |
| +// rootReadChainBefore is rootReadChain without the manifest's last overlay |
| +// when that overlay is the file named last (a census-propose re-run |
| +// proposing against the chain before its own overlay); "" loads it all. |
| +func rootReadChainBefore(t testing.TB, root, last string) (rootReadInventory, string) { |
| + t.Helper() |
| + var err error |
| inv := rootReadInventory{Entries: map[string]classifiedRootRead{}, Coverage: map[string]string{}} |
| if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads.tsv"), &inv, false); err != nil { |
| t.Fatal(err) |
| @@ -1456,32 +1486,7 @@ func TestPlan14RootReadCensus(t *testing.T) { |
| t.Fatal(err) |
| } |
| seal = "09e20ea4bbc534710334e632e903878656dabfb5817d4d96c6e7c56508172148" |
| - // Census overlay sec-deps-dos-bumps (golang.org/x/net v0.60.0, |
| - // golang.org/x/text v0.42.0 and github.com/pion/stun/v3 v3.1.5 for |
| - // govulncheck's reachable DoS findings), loaded last, after |
| - // wake-latency-1. Envelope-only: go.mod and go.sum re-pin the root |
| - // package's envelope and nothing else; no read is added, removed or |
| - // changed, so the seal does not move (wake-latency-1's is checked first). PROPOSED-sec-deps-dos-bumps, asked |
| - // of the census owner on #280 |
| - // (docs/superpowers/notes/census-proposal-sec-deps-dos-bumps.md). |
| - if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 { |
| - t.Fatal(errs) |
| - } |
| - if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-sec-deps-dos-bumps.tsv"), &inv, true); err != nil { |
| - t.Fatal(err) |
| - } |
| - errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) |
| - errs = append(errs, checkRootReadDeclarations(got, inv)...) |
| - if len(errs) > 0 { |
| - t.Fatalf("root read census: %d discrepancies\n%s", len(errs), strings.Join(errs, "\n")) |
| - } |
| - counts := map[string]int{} |
| - sites := 0 |
| - for _, e := range inv.Entries { |
| - counts[e.Class]++ |
| - sites += len(strings.Split(e.Kinds, ",")) |
| - } |
| - t.Logf("%d units, %d sites, %d source envelopes; classes %v", len(inv.Entries), sites, len(inv.Coverage), counts) |
| + return rootReadManifest(t, root, inv, seal, last) |
| } |
| func TestPlan14RootReadCensusRejectsDrift(t *testing.T) { |
| @@ -1740,3 +1745,60 @@ func TestPlan14ActivationToolsRemovalRequiresAbsentDeclaration(t *testing.T) { |
| t.Fatal("unruled removal accepted") |
| } |
| } |
| + |
| +// rootReadChainManifest lists the overlays loaded after the chain written |
| +// out in rootReadChain, in order, one per line: the overlay's file name |
| +// under testdata and the seal it leaves. census-propose appends to it |
| +// (rootReadPropose), so a new overlay needs no edit to this file. |
| +const rootReadChainManifest = "internal/provision/testdata/plan14-root-reads-chain.tsv" |
| + |
| +type rootReadChainLink struct{ File, Seal string } |
| + |
| +func readRootReadChainManifest(path string) ([]rootReadChainLink, error) { |
| + raw, err := os.ReadFile(path) |
| + if err != nil { |
| + return nil, err |
| + } |
| + var links []rootReadChainLink |
| + for i, line := range strings.Split(strings.TrimSpace(string(raw)), "\n") { |
| + if line == "" || strings.HasPrefix(line, "#") { |
| + continue |
| + } |
| + col := strings.Split(line, "\t") |
| + if len(col) != 2 || len(col[1]) != 64 || strings.ContainsAny(col[0], "/\\") { |
| + return nil, fmt.Errorf("%s:%d: want <overlay file name>\\t<seal>", path, i+1) |
| + } |
| + if _, err := hex.DecodeString(col[1]); err != nil { |
| + return nil, fmt.Errorf("%s:%d: seal: %v", path, i+1, err) |
| + } |
| + links = append(links, rootReadChainLink{col[0], col[1]}) |
| + } |
| + return links, nil |
| +} |
| + |
| +// rootReadManifest loads the manifest's overlays onto inv, each after the |
| +// seal before it is checked, exactly as rootReadChain loads its own. A |
| +// non-empty last leaves that overlay out, and only when it is the last. |
| +func rootReadManifest(t testing.TB, root string, inv rootReadInventory, seal, last string) (rootReadInventory, string) { |
| + t.Helper() |
| + links, err := readRootReadChainManifest(filepath.Join(root, rootReadChainManifest)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + for i, l := range links { |
| + if last != "" && l.File == last { |
| + if i != len(links)-1 { |
| + t.Fatalf("%s is not the last overlay in %s: propose under a new name", last, rootReadChainManifest) |
| + } |
| + break |
| + } |
| + if errs := checkRootReadLegacy(inv, seal); len(errs) != 0 { |
| + t.Fatalf("before overlay %s: %v", l.File, errs) |
| + } |
| + if err := loadRootReadInventory(filepath.Join(root, "internal/provision/testdata", l.File), &inv, true); err != nil { |
| + t.Fatal(err) |
| + } |
| + seal = l.Seal |
| + } |
| + return inv, seal |
| +} |
diff --git a/internal/provision/testdata/plan14-root-reads-chain.tsv b/internal/provision/testdata/plan14-root-reads-chain.tsv| new file mode 100644 |
| index 000000000..786c30395 |
| --- /dev/null |
| +++ b/internal/provision/testdata/plan14-root-reads-chain.tsv |
| @@ -0,0 +1,3 @@ |
| +# Census chain manifest: overlays loaded after rootReadChain's own, in order. |
| +# One per line: <overlay file name under testdata>\t<the seal it leaves>. |
| +# make census-propose appends here; nothing else edits this file. |
diff --git a/internal/provision/testdata/plan14-root-reads-sec-deps-dos-bumps.tsv b/internal/provision/testdata/plan14-root-reads-sec-deps-dos-bumps.tsv| deleted file mode 100644 |
| index cfe83b629..000000000 |
| --- a/internal/provision/testdata/plan14-root-reads-sec-deps-dos-bumps.tsv |
| +++ /dev/null |
| @@ -1,3 +0,0 @@ |
| -# Census overlay sec-deps-dos-bumps (go.mod and go.sum move golang.org/x/net to v0.60.0, golang.org/x/text to v0.42.0 and github.com/pion/stun/v3 to v3.1.5 for govulncheck's HTTP/2 DoS, idna infinite-loop and STUN panic findings; x/net v0.60.0 also raises the go directive from 1.26 to 1.26.0 and pulls x/crypto, x/sys, x/sync, x/mod, x/tools, pion/dtls and pion/transport/v4 forward), loaded LAST in TestPlan14RootReadCensus, after wake-latency-1. PROPOSED: rev3=PROPOSED-sec-deps-dos-bumps, ruling asked of the census owner on #280 (docs/superpowers/notes/census-proposal-sec-deps-dos-bumps.md). |
| -# Envelope-only. go.mod and go.sum are in the root package's source envelope, so the bump re-pins `.` from main's cc17fa0a… (db-region-forward-3) to the value below. No source file changes: no root-read unit is added, changed or removed, and the legacy seal does not move. |
| -@coverage . ff7e60e8734238290264ba3b27931d9477811d9a37921a205b1b7244a610bef0 |