amadan
docs: region HA plan 1b-next rev 7 proves a thread's descriptor is the leader's with kcmp
Astra round 6 finding 1 (P1), accepted. The writers-stopped scan counts a thread's descriptor as the leader's only if the kernel confirms it: kcmp(2) KCMP_FILES shows the thread shares the leader's table, and KCMP_FILE shows the descriptor is the same open file description. x/sys has only SYS_KCMP, so this is a raw syscall. Any kcmp error other than ESRCH or EBADF is ErrIncomplete. The tests run on the real kernel with LockOSThread and unshare(CLONE_FILES): one thread keeps the copied descriptor, the other dup3s an independent open onto the leader's number. Dry-run on 3407d114; 3 guards removed in turn, each failing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+311
−16
docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md+311 −16
diff --git a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md| index 24ef7beb0..e07aec0ac 100644 |
| --- a/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| +++ b/docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md |
| @@ -1,15 +1,25 @@ |
| -# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 6) |
| +# Region HA 1b-next: Relocation, Row Cases, Create Choice and Fences Implementation Plan (rev 7) |
| > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| **Goal:** Build what region HA spec 1 had to leave until the host lifecycle claim was on main: the `relocating` operation within one region, released by the source's stop evidence (drain) or by its termination fence (failover); reconcile's row cases for it; create's once-only choice of executor box with `boxchoice.Choose`; and every claim CAS that consumes a termination fence. All of it dormant behind `CARLOS_REGION_HA`, and none of it mergeable until the joint two-box simulation round passes (L18). |
| -**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 5 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| +**Architecture:** Two stages (ruling 24). **Stage A, build now:** Tasks 1 to 7 and 9a, buildable on main today and dry-run on every revision: the claim library's relocation transitions and `released` coverage (`internal/hostclaim`), create's executor selection (`internal/hostclaim`, `internal/hostops`), the row and wake decisions (`internal/hostbox`), the writers-stopped observation (`internal/stopevidence`), the regional worker and the participant driver (new `internal/hostrelocate`), and the box-side release adapter with its receipt verifier (`internal/host`). **Stage B, gated:** Tasks 8a, 8b, 8c, 9 and 10 bind to claim Plan 6b and moves Plans 1 and 2, none merged. They keep their design and contracts here; their code is rewritten against the real 6b code after it merges and re-reviewed then (the "Stage B re-plan" step before Task 8a lists every obligation rounds 1 to 6 placed on them). What those need and no plan declares is a numbered request, never an assumption. The last task is the merge gate. The worker only reads termination fences: `hostfence.TerminationWorker` writes them, and `hostactors.Directory` is the only roster reader. |
| **Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `release.Store` (create-only and ETag compare-and-swap puts); the claim library (`hostclaim`), roster (`hostactors`), fences (`hostfence`, `termfence`), box choice (`boxchoice`), files (`hostfiles`), generation selection (`genselect`), stop evidence (`stopevidence`, `internal/host` stop receipts), execution controller (`hostexec`). |
| **Spec:** `docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md`, rev 5.4 with §24 (rulings H1 to H19 final). Read §0, §2, §5.2, §5.3, §5.8, §5.9, §5.10, §9 (L1 to L20, accepted) and §24 first. Then the claim spec rev 6.30 on `origin/design-replica-ownership` (`docs/superpowers/specs/2026-10-02-host-lifecycle-claim-design.md`): §1.1's activation gate, §4 ("Same-region relocation (L1)" and the relocation exception to CC11), §5.1 and P5-D2/P5-D6 (the create intent), §6.1 ("Release before full stop (L10)"), §6.4a item 2 (`released`), §7.3 D2 ("Automatic termination (L2, L4)", "The fence's journal reference") and D4 ("Automatic loss persistence"). Then moves rev 7.22 on `origin/design-moves-any-region` §7.6 ("One install routine, two sources", L16 and L17) and §7.10 (the pre-flip conversion). The contracts note `docs/superpowers/notes/2026-10-07-region-ha-contracts.md` gives `hostfiles` v2, `CopyManifest`, `Publish` and `boxchoice`. Plan 1b (`docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`) lists in "Waits for the claim" what this plan now builds. |
| +## Rev 7: round 6 dispositions |
| + |
| +Astra's round 6 (`CODEX-PLAN-REGION-HA-1BN-R6.md`, against `222ff1c6`) resolved all six of round 5's findings and added one. It was checked against the plan and the kernel before anything changed, and is accepted; none is declined. The fix is stage A code, dry-run again on a clean 3407d114 (build; vet for darwin, windows and linux/arm64; cross-builds; the stopevidence tests run five times against the real kernel), with 3 new guards removed in turn (listed in the Self-review). Stage B is unchanged: R3's root endpoint calls the same `WritersStopped`. |
| + |
| +| # | Stage | Evidence checked | Disposition | What changed | Where | |
| +|---|---|---|---|---|---| |
| +| 1 (P1) | A | `scanOne` exempted a thread's descriptor when the leader showed the same number and path. Confirmed on this kernel: a thread that runs `unshare(CLONE_FILES)` keeps a copied table with the database at the same number, and it can `dup3` an independent open onto that number; `/proc/<pid>/task/<tid>/fd/<n>` reads the same path either way. kcmp(2) is the positive check: `KCMP_FILES` compares two tasks' descriptor tables and `KCMP_FILE` compares two descriptors' open file descriptions. golang.org/x/sys v0.46.0 has `unix.SYS_KCMP` on all 15 linux architectures but no `unix.Kcmp` and no constants, so the plan makes a raw `Syscall6` with `KCMP_FILE = 0` and `KCMP_FILES = 2` from linux/kcmp.h. kcmp needs ptrace read access to both tasks. The agent always has it within its own thread group, and root's complete scan has it through `CAP_SYS_PTRACE`. This box runs `CONFIG_KCMP=y` and Yama scope 0. | accepted | A thread's descriptor is the leader's only if three things hold: the leader shows the same number and path, `KCMP_FILES(pid, tid)` is 0 (the thread shares the leader's table), and `KCMP_FILE(pid, tid, n, n)` is 0 (the same open file description). Otherwise it is reported with FD -1, which no allow-list exempts. A copied table counts as foreign even when the description is shared, because the thread could still use that description after the replicator closes. If a task exits mid-scan (ESRCH), it is skipped as before. EBADF is treated as "not the leader's". Any other kcmp error, including ENOSYS and EPERM, is `ErrIncomplete`. kcmp takes pids in the caller's pid namespace, so it is used only for the caller's own `/proc`. Any other root fails closed unless a test replaces `kcmpFor`. The `//go:build linux` file and the `!linux` stub stay as they were. | Task 4 (`TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity` on the real kernel: a shared locked thread adds nothing; an unshared thread with the copied descriptor, and one that reopened the database and `dup3`'d it onto the leader's number, are each reported FD -1. `TestWritersStoppedRefusesAnyOtherHolder`'s new fixture cases: an unshared thread at the leader's number and path, a shared table whose descriptor is another description, and kcmp unavailable (ENOSYS, EPERM) as `ErrIncomplete` for both scans) | |
| + |
| +Astra's readings of rulings 1 to 24 stand. |
| + |
| ## Rev 6: round 5 dispositions |
| Astra's round 5 (`CODEX-PLAN-REGION-HA-1BN-R5.md`, against `996b24c3`) left R4-5 partly resolved and added five findings. Each was checked against the tree at 3407d114 and the plan before anything changed; all six are accepted and none is declined. Every one lands in stage A code, dry-run again on a clean 3407d114 (build, vet for darwin and windows, cross-builds for darwin, windows and linux/arm64, targeted tests), with 11 new guards removed in turn (listed in the Self-review); R4-5's restart obligations are carried into the stage B re-plan as well. |
| @@ -177,7 +187,7 @@ Left out, with where it goes: |
| - **Go environment.** `GOFLAGS=-buildvcs=false`. Leave `GOCACHE` at its default; if the sandbox refuses it, `GOCACHE=$TMPDIR/gocache` for that command. Socket tests need `TMPDIR=/tmp/k` (`mkdir -p /tmp/k`). Run tests niced (`nice -n 19`), targeted with `-run`; never the full `-race` suite, and no simulation sweeps. Scratch goes under `$TMPDIR`. |
| - **Gate before each commit:** `go build ./... && go vet ./<pkgs>/ && gofmt -l <dirs>` (empty), the task's targeted tests, then `GOOS=darwin go build ./... && GOOS=windows go build ./...`. (`GOOS=windows go vet ./internal/...` fails on main's own test files, `release/versions_test.go`, `appconfig/syncexec_test.go` and `activator/activator_test.go`; vet the task's packages instead.) |
| - Commit messages: lowercase `area:` prefix and a sentence stating the behaviour, then a blank line and `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. Work on a branch cut from `origin/main` (suggested `region-ha-1b-next`). Merging is Paul's call, through `amadan branch merge`, and only after Task 11. |
| -- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven and rev 6's eleven (listed in the Self-review). Then the code was removed; only this document is committed. |
| +- **Dry run.** Tasks 1 to 7 were applied in order to `origin/main` at 3407d114, built, vetted, cross-built for darwin, windows and linux/arm64, and their tests run; nineteen guards were each removed in turn and their test watched to fail (rev 1), then rev 2's twenty-five, rev 3's eleven, rev 4's eight, rev 5's seven, rev 6's eleven and rev 7's three (listed in the Self-review). Then the code was removed; only this document is committed. |
| ## Review Focus |
| @@ -185,7 +195,7 @@ Left out, with where it goes: |
| 2. **P opened twice.** A restarted probe that restores P again loses everything attempt two replicated (B1-1). Expect every attempt after the first to take a fresh generation, and P refused. Task 1, `TestRelocationAdmissionRules/a_third_attempt_reopens_P`. |
| 3. **A failover pinned too early.** Requests S3 accepted from the terminated box can still land; a pin taken before `observed_at` + 30 s widens H3's window. Expect the worker to wait and the claim to refuse. Task 1 `TestRelocationFailoverCitesTheFenceAndDeclaresTheLoss`, Task 5 `TestWorkerPinsNoEarlierThanThirtySecondsAfterTheFence`. |
| 4. **A fence that cannot be read taken as no fence.** A garbled `control/box-fences/` object must stop the drain and the create choice, not let them proceed as if the box were alive. Task 2 `TestExecutorChooserNeverNamesAFencedLostOrUnknownBox/an_unreadable_fence`, Task 5 `TestWorkerTreatsAnUnreadableFenceAsAnErrorNeverAsNoFence`. |
| -5. **A database descriptor in an unshared thread exempted as the replicator's.** Writers-stopped allows only the replicator's identified descriptors; a thread whose table no longer matches the leader's must count as a writer. Task 4, `TestWritersStoppedRefusesAnyOtherHolder/a_thread's_descriptor_the_leader's_table_does_not_show`. |
| +5. **A database descriptor in an unshared thread exempted as the replicator's.** Writers-stopped allows only the replicator's identified descriptors; a thread whose table no longer matches the leader's must count as a writer. A matching number and path is not enough: the kernel must say the table and the description are shared (rev 7). Task 4, `TestWritersStoppedRefusesAnyOtherHolder/a_thread's_descriptor_the_leader's_table_does_not_show`, `…/an_unshared_thread_at_the_leader's_number_and_path`, `TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity`. |
| ## File Structure |
| @@ -3810,14 +3820,14 @@ Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>" |
| ### Task 4: Writers stopped, with the replicator still open (L10) |
| -Claim §6.1, "Release before full stop": release step 1 needs the admission fence, settled spawns and drained work (the drain's own facts), an empty cgroup with no members, and a complete descriptor scan in which the only holders of the database, WAL and SHM files are descriptors positively identified as the replicator's: its process (PID and start time, field 22 of `/proc/<pid>/stat`) and each descriptor number and target. A descriptor seen in a thread whose table the leader's does not match is never exempt. `Collect` is unchanged: full stop evidence follows once the replicator closes. |
| +Claim §6.1, "Release before full stop": release step 1 needs the admission fence, settled spawns and drained work (the drain's own facts), an empty cgroup with no members, and a complete descriptor scan in which the only holders of the database, WAL and SHM files are descriptors positively identified as the replicator's: its process (PID and start time, field 22 of `/proc/<pid>/stat`) and each descriptor number and target. A descriptor seen in a thread whose table the leader's does not match is never exempt. *Rev 7:* that match is the kernel's word, never a number and a path. kcmp(2) `KCMP_FILES` must show the thread shares the leader's table, and `KCMP_FILE` must show the descriptor is the leader's open file description. A comparison that cannot be made fails closed as `ErrIncomplete`. `Collect` is unchanged: full stop evidence follows once the replicator closes. |
| **Files:** |
| - Create: `internal/stopevidence/writers.go`, `internal/stopevidence/writers_linux.go`, `internal/stopevidence/writers_other.go` |
| - Test: `internal/stopevidence/writers_linux_test.go` |
| **Interfaces:** |
| -- Consumes: `realFDOps`, `readProcDir`, `procGone`, `ErrIncomplete`, `ErrInvalidInput`, `ErrUnsupported`, `ErrNoCgroupEvidence`, `ErrPopulated`, `ErrMembers` (all on main). |
| +- Consumes: `realFDOps`, `readProcDir`, `procGone`, `ErrIncomplete`, `ErrInvalidInput`, `ErrUnsupported`, `ErrNoCgroupEvidence`, `ErrPopulated`, `ErrMembers` (all on main); `unix.SYS_KCMP`, `unix.Syscall6`, `unix.Unshare`, `unix.CLONE_FILES`, `unix.Dup3` (golang.org/x/sys v0.46.0, already a dependency). |
| - Produces: `stopevidence.Descriptor{PID, FD int; Target string}`, `stopevidence.WritersInputs`, `stopevidence.WritersEvidence`, `stopevidence.WritersStopped(ctx, WritersInputs) (WritersEvidence, error)`, `stopevidence.ProcessDescriptors(ctx, procRoot string, pid int, dbs []string) ([]Descriptor, error)` (one process's descriptors on the database, reading only `/proc/<pid>`, taken by the agent for its own pid before and after the replicator opens: R3's agent half), `ErrWorkOpen`, `ErrForeignHolder`. `WritersStopped`'s complete scan of every process is root's (ruling 22). |
| - [ ] **Step 1: Write the failing tests** |
| @@ -3833,15 +3843,68 @@ import ( |
| "errors" |
| "os" |
| "path/filepath" |
| + "runtime" |
| + "slices" |
| "strconv" |
| "testing" |
| + |
| + "golang.org/x/sys/unix" |
| +) |
| + |
| +// fixtureUnshared names the fixture threads that do not share their |
| +// leader's descriptor table; fixtureKcmpErr makes every comparison fail. |
| +var ( |
| + fixtureUnshared map[int]bool |
| + fixtureKcmpErr error |
| ) |
| +// fixtureKcmp answers kcmp(2) for a /proc fixture as the kernel would: a |
| +// thread shares the leader's table unless fixtureUnshared names it, and a |
| +// shared table's descriptor n is the leader's n; a descriptor missing from |
| +// either table is EBADF. |
| +func fixtureKcmp(root string) kcmpFunc { |
| + return func(pid, tid, typ int, idx1, idx2 uintptr) (int, error) { |
| + if fixtureKcmpErr != nil { |
| + return 0, fixtureKcmpErr |
| + } |
| + switch typ { |
| + case kcmpFiles: |
| + if fixtureUnshared[tid] { |
| + return 1, nil |
| + } |
| + return 0, nil |
| + case kcmpFile: |
| + fd := func(task int, n uintptr) (string, error) { |
| + return os.Readlink(filepath.Join(root, strconv.Itoa(pid), "task", strconv.Itoa(task), "fd", strconv.Itoa(int(n)))) |
| + } |
| + a, errA := fd(pid, idx1) |
| + b, errB := fd(tid, idx2) |
| + if errA != nil || errB != nil { |
| + return 0, unix.EBADF |
| + } |
| + if fixtureUnshared[tid] || a != b { |
| + return 1, nil |
| + } |
| + return 0, nil |
| + } |
| + return 0, unix.EINVAL |
| + } |
| +} |
| + |
| // writersProc builds a /proc fixture: the agent (pid 20, threads 20 and |
| // 7, sharing the table) holds the database on fds 5 and 6. |
| func writersProc(t *testing.T) (string, string) { |
| t.Helper() |
| root, db := t.TempDir(), filepath.Join(t.TempDir(), "data.db") |
| + fixtureUnshared, fixtureKcmpErr = map[int]bool{}, nil |
| + prior := kcmpFor |
| + kcmpFor = func(r string) kcmpFunc { |
| + if r == root { |
| + return fixtureKcmp(root) |
| + } |
| + return prior(r) |
| + } |
| + t.Cleanup(func() { kcmpFor = prior }) |
| for _, tid := range []string{"20", "7"} { |
| dir := filepath.Join(root, "20", "task", tid, "fd") |
| if err := os.MkdirAll(dir, 0700); err != nil { |
| @@ -3969,6 +4032,45 @@ func TestWritersStoppedRefusesAnyOtherHolder(t *testing.T) { |
| t.Fatalf("got %v", err) |
| } |
| }) |
| + // Round 6 finding 1: a thread holding the database at the leader's |
| + // number and path is the leader's only on the kernel's word. |
| + t.Run("an unshared thread at the leader's number and path", func(t *testing.T) { |
| + root, db := writersProc(t) |
| + fixtureUnshared[7] = true |
| + if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrForeignHolder) { |
| + t.Fatalf("got %v", err) |
| + } |
| + }) |
| + t.Run("a shared thread whose descriptor is another description", func(t *testing.T) { |
| + // The table compared shared, then the descriptor did not: the |
| + // thread unshared and reopened between the two comparisons. |
| + root, db := writersProc(t) |
| + prior := kcmpFor |
| + kcmpFor = func(string) kcmpFunc { |
| + return func(pid, tid, typ int, a, b uintptr) (int, error) { |
| + if typ == kcmpFile { |
| + return 2, nil |
| + } |
| + return 0, nil |
| + } |
| + } |
| + defer func() { kcmpFor = prior }() |
| + if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrForeignHolder) { |
| + t.Fatalf("got %v", err) |
| + } |
| + }) |
| + for _, kerr := range []error{unix.ENOSYS, unix.EPERM} { |
| + t.Run("kcmp unavailable: "+kerr.Error(), func(t *testing.T) { |
| + root, db := writersProc(t) |
| + fixtureKcmpErr = kerr |
| + if _, err := WritersStopped(t.Context(), writersInputs(root, db)); !errors.Is(err, ErrIncomplete) { |
| + t.Fatalf("got %v", err) |
| + } |
| + if _, err := ProcessDescriptors(t.Context(), root, 20, []string{db}); !errors.Is(err, ErrIncomplete) { |
| + t.Fatalf("own descriptors: %v", err) |
| + } |
| + }) |
| + } |
| t.Run("an unreadable descriptor directory", func(t *testing.T) { |
| root, db := writersProc(t) |
| if _, err := WritersStopped(t.Context(), writersInputs(filepath.Join(root, "missing"), db)); !errors.Is(err, ErrIncomplete) { |
| @@ -3976,11 +4078,112 @@ func TestWritersStoppedRefusesAnyOtherHolder(t *testing.T) { |
| } |
| }) |
| } |
| + |
| +// threadWith runs act on a new OS thread that has unshared its descriptor |
| +// table (CLONE_FILES) when unshare is set, and keeps the thread alive |
| +// until the returned release is called. The thread is never unlocked, so |
| +// it exits with its goroutine and the runtime never reuses its table. |
| +func threadWith(t *testing.T, unshare bool, act func() error) (tid int, release func()) { |
| + t.Helper() |
| + type started struct { |
| + tid int |
| + err error |
| + } |
| + ready, done := make(chan started), make(chan struct{}) |
| + var run func() |
| + run = func() { |
| + runtime.LockOSThread() |
| + if unix.Gettid() == os.Getpid() { |
| + // The leader's own thread: unsharing it would move the |
| + // leader's table, not a thread's. Hold it and start again on |
| + // another thread. |
| + go run() |
| + <-done |
| + runtime.UnlockOSThread() |
| + return |
| + } |
| + if unshare { |
| + if err := unix.Unshare(unix.CLONE_FILES); err != nil { |
| + ready <- started{err: err} |
| + return |
| + } |
| + } |
| + if err := act(); err != nil { |
| + ready <- started{err: err} |
| + return |
| + } |
| + ready <- started{tid: unix.Gettid()} |
| + <-done |
| + } |
| + go run() |
| + s := <-ready |
| + if s.err != nil { |
| + t.Fatal(s.err) |
| + } |
| + return s.tid, func() { close(done) } |
| +} |
| + |
| +// The real kernel, on this test process: a thread |
| +// sharing the table adds nothing; a thread that unshared it holds the |
| +// database at the leader's number and path, either by the copied |
| +// description or by reopening it independently there, and is reported |
| +// with FD -1, never as the leader's. |
| +func TestProcessDescriptorsTellsSharedFromUnsharedThreadsByKernelIdentity(t *testing.T) { |
| + db := filepath.Join(t.TempDir(), "data.db") |
| + f, err := os.OpenFile(db, os.O_RDWR|os.O_CREATE, 0o600) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + defer f.Close() |
| + n, pid := int(f.Fd()), os.Getpid() |
| + own := Descriptor{PID: pid, FD: n, Target: db} |
| + scan := func() []Descriptor { |
| + t.Helper() |
| + // scanOne through the one-process scan: the complete scan root |
| + // makes (scanDescriptors) reads each process the same way. |
| + got, err := scanProcess(t.Context(), "/proc", pid, []string{db}) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return got |
| + } |
| + tid, release := threadWith(t, false, func() error { return nil }) |
| + if tid == pid { |
| + t.Fatal("the helper thread is the leader") |
| + } |
| + if got := scan(); !slices.Equal(got, []Descriptor{own}) { |
| + t.Fatalf("a shared thread: %+v", got) |
| + } |
| + release() |
| + for _, tc := range []struct { |
| + name string |
| + act func() error |
| + }{ |
| + {"the copied description", func() error { return nil }}, |
| + {"reopened independently at the same number", func() error { |
| + fd, err := unix.Open(db, unix.O_RDWR|unix.O_CLOEXEC, 0) |
| + if err != nil { |
| + return err |
| + } |
| + defer unix.Close(fd) |
| + return unix.Dup3(fd, n, unix.O_CLOEXEC) |
| + }}, |
| + } { |
| + t.Run(tc.name, func(t *testing.T) { |
| + _, release := threadWith(t, true, tc.act) |
| + defer release() |
| + got := scan() |
| + if !slices.Contains(got, own) || !slices.Contains(got, Descriptor{PID: pid, FD: -1, Target: db}) { |
| + t.Fatalf("an unshared thread passed as the leader's: %+v", got) |
| + } |
| + }) |
| + } |
| +} |
| ``` |
| - [ ] **Step 2: Run them to see them fail** |
| -Run: `nice -n 19 go test ./internal/stopevidence/ -run 'Writers' -count=1` |
| +Run: `nice -n 19 go test ./internal/stopevidence/ -run 'Writers|ProcessDescriptors' -count=1` |
| Expected: FAIL to compile: `undefined: WritersInputs`. |
| - [ ] **Step 3: The observation** |
| @@ -4171,14 +4374,52 @@ package stopevidence |
| import ( |
| "context" |
| + "errors" |
| "fmt" |
| "os" |
| "path/filepath" |
| "sort" |
| "strconv" |
| "strings" |
| + |
| + "golang.org/x/sys/unix" |
| +) |
| + |
| +// kcmp(2)'s comparison types (linux/kcmp.h). golang.org/x/sys has the |
| +// syscall number (unix.SYS_KCMP) on every linux architecture but no |
| +// wrapper or constants. |
| +const ( |
| + kcmpFile = 0 // the same open file description: idx1, idx2 are fds |
| + kcmpFiles = 2 // the same file descriptor table |
| ) |
| +// kcmpFunc is kcmp(2): 0 when the two tasks' resources are the same, a |
| +// positive ordering (1, 2) or 3 (not comparable) when they are not. |
| +type kcmpFunc func(pid1, pid2, typ int, idx1, idx2 uintptr) (int, error) |
| + |
| +func sysKcmp(pid1, pid2, typ int, idx1, idx2 uintptr) (int, error) { |
| + r, _, errno := unix.Syscall6(unix.SYS_KCMP, uintptr(pid1), uintptr(pid2), uintptr(typ), idx1, idx2, 0) |
| + if errno != 0 { |
| + return 0, errno |
| + } |
| + return int(r), nil |
| +} |
| + |
| +// kcmpFor is the kernel's identity check for the tasks under root. |
| +// kcmp(2) takes pids in the caller's own pid namespace, so it is used |
| +// only for the caller's /proc; any other root (a test fixture) has none, |
| +// which fails closed as ErrIncomplete. Tests replace it. kcmp needs |
| +// ptrace read access to both tasks: always granted within the caller's |
| +// own process (ProcessDescriptors), and to root's complete scan through |
| +// CAP_SYS_PTRACE; an EPERM (a Yama scope that forbids it), or ENOSYS (a |
| +// kernel built without CONFIG_KCMP), is an incomplete observation. |
| +var kcmpFor = func(root string) kcmpFunc { |
| + if filepath.Clean(root) == "/proc" { |
| + return sysKcmp |
| + } |
| + return func(int, int, int, uintptr, uintptr) (int, error) { return 0, unix.ENOSYS } |
| +} |
| + |
| // scanDescriptors lists every descriptor under root that resolves to one |
| // of paths, deduplicated by process and descriptor number. A descriptor |
| // seen only in a task that does not share the leader's table (an unshared |
| @@ -4202,7 +4443,7 @@ func scanDescriptors(ctx context.Context, root string, paths []string) ([]Descri |
| if err != nil || pid <= 0 { |
| continue |
| } |
| - held, err := scanOne(ctx, root, pid, paths, ops) |
| + held, err := scanOne(ctx, root, pid, paths, ops, kcmpFor(root)) |
| if err != nil { |
| return nil, err |
| } |
| @@ -4229,7 +4470,7 @@ func scanProcess(ctx context.Context, root string, pid int, paths []string) ([]D |
| // or unreadable /proc/<pid> is an incomplete observation. |
| return nil, fmt.Errorf("%w: pid %d: %w", ErrIncomplete, pid, err) |
| } |
| - out, err := scanOne(ctx, root, pid, paths, ops) |
| + out, err := scanOne(ctx, root, pid, paths, ops, kcmpFor(root)) |
| if err != nil { |
| return nil, err |
| } |
| @@ -4238,11 +4479,18 @@ func scanProcess(ctx context.Context, root string, pid int, paths []string) ([]D |
| } |
| // scanOne lists pid's descriptors on paths, deduplicated by descriptor |
| -// number. A descriptor seen only in a task that does not share the |
| -// leader's table (an unshared thread) is reported with FD -1, so no |
| -// allow-list can exempt it. A process that vanished mid-scan is skipped; |
| -// an unreadable table is ErrIncomplete. |
| -func scanOne(ctx context.Context, root string, pid int, paths []string, ops fdOps) ([]Descriptor, error) { |
| +// number. A thread's descriptor counts as the leader's only on positive |
| +// kernel evidence (round 6 finding 1): the leader shows the same number |
| +// for the same path, kcmp(2) KCMP_FILES says the thread shares the |
| +// leader's descriptor table, and KCMP_FILE says that descriptor is the |
| +// leader's open file description. A number and a path alone prove |
| +// nothing: a thread that unshared its table (CLONE_FILES) can reopen the |
| +// database at the same number. Any other thread's descriptor is reported |
| +// with FD -1, so no allow-list can exempt it. A comparison that cannot be |
| +// made (ENOSYS, EPERM, anything but a vanished task) is ErrIncomplete. A |
| +// process or thread that vanished mid-scan is skipped; an unreadable |
| +// table is ErrIncomplete. |
| +func scanOne(ctx context.Context, root string, pid int, paths []string, ops fdOps, cmp kcmpFunc) ([]Descriptor, error) { |
| wanted := make(map[string]bool, len(paths)) |
| for _, p := range paths { |
| wanted[filepath.Clean(p)] = true |
| @@ -4273,6 +4521,8 @@ func scanOne(ctx context.Context, root string, pid int, paths []string, ops fdOp |
| if err != nil { |
| return nil, fmt.Errorf("%w: pid %d tid %d: %w", ErrIncomplete, pid, tid, err) |
| } |
| + shared := -1 // this thread's KCMP_FILES verdict, asked once, only if needed |
| + descriptors: |
| for _, fd := range fds { |
| n, err := strconv.Atoi(fd.Name()) |
| if err != nil { |
| @@ -4292,8 +4542,17 @@ func scanOne(ctx context.Context, root string, pid int, paths []string, ops fdOp |
| d := Descriptor{PID: pid, FD: n, Target: clean} |
| if tid == pid { |
| leader[n] = clean |
| - } else if leader[n] != clean { |
| - d.FD = -1 // not provably the leader's shared table |
| + } else { |
| + ok, gone, err := leadersDescriptor(cmp, pid, tid, n, leader[n] == clean, &shared) |
| + if err != nil { |
| + return nil, fmt.Errorf("%w: pid %d tid %d fd %d: kcmp: %w", ErrIncomplete, pid, tid, n, err) |
| + } |
| + if gone { |
| + break descriptors // the thread exited mid-scan |
| + } |
| + if !ok { |
| + d.FD = -1 // not provably the leader's descriptor |
| + } |
| } |
| if !seen[d] { |
| seen[d] = true |
| @@ -4304,6 +4563,41 @@ func scanOne(ctx context.Context, root string, pid int, paths []string, ops fdOp |
| return out, nil |
| } |
| +// leadersDescriptor is the kernel's evidence that thread tid's descriptor |
| +// n is the leader's own: the same table (KCMP_FILES, asked once per thread |
| +// and kept in *shared) and the same open file description (KCMP_FILE). |
| +// sameName is the leader showing n on the same path. gone reports a task |
| +// that exited (ESRCH); EBADF (n closed in one of them since its listing) |
| +// is "not the leader's"; any other error is returned. |
| +func leadersDescriptor(cmp kcmpFunc, pid, tid, n int, sameName bool, shared *int) (ok, gone bool, err error) { |
| + if !sameName { |
| + return false, false, nil |
| + } |
| + if *shared < 0 { |
| + r, err := cmp(pid, tid, kcmpFiles, 0, 0) |
| + if errors.Is(err, unix.ESRCH) { |
| + return false, true, nil |
| + } |
| + if err != nil { |
| + return false, false, err |
| + } |
| + *shared = r |
| + } |
| + if *shared != 0 { |
| + return false, false, nil |
| + } |
| + r, err := cmp(pid, tid, kcmpFile, uintptr(n), uintptr(n)) |
| + switch { |
| + case errors.Is(err, unix.ESRCH): |
| + return false, true, nil |
| + case errors.Is(err, unix.EBADF): |
| + return false, false, nil |
| + case err != nil: |
| + return false, false, err |
| + } |
| + return r == 0, false, nil |
| +} |
| + |
| // processStart is field 22 of /proc/<pid>/stat: the process start time in |
| // clock ticks since boot, which with the boot ID names one process. |
| func processStart(root string, pid int) (string, error) { |
| @@ -10596,3 +10890,4 @@ Expected: exit 0. Exit 2 is pending: wait and read again. Exit 1: read the log a |
| - **Rev 4 guards**, each removed in turn and its test watched to fail: the acknowledgement barrier on `releasing → pinning`; `FenceExhaustedTarget`'s exhaustion check; the worker's use of it over `MarkStuck`; the serving branch's obsolete-hold clearance; `RelocationPolicy`'s executor check; `AdmissionFenced`'s active fence; `Finish` keeping its entry until `Close` succeeds; the recorded directory revision after a restart. Eight in all. |
| - **Rev 5 guards**, each removed in turn and its test watched to fail: the lock-free fence observer (re-locking deadlocks the bounded test); the `probe` action in `Policy.Check`; the HA gate on the obsolete-hold clearance; `Finish`'s durable clear; the journal's retry after a restart; the hold-only path for a fenced executor; the observer error keeping rows. Seven in all. |
| - **Rev 6 guards**, each removed in turn and its test watched to fail: `Finish`'s repeated scoped cleanup before every clear (the real hostexec test's sweep after a failed cleanup clears); the participation journaled before the open; the journal listing's `.tmp` exclusion; a damaged committed entry keeping the readable ones; `Sweep` reporting `Retained`'s error; the env file's staged deletion; `replaceTargetSlot`'s `park-local` refusal; the worker's `park-local` exclusion; the chooser's `park-local` exclusion; `fenceBox`'s regional reassignment; Task 9a's serving-only check with HA off. Eleven in all. |
| +- **Rev 7 guards**, each removed in turn and its test watched to fail: the `KCMP_FILES` shared-table check (the real kernel's unshared thread with the copied descriptor passes as the leader's); the `KCMP_FILE` description check (the fixture's shared table with another description passes); an unavailable kcmp failing closed (ENOSYS and EPERM pass instead of `ErrIncomplete`). Three in all. |