carlos / platform Public

isolated exec: the probe goes ready (#126)

The first roll's probe answered ready: false on every box, for two
reasons neither of which touched the flat path.

RestrictSUIDSGID=yes makes openat2 fail with ENOSYS on systemd 257:
systemd cannot read the mode inside open_how, so it blocks the whole
call, and every instancefs open inside the namespace is an openat2. The
line is dropped from the four templates. NoNewPrivileges=yes already
voids a set-id bit on exec in there, and every tree an instance can
write (the three bind mounts, PrivateTmp, /dev/shm) is nosuid.

control-unreachable checked Lstat, but InaccessiblePaths= leaves a mode
000 root-owned node at the path. The check now does what a tenant
could: list the directory and dial the socket. It passes only when both
fail.

Verified on dev (i-046c7fe43620944bc, c1d40ee) with this build and the
new probe unit under a scratch name: ready: true, imds-v6 unverified as
before on a v4-only box. Spec revision 15, §13 R15.

Human

Prompt

Can we fix platform#126 ?

From this commit's own message. The prompt ledger has no entries for it — which is what a squashed, rebased or amended commit looks like, and what every commit looks like in a repo without a ledger.

Summary

Drops RestrictSUIDSGID from the instance templates so openat2 works, and makes the probe's control check dial rather than Lstat.

Agent notes

Scratch probe run on dev over SSM; box cleaned up. Plans 4 and 5 remain unbuilt.

claude 1 month ago 6b54db786252e3f47ac1248d2927448130244ff7 parent 4d5eab8

8 files changed, +117 −14

docs/superpowers/specs/2026-09-11-isolated-exec-design.md +5 −3
internal/host/units/carlos-instance-probe.service +5 −1
internal/host/units/carlos-instance-seed@.service +5 −1
internal/host/units/carlos-instance-sidecar@.service +5 −1
internal/host/units/carlos-instance@.service +5 −1
internal/host/units_isolation_test.go +7 −0
internal/instanceexec/probe.go +23 −6
internal/instanceexec/probe_test.go +62 −1

diff --git a/docs/superpowers/specs/2026-09-11-isolated-exec-design.md b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md

index 03c033f38..1a3eb7253 100644

--- a/docs/superpowers/specs/2026-09-11-isolated-exec-design.md

+++ b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md

@@ -1,11 +1,12 @@

# Isolated exec — every exec-backed instance in its own namespace

-**Date:** 2026-09-11 · **Status:** revision 14; codex verdict after

+**Date:** 2026-09-11 · **Status:** revision 15; codex verdict after

round 11 is **accept with corrections**, and the one correction is

applied here (54 findings over 11 rounds, all accepted; §13); a

whole-branch final review (plan 2) found six further items, fixed in

revision 13 (§13); plan 3's task 4 review settled the requested-stop

-close, recorded in revision 14 (§6.5, §13 R13); awaiting Paul's review · **Requested by:** discussion

+close, recorded in revision 14 (§6.5, §13 R13); the first roll's probe

+found two defects, fixed in revision 15 (§4.1, §8.5, §13 R15); awaiting Paul's review · **Requested by:** discussion

#126 (Tito, with box evidence), approved in direction by Paul on

2026-09-11

@@ -650,7 +651,7 @@ ProtectHostname=yes

RemoveIPC=yes

LockPersonality=yes

RestrictRealtime=yes

-RestrictSUIDSGID=yes

+# No RestrictSUIDSGID=: it refuses openat2 outright (R15).

MemoryDenyWriteExecute=yes

SystemCallArchitectures=native

@@ -2131,3 +2132,4 @@ rollback runs the old binary).

R11-1 the probe fixture comes from tmpfiles at bootstrap, and the control service verifies its ownership and mode before accepting a verdict (§3.2, §8.5).

- R12 (final review, plan 2): cgroup-ro targets the edge's delegated cgroup.procs; DAC precedes the mount flag so a root-owned file answers EACCES.

- R13 (task 4 review, plan 3): after a stop the activator requested, a unit reporting a foreign or absent invocation proves the old invocation over without the cgroup gate (§6.5); the observer never stops an invocation it did not start once its own stop was requested.

+- R15 (first roll, 2026-09-12; fixed 2026-09-25): the probe on dev's boxes answered `ready: false` twice. (a) `RestrictSUIDSGID=yes` makes `openat2` fail with ENOSYS on systemd 257, because systemd cannot read the mode inside `open_how` and so blocks the whole call; every `instancefs` open in the namespace is an `openat2`. The line is dropped from all four templates: `NoNewPrivileges=yes` already voids a set-id bit on exec inside the namespace, and every tree an instance can write is `nosuid` (the three bind mounts, `PrivateTmp=`, `/dev/shm`). (b) `control-unreachable` checked `Lstat`, but `InaccessiblePaths=` leaves a mode 000 root-owned node at the path; the check is now what a tenant could do, a listing of the directory and a dial of the socket, and passes only when both fail (§4.1, §8.5).

diff --git a/internal/host/units/carlos-instance-probe.service b/internal/host/units/carlos-instance-probe.service

index f8878fe1e..a6392a0c4 100644

--- a/internal/host/units/carlos-instance-probe.service

+++ b/internal/host/units/carlos-instance-probe.service

@@ -66,7 +66,11 @@ ProtectHostname=yes

RemoveIPC=yes

LockPersonality=yes

RestrictRealtime=yes

-RestrictSUIDSGID=yes

+# No RestrictSUIDSGID=: systemd cannot read the mode inside openat2's

+# open_how, so it refuses openat2 outright (ENOSYS), and every instancefs

+# open in here is an openat2 (#126, first roll; systemd 257). Nothing it

+# guarded is left: NoNewPrivileges= voids a set-id bit on exec in here,

+# and every tree an instance can write is mounted nosuid.

MemoryDenyWriteExecute=yes

SystemCallArchitectures=native

diff --git a/internal/host/units/carlos-instance-seed@.service b/internal/host/units/carlos-instance-seed@.service

index 711b51b24..386b8b19f 100644

--- a/internal/host/units/carlos-instance-seed@.service

+++ b/internal/host/units/carlos-instance-seed@.service

@@ -93,7 +93,11 @@ ProtectHostname=yes

RemoveIPC=yes

LockPersonality=yes

RestrictRealtime=yes

-RestrictSUIDSGID=yes

+# No RestrictSUIDSGID=: systemd cannot read the mode inside openat2's

+# open_how, so it refuses openat2 outright (ENOSYS), and every instancefs

+# open in here is an openat2 (#126, first roll; systemd 257). Nothing it

+# guarded is left: NoNewPrivileges= voids a set-id bit on exec in here,

+# and every tree an instance can write is mounted nosuid.

MemoryDenyWriteExecute=yes

SystemCallArchitectures=native

diff --git a/internal/host/units/carlos-instance-sidecar@.service b/internal/host/units/carlos-instance-sidecar@.service

index 8b65c85f0..00a32615f 100644

--- a/internal/host/units/carlos-instance-sidecar@.service

+++ b/internal/host/units/carlos-instance-sidecar@.service

@@ -89,7 +89,11 @@ ProtectHostname=yes

RemoveIPC=yes

LockPersonality=yes

RestrictRealtime=yes

-RestrictSUIDSGID=yes

+# No RestrictSUIDSGID=: systemd cannot read the mode inside openat2's

+# open_how, so it refuses openat2 outright (ENOSYS), and every instancefs

+# open in here is an openat2 (#126, first roll; systemd 257). Nothing it

+# guarded is left: NoNewPrivileges= voids a set-id bit on exec in here,

+# and every tree an instance can write is mounted nosuid.

MemoryDenyWriteExecute=yes

SystemCallArchitectures=native

diff --git a/internal/host/units/carlos-instance@.service b/internal/host/units/carlos-instance@.service

index 01b3063a1..8911a22fe 100644

--- a/internal/host/units/carlos-instance@.service

+++ b/internal/host/units/carlos-instance@.service

@@ -89,7 +89,11 @@ ProtectHostname=yes

RemoveIPC=yes

LockPersonality=yes

RestrictRealtime=yes

-RestrictSUIDSGID=yes

+# No RestrictSUIDSGID=: systemd cannot read the mode inside openat2's

+# open_how, so it refuses openat2 outright (ENOSYS), and every instancefs

+# open in here is an openat2 (#126, first roll; systemd 257). Nothing it

+# guarded is left: NoNewPrivileges= voids a set-id bit on exec in here,

+# and every tree an instance can write is mounted nosuid.

MemoryDenyWriteExecute=yes

SystemCallArchitectures=native

diff --git a/internal/host/units_isolation_test.go b/internal/host/units_isolation_test.go

index 25dd4d829..3fbec1642 100644

--- a/internal/host/units_isolation_test.go

+++ b/internal/host/units_isolation_test.go

@@ -228,6 +228,13 @@ func TestInstanceTemplatesShareOneNamespace(t *testing.T) {

t.Errorf("namespace block lacks %q", want)

}

+ // RestrictSUIDSGID= makes openat2 answer ENOSYS, and every instancefs

+ // open inside the namespace is an openat2 (#126, first roll).

+ for _, line := range strings.Split(ref, "\n") {

+ if strings.HasPrefix(line, "RestrictSUIDSGID=") {

+ t.Errorf("namespace block carries %q, which breaks openat2", line)

+ }

for name, wants := range map[string][]string{

"carlos-instance@.service": {"ExecStart=/opt/carlos/carlos instance-exec run %I", "ExecStopPost=/opt/carlos/carlos instance-exit run %I",

"BindPaths=/data/instances/%I /run/carlos/instances/%I", "StandardOutput=file:/run/carlos/logs/%I.run.sock",

diff --git a/internal/instanceexec/probe.go b/internal/instanceexec/probe.go

index 21d41e3e4..08fe77626 100644

--- a/internal/instanceexec/probe.go

+++ b/internal/instanceexec/probe.go

@@ -7,6 +7,7 @@ import (

"errors"

"flag"

"fmt"

+ "net"

"os"

"path/filepath"

"sort"

@@ -15,6 +16,7 @@ import (

"time"

"github.com/carlosframework/platform/internal/cliflag"

+ "github.com/carlosframework/platform/internal/controlsock"

"github.com/carlosframework/platform/internal/instancectl"

"github.com/carlosframework/platform/internal/instancefs"

)

@@ -29,7 +31,8 @@ type probeEnv struct {

varLib string // /var/lib

etcCarlos string // /etc/carlos

procRoot string // /proc

- controlDir string // /run/carlos-control, which must be invisible from in here

+ controlDir string // /run/carlos-control, which must be unusable from in here

+ controlSock string // the control socket inside it, which must not dial

cgroupProcs string // the edge's delegated cgroup.procs, to test writability on

pid func() int

writable func(path string) error

@@ -76,7 +79,7 @@ func RunProbe(args []string) {

func productionProbeEnv() probeEnv {

return probeEnv{

dataRoot: "/data", runRoot: "/run/carlos", varCache: "/var/cache", varLib: "/var/lib",

- etcCarlos: "/etc/carlos", procRoot: "/proc", controlDir: "/run/carlos-control",

+ etcCarlos: "/etc/carlos", procRoot: "/proc", controlDir: controlsock.Dir, controlSock: controlsock.Path,

// The edge's own delegated cgroup.procs, not the probe's: Delegate=

// on carlos-edge.service chowns this file to carlos, so DAC passes

// and only the mount's ro flag can refuse the write, making EROFS

@@ -175,12 +178,16 @@ func runChecks(e probeEnv) instancectl.ProbeReport {

mro, mroErr := e.mountOption(cgroupMountpoint, "ro")

pass("cgroup-ro-mount", fmt.Sprintf("MountOptionOf(%s, ro) = %v, %v", cgroupMountpoint, mro, mroErr), mro && mroErr == nil)

- // The control socket directory is invisible from in here: InaccessiblePaths=

+ // The control socket is unreachable from in here: InaccessiblePaths=

// in the shared namespace block (design §5.2) makes a connect from

// inside any instance impossible outright, not merely refused by the

- // peer check on the other end.

- _, cerr := os.Lstat(e.controlDir)

- pass("control-unreachable", "control socket directory visible", cerr != nil)

+ // peer check on the other end. The path itself still answers Lstat,

+ // because systemd hides it under a mode 000 root-owned node rather

+ // than removing it, so the check is what a tenant could actually do:

+ // list the directory, or dial the socket (#126, first roll).

+ _, cerr := os.ReadDir(e.controlDir)

+ kerr := dialUnix(e.controlSock)

+ pass("control-unreachable", fmt.Sprintf("readdir=%v dial=%v", cerr, kerr), cerr != nil && kerr != nil)

// /etc/carlos cannot be opened and its fixture cannot be read.

_, derr := os.ReadDir(e.etcCarlos)

@@ -313,3 +320,13 @@ func realSymlinkCheck(dir string) error {

}

return err

}

+// dialUnix connects to a unix socket and closes the connection at once,

+// returning nil only if the connect succeeded.

+func dialUnix(path string) error {

+ c, err := net.DialTimeout("unix", path, 2*time.Second)

+ if err != nil {

+ return err

+ }

+ return c.Close()

diff --git a/internal/instanceexec/probe_test.go b/internal/instanceexec/probe_test.go

index e03e54f12..a78810274 100644

--- a/internal/instanceexec/probe_test.go

+++ b/internal/instanceexec/probe_test.go

@@ -4,6 +4,7 @@ package instanceexec

import (

"errors"

+ "net"

"os"

"path/filepath"

"strings"

@@ -39,7 +40,8 @@ func goodEnv(t *testing.T) probeEnv {

mk("proc/self")

return probeEnv{

dataRoot: data, runRoot: run, varCache: varCache, varLib: varLib, etcCarlos: etc, procRoot: proc,

- controlDir: filepath.Join(root, "run", "carlos-control"), // deliberately absent: Lstat fails

+ controlDir: filepath.Join(root, "run", "carlos-control"), // deliberately absent: ReadDir and dial fail

+ controlSock: filepath.Join(root, "run", "carlos-control", "control.sock"),

cgroupProcs: filepath.Join(root, "sys", "cgroup.procs"),

pid: func() int { return 1 },

writable: func(string) error { return syscall.EROFS },

@@ -142,6 +144,65 @@ func TestProbeFailsWhenTheControlSocketDirectoryIsVisible(t *testing.T) {

}

+// TestProbePassesOnAnInaccessibleControlNode pins the shape

+// InaccessiblePaths= really leaves: a mode 000 directory that still

+// answers Lstat. The first probe on a box failed here because it checked

+// Lstat; what a tenant cannot do is list it or dial through it (#126).

+func TestProbePassesOnAnInaccessibleControlNode(t *testing.T) {

+ if os.Geteuid() == 0 {

+ t.Skip("root reads a mode 000 directory")

+ }

+ e := goodEnv(t)

+ if err := os.MkdirAll(e.controlDir, 0o700); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.Chmod(e.controlDir, 0); err != nil {

+ t.Fatal(err)

+ }

+ t.Cleanup(func() { os.Chmod(e.controlDir, 0o700) })

+ if _, err := os.Lstat(e.controlDir); err != nil {

+ t.Fatalf("fixture: Lstat = %v, want the node to exist", err)

+ }

+ r := runChecks(e)

+ if v := verdict(r, "control-unreachable"); v != "ok" {

+ t.Errorf("control-unreachable = %s on a mode 000 node: %+v", v, r.Lines)

+ }

+// TestProbeFailsWhenTheControlSocketDials: a directory that refuses a

+// listing but still lets a connect through is not unreachable.

+func TestProbeFailsWhenTheControlSocketDials(t *testing.T) {

+ if os.Geteuid() == 0 {

+ t.Skip("root reads a mode 0300 directory")

+ }

+ e := goodEnv(t)

+ if err := os.MkdirAll(e.controlDir, 0o700); err != nil {

+ t.Fatal(err)

+ }

+ l, err := net.Listen("unix", e.controlSock)

+ if err != nil {

+ t.Fatalf("listen at %s (a socket test wants a short TMPDIR): %v", e.controlSock, err)

+ }

+ defer l.Close()

+ go func() {

+ for {

+ c, err := l.Accept()

+ if err != nil {

+ return

+ }

+ c.Close()

+ }

+ }()

+ if err := os.Chmod(e.controlDir, 0o300); err != nil {

+ t.Fatal(err)

+ }

+ t.Cleanup(func() { os.Chmod(e.controlDir, 0o700) })

+ r := runChecks(e)

+ if r.Ready || verdict(r, "control-unreachable") != "fail" {

+ t.Errorf("dialable control socket yet ready=%v control-unreachable=%s", r.Ready, verdict(r, "control-unreachable"))

+ }

func TestProbeFailsWhenEtcCarlosIsReadable(t *testing.T) {

e := goodEnv(t)

if err := os.MkdirAll(e.etcCarlos, 0o755); err != nil {