carlos / platform Public

Nested instances: the trampoline stays as PID 1 and runs the app as its child (#126)

What was wrong. Nested instances run with PrivatePIDs=yes, so the trampoline's exec put the app at PID 1 of its own PID namespace. The kernel gives a namespace's PID 1 no default signal actions. A Go app with no SIGTERM handler can't be stopped by SIGTERM there: the runtime tries to die by the signal, the kernel drops it, and the app exits 2. An app with no handler at all waits out TimeoutStopSec=20 for the SIGKILL. Either way, every hibernate, restart, layout move and deploy left the unit failed. This was found on the dev fleet. Flat tenants were never PID 1.

What changes.

• instance-exec now forks the app instead of exec'ing it, and stays behind as a minimal init (internal/instanceexec/init_linux.go): - It forwards SIGTERM, SIGINT, SIGHUP, SIGQUIT, SIGUSR1 and SIGUSR2 to the app. - It reaps every orphan. - It exits with the app's status, or 128+n if a signal n killed the app. - The signal path allocates nothing. Escape analysis confirms this.
• The Landlock abstract-socket scope is applied on the locked thread that does the fork. A fork copies that thread's Landlock domain and exec keeps it, so the app is still scoped. A test confirms this.
• The app gets the same argv, env, cwd, stdout/stderr and inherited fds as before. Setsid gives it its own session, as systemd gave the main process.
• KillMode=mixed is unchanged. SIGTERM goes to the main process (the init), which forwards it. After the timeout, SIGKILL goes to the whole cgroup, and a SIGKILL from outside the namespace always kills its PID 1, which takes the rest of the namespace with it.
• The run and sidecar templates gain SuccessExitStatus=129 130 141 143. These are the signals (HUP, INT, PIPE, TERM) that systemd counts as a clean death of a main process, so a stop still leaves the unit inactive. The seed template deliberately doesn't get this line, because a seed that a signal ended has not finished seeding. A test pins that.
• The probe is unchanged. It is platform code, it has to be PID 1 itself (its pid1 check), and it runs no tenant code.
• docs/site/cli.md (carlos instance-exec) now describes what the init does. NoNewPrivileges and every other restriction are unchanged.

How it was tested. internal/instanceexec/init_test.go runs the real init in a re-executed test binary. It covers:

• forwarding of all six signals;
• an app with no handler dying by SIGTERM, SIGINT or SIGHUP through the init (exit 143, 130 or 129);
• exit-code passthrough and the 128+n mapping, end to end and as a table;
• orphan reaping: the grandchild must be adopted by the init and never left a zombie;
• env, cwd, fd 3 and session passthrough;
• Landlock still enforced in the app: the dial answers EPERM with the scope and connects without it.

One more test runs in an unprivileged user and PID namespace, and is skipped where there isn't one. It reproduces the finding (as PID 1 the app exits 2 on SIGTERM) and shows the fix (through the init it dies by SIGTERM in about 1 ms). All of these tests failed before the fix.

Gate results: gofmt -l . is clean, and go build ./... (also for GOOS=darwin) and go vet ./... pass. make test: 69 packages pass. The 11 failing tests are all known environment failures on this machine: 10 hit the socket-path length limit (bind: invalid argument in activator, edge and instanceexec), and the root package reads ~/.carlos/config.

What this doesn't do.

• A crash by signal now shows in the member event line as instance exited: exit 139 (exit-code) rather than signal SEGV. A signal-ended seed now reports exit 143 (exit-code). A stop still reads instance exited (exit 0), as before. Rewording exitLine to recognise 128+n would be a guess, because an app can exit 139 itself, so it is left alone.
• The init adds a few Go runtime threads (against TasksMax=256) and a few MB of memory to each unit.
• Nothing has been rolled out. It still needs verifying on the dev fleet: a stop of a nested instance whose app has no SIGTERM handler should end inactive, not failed, well inside 20 s.