218b74547 · carlos/platform

diff --git a/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-2-R2.md b/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-2-R2.md

new file mode 100644

index 000000000..feabd8954

--- /dev/null

+++ b/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-2-R2.md

@@ -0,0 +1,58 @@

+# Astra (codex, GPT, reasoning high) scoped re-review of the zdd plan 2 final fix wave, round 2

+

+Fix range `cab0dbd4..5263503b`. Brief: ~/.cache/zdd-impl/plan2/brief-r2.md. Run read-only, 2026-10-08.

+

+**ACCEPT WITH FIXES**

+

+Reviewed `cab0dbd4..5263503b` against spec rev 17. Read-only: no builds or tests run. `git diff --check` passed; test results below refer to the supplied report.

+

+### Finding verdicts

+

+| Finding | Verdict | Evidence |

+|---|---|---|

+| D1 | **ADDRESSED** | [cleanup_linux.go:251](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/cgroup/cleanup_linux.go:251) reads 64 entries at a time and enforces the shared group budget during enumeration. Tests cover 1,025 children refusing and 1,024 succeeding. |

+| D2 | **ADDRESSED** | [gen_kernel_linux_test.go:215](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/cgroup/gen_kernel_linux_test.go:215) registers cleanup with direct-process kill, pipe closure and bounded reaping. Spawns receive cleanup immediately; confinement errors return through it. Independent kill budgets replace the shared context. |

+| L1 | **NOT ADDRESSED** | Server cleanup and backend resource retention are implemented, but the failed-kill path still returns a nil process at [backend_exec.go:1534](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/backend_exec.go:1534). The existing wake caller releases the lease for that result. Details below. |

+| L2 | **ADDRESSED** | [containment.go:166](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/containment.go:166) refuses a missing manager with `cgroup.ErrNoDelegation`. The manager’s legacy contract is unchanged. |

+| L3 | **ADDRESSED** | [unitproc.go:156](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/unitproc.go:156) finalises proven natural and foreign-invocation exits; [backend_exec.go:1536](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/backend_exec.go:1536) removes launch files after proven failed-start cleanup; [gen.go:279](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/instancectl/gen.go:279) recovers inactive, empty generation leftovers. |

+| T8 | **ADDRESSED** | [exec_test.go:185](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/instanceexec/exec_test.go:185) requires parser refusal and zero executions, includes traversal and extra-colon cases, and verifies a valid launch succeeds. Missing files or host mismatches can no longer satisfy the malformed-case assertions. |

+| T12 | **ADDRESSED** | [claimspawn.go:25](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/claimspawn.go:25) and the corresponding test comment accurately describe the closed helper and the missing isolated root protocol. |

+

+**L1 remains High, latent behind `GenContainment`; fix before merge.**

+

+Failing sequence: the generation starts, invocation lookup fails, and neither cleanup attempt proves termination. The backend keeps listeners and files but returns `nil, errGenUnproven`. At [activator.go:1326](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/activator.go:1326), that nil process selects `lease.Release(ctx)` without inspecting the sentinel. With a still-valid caller context, the lease can be released while the generation continues writing.

+

+The server’s 500-to-`instancectl.ErrGenUnproven` mapping works. The backend’s separate sentinel also works. Neither establishes ownership retention in the current caller. This is the original failing sequence, so implementing the controller’s sentinel suggestion alone does not close L1.

+

+**Smallest complete fix:** retain the failed generation in a cleanup handle and make failed-wake cleanup retain ownership and the lease until termination is proven. Returning a handle alone is insufficient: [maintenance.go:252](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/maintenance.go:252) currently releases the lease even when `Proc.Stop` fails. Add a service-level regression asserting no lease release after an unproven kill, followed by eventual cleanup and release after proof.

+

+### New breakage in the fix diff (severity, file:line, failing sequence, smallest fix, fix-before-merge or defer)

+

+No separate new regression established.

+

+- **Flag off:** replacing `removeLaunch()` with `finaliseGen()` preserves legacy behavior: both skip generation zero. The additional drop-in scan is a compatibility change, not byte-identical dormancy.

+- **Sweep/start race:** the new leftover pass holds `genMu` across state checks and removal; `startGen` marks the unit before creating files. This protects against a concurrent fresh start through this server. Duplicate starts of the same generation remain outside the established fresh-generation-only contract.

+- **Unproven termination:** the new failed-start paths retain files and listeners when kill proof fails. Ownership release remains the unresolved L1 finding above.

+

+### Residual concerns

+

+1. **Unproven start leaves listeners without a handle — agree; disagree with deferring the complete problem to plan 3.** The consequence includes lease release, not just leaked descriptors. L1 remains a merge blocker.

+

+2. **Server cleanup can take 60 seconds while the client waits 30 — agree.** Every start error, including transport timeout, triggers the backend’s independent kill attempt. Thus timeout does not bypass cleanup. A failed second attempt still reaches unresolved L1; the timeout handling cannot be described as safe end to end yet.

+

+3. **Flag-off sweep only lists the directory and finds nothing — qualified agreement.** That holds on a box that has never created generation drop-ins. After rollback, leftovers may exist and be removed after inactive/empty checks. Document that compatibility behavior.

+

+4. **Drop-in-only recovery and no daemon-reload — agree with deferral.** Executable-only leftovers are missed. They can also arise when drop-in removal succeeds but executable removal fails, since removal errors are ignored. Skipping reload is acceptable for cleanup of retired names; subsequent generation starts reload before execution.

+

+5. **Repeated stops from observer/Stop races — agree.** Under the fresh-generation-only contract, repeated stops target the same retired generation and are harmless.

+

+### Out-of-scope

+

+Unchanged lifecycle code was inspected only to trace L1’s existing caller; this was not a broader lifecycle re-review. Other round-1 deferrals, plan-3 integration, activation prerequisites, cache-policy compliance and deployment validation remain outside this fix review. The supplied package, kernel and build results were not independently rerun.

+---

+

+## Controller adjudication (no second fix wave, per process)

+

+- **L1 parked to plan 3 Task 2, and flagged to Paul as Astra's merge blocker.** Real and High, but latent: `ExecBackend.GenContainment` is false everywhere outside tests until plan 5 sets it. Plan 3 Task 2 already owns the mechanism: `cleanupFailedWake` keeps the lease of a failed wake whose generation is unproven (`unprovenWakes`, `errWakeUnproven`, `releasable`). Plan 3 must (a) return or keep a handle for the unproven generation from `startUnitAt`/`StartGen` instead of a nil `Proc`; (b) have `cleanupFailedWake` treat `errGenUnproven` / `instancectl.ErrGenUnproven` as unproven; (c) stop `maintenance.go`'s release after a failed `Stop` for generations. It needs a service-level test: no release after an unproven kill, and a release after a later proof. Paul decides whether plan 2 merges before plan 3 lands this.

+- **Exe-layout-only leftovers:** deferred, as Astra agrees.

+- **Dormancy wording:** the flag-off compatibility exceptions Astra lists in rounds 1 and 2 are accepted. They belong in the index's dormancy text.

diff --git a/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-2.md b/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-2.md

new file mode 100644

index 000000000..27cd4f910

--- /dev/null

+++ b/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-2.md

@@ -0,0 +1,105 @@

+# Astra (codex, GPT, reasoning high) final review of zdd plan 2 (containment), round 1

+

+Range `98d09867..1f75ae65` (origin/zdd-1-generations..zdd-2-containment). Brief: ~/.cache/zdd-impl/plan2/brief-r1.md. Run read-only, 2026-10-08.

+

+**ACCEPT WITH FIXES**

+

+Reviewed `98d09867..1f75ae65` against spec rev 17, the plan, index and execution ledger. Read-only review; no builds or tests run. `git diff --check` passed.

+

+**Findings affecting the dormant branch (flag off)**

+

+1. **Medium — subtree enumeration is not actually bounded.** [cleanup_linux.go:212](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/cgroup/cleanup_linux.go:212). A flat tenant can leave many empty sub-cgroups. After termination, `Members()` reads every directory entry into memory before checking `maxCleanupGroups`. The 1,024-group limit therefore does not bound allocation or enumeration work, and this path runs without `GenContainment`. **Smallest fix:** read directory entries in bounded batches and enforce the group budget during enumeration; add a width-bound regression test. **Fix before merge.**

+

+2. **Medium — kernel-test cleanup can hang on the failure it should contain.** [gen_kernel_linux_test.go:209](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/cgroup/gen_kernel_linux_test.go:209). If `OpenCleanupChild` fails—or `KillAndEmpty` fails while the incumbent remains alive—the cleanup calls `incumbent.Wait()` without killing it or closing its input. The echo child remains blocked, so the test hangs. The strict kernel test also lacks failure-path cleanup after starting children. These opt-in tests do not depend on the backend flag. **Smallest fix:** register cleanup immediately after each spawn, provide direct-process termination when cgroup cleanup fails, close pipes, and bound reaping. **Fix before merge.**

+

+Dormancy is otherwise preserved for ordinary spawn selection, names, launch files and ordering. However, **literal byte-identical behaviour is not true across the whole branch**: flag-off `/sweep` adds the generation glob; startup sweeping removes recognised child groups; `Members()` descends and sorts; the sampler includes populated subtrees; installation adds the template and widens the control unit’s writable paths. These are requested compatibility changes, including accepted D1/D5 decisions. Retain them, but qualify the blanket dormancy claim.

+

+**Findings latent until the flag is on**

+

+1. **High — a failed generation-start response can discard a live process.** [gen.go:157](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/instancectl/gen.go:157), [backend_exec.go:1527](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/backend_exec.go:1527). `systemctl start` succeeds; the subsequent invocation lookup fails or its context expires; the server returns 502 without stopping the generation. The activator closes its listeners and returns a nil process. The generation can continue writing its database while wake failure releases ownership. The earlier start-error branch also removes files after a stop whose failure is only logged.

+

+ **Smallest fix:** handle uncertain generation starts in this branch: stop and prove the named generation empty using an independent bounded context, or retain and return an unresolved cleanup handle. Do not discard ownership or files while termination is uncertain. Add successful-start/failed-show and failed-cleanup tests. The ledger’s “future caller contract” is insufficient because the caller already exists here. **Fix before merge.**

+

+2. **High — `ParentLimit` reports success when containment is unavailable.** [containment.go:158](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/containment.go:158). When cgroup setup returns nil, flag-on `ParentLimit` returns nil through the manager’s nil-safe method. A caller can therefore proceed past its overlap admission check, and `StartChild` starts the successor without a generation cgroup. `KillGen` subsequently returns `ErrNoDelegation`. No parent was raised and the promised per-generation termination mechanism does not exist.

+

+ **Smallest fix:** make the backend’s flat/foreign `ParentLimit` refuse a nil manager with `ErrNoDelegation`; preserve the legacy manager contract if other callers require it. Test that refusal before admitting a successor. **Fix before merge.**

+

+3. **Medium — generation files accumulate on ordinary exits and failed starts.** [unitproc.go:102](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/unitproc.go:102), [backend_exec.go:1511](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/zdd-2-containment/internal/activator/backend_exec.go:1511). A generation exits naturally; observation marks it exited; later `Stop()` skips `StopGen`, leaving its runtime drop-in and executable directory. Clean inactive units are also skipped by `/sweep`. Separately, a refused start after `writeLaunchFor` leaves `launch.gen.N.json`, and the foreign-invocation exit branch omits launch cleanup. Successive generation numbers turn these into accumulating files rather than overwritten leftovers.

+

+ **Smallest fix:** finalise generation files after proven termination, including natural and foreign-invocation exits; remove launch files on proven pre-execution failure. Give inactive generation leftovers a recovery cleanup path. Coordinate uncertain starts with finding 1. **Fix before merge.**

+

+**Rulings**

+

+- **Branch from plan 1 — agree.** This matches the controller’s explicit sequencing and the dependency being reviewed. Reconcile with landed main before merge.

+- **Override `GOCACHE` — disagree as documented.** The supplied machine instructions expressly require the default cache and say it is writable. The ledger’s contrary memory-based rule does not substantiate this exception. This does not itself invalidate the code review.

+- **Defer the dev-box kernel run — agree.** Scoped production approval is absent. DOCS-T1 results do not discharge the later dev requirement.

+- **Push and describe after final review — agree.** This follows the controller’s requested order.

+- **Extract shared `pinChild` — agree.** The final implementation preserves the role path’s intent-before-mkdir ordering and its refusal of `EEXIST`, including the intervening-directory case.

+- **Add `SuccessExitStatus` — agree.** The generation template uses the same signal-translating trampoline as the existing templates.

+- **Return the settlement callback from `spawnInCgroupAt` — agree.** Dropping it would break cleanup obligations inherited from the legacy strict path.

+- **Preserve strict-host/owner gates and strict-aware cleanup — agree.** Generation placement must not bypass existing ownership restrictions.

+- **Keep `spawnClaimedGen` closed — agree.** Reintroducing in-process tenant execution would reverse the isolation decision. However, `RoleLaunch.Start` is also closed at this tip: routing to it alone does not implement claimed generation spawning.

+

+The burst-allocation kernel-test adjustment is reasonable. It proves containment under that workload, not that gradual growth promptly produces an OOM kill.

+

+**Cross-plan consequences**

+

+- **Plan 3 Task 1:** change `spawnInCgroupAtTier`, forwarding helpers, callers and tests to return `(*exec.Cmd, func(), error)`. Preserve the strict gate and call settlement after reaping.

+- **Plan 3 Task 10:** replace the assumed working `spawnClaimedGen` body with an explicitly designed isolated root launch protocol carrying reservations and the final authority check. `RoleLaunch.Start` currently refuses execution too.

+- **Plan 5:** keep claimed generation activation unavailable until that protocol exists. Require usable containment before overlap, route instance wakes through nonzero `StartGen`, and prevent switching a populated legacy parent directly into the generation layout.

+- **Plans 3 and 5:** preserve cleanup ownership after ambiguous start errors. Never retry a running generation’s start as recovery.

+- **Index:** correct the three-result signature, document the closed claimed-spawn contract and missing root integration, record the activation prerequisites, and qualify dormancy’s shared compatibility exceptions.

+- Keep the accepted dev kernel run and D1 reload-cost measurement as prerequisites to enabling the flag.

+

+**Deferred-minor triage**

+

+“Defer” below means it does not block this dormant merge; any stated activation prerequisite still applies.

+

+| Item | Disposition |

+|---|---|

+| T1: initially unused `isGenName` | Defer/closed: `CloseRemove` consumes it. |

+| T1: failed spawn leaves empty parent | Defer: sampler excludes it; later cold wake resets it. |

+| T1: fallback placement lacks a direct assertion | Defer; add when exercising generation spawn integration. |

+| T1: populated legacy parent skips delegation | Defer to plan 5; migration/drain is required before activation. |

+| T1/T3/T6: reconstructed or placeholder RED evidence | **Fix before merge:** correct the evidence record; do not present reconstructed text as captured output. |

+| T2: deterministic member sorting | Defer: harmless ordering change; acknowledge it in dormancy wording. |

+| T2: missing width-cap test | **Fix before merge**, with dormant finding 1. |

+| T2: missing ENOENT, replaced-child identity and public-wrapper tests | Defer: no demonstrated incorrect result in those paths. |

+| T2: non-atomic member walk | Defer: it remains supplementary evidence; retain the fence and kernel population checks. |

+| T2/T5: attribution and trailer formatting | Defer: correct the final squash attribution. |

+| T3: strict-parent success branch lacks fixture coverage | Defer/closed for success: covered by the reported kernel run; refusal coverage can follow. |

+| T3: write-only `Prepared.tier` | Defer. |

+| T3: invalid `CloseRemove` drains a parent | Defer: fail-closed API misuse; validate handle kind before mutation in a follow-up. |

+| T3: `_ = g1` and inaccurate report line count | Defer; remove cosmetic debris and correct reporting. |

+| T4: missing g2/strict-test cleanup; potentially hanging incumbent cleanup | **Fix before merge**, with dormant finding 2. |

+| T4: swallowed mmap error | Defer/closed: tip checks the error and exits nonzero. |

+| T4: discarded strict-child wait statuses | Defer: emptiness and surviving-sibling checks establish the containment result. |

+| T4: shared three-second cleanup context | **Fix before merge** with cleanup work; give independent cleanup operations fresh budgets. |

+| T4: swap-dependent OOM timeout; gradual-growth observation | Defer to rollout notes; accurately describe the burst-allocation evidence. |

+| T4: ignored `StdinPipe` error and `cgroupOf` fatal bypass | **Fix before merge** with unconditional child cleanup. |

+| T4: g2 swap limit not asserted | Defer; strengthen the next kernel validation. |

+| T5: stale sampler comment | Defer. |

+| T5: substring population match; missing-events coverage | Defer: kernel-generated values are constrained; exact field parsing would be clearer. |

+| T6: escaped input passed to `ParseGenInstance` | Defer: filesystem cleanup separately unescapes and rejects unsafe components; trampoline parsing uses unescaped input. |

+| T6: three-sample key-collision test | Defer: hostname validation and the identifier construction establish `@` exclusion. |

+| T6: generation sweep glob | Defer/closed: present. |

+| T7: permissive template-specifier test | Defer: inspected template has the intended split. |

+| T7: stale template count | Defer/closed: changed hardening comment now says four templates plus probe. |

+| T8: malformed-generation test passes for unrelated reasons | **Fix before merge:** plant otherwise-valid launch files and assert parser refusals, including traversal and extra-colon cases. |

+| T8: missing malformed exit-record cases | Defer; shared parser already has direct validation coverage. |

+| T9: starting an already-running generation | Defer as an explicit fresh-generation-only contract; plans 3/5 must enforce it. |

+| T9: successful start followed by failed invocation lookup | **Fix before merge**, latent finding 1. |

+| T9: naturally exited generation retains files | **Fix before merge**, latent finding 3. |

+| T9: missing write/reload-failure cleanup tests | **Fix before merge** alongside start-failure cleanup. |

+| T9: long-host cleanup round trip; empty test-only `ExeRoot` | Defer. |

+| T10: duplicated strict-gate mapping and different error wrapping | Defer: refusal remains intact. |

+| T10: missing end-to-end flag-on spawn cleanup coverage | Defer to integration before activation. |

+| T10: redundant removal survives mutation | Defer: redundant cleanup need not be independently observable. |

+| T10: nil-manager `ParentLimit` success | **Fix before merge**, latent finding 2. |

+| T10: weak no-cgroup error assertion; stale seed comment | Defer. |

+| T11: record-read and legacy-log non-interference assertions; missing comment | Defer. |

+| T11: foreign-invocation launch-file leak | **Fix before merge**, latent finding 3. |

+| T11: old `exit.run.*` records retained | Defer to migration cleanup; generation preparation must not delete another live role’s records. |

+| T11: flag-on `Start` rejects generation zero | Defer/intentional: plan 5 must use `StartGen`. |

+| T12: stale commit subject and helper/test descriptions | **Fix before merge:** describe a closed helper accurately, particularly in the index and final squash. |

+| T13: literal replacement coupling in `claimUnitContent` | Defer: current replacement and its test agree; an explicit no-match failure would strengthen future edits. |

\ No newline at end of file

diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go

index 7e9745ee4..fa22b4cf1 100644

--- a/internal/activator/backend_exec.go

+++ b/internal/activator/backend_exec.go

@@ -112,6 +112,13 @@ type ExecBackend struct {

// tier. Instance, seed and sidecar share the cgroup, so they share the

// tier — a seed's first-boot cost is that instance's work.

Tier func(registry.Route) cgroup.Limit

+ // GenContainment selects the generation layout (zero-downtime spec

+ // §7.3, §7.6): each instance spawn gets its own child cgroup g<n> (or

+ // the carlos-instance-gen@<id>:<n> unit for a nested host) under a

+ // parent that holds no processes. False, the zero value, is today's

+ // layout byte for byte: one cgroup per host and carlos-instance@. Set

+ // only by plan 5's CARLOS_SWAP wiring.

+ GenContainment bool

// Logs, when non-nil, gets each instance and seed child's stdout/stderr

// teed in line-wise for `carlos logs` (spec §5) — journald keeps every

// byte it got before. Nil (tests, a box without the shipping loop)

@@ -799,7 +806,11 @@ func (b *ExecBackend) seedChild(ctx context.Context, rt registry.Route) error {

// cost is that instance's work. No CleanupHost here — the instance spawn

// reuses the dir moments later, and its exit path (or the next agent

// start's sweep) removes it.

- cmd, settle, err := b.spawnInCgroup(rt, func() *exec.Cmd {

+ child, _, perr := b.placeFor(spawnSeed, 0)

+ if perr != nil {

+ return perr

+ }

+ cmd, settle, err := b.spawnInCgroupAt(rt, child, false, func() *exec.Cmd {

cmd := exec.CommandContext(ctx, b.liveLink(host), argv...)

// The control-channel vars are agent-owned in BOTH directions here too

// (Start's discipline, see withoutVars), and here they are always

@@ -836,6 +847,7 @@ func (b *ExecBackend) seedChild(ctx context.Context, rt registry.Route) error {

if err == nil {

err = cmd.Wait()

settle()

+ b.Cgroups.RemoveChild(host, child) // the seed's own child only; no-op for "" (flag off)

}

if err != nil {

// WaitDelay reports on the SUCCESS path too: a seed that exits 0 after

@@ -1139,11 +1151,11 @@ func (b *ExecBackend) Start(ctx context.Context, rt registry.Route) (Proc, strin

p := b.pathsFor(rt)

switch p.layout {

case LayoutNested:

- return b.startUnitAt(ctx, rt, rt.Socket)

+ return b.startUnitAt(ctx, rt, rt.Socket, 0)

case LayoutFenced, LayoutConflict:

return nil, "", fmt.Errorf("%s: refusing to wake: %s", rt.Host, p.reason)

}

- return b.startChildAt(ctx, rt, rt.Socket)

+ return b.startChildAt(ctx, rt, rt.Socket, 0)

}

// StartGen is Start for generation n (zero-downtime spec §7.1). perGen false

@@ -1177,7 +1189,7 @@ func (b *ExecBackend) StartGen(ctx context.Context, rt registry.Route, n uint64,

switch p.layout {

case LayoutNested:

// prepareAt removes the stale file at sock for the nested layout.

- proc, ver, err = b.startUnitAt(ctx, rt, sock)

+ proc, ver, err = b.startUnitAt(ctx, rt, sock, n)

case LayoutFenced, LayoutConflict:

return nil, Gen{}, fmt.Errorf("%s: refusing to wake: %s", rt.Host, p.reason)

default:

@@ -1193,7 +1205,7 @@ func (b *ExecBackend) StartGen(ctx context.Context, rt registry.Route, n uint64,

return nil, Gen{}, fmt.Errorf("%s: generation socket: %w", rt.Host, err)

}

- proc, ver, err = b.startChildAt(ctx, rt, sock)

+ proc, ver, err = b.startChildAt(ctx, rt, sock, n)

}

if err != nil {

return proc, Gen{}, err

@@ -1225,11 +1237,17 @@ func clearOwnGenSocket(sock string) error {

// startChildAt is Start's exec-child body (flat and foreign layouts):

// exec of the versioned binary directly, exactly as before nested layouts

// existed. sock is the app socket this spawn binds: rt.Socket, or a

-// generation socket (StartGen).

-func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock string) (Proc, string, error) {

+// generation socket (StartGen). n is the generation this spawn is (StartGen's

+// n; 0 from Start). It only places the spawn's cgroup, and only under

+// GenContainment (placeFor).

+func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock string, n uint64) (Proc, string, error) {

host := rt.Host

paths := b.pathsFor(rt)

dbPath := paths.dbPath

+ child, ownTier, err := b.placeFor(spawnInstance, n)

+ if err != nil {

+ return nil, "", err

+ }

// Exec exactly what a systemd unit would: the per-instance live symlink

// (<OptDir>/live/<host> → a staged version), same --socket/--db flags,

@@ -1342,7 +1360,7 @@ func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock

// carlos-exec@<host> and the sampler can account it. build may run

// twice — the manager's clone3-refusal fallback — so everything about

// the Cmd is configured inside it.

- cmd, settle, err := b.spawnInCgroup(rt, func() *exec.Cmd {

+ cmd, settle, err := b.spawnInCgroupAt(rt, child, ownTier, func() *exec.Cmd {

cmd := exec.Command(bin, "--socket", sock, "--db", dbPath)

cmd.Env = childEnv(childVars)

// The child logs through the agent's own stderr/stdout (journald

@@ -1387,6 +1405,7 @@ func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock

// A strict child's group is drained and closed before anything

// reads this exit as a stop (p.done closes below).

settle()

+ b.Cgroups.RemoveChild(host, child) // no-op for "" (flag off)

// Last exiter wins: rmdir refuses while the sidecar (same host

// cgroup) is still alive, and that failure is tolerated.

b.cleanupHost(host)

@@ -1485,14 +1504,22 @@ func (b *ExecBackend) normalizeSocketMode(rt registry.Route, sock string, proc P

// started by the root instance-control channel from a launch file rather

// than exec'd directly. The agent has no privilege to enter the tenant's

// namespace, only the control service does.

-func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock string) (Proc, string, error) {

+//

+// n is the generation this spawn is (StartGen's n; 0 from Start). Under

+// GenContainment it selects the carlos-instance-gen@<id>:<n> unit and its

+// generation-keyed files (unitGen); otherwise it is ignored.

+func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock string, n uint64) (Proc, string, error) {

if err := b.nestedGate(rt); err != nil {

return nil, "", err

}

+ gen, err := b.unitGen(n)

+ if err != nil {

+ return nil, "", err

+ }

if err := b.ensureLiveLink(ctx, rt.Host); err != nil {

return nil, "", err

}

- p, ctl, logs, fileEnv, err := b.prepareAt(rt, instancectl.RoleRun, sock)

+ p, ctl, logs, fileEnv, err := b.prepareAt(rt, instancectl.RoleRun, sock, gen)

if err != nil {

return nil, "", err

}

@@ -1505,14 +1532,25 @@ func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock s

}

childVars := b.agentOwnedVars(rt, sock+".control", fileEnv, ctl != nil)

argv := []string{b.liveLink(rt.Host), "--socket", sock, "--db", p.dbPath}

- if err := b.writeLaunch(p, instancectl.RoleRun, rt.Host, argv, childEnv(childVars)); err != nil {

+ if err := b.writeLaunchFor(p, instancectl.RoleRun, gen, rt.Host, argv, childEnv(childVars)); err != nil {

return fail(err)

}

// Read the symlink immediately before the start, same fail-safe

// direction as startChild: a concurrent repoint at worst records the

// older version (one extra convergence cycle), never the newer.

spawnVer, _ := b.LiveVersion(rt.Host)

- inv, err := b.Control.Start(ctx, rt.Host, instancectl.RoleRun, b.tier(rt).MemoryMax)

+ inv, err := b.startRole(ctx, rt, instancectl.RoleRun, gen)

+ if gen != 0 && (err != nil || inv == "") {

+ if err == nil {

+ err = fmt.Errorf("%s exited before it was observed", unitName(instancectl.RoleRun, rt.Host, gen))

+ }

+ if kerr := b.killFailedGen(ctx, rt, gen); kerr != nil {

+ // Nothing is released: the generation may still be running.

+ return nil, "", fmt.Errorf("%s: start unit: %w; %w: %v", rt.Host, err, errGenUnproven, kerr)

+ }

+ _ = os.Remove(filepath.Join(p.dataDir, instancectl.GenLaunchFileName(gen)))

+ return fail(fmt.Errorf("%s: start unit: %w", rt.Host, err))

+ }

if err != nil {

return fail(fmt.Errorf("%s: start unit: %w", rt.Host, err))

}

@@ -1520,10 +1558,10 @@ func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock s

// The server read the invocation after the start and found none:

// the unit already exited (plan 2 final review). Nothing is

// running, so there is nothing to stop; the wake failed.

- return fail(fmt.Errorf("%s: %s exited before it was observed", rt.Host, instancectl.UnitName(instancectl.RoleRun, rt.Host)))

+ return fail(fmt.Errorf("%s: %s exited before it was observed", rt.Host, unitName(instancectl.RoleRun, rt.Host, gen)))

}

- log.Printf("activator: %s: started %s memory.max=%d", rt.Host, instancectl.UnitName(instancectl.RoleRun, rt.Host), b.tier(rt).MemoryMax)

- up := b.newUnitProc(rt, instancectl.RoleRun, p, inv, ctl, logs)

+ log.Printf("activator: %s: started %s memory.max=%d", rt.Host, unitName(instancectl.RoleRun, rt.Host, gen), b.tier(rt).MemoryMax)

+ up := b.newUnitProcFor(rt, instancectl.RoleRun, gen, p, inv, ctl, logs)

if b.Cache != nil {

// A fresh process for this host is exactly the moment its cached

// content becomes suspect, same rule as startChild.

@@ -1556,6 +1594,25 @@ func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock s

return up, spawnVer, nil

}

+// killFailedGen ends a generation whose start failed or answered no

+// invocation. The start may have succeeded with its answer lost, so the

+// generation is killed and proven empty (StopGen with kill) on a budget the

+// caller's context cannot cancel. nil is that proof. With no generation

+// channel StartGen was never sent, so nothing was started.

+func (b *ExecBackend) killFailedGen(ctx context.Context, rt registry.Route, gen uint64) error {

+ gc, err := b.genControl()

+ if err != nil {

+ return nil

+ }

+ kctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), instancectl.DeadlineControl)

+ defer cancel()

+ if err := gc.StopGen(kctx, rt.Host, gen, true); err != nil {

+ log.Printf("activator: %s: kill %s after a failed start: %v", rt.Host, unitName(instancectl.RoleRun, rt.Host, gen), err)

+ return err

+ }

+ return nil

+}

+

// nestedGate is the pre-flight check every nested spawn path runs first: no

// root instance-control channel configured, or the box itself is not fit

// to run an isolated instance (NestedOK's own mounts and /ready check),

@@ -1794,10 +1851,14 @@ func (b *ExecBackend) startSidecarChild(_ context.Context, rt registry.Route) (P

if err != nil {

return nil, fmt.Errorf("read sidecar env: %w", err)

}

+ child, _, err := b.placeFor(spawnSidecar, 0)

+ if err != nil {

+ return nil, err

+ }

// Same host cgroup as the instance: a sidecar's load is that instance's

// work, and attributing it elsewhere would be a lie in the other

// direction.

- cmd, settle, err := b.spawnInCgroup(rt, func() *exec.Cmd {

+ cmd, settle, err := b.spawnInCgroupAt(rt, child, false, func() *exec.Cmd {

cmd := exec.Command(b.liveLink(rt.Host), "sidecar", "run")

cmd.Env = childEnv(fileEnv)

// Like the instance child: logs through the agent's own streams

@@ -1813,6 +1874,7 @@ func (b *ExecBackend) startSidecarChild(_ context.Context, rt registry.Route) (P

go func() {

p.err = cmd.Wait()

settle()

+ b.Cgroups.RemoveChild(rt.Host, child) // no-op for "" (flag off)

b.cleanupHost(rt.Host)

close(p.done)

}()

diff --git a/internal/activator/backend_unit.go b/internal/activator/backend_unit.go

index 5bba2b772..96495f493 100644

--- a/internal/activator/backend_unit.go

+++ b/internal/activator/backend_unit.go

@@ -345,6 +345,24 @@ func (b *SplitBackend) NextDueVia(ctx context.Context, rt registry.Route, socket

return b.pick(rt).NextDue(ctx, rt)

}

+// KillGen dispatches like Start, for an exec row only (zero-downtime spec

+// §8.2 step 1). A unit-backed row has no generation child to kill: systemd

+// owns the unit tenant's one process tree. An exec half without KillGen (a

+// test double) cannot prove a kill either. Both refuse, wrapping

+// errNoGenContainment, so a caller never believes a kill it could not make.

+func (b *SplitBackend) KillGen(ctx context.Context, rt registry.Route, n uint64) error {

+ if rt.Backing == registry.BackingUnit {

+ return fmt.Errorf("%s: a unit-backed row has no generation %d to kill: %w", rt.Host, n, errNoGenContainment)

+ }

+ k, ok := b.Exec.(interface {

+ KillGen(ctx context.Context, rt registry.Route, n uint64) error

+ })

+ if !ok {

+ return fmt.Errorf("%s: this backend cannot kill generation %d: %w", rt.Host, n, errNoGenContainment)

+ }

+ return k.KillGen(ctx, rt, n)

+}

+

// Layout implements Classifier for the composed backend: a unit-backed row

// is always today's flat unit shape (the exec-only layouts do not apply to

// it), and an exec row defers to the exec side's own classification.

diff --git a/internal/activator/claimspawn.go b/internal/activator/claimspawn.go

index 60b5cc0c8..09a6c7a83 100644

--- a/internal/activator/claimspawn.go

+++ b/internal/activator/claimspawn.go

@@ -21,3 +21,20 @@ func spawnClaimedBefore(ctx context.Context, enabled bool, manager *cgroup.Manag

// Keep this old helper closed even if a future caller reintroduces it.

return nil, nil, ErrClaimIsolationUnavailable

}

+

+// spawnClaimedGen is the claims-on spawn of a generation child under the

+// entry's parent (zero-downtime spec section 7.3), and it is closed: it

+// pins nothing, reserves nothing, never builds the command and never

+// enters the host write. Invalid arguments answer hostwrite.ErrUnavailable;

+// every other call answers ErrClaimIsolationUnavailable, like

+// spawnClaimedBefore. Main's RoleLaunch.Start is closed too, so routing a

+// claimed generation spawn there would not open it either. A later plan

+// needs an isolated root launch protocol that carries the claim

+// reservations and the final authority check, rather than a re-route or

+// an in-process path.

+func spawnClaimedGen(ctx context.Context, parent *cgroup.Prepared, n uint64, tier cgroup.Limit, selected hostwrite.Work, identity hostwrite.Identity, build func() *exec.Cmd, before func() error) (*exec.Cmd, *cgroup.Prepared, error) {

+ if parent == nil || selected == nil || build == nil || n == 0 {

+ return nil, nil, hostwrite.ErrUnavailable

+ }

+ return nil, nil, ErrClaimIsolationUnavailable

+}

diff --git a/internal/activator/claimspawn_test.go b/internal/activator/claimspawn_test.go

index a6933ccca..0d17f4672 100644

--- a/internal/activator/claimspawn_test.go

+++ b/internal/activator/claimspawn_test.go

@@ -39,3 +39,40 @@ func TestClaimSpawnNeverFallsBack(t *testing.T) {

t.Fatal(builds, calls)

}

+

+// spawnClaimedGen is closed: an invalid argument answers

+// hostwrite.ErrUnavailable, and every other call answers

+// ErrClaimIsolationUnavailable, whatever the work would decide. It never

+// enters the host write, never calls before and never builds the command.

+func TestClaimSpawnGenNeverFallsBack(t *testing.T) {

+ calls, builds := 0, 0

+ beforeCalled := false

+ build := func() *exec.Cmd { builds++; return exec.Command("must-not-execute") }

+ before := func() error { beforeCalled = true; return nil }

+

+ if _, _, err := spawnClaimedGen(context.Background(), nil, 1, cgroup.TierDefault, claimSpawnWork{false, &calls}, hostwrite.Identity{}, build, before); !errors.Is(err, hostwrite.ErrUnavailable) {

+ t.Fatalf("nil parent: %v", err)

+ }

+ if _, _, err := spawnClaimedGen(context.Background(), &cgroup.Prepared{}, 0, cgroup.TierDefault, claimSpawnWork{false, &calls}, hostwrite.Identity{}, build, before); !errors.Is(err, hostwrite.ErrUnavailable) {

+ t.Fatalf("generation 0: %v", err)

+ }

+ if calls != 0 {

+ t.Fatalf("host write entered %d times for invalid arguments", calls)

+ }

+ for _, deny := range []bool{true, false} {

+ beforeCalled = false

+ cmd, child, err := spawnClaimedGen(context.Background(), &cgroup.Prepared{}, 7, cgroup.TierDefault, claimSpawnWork{deny, &calls}, hostwrite.Identity{}, build, before)

+ if cmd != nil || child != nil {

+ t.Fatalf("deny=%v: cmd=%v child=%v", deny, cmd, child)

+ }

+ if !errors.Is(err, ErrClaimIsolationUnavailable) {

+ t.Fatalf("deny=%v: err=%v, want ErrClaimIsolationUnavailable", deny, err)

+ }

+ if beforeCalled {

+ t.Fatalf("deny=%v: before was called", deny)

+ }

+ if builds != 0 || calls != 0 {

+ t.Fatalf("builds=%d calls=%d, want 0 and 0", builds, calls)

+ }

+}

diff --git a/internal/activator/containment.go b/internal/activator/containment.go

new file mode 100644

index 000000000..ec0a824ec

--- /dev/null

+++ b/internal/activator/containment.go

@@ -0,0 +1,208 @@

+package activator

+

+import (

+ "context"

+ "errors"

+ "fmt"

+ "os/exec"

+

+ "github.com/carlosframework/platform/internal/cgroup"

+ "github.com/carlosframework/platform/internal/instancectl"

+ "github.com/carlosframework/platform/internal/registry"

+)

+

+var (

+ errNoGenContainment = errors.New("activator: generation containment is off")

+ errNoGenControl = errors.New("activator: the instance control channel cannot run generation units")

+ // errGenUnproven is a generation start that failed and whose kill did

+ // not prove the generation gone. It may still be running: its listeners

+ // and launch file are kept, and a caller must keep whatever it owns for

+ // the host (a later plan's swap keeps the instance's ownership).

+ errGenUnproven = errors.New("activator: generation start failed and its termination is unproven")

+)

+

+// spawnKind is which child shape a spawn is, for placeFor.

+type spawnKind int

+

+const (

+ spawnInstance spawnKind = iota

+ spawnSidecar // the legacy co-spawned sidecar (<host>-sidecar.env)

+ spawnSeed

+ spawnWorker // workers and native sidecar members (StartWorker)

+)

+

+// placeFor names the child cgroup a spawn lands in under the generation

+// layout (zero-downtime spec §7.3, plan 2 D4): g<n> with the tier's full

+// limits for an instance generation; role-sidecar and role-seed, with no

+// limit of their own, for the legacy co-spawned sidecar and the seed, which

+// share the instance's host cgroup and so cannot sit in a parent that holds

+// no processes. Workers keep one cgroup per host because they never overlap

+// (§7.9). "" (today's single-level cgroup) for every kind while

+// GenContainment is off.

+func (b *ExecBackend) placeFor(kind spawnKind, n uint64) (child string, ownTier bool, err error) {

+ if !b.GenContainment {

+ return "", false, nil

+ }

+ switch kind {

+ case spawnInstance:

+ if n == 0 {

+ return "", false, errors.New("activator: generation 0 is not a generation")

+ }

+ return cgroup.GenChild(n), true, nil

+ case spawnSidecar:

+ return cgroup.RoleChild("sidecar"), false, nil

+ case spawnSeed:

+ return cgroup.RoleChild("seed"), false, nil

+ }

+ return "", false, nil

+}

+

+// spawnInCgroupAt is spawnInCgroup for a placed child. With no child it IS

+// spawnInCgroup, so a flag-off spawn is byte for byte today's.

+//

+// A placed spawn is never strict: strict containment of a generation layout

+// is the claim path's (PrepareStrictParent, plan 5). It still honours the

+// host gate, though, so a placed spawn can never start beside a strict

+// owner's group, and a host that is strict is refused rather than started

+// outside its strict group. The returned settle is spawnInCgroup's: a no-op

+// here. A spawn that fails removes the child it may have created.

+func (b *ExecBackend) spawnInCgroupAt(rt registry.Route, child string, ownTier bool, build func() *exec.Cmd) (*exec.Cmd, func(), error) {

+ if child == "" {

+ return b.spawnInCgroup(rt, build)

+ }

+ start := func() (*exec.Cmd, func(), error) {

+ tier := b.tier(rt)

+ own := cgroup.Limit{}

+ if ownTier {

+ own = tier

+ }

+ cmd, err := b.Cgroups.StartChild(rt.Host, child, tier, own, build)

+ if err != nil {

+ b.Cgroups.RemoveChild(rt.Host, child)

+ return nil, nil, err

+ }

+ return cmd, func() {}, nil

+ }

+ if b.Strict == nil {

+ return start()

+ }

+ g := b.gate(rt.Host)

+ g.mu.Lock()

+ defer g.mu.Unlock()

+ if b.Strict(rt.Host) {

+ return nil, nil, fmt.Errorf("%s: %w: a generation layout is not started strict here", rt.Host, ErrStrictRequired)

+ }

+ if o := g.owner; o != nil {

+ cause := ErrStrictGroupInUse

+ if o.pending {

+ cause = ErrStrictCleanupPending

+ }

+ return nil, nil, fmt.Errorf("%s: %w", rt.Host, cause)

+ }

+ return start()

+}

+

+// genControl is Control's generation half, or errNoGenControl.

+func (b *ExecBackend) genControl() (GenControl, error) {

+ if b.Control == nil {

+ return nil, errNoGenControl

+ }

+ gc, ok := b.Control.(GenControl)

+ if !ok {

+ return nil, errNoGenControl

+ }

+ return gc, nil

+}

+

+// unitGen is the generation a nested run unit is started as: 0 (the

+// original template) while GenContainment is off, n otherwise. n of 0

+// with containment on is refused.

+func (b *ExecBackend) unitGen(n uint64) (uint64, error) {

+ if !b.GenContainment {

+ return 0, nil

+ }

+ if n == 0 {

+ return 0, errors.New("activator: generation 0 is not a generation")

+ }

+ return n, nil

+}

+

+// startRole starts one role's unit and returns its invocation: the

+// original template through Control, a generation through GenControl.

+func (b *ExecBackend) startRole(ctx context.Context, rt registry.Route, role instancectl.Role, gen uint64) (string, error) {

+ if gen == 0 {

+ return b.Control.Start(ctx, rt.Host, role, b.tier(rt).MemoryMax)

+ }

+ gc, err := b.genControl()

+ if err != nil {

+ return "", err

+ }

+ return gc.StartGen(ctx, rt.Host, gen, b.tier(rt).MemoryMax)

+}

+

+// ParentLimit sets rt's generation parent to l: carlos-exec@<host> for a

+// flat or foreign host, the host slice for a nested one (spec §7.3). The

+// swap raises it to cgroup.OverlapLimit before the successor spawns and

+// restores the tier once the retiring generation's child is empty. With

+// GenContainment off there is no parent, and it refuses, so a caller can

+// never believe it raised a limit that does not exist.

+func (b *ExecBackend) ParentLimit(ctx context.Context, rt registry.Route, l cgroup.Limit) error {

+ if !b.GenContainment {

+ return errNoGenContainment

+ }

+ switch p := b.pathsFor(rt); p.layout {

+ case LayoutNested:

+ gc, err := b.genControl()

+ if err != nil {

+ return err

+ }

+ return gc.Parent(ctx, rt.Host, l.MemoryMax)

+ case LayoutFenced, LayoutConflict:

+ return fmt.Errorf("%s: refusing to set a parent limit: %s", rt.Host, p.reason)

+ }

+ // The manager's SetParentLimit answers nil with no manager; here that

+ // would admit an overlap with no parent raised and no generation child

+ // to kill, so it refuses.

+ if b.Cgroups == nil {

+ return cgroup.ErrNoDelegation

+ }

+ return b.Cgroups.SetParentLimit(rt.Host, l)

+}

+

+// KillGen kills generation n outright and returns once its child is empty

+// (spec §7.3, §8.2 step 1). It acts on that generation's child only.

+func (b *ExecBackend) KillGen(ctx context.Context, rt registry.Route, n uint64) error {

+ if !b.GenContainment || n == 0 {

+ return errNoGenContainment

+ }

+ switch p := b.pathsFor(rt); p.layout {

+ case LayoutNested:

+ gc, err := b.genControl()

+ if err != nil {

+ return err

+ }

+ return gc.StopGen(ctx, rt.Host, n, true)

+ case LayoutFenced, LayoutConflict:

+ return fmt.Errorf("%s: refusing to kill generation %d: %s", rt.Host, n, p.reason)

+ }

+ if b.Cgroups == nil {

+ return cgroup.ErrNoDelegation

+ }

+ child := cgroup.GenChild(n)

+ c, err := b.Cgroups.OpenCleanupChild(rt.Host, child)

+ if errors.Is(err, cgroup.ErrNoCgroup) {

+ return nil // already removed, so already empty

+ }

+ if err != nil {

+ return err

+ }

+ if err := c.KillAndEmpty(ctx); err != nil {

+ _ = c.Close()

+ return err

+ }

+ if err := c.Close(); err != nil {

+ return err

+ }

+ b.Cgroups.RemoveChild(rt.Host, child)

+ return nil

+}

diff --git a/internal/activator/containment_test.go b/internal/activator/containment_test.go

new file mode 100644

index 000000000..a66c53e47

--- /dev/null

+++ b/internal/activator/containment_test.go

@@ -0,0 +1,275 @@

+package activator

+

+import (

+ "context"

+ "errors"

+ "fmt"

+ "os"

+ "os/exec"

+ "path/filepath"

+ "testing"

+

+ "github.com/carlosframework/platform/internal/cgroup"

+ "github.com/carlosframework/platform/internal/instancectl"

+ "github.com/carlosframework/platform/internal/registry"

+)

+

+// fakeGenControl adds the generation verbs to fakeControl (unitproc_test.go).

+type fakeGenControl struct{ *fakeControl }

+

+func (f fakeGenControl) StartGen(_ context.Context, host string, n uint64, max int64) (string, error) {

+ f.rec(fmt.Sprintf("startgen %s %d %d", host, n, max))

+ if f.onStart != nil {

+ f.onStart()

+ }

+ return f.inv, f.startErr

+}

+func (f fakeGenControl) StopGen(_ context.Context, host string, n uint64, kill bool) error {

+ f.rec(fmt.Sprintf("stopgen %s %d kill=%v", host, n, kill))

+ if f.stopHook != nil {

+ f.stopHook(host, instancectl.RoleRun)

+ }

+ return f.stopGenErr

+}

+func (f fakeGenControl) ResetGen(_ context.Context, host string, n uint64) error {

+ f.rec(fmt.Sprintf("resetgen %s %d", host, n))

+ return nil

+}

+func (f fakeGenControl) Parent(_ context.Context, host string, max int64) error {

+ f.rec(fmt.Sprintf("parent %s %d", host, max))

+ return nil

+}

+

+func TestPlaceForIsSingleLevelWithContainmentOff(t *testing.T) {

+ b := testExecBackend(t)

+ for _, kind := range []spawnKind{spawnInstance, spawnSidecar, spawnSeed, spawnWorker} {

+ child, own, err := b.placeFor(kind, 7)

+ if child != "" || own || err != nil {

+ t.Errorf("kind %d with containment off: %q %v %v", kind, child, own, err)

+ }

+}

+

+func TestPlaceForUnderContainment(t *testing.T) {

+ b := testExecBackend(t)

+ b.GenContainment = true

+ for _, tc := range []struct {

+ kind spawnKind

+ child string

+ own bool

+ }{

+ {spawnInstance, "g7", true},

+ {spawnSidecar, "role-sidecar", false},

+ {spawnSeed, "role-seed", false},

+ {spawnWorker, "", false},

+ } {

+ child, own, err := b.placeFor(tc.kind, 7)

+ if child != tc.child || own != tc.own || err != nil {

+ t.Errorf("kind %d: %q %v %v, want %q %v", tc.kind, child, own, err, tc.child, tc.own)

+ }

+ if _, _, err := b.placeFor(spawnInstance, 0); err == nil {

+ t.Error("an instance spawn with generation 0 was placed")

+ }

+}

+

+func delegatedBackend(t *testing.T, host string) (*ExecBackend, string) {

+ t.Helper()

+ root := t.TempDir()

+ for _, f := range []string{"cgroup.procs", "cgroup.subtree_control"} {

+ if err := os.WriteFile(filepath.Join(root, f), nil, 0o644); err != nil {

+ t.Fatal(err)

+ }

+ m, err := cgroup.SetupAt(root, t.Logf)

+ if err != nil {

+ t.Fatal(err)

+ }

+ parent := filepath.Join(root, cgroup.ExecName(host))

+ if err := os.MkdirAll(parent, 0o755); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.WriteFile(filepath.Join(parent, "cgroup.controllers"), []byte("cpu io memory pids\n"), 0o644); err != nil {

+ t.Fatal(err)

+ }

+ b := testExecBackend(t)

+ b.Cgroups = m

+ return b, parent

+}

+

+func TestSpawnInCgroupAtPutsAGenerationUnderItsParent(t *testing.T) {

+ const host = "h.example.com"

+ b, parent := delegatedBackend(t, host)

+ b.GenContainment = true

+ cmd, _, err := b.spawnInCgroupAt(registry.Route{Host: host}, "g7", true, func() *exec.Cmd { return exec.Command("/bin/true") })

+ if err != nil {

+ t.Fatal(err)

+ }

+ _ = cmd.Wait()

+ for file, want := range map[string]string{

+ filepath.Join(parent, "g7", "memory.max"): "536870912",

+ filepath.Join(parent, "g7", "memory.oom.group"): "1",

+ filepath.Join(parent, "memory.oom.group"): "0",

+ } {

+ if got, err := os.ReadFile(file); err != nil || string(got) != want {

+ t.Errorf("%s = %q, %v; want %q", file, got, err, want)

+ }

+}

+

+// Flag off: the spawn lands in the host cgroup itself, as today.

+func TestSpawnInCgroupAtWithNoChildKeepsTheHostCgroup(t *testing.T) {

+ const host = "h.example.com"

+ b, parent := delegatedBackend(t, host)

+ cmd, _, err := b.spawnInCgroupAt(registry.Route{Host: host}, "", false, func() *exec.Cmd { return exec.Command("/bin/true") })

+ if err != nil {

+ t.Fatal(err)

+ }

+ _ = cmd.Wait()

+ if got, err := os.ReadFile(filepath.Join(parent, "memory.oom.group")); err != nil || string(got) != "1" {

+ t.Errorf("host cgroup memory.oom.group = %q, %v; want 1", got, err)

+ }

+ entries, _ := os.ReadDir(parent)

+ for _, e := range entries {

+ if e.IsDir() {

+ t.Errorf("a flag-off spawn made child cgroup %s", e.Name())

+ }

+}

+

+func TestParentLimitRefusesWithContainmentOff(t *testing.T) {

+ b, _ := delegatedBackend(t, "h.example.com")

+ if err := b.ParentLimit(context.Background(), registry.Route{Host: "h.example.com"}, cgroup.TierDefault); !errors.Is(err, errNoGenContainment) {

+ t.Fatalf("ParentLimit with containment off: %v", err)

+ }

+ if err := b.KillGen(context.Background(), registry.Route{Host: "h.example.com"}, 7); !errors.Is(err, errNoGenContainment) {

+ t.Fatalf("KillGen with containment off: %v", err)

+ }

+}

+

+func TestParentLimitFlatWritesTheParent(t *testing.T) {

+ const host = "h.example.com"

+ b, parent := delegatedBackend(t, host)

+ b.GenContainment = true

+ if err := b.ParentLimit(context.Background(), registry.Route{Host: host}, cgroup.OverlapLimit(cgroup.TierDefault, cgroup.TierDefault)); err != nil {

+ t.Fatal(err)

+ }

+ if got, err := os.ReadFile(filepath.Join(parent, "memory.max")); err != nil || string(got) != "1073741824" {

+ t.Errorf("parent memory.max = %q, %v", got, err)

+ }

+}

+

+// No cgroup manager (no delegation): there is no parent to raise, so a

+// flat ParentLimit refuses rather than let a swap admit an overlap with

+// nothing raised.

+func TestParentLimitFlatRefusesWithoutACgroupManager(t *testing.T) {

+ b := testExecBackend(t)

+ b.GenContainment = true

+ b.Cgroups = nil

+ if err := b.ParentLimit(context.Background(), registry.Route{Host: "h.example.com"}, cgroup.OverlapLimit(cgroup.TierDefault, cgroup.TierDefault)); !errors.Is(err, cgroup.ErrNoDelegation) {

+ t.Fatalf("ParentLimit with no cgroup manager: %v, want cgroup.ErrNoDelegation", err)

+ }

+}

+

+// A fixture is not cgroup v2: KillGen must refuse rather than claim an

+// empty child it never proved.

+func TestKillGenFlatRefusesWithoutCgroupV2(t *testing.T) {

+ const host = "h.example.com"

+ b, parent := delegatedBackend(t, host)

+ b.GenContainment = true

+ if err := os.Mkdir(filepath.Join(parent, "g7"), 0o755); err != nil {

+ t.Fatal(err)

+ }

+ if err := b.KillGen(context.Background(), registry.Route{Host: host}, 7); err == nil {

+ t.Fatal("KillGen reported an unproven kill as done")

+ }

+}

+

+func TestParentLimitAndKillGenNestedAskTheControlService(t *testing.T) {

+ ctl := &fakeControl{}

+ b, rt := nestedBackend(t, ctl, &fakeUnit{})

+ b.GenContainment = true

+ b.Control = fakeGenControl{ctl}

+ if err := b.ParentLimit(context.Background(), rt, cgroup.OverlapLimit(cgroup.TierDefault, cgroup.TierDefault)); err != nil {

+ t.Fatal(err)

+ }

+ if err := b.KillGen(context.Background(), rt, 7); err != nil {

+ t.Fatal(err)

+ }

+ if want := "parent h.example.com 1073741824\nstopgen h.example.com 7 kill=true"; ctl.joined() != want {

+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)

+ }

+ b.Control = ctl // a Control without the generation verbs

+ if err := b.ParentLimit(context.Background(), rt, cgroup.TierDefault); !errors.Is(err, errNoGenControl) {

+ t.Errorf("ParentLimit through a Control without generations: %v", err)

+ }

+}

+

+// SplitBackend forwards KillGen to the exec half for an exec row. A

+// unit-backed row, and an exec half that cannot kill a generation, refuse:

+// a caller must never believe a kill it could not make.

+func TestSplitBackendKillGen(t *testing.T) {

+ ctl := &fakeControl{}

+ b, rt := nestedBackend(t, ctl, &fakeUnit{})

+ b.GenContainment = true

+ b.Control = fakeGenControl{ctl}

+ sb := &SplitBackend{Exec: b, Unit: &UnitBackend{}}

+ if err := sb.KillGen(context.Background(), rt, 7); err != nil {

+ t.Fatal(err)

+ }

+ if want := "stopgen h.example.com 7 kill=true"; ctl.joined() != want {

+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)

+ }

+ unitRt := rt

+ unitRt.Backing = registry.BackingUnit

+ if err := sb.KillGen(context.Background(), unitRt, 7); !errors.Is(err, errNoGenContainment) {

+ t.Errorf("KillGen of a unit-backed row: %v", err)

+ }

+ if want := "stopgen h.example.com 7 kill=true"; ctl.joined() != want {

+ t.Errorf("a unit-backed KillGen reached the control service:\n%s", ctl.joined())

+ }

+ plain := &SplitBackend{Exec: &fakeBackend{}, Unit: &UnitBackend{}}

+ if err := plain.KillGen(context.Background(), rt, 7); !errors.Is(err, errNoGenContainment) {

+ t.Errorf("KillGen through an exec half without KillGen: %v", err)

+ }

+}

+

+// A placed spawn that fails to start leaves no child cgroup behind.

+func TestSpawnInCgroupAtRemovesTheChildOfAFailedSpawn(t *testing.T) {

+ const host = "h.example.com"

+ b, parent := delegatedBackend(t, host)

+ b.GenContainment = true

+ // A zero tier writes no limit files, so the fixture's child directory is

+ // as empty as a real cgroup's after its control files vanish.

+ b.Tier = func(registry.Route) cgroup.Limit { return cgroup.Limit{} }

+ _, _, err := b.spawnInCgroupAt(registry.Route{Host: host}, "g7", true, func() *exec.Cmd {

+ return exec.Command(filepath.Join(t.TempDir(), "no-such-binary"))

+ })

+ if err == nil {

+ t.Fatal("a spawn of a missing binary succeeded")

+ }

+ if _, err := os.Stat(filepath.Join(parent, "g7")); !os.IsNotExist(err) {

+ t.Errorf("the failed spawn's child cgroup is still there: %v", err)

+ }

+}

+

+// A placed spawn never starts outside a strict host's group: a host that is

+// strict, or one whose group a strict owner holds, is refused.

+func TestSpawnInCgroupAtRefusesAStrictHost(t *testing.T) {

+ const host = "h.example.com"

+ b, parent := delegatedBackend(t, host)

+ b.GenContainment = true

+ b.Strict = func(string) bool { return true }

+ build := func() *exec.Cmd { return exec.Command("/bin/true") }

+ if _, _, err := b.spawnInCgroupAt(registry.Route{Host: host}, "g7", true, build); !errors.Is(err, ErrStrictRequired) {

+ t.Fatalf("a strict host's placed spawn: %v", err)

+ }

+ b.Strict = func(string) bool { return false }

+ b.gate(host).owner = &strictOwner{}

+ if _, _, err := b.spawnInCgroupAt(registry.Route{Host: host}, "role-seed", false, build); !errors.Is(err, ErrStrictGroupInUse) {

+ t.Fatalf("a placed spawn beside a strict owner: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(parent, "g7")); !os.IsNotExist(err) {

+ t.Errorf("a refused spawn made a child cgroup: %v", err)

+ }

+}

diff --git a/internal/activator/control.go b/internal/activator/control.go

index 3fe7bf79c..51d6fd9c8 100644

--- a/internal/activator/control.go

+++ b/internal/activator/control.go

@@ -25,6 +25,21 @@ type Control interface {

var _ Control = (*instancectl.Client)(nil)

+// GenControl is the generation half of the instance-control channel

+// (zero-downtime spec §7.6): the carlos-instance-gen@<id>:<n> template's

+// start, stop and reset, and the host slice's parent limit (§7.3). It is a

+// separate interface so a Control without it (a fake, an older client)

+// still satisfies Control. The backend asks for it only while

+// GenContainment is on.

+type GenControl interface {

+ StartGen(ctx context.Context, host string, n uint64, memoryMax int64) (string, error)

+ StopGen(ctx context.Context, host string, n uint64, kill bool) error

+ ResetGen(ctx context.Context, host string, n uint64) error

+ Parent(ctx context.Context, host string, memoryMax int64) error

+}

+

+var _ GenControl = (*instancectl.Client)(nil)

+

// UnitState answers `systemctl show -p ActiveState,InvocationID <unit>`,

// which needs no privilege. An unloaded unit answers inactive with an

// empty invocation.

diff --git a/internal/activator/prepare.go b/internal/activator/prepare.go

index f8ac05fcb..929aea921 100644

--- a/internal/activator/prepare.go

+++ b/internal/activator/prepare.go

@@ -31,14 +31,17 @@ import (

// and stale records cleared. It returns the listeners so the caller can

// hand them to the proc or close them on a failed start.

func (b *ExecBackend) prepare(rt registry.Route, role instancectl.Role) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) {

- return b.prepareAt(rt, role, rt.Socket)

+ return b.prepareAt(rt, role, rt.Socket, 0)

}

// prepareAt is prepare for a run role whose app socket is sock: rt.Socket,

// or a generation socket (StartGen, zero-downtime spec §7.1). Only the

// file at sock is removed, never a sibling generation's, and the control

// listener binds sock + ".control". The seed and sidecar roles ignore sock.

-func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock string) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) {

+// gen is the generation unit being prepared (0 for the original templates):

+// it keys the log socket and the exit records cleared (zero-downtime spec

+// §7.6).

+func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock string, gen uint64) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) {

p := b.pathsFor(rt)

if err := os.MkdirAll(p.dataDir, 0o700); err != nil {

return p, nil, nil, nil, err

@@ -46,7 +49,7 @@ func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock s

if err := os.MkdirAll(p.runDir, 0o750|os.ModeSetgid); err != nil {

return p, nil, nil, nil, err

}

- if err := os.MkdirAll(filepath.Dir(b.logSocket(rt.Host, role)), 0o700); err != nil {

+ if err := os.MkdirAll(filepath.Dir(b.logSocketFor(rt.Host, role, gen)), 0o700); err != nil {

return p, nil, nil, nil, err

}

var env map[string]string

@@ -75,7 +78,7 @@ func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock s

// this one.

if entries, err := os.ReadDir(p.runDir); err == nil {

for _, e := range entries {

- if strings.HasPrefix(e.Name(), "exit."+string(role)+".") {

+ if strings.HasPrefix(e.Name(), exitPrefix(role, gen)) {

_ = os.Remove(filepath.Join(p.runDir, e.Name()))

}

@@ -99,7 +102,7 @@ func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock s

_ = os.Remove(sock)

ctl = b.listenControl(rt, sock+".control", p.envFile)

}

- logs, err := listenLogs(b.logSocket(rt.Host, role), b.logEmitter(rt))

+ logs, err := listenLogs(b.logSocketFor(rt.Host, role, gen), b.logEmitter(rt))

if err != nil {

if ctl != nil {

ctl.Close()

@@ -109,6 +112,17 @@ func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock s

return p, ctl, logs, env, nil

}

+// exitPrefix is the exit records prepare clears before a start: this

+// role's for the original templates, this generation's alone otherwise.

+// A sibling generation's record is never touched (zero-downtime spec

+// §7.6).

+func exitPrefix(role instancectl.Role, gen uint64) string {

+ if gen == 0 {

+ return "exit." + string(role) + "."

+ }

+ return instancectl.GenExitPrefix(gen)

+}

+

// listenControl binds the control socket exactly as startChild does for an

// exec child (the tokenReader and schedule mount are unchanged, aside from

// reading beneath the instance directory rather than following a plain

@@ -134,9 +148,21 @@ func (b *ExecBackend) listenControl(rt registry.Route, control, envPath string)

return ctl

}

-// logSocket is the role's log socket path, honouring the test override of

-// the run directory (RunDirOverride).

+// logSocket is logSocketFor for the original templates.

func (b *ExecBackend) logSocket(host string, role instancectl.Role) string {

+ return b.logSocketFor(host, role, 0)

+}

+

+// logSocketFor is the role's log socket path, keyed by generation for a

+// generation unit, honouring the test override of the run directory

+// (RunDirOverride).

+func (b *ExecBackend) logSocketFor(host string, role instancectl.Role, gen uint64) string {

+ if gen != 0 {

+ if b.RunDirOverride != "" {

+ return filepath.Join(b.RunDirOverride, "logs", instancectl.GenKey(host, gen)+".run.sock")

+ }

+ return instancectl.GenLogSocket(host, gen)

+ }

if b.RunDirOverride != "" {

return filepath.Join(b.RunDirOverride, "logs", provision.InstanceID(host)+"."+string(role)+".sock")

}

@@ -151,15 +177,25 @@ func (b *ExecBackend) logEmitter(rt registry.Route) func(string) {

return func(line string) { b.Logs.App(release.RouteAccount(rt), rt.App, rt.Host, line) }

}

-// writeLaunch writes launch.<role>.json atomically into the instance

-// directory (design §6.2). The env is childEnv's deduplicated list; the

-// trampoline execs it as is.

+// writeLaunch is writeLaunchFor for the original templates.

func (b *ExecBackend) writeLaunch(p instancePaths, role instancectl.Role, host string, argv, env []string) error {

+ return b.writeLaunchFor(p, role, 0, host, argv, env)

+}

+

+// writeLaunchFor writes the launch file atomically into the instance

+// directory (design §6.2): launch.<role>.json, or launch.gen.<n>.json for a

+// generation (zero-downtime spec §7.6). The env is childEnv's deduplicated

+// list; the trampoline execs it as is.

+func (b *ExecBackend) writeLaunchFor(p instancePaths, role instancectl.Role, gen uint64, host string, argv, env []string) error {

raw, err := json.MarshalIndent(instancectl.Launch{Schema: instancectl.LaunchSchema, Host: host, Argv: argv, Env: env}, "", " ")

if err != nil {

return err

}

- return instancefs.WriteAtomic(p.dataDir, instancectl.LaunchFileName(role), append(raw, '\n'), 0o600)

+ name := instancectl.LaunchFileName(role)

+ if gen != 0 {

+ name = instancectl.GenLaunchFileName(gen)

+ }

+ return instancefs.WriteAtomic(p.dataDir, name, append(raw, '\n'), 0o600)

}

// readEnvFileBeneath is readEnvFile through instancefs: a nested host's env

diff --git a/internal/activator/unitproc.go b/internal/activator/unitproc.go

index 480a6a3a0..e8b6de72b 100644

--- a/internal/activator/unitproc.go

+++ b/internal/activator/unitproc.go

@@ -35,11 +35,15 @@ type unitProc struct {

unit, slice string

invocation string

runDir string

- ctl *edge.ControlListener

- logs *logListener

- exited atomic.Bool

- done chan struct{}

- closeOnce atomic.Bool

+ // gen is the generation of a carlos-instance-gen@ unit, 0 for the

+ // original templates; dataDir holds its launch file.

+ gen uint64

+ dataDir string

+ ctl *edge.ControlListener

+ logs *logListener

+ exited atomic.Bool

+ done chan struct{}

+ closeOnce atomic.Bool

// listenersArmed guards the done-watcher goroutine Stop's ctx.Done

// branch starts: several callers bound their wait on Stop shorter than

// the unit may actually take to die (a sidecar co-stop, a discarded

@@ -53,15 +57,46 @@ type unitProc struct {

// stop on its own. It is instead the proof that our own invocation is

// already over (see observe).

stopRequested atomic.Bool

+ // genFilesGone is set once a stop of this generation succeeded: the

+ // control service removed the unit's drop-in and exe layout with it,

+ // so finaliseGen need not ask again.

+ genFilesGone atomic.Bool

}

func (b *ExecBackend) newUnitProc(rt registry.Route, role instancectl.Role, p instancePaths, inv string, ctl *edge.ControlListener, logs *logListener) *unitProc {

- up := &unitProc{b: b, rt: rt, role: role, unit: instancectl.UnitName(role, rt.Host), slice: instancectl.SliceName(rt.Host),

- invocation: inv, runDir: p.runDir, ctl: ctl, logs: logs, done: make(chan struct{})}

+ return b.newUnitProcFor(rt, role, 0, p, inv, ctl, logs)

+}

+

+func (b *ExecBackend) newUnitProcFor(rt registry.Route, role instancectl.Role, gen uint64, p instancePaths, inv string, ctl *edge.ControlListener, logs *logListener) *unitProc {

+ up := &unitProc{b: b, rt: rt, role: role, gen: gen, unit: unitName(role, rt.Host, gen), slice: instancectl.SliceName(rt.Host),

+ invocation: inv, runDir: p.runDir, dataDir: p.dataDir, ctl: ctl, logs: logs, done: make(chan struct{})}

go b.observe(up)

return up

}

+// unitName is the unit a role of host runs as: the original template's, or

+// generation gen's (zero-downtime spec §7.6).

+func unitName(role instancectl.Role, host string, gen uint64) string {

+ if gen == 0 {

+ return instancectl.UnitName(role, host)

+ }

+ return instancectl.GenUnitName(host, gen)

+}

+

+// stopRole stops one role's unit: the original template's through Control,

+// a generation's through GenControl. A generation stop never touches a

+// sibling generation.

+func (b *ExecBackend) stopRole(ctx context.Context, host string, role instancectl.Role, gen uint64) error {

+ if gen == 0 {

+ return b.Control.Stop(ctx, host, role)

+ }

+ gc, err := b.genControl()

+ if err != nil {

+ return err

+ }

+ return gc.StopGen(ctx, host, gen, false)

+}

+

func (p *unitProc) Exited() bool { return p.exited.Load() }

// Stop asks the control service to stop the unit and waits for termination

@@ -75,7 +110,7 @@ func (p *unitProc) Stop(ctx context.Context) error {

// still fire its own foreign-invocation stop for the same unit.

p.stopRequested.Store(true)

sctx, cancel := context.WithTimeout(ctx, instancectl.DeadlineControl)

- err := p.b.Control.Stop(sctx, p.rt.Host, p.role)

+ err := p.stopUnit(sctx)

cancel()

if err != nil {

log.Printf("activator: stop %s %s: %v", p.rt.Host, p.role, err)

@@ -103,6 +138,36 @@ func (p *unitProc) Stop(ctx context.Context) error {

}

+// stopUnit asks the control service to stop this proc's unit. For a

+// generation, a stop that succeeded also removed the unit's files there.

+func (p *unitProc) stopUnit(ctx context.Context) error {

+ err := p.b.stopRole(ctx, p.rt.Host, p.role, p.gen)

+ if err == nil && p.gen != 0 {

+ p.genFilesGone.Store(true)

+ }

+ return err

+}

+

+// finaliseGen removes a generation's files once observe has proven it

+// gone (generation numbers are never reused, so nothing else would): its

+// launch file here, and the control service's drop-in and exe layout

+// through a stop of the inactive unit, unless a stop of ours already

+// removed them. Never called while termination is unproven.

+func (p *unitProc) finaliseGen() {

+ if p.gen == 0 {

+ return

+ }

+ p.removeLaunch()

+ if p.genFilesGone.Load() {

+ return

+ }

+ ctx, cancel := context.WithTimeout(context.Background(), instancectl.DeadlineControl)

+ defer cancel()

+ if err := p.stopUnit(ctx); err != nil {

+ log.Printf("activator: %s: remove the files of %s: %v", p.rt.Host, p.unit, err)

+ }

+}

+

func (p *unitProc) closeListeners() {

if p.closeOnce.Swap(true) {

return

@@ -171,7 +236,7 @@ func (b *ExecBackend) observe(p *unitProc) {

if foreign && !foreignStopped && !p.stopRequested.Load() {

log.Printf("activator: %s: unit restarted outside the activator (invocation %s, ours %s); stopping it", p.rt.Host, inv, p.invocation)

sctx, scancel := context.WithTimeout(context.Background(), instancectl.DeadlineControl)

- if err := b.Control.Stop(sctx, p.rt.Host, p.role); err != nil {

+ if err := p.stopUnit(sctx); err != nil {

log.Printf("activator: %s: stop foreign invocation: %v", p.rt.Host, err)

}

scancel()

@@ -208,6 +273,7 @@ func (b *ExecBackend) observe(p *unitProc) {

} else {

emitEvent(b.Logs, p.rt, "warn", "instance exited (status unknown: restarted outside the activator)")

}

+ p.finaliseGen()

p.exited.Store(true)

// Stop closes the listeners when it observes done; a crash

// nobody is stopping yet leaves them for the sweeper's

@@ -268,8 +334,15 @@ func exitLevel(rec instancectl.ExitRecord, ok bool) string {

return "warn"

}

+func (p *unitProc) recordName() string {

+ if p.gen == 0 {

+ return instancectl.ExitRecordName(p.role, p.invocation)

+ }

+ return instancectl.GenExitRecordName(p.gen, p.invocation)

+}

+

func (p *unitProc) readRecord() (instancectl.ExitRecord, bool) {

- raw, err := instancefs.ReadBeneathLimit(p.runDir, instancectl.ExitRecordName(p.role, p.invocation), 4096)

+ raw, err := instancefs.ReadBeneathLimit(p.runDir, p.recordName(), 4096)

if err != nil {

return instancectl.ExitRecord{}, false

}

@@ -286,7 +359,17 @@ func (p *unitProc) readRecord() (instancectl.ExitRecord, bool) {

// the remove, leaves nothing worse than what prepare's own stale-record

// sweep would have cleared on the NEXT invocation anyway.

func (p *unitProc) removeRecord() {

- _ = os.Remove(filepath.Join(p.runDir, instancectl.ExitRecordName(p.role, p.invocation)))

+ _ = os.Remove(filepath.Join(p.runDir, p.recordName()))

+}

+

+// removeLaunch deletes a generation's launch file once the generation is

+// proven gone. Generation numbers are never reused, so nothing else would.

+// The original templates' launch file is rewritten by every start and is

+// left alone. os.Remove never follows the final component.

+func (p *unitProc) removeLaunch() {

+ if p.gen != 0 && p.dataDir != "" {

+ _ = os.Remove(filepath.Join(p.dataDir, instancectl.GenLaunchFileName(p.gen)))

+ }

}

// cgroupRoot is CgroupRoot with the nil default applied.

diff --git a/internal/activator/unitproc_gen_test.go b/internal/activator/unitproc_gen_test.go

new file mode 100644

index 000000000..552a09c7b

--- /dev/null

+++ b/internal/activator/unitproc_gen_test.go

@@ -0,0 +1,346 @@

+package activator

+

+import (

+ "context"

+ "encoding/json"

+ "errors"

+ "os"

+ "path/filepath"

+ "strings"

+ "sync"

+ "testing"

+ "time"

+

+ "github.com/carlosframework/platform/internal/instancectl"

+ "github.com/carlosframework/platform/internal/provision"

+ "github.com/carlosframework/platform/internal/registry"

+)

+

+func TestPrepareAtAGenerationKeysItsFilesByGeneration(t *testing.T) {

+ ctl := &fakeControl{}

+ b, rt := nestedBackend(t, ctl, &fakeUnit{})

+ b.GenContainment = true

+ p := b.pathsFor(rt)

+ if err := os.MkdirAll(p.runDir, 0o750); err != nil {

+ t.Fatal(err)

+ }

+ for _, name := range []string{"exit.run.aaa.json", "exit.gen.7.bbb.json", "exit.gen.8.ccc.json"} {

+ if err := os.WriteFile(filepath.Join(p.runDir, name), []byte("{}"), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ p2, ctlL, logs, _, err := b.prepareAt(rt, instancectl.RoleRun, nestedGenSocketIn(p.runDir, 7), 7)

+ if err != nil {

+ t.Fatal(err)

+ }

+ defer logs.Close()

+ if ctlL != nil {

+ defer ctlL.Close()

+ }

+ for name, want := range map[string]bool{"exit.run.aaa.json": true, "exit.gen.7.bbb.json": false, "exit.gen.8.ccc.json": true} {

+ if _, err := os.Stat(filepath.Join(p.runDir, name)); (err == nil) != want {

+ t.Errorf("%s present=%v, want %v", name, err == nil, want)

+ }

+ wantSock := filepath.Join(b.RunDirOverride, "logs", provision.InstanceID(rt.Host)+"@7.run.sock")

+ if got := b.logSocketFor(rt.Host, instancectl.RoleRun, 7); got != wantSock {

+ t.Errorf("log socket = %q, want %q", got, wantSock)

+ }

+ if _, err := os.Stat(wantSock); err != nil {

+ t.Errorf("generation log listener not bound: %v", err)

+ }

+ if err := b.writeLaunchFor(p2, instancectl.RoleRun, 7, rt.Host, []string{"/opt/carlos/live/" + rt.Host}, nil); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := os.Stat(filepath.Join(p2.dataDir, "launch.gen.7.json")); err != nil {

+ t.Errorf("launch.gen.7.json: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(p2.dataDir, "launch.run.json")); !os.IsNotExist(err) {

+ t.Errorf("a generation wrote the legacy launch file: %v", err)

+ }

+}

+

+func TestGenerationUnitProcStopsAndWatchesItsOwnUnit(t *testing.T) {

+ ctl := &fakeControl{}

+ unit := &fakeUnit{}

+ b, rt := nestedBackend(t, ctl, unit)

+ b.GenContainment = true

+ b.Control = fakeGenControl{ctl}

+ var mu sync.Mutex

+ var seen []string

+ b.UnitState = func(ctx context.Context, name string) (string, string, error) {

+ mu.Lock()

+ seen = append(seen, name)

+ mu.Unlock()

+ return unit.state(ctx, name)

+ }

+ unit.set("active", "inv7", true, true)

+ ctl.stopHook = func(string, instancectl.Role) { unit.set("inactive", "inv7", false, false) }

+ up := b.newUnitProcFor(rt, instancectl.RoleRun, 7, b.pathsFor(rt), "inv7", nil, nil)

+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)

+ defer cancel()

+ if err := up.Stop(ctx); err != nil {

+ t.Fatal(err)

+ }

+ if got := ctl.joined(); !strings.Contains(got, "stopgen h.example.com 7 kill=false") || strings.Contains(got, "stop h.example.com run") {

+ t.Errorf("calls =\n%s", got)

+ }

+ want := instancectl.GenUnitName(rt.Host, 7)

+ if up.unit != want {

+ t.Errorf("unit = %q, want %q", up.unit, want)

+ }

+ mu.Lock()

+ defer mu.Unlock()

+ if len(seen) == 0 {

+ t.Fatal("the observer never asked about the unit")

+ }

+ for _, name := range seen {

+ if name != want {

+ t.Errorf("the observer asked about %s, want %s", name, want)

+ }

+}

+

+func TestGenerationUnitProcConsumesOnlyItsOwnRecordAndLaunchFile(t *testing.T) {

+ ctl := &fakeControl{}

+ unit := &fakeUnit{}

+ b, rt := nestedBackend(t, ctl, unit)

+ b.GenContainment = true

+ b.Control = fakeGenControl{ctl}

+ p := b.pathsFor(rt)

+ for _, d := range []string{p.runDir, p.dataDir} {

+ if err := os.MkdirAll(d, 0o750); err != nil {

+ t.Fatal(err)

+ }

+ rec, _ := json.Marshal(instancectl.ExitRecord{Invocation: "inv7", Result: "success", Code: "exited", Status: "0"})

+ for dir, names := range map[string][]string{

+ p.runDir: {"exit.gen.7.inv7.json", "exit.run.inv7.json"},

+ p.dataDir: {"launch.gen.7.json", "launch.gen.8.json"},

+ } {

+ for _, name := range names {

+ if err := os.WriteFile(filepath.Join(dir, name), rec, 0o600); err != nil {

+ t.Fatal(err)

+ }

+ unit.set("active", "inv7", true, true)

+ up := b.newUnitProcFor(rt, instancectl.RoleRun, 7, p, "inv7", nil, nil)

+ unit.set("inactive", "inv7", false, false)

+ waitFor(t, "the generation to exit", up.Exited)

+ for path, want := range map[string]bool{

+ filepath.Join(p.runDir, "exit.gen.7.inv7.json"): false,

+ filepath.Join(p.runDir, "exit.run.inv7.json"): true,

+ filepath.Join(p.dataDir, "launch.gen.7.json"): false,

+ filepath.Join(p.dataDir, "launch.gen.8.json"): true,

+ } {

+ if _, err := os.Stat(path); (err == nil) != want {

+ t.Errorf("%s present=%v, want %v", path, err == nil, want)

+ }

+}

+

+func TestStartRoleDispatchesByGeneration(t *testing.T) {

+ ctl := &fakeControl{inv: "inv"}

+ b, rt := nestedBackend(t, ctl, &fakeUnit{})

+ b.Control = fakeGenControl{ctl}

+ if _, err := b.startRole(context.Background(), rt, instancectl.RoleRun, 0); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := b.startRole(context.Background(), rt, instancectl.RoleRun, 7); err != nil {

+ t.Fatal(err)

+ }

+ if want := "start h.example.com run\nstartgen h.example.com 7 536870912"; ctl.joined() != want {

+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)

+ }

+}

+

+func TestUnitGen(t *testing.T) {

+ b := testExecBackend(t)

+ if n, err := b.unitGen(7); n != 0 || err != nil {

+ t.Errorf("containment off: %d %v, want 0", n, err)

+ }

+ b.GenContainment = true

+ if n, err := b.unitGen(7); n != 7 || err != nil {

+ t.Errorf("containment on: %d %v, want 7", n, err)

+ }

+ if _, err := b.unitGen(0); err == nil {

+ t.Error("generation 0 accepted")

+ }

+}

+

+// Flag off: a nested wake starts the original template, with the original

+// launch file, log socket and unit name.

+func TestStartUnitWithContainmentOffKeepsTheOriginalTemplate(t *testing.T) {

+ ctl := &fakeControl{inv: "inv1"}

+ unit := &fakeUnit{}

+ b, rt := nestedBackend(t, ctl, unit)

+ var stopServing func()

+ ctl.onStart = func() { stopServing = serveOK(t, rt.Socket) }

+ defer func() {

+ if stopServing != nil {

+ stopServing()

+ }

+ }()

+ unit.set("active", "inv1", true, true)

+ p, _, err := b.Start(context.Background(), rt)

+ if err != nil {

+ t.Fatalf("Start: %v", err)

+ }

+ up := p.(*unitProc)

+ if up.unit != "carlos-instance@h.example.com.service" || up.gen != 0 {

+ t.Errorf("unit = %q gen %d", up.unit, up.gen)

+ }

+ if strings.Contains(ctl.joined(), "startgen") {

+ t.Errorf("a flag-off wake used the generation verb:\n%s", ctl.joined())

+ }

+ if _, err := os.Stat(b.logSocket(rt.Host, instancectl.RoleRun)); err != nil {

+ t.Errorf("legacy log socket: %v", err)

+ }

+ unit.set("inactive", "inv1", false, false)

+ waitFor(t, "the unit to exit", p.Exited)

+}

+

+// genStartFixture is a nested backend with containment on, ready for a

+// generation-7 start whose answer the test chooses.

+func genStartFixture(t *testing.T, ctl *fakeControl) (*ExecBackend, registry.Route, string, string) {

+ t.Helper()

+ b, rt := nestedBackend(t, ctl, &fakeUnit{})

+ b.GenContainment = true

+ b.Control = fakeGenControl{ctl}

+ launch := filepath.Join(b.pathsFor(rt).dataDir, instancectl.GenLaunchFileName(7))

+ return b, rt, launch, b.logSocketFor(rt.Host, instancectl.RoleRun, 7)

+}

+

+// A generation start that failed may have started the unit (its answer

+// was lost): the backend kills it through StopGen on its own budget, and

+// only then closes its listeners and removes its launch file.

+func TestStartGenFailureKillsTheGenerationBeforeReleasingIt(t *testing.T) {

+ ctl := &fakeControl{startErr: errors.New("instance control /start: 502: show: bus timeout")}

+ b, rt, launch, logSock := genStartFixture(t, ctl)

+ var launchAtStop bool

+ ctl.stopHook = func(string, instancectl.Role) { _, err := os.Stat(launch); launchAtStop = err == nil }

+ _, _, err := b.StartGen(context.Background(), rt, 7, false)

+ if err == nil || errors.Is(err, errGenUnproven) {

+ t.Fatalf("StartGen = %v, want an ordinary start failure", err)

+ }

+ if want := "startgen h.example.com 7 536870912\nstopgen h.example.com 7 kill=true"; ctl.joined() != want {

+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)

+ }

+ if !launchAtStop {

+ t.Error("the launch file was gone before the kill")

+ }

+ if _, err := os.Stat(launch); !os.IsNotExist(err) {

+ t.Errorf("launch file left after a proven stop: %v", err)

+ }

+ if _, err := os.Stat(logSock); !os.IsNotExist(err) {

+ t.Errorf("log listener still bound after a proven stop: %v", err)

+ }

+}

+

+// The kill that follows a failed start fails too: termination is

+// unproven, so the generation's files stay and the error says so.

+func TestStartGenFailureWithAnUnprovenKillKeepsTheGeneration(t *testing.T) {

+ ctl := &fakeControl{startErr: errors.New("instance control socket: context deadline exceeded"), stopGenErr: errors.New("instance control /stop: 502: stop: processes remain")}

+ b, rt, launch, logSock := genStartFixture(t, ctl)

+ _, _, err := b.StartGen(context.Background(), rt, 7, false)

+ if !errors.Is(err, errGenUnproven) {

+ t.Fatalf("StartGen = %v, want errGenUnproven", err)

+ }

+ if _, err := os.Stat(launch); err != nil {

+ t.Errorf("launch file removed while termination is unproven: %v", err)

+ }

+ if _, err := os.Stat(logSock); err != nil {

+ t.Errorf("log listener released while termination is unproven: %v", err)

+ }

+}

+

+// A start that answered no invocation is just as uncertain for a

+// generation: it is killed and proven gone before its files go.

+func TestStartGenWithNoInvocationKillsTheGeneration(t *testing.T) {

+ ctl := &fakeControl{}

+ b, rt, launch, _ := genStartFixture(t, ctl)

+ if _, _, err := b.StartGen(context.Background(), rt, 7, false); err == nil {

+ t.Fatal("a start with no invocation succeeded")

+ }

+ if !strings.Contains(ctl.joined(), "stopgen h.example.com 7 kill=true") {

+ t.Errorf("calls =\n%s", ctl.joined())

+ }

+ if _, err := os.Stat(launch); !os.IsNotExist(err) {

+ t.Errorf("launch file left after a proven stop: %v", err)

+ }

+}

+

+// genProcFixture is a generation-7 unitProc over fakes, with its launch

+// file on disk.

+func genProcFixture(t *testing.T) (*fakeControl, *fakeUnit, *ExecBackend, registry.Route, string) {

+ t.Helper()

+ ctl, unit := &fakeControl{}, &fakeUnit{}

+ b, rt := nestedBackend(t, ctl, unit)

+ b.GenContainment = true

+ b.Control = fakeGenControl{ctl}

+ p := b.pathsFor(rt)

+ if err := os.MkdirAll(p.dataDir, 0o750); err != nil {

+ t.Fatal(err)

+ }

+ launch := filepath.Join(p.dataDir, instancectl.GenLaunchFileName(7))

+ if err := os.WriteFile(launch, []byte("{}"), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ return ctl, unit, b, rt, launch

+}

+

+// A generation that exits on its own is finalised once observe proves it

+// gone: its launch file here, and the control service's drop-in and exe

+// layout through a stop of the inactive unit. A later Stop sends nothing.

+func TestGenerationNaturalExitFinalisesItsFiles(t *testing.T) {

+ ctl, unit, b, rt, launch := genProcFixture(t)

+ unit.set("active", "inv7", true, true)

+ up := b.newUnitProcFor(rt, instancectl.RoleRun, 7, b.pathsFor(rt), "inv7", nil, nil)

+ unit.set("inactive", "inv7", false, false)

+ waitFor(t, "the generation to exit", up.Exited)

+ if want := "stopgen h.example.com 7 kill=false"; ctl.joined() != want {

+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)

+ }

+ if _, err := os.Stat(launch); !os.IsNotExist(err) {

+ t.Errorf("launch file left after a natural exit: %v", err)

+ }

+ if err := up.Stop(context.Background()); err != nil {

+ t.Fatal(err)

+ }

+ if want := "stopgen h.example.com 7 kill=false"; ctl.joined() != want {

+ t.Errorf("a stop after the exit sent more:\n%s", ctl.joined())

+ }

+}

+

+// A generation restarted outside the activator is stopped, and once that

+// is proven its files go too. The stop that proved it already removed the

+// control service's files, so it is not asked twice.

+func TestGenerationForeignInvocationExitFinalisesItsFiles(t *testing.T) {

+ ctl, unit, b, rt, launch := genProcFixture(t)

+ ctl.stopHook = func(string, instancectl.Role) { unit.set("inactive", "", false, false) }

+ unit.set("active", "inv7", true, true)

+ up := b.newUnitProcFor(rt, instancectl.RoleRun, 7, b.pathsFor(rt), "inv7", nil, nil)

+ unit.set("active", "someone-else", true, true)

+ waitFor(t, "the generation to exit", up.Exited)

+ if want := "stopgen h.example.com 7 kill=false"; ctl.joined() != want {

+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)

+ }

+ if _, err := os.Stat(launch); !os.IsNotExist(err) {

+ t.Errorf("launch file left after a foreign invocation was stopped: %v", err)

+ }

+}

+

+// With no generation channel StartGen is never sent, so nothing ran: the

+// launch file written for it goes.

+func TestStartGenRefusedBeforeSendingRemovesTheLaunchFile(t *testing.T) {

+ ctl := &fakeControl{inv: "inv7"}

+ b, rt, launch, _ := genStartFixture(t, ctl)

+ b.Control = ctl // a Control without the generation verbs

+ if _, _, err := b.StartGen(context.Background(), rt, 7, false); !errors.Is(err, errNoGenControl) {

+ t.Fatalf("StartGen = %v, want errNoGenControl", err)

+ }

+ if _, err := os.Stat(launch); !os.IsNotExist(err) {

+ t.Errorf("launch file left by a start never sent: %v", err)

+ }

+}

diff --git a/internal/activator/unitproc_test.go b/internal/activator/unitproc_test.go

index 7b867e7c5..11f55352b 100644

--- a/internal/activator/unitproc_test.go

+++ b/internal/activator/unitproc_test.go

@@ -27,8 +27,10 @@ type fakeControl struct {

inv string

seed instancectl.SeedResponse

startErr error

- stopHook func(host string, role instancectl.Role)

- readyErr error

+ // stopGenErr is what fakeGenControl.StopGen answers.

+ stopGenErr error

+ stopHook func(host string, role instancectl.Role)

+ readyErr error

// onStart runs inside Start, in the fake control's own goroutine, the

// same place the real control service's trampoline would exec the app

// and the app would bind its socket: prepare's stale-socket removal

diff --git a/internal/cgroup/cgroup.go b/internal/cgroup/cgroup.go

index 6194feb67..5c2a74457 100644

--- a/internal/cgroup/cgroup.go

+++ b/internal/cgroup/cgroup.go

@@ -244,7 +244,20 @@ func (m *Manager) sweepStale() {

if !e.IsDir() || !IsExecName(e.Name()) {

continue

}

- if err := os.Remove(filepath.Join(m.root, e.Name())); err != nil {

+ dir := filepath.Join(m.root, e.Name())

+ // A generation parent's children go first: rmdir refuses a cgroup

+ // that has child cgroups, empty or not (zero-downtime spec §7.3).

+ if kids, err := os.ReadDir(dir); err == nil {

+ for _, k := range kids {

+ if !k.IsDir() || !isChildName(k.Name()) {

+ continue

+ }

+ if err := os.Remove(filepath.Join(dir, k.Name())); err != nil {

+ m.logf("cgroup: stale %s/%s not removed: %v", e.Name(), k.Name(), err)

+ }

+ if err := os.Remove(dir); err != nil {

m.logf("cgroup: stale %s not removed: %v", e.Name(), err)

}

@@ -284,7 +297,13 @@ func (m *Manager) Start(host string, limit Limit, build func() *exec.Cmd) (*exec

cmd := build()

return cmd, startCmd(cmd)

}

- dir := m.hostDir(host)

+ return m.startIn(host, m.hostDir(host), limit, build)

+}

+

+// startIn is Start against an explicit cgroup directory: the host's own

+// (Start) or one of its children (StartChild). The clone3 attempt, the

+// fallback and its latch are Start's, unchanged.

+func (m *Manager) startIn(host, dir string, limit Limit, build func() *exec.Cmd) (*exec.Cmd, error) {

if err := os.MkdirAll(dir, 0o755); err != nil {

m.logf("cgroup: %s: create cgroup: %v — spawning unaccounted", host, err)

cmd := build()

@@ -295,7 +314,7 @@ func (m *Manager) Start(host string, limit Limit, build func() *exec.Cmd) (*exec

f, err := os.Open(dir)

if err != nil {

m.logf("cgroup: %s: open cgroup dir: %v — using PID-move fallback for this spawn", host, err)

- return m.startAndMove(host, build)

+ return m.startAndMoveIn(host, dir, build)

}

cmd := build()

setInto(cmd, int(f.Fd()))

@@ -306,9 +325,9 @@ func (m *Manager) Start(host string, limit Limit, build func() *exec.Cmd) (*exec

return cmd, nil

}

firstErr := err

- retried, retryErr := m.startAndMove(host, build)

+ retried, retryErr := m.startAndMoveIn(host, dir, build)

if retryErr != nil {

- // startAndMove already dropped the ghost host dir on its own

+ // startAndMoveIn already dropped the ghost dir on its own

// failure; join both errors so the operator can see a clone3

// attempt happened at all, not just that the retry failed.

return nil, errors.Join(firstErr, retryErr)

@@ -321,7 +340,7 @@ func (m *Manager) Start(host string, limit Limit, build func() *exec.Cmd) (*exec

}

return retried, nil

}

- return m.startAndMove(host, build)

+ return m.startAndMoveIn(host, dir, build)

}

// applyLimit writes limit into dir. Best-effort in the package's usual

@@ -365,27 +384,29 @@ func (m *Manager) applyLimit(host, dir string, limit Limit) {

m.logf("cgroup: %s: memory.max=%dM memory.high=%dM", host, limit.MemoryMax>>20, HighOf(limit.MemoryMax)>>20)

}

-// startAndMove is the fallback spawn: plain start, then write the child's

-// PID into the host cgroup. A failed move is logged and tolerated — the

-// child runs (unaccounted), which beats not running.

func (m *Manager) startAndMove(host string, build func() *exec.Cmd) (*exec.Cmd, error) {

+ return m.startAndMoveIn(host, m.hostDir(host), build)

+}

+

+// startAndMoveIn is startAndMove into an explicit cgroup directory.

+func (m *Manager) startAndMoveIn(host, dir string, build func() *exec.Cmd) (*exec.Cmd, error) {

cmd := build()

if err := startCmd(cmd); err != nil {

- // The spawn never happened: drop the now-ownerless host dir so a

- // failed wake doesn't leave the sampler reporting permanent zeros

- // for a host with nothing running. rmdir-safe — os.Remove refuses a

+ // The spawn never happened: drop the now-ownerless dir so a failed

+ // wake doesn't leave the sampler reporting permanent zeros for a

+ // host with nothing running. rmdir-safe: os.Remove refuses a

// non-empty dir, so this can never take a live sibling's dir.

- _ = os.Remove(m.hostDir(host))

+ _ = os.Remove(dir)

return nil, err

}

if cmd.Process == nil {

// A broken startCmd seam reporting success with no process: this

// package must never panic the agent over it. Log and skip the

- // move — the child, if any, simply stays unaccounted.

+ // move; the child, if any, simply stays unaccounted.

m.logf("cgroup: %s: start reported success with no process — skipping pid move", host)

return cmd, nil

}

- procs := filepath.Join(m.hostDir(host), "cgroup.procs")

+ procs := filepath.Join(dir, "cgroup.procs")

if err := os.WriteFile(procs, []byte(strconv.Itoa(cmd.Process.Pid)), 0o644); err != nil {

m.logf("cgroup: %s: move pid %d: %v — child stays unaccounted", host, cmd.Process.Pid, err)

}

diff --git a/internal/cgroup/cleanup_linux.go b/internal/cgroup/cleanup_linux.go

index b5ceee2db..f11b99e96 100644

--- a/internal/cgroup/cleanup_linux.go

+++ b/internal/cgroup/cleanup_linux.go

@@ -7,7 +7,9 @@ import (

"context"

"errors"

"fmt"

+ "io"

"os"

+ "slices"

"strconv"

"strings"

"time"

@@ -34,14 +36,26 @@ func cleanupFilesystem(root *os.File) error {

}

func (m *Manager) OpenCleanup(host string) (*Cleanup, error) {

- return m.openCleanup(host, cleanupFilesystem)

+ return m.openCleanupAt(host, "", cleanupFilesystem)

}

-func (m *Manager) openCleanup(host string, checkFS func(*os.File) error) (out *Cleanup, err error) {

+// OpenCleanupChild pins one child of host's generation parent (zero-

+// downtime spec §7.3): its KillAndEmpty writes that child's cgroup.kill

+// and proves that child empty, and never touches a sibling. ErrNoCgroup

+// when the child is not there.

+func (m *Manager) OpenCleanupChild(host, child string) (*Cleanup, error) {

+ return m.openCleanupAt(host, child, cleanupFilesystem)

+}

+

+func (m *Manager) openCleanup(host string, checkFS func(*os.File) error) (*Cleanup, error) {

+ return m.openCleanupAt(host, "", checkFS)

+}

+

+func (m *Manager) openCleanupAt(host, child string, checkFS func(*os.File) error) (out *Cleanup, err error) {

if m == nil || m.root == "" {

return nil, ErrNoDelegation

}

- if !controlname.ValidHostShape(host) {

+ if !controlname.ValidHostShape(host) || (child != "" && !isChildName(child)) {

return nil, ErrStrictRefused

}

root, err := os.OpenFile(m.root, os.O_RDONLY|unix.O_DIRECTORY|unix.O_NOFOLLOW, 0)

@@ -68,6 +82,18 @@ func (m *Manager) openCleanup(host string, checkFS func(*os.File) error) (out *C

if err != nil {

return nil, err

}

+ if child != "" {

+ // The child is pinned beneath the parent's own descriptor, and its

+ // identity is checked against that descriptor from now on.

+ p.entryParent, p.entryName = p.dir, child

+ p.dir, err = strictOpen(p.entryParent, child, unix.O_RDONLY|unix.O_DIRECTORY)

+ if errors.Is(err, unix.ENOENT) {

+ return nil, ErrNoCgroup

+ }

+ if err != nil {

+ return nil, err

+ }

p.identity, err = strictID(p.dir)

if err != nil {

return nil, err

@@ -125,32 +151,132 @@ func (c *Cleanup) Members() ([]int, error) {

return c.membersLocked()

}

+// Cleanup's walk is bounded: a tree deeper or wider than this is refused

+// rather than vouched for unread.

+const (

+ maxCleanupDepth = 8

+ maxCleanupGroups = 1024

+)

+

+// membersLocked lists every process in the pinned cgroup AND in every

+// cgroup below it. A generation parent holds no process itself (zero-

+// downtime spec §7.3), so stop evidence taken on it must read its

+// generations'. A flat tenant runs as the agent's uid and can make cgroups

+// of its own under its generation, so the walk goes to any depth, bounded.

func (c *Cleanup) membersLocked() ([]int, error) {

p := c.state.pin

if err := p.identityOK(); err != nil {

return nil, err

}

- if _, err := c.state.procs.Seek(0, 0); err != nil {

+ members, err := readProcs(c.state.procs)

+ if err != nil {

+ return nil, err

+ }

+ groups := 0

+ if err := walkMembers(p.dir, 1, &groups, &members); err != nil {

+ return nil, err

+ }

+ slices.Sort(members)

+ if err := p.identityOK(); err != nil {

+ return nil, err

+ }

+ return members, nil

+}

+

+// readProcs reads every PID in one cgroup.procs. It may exceed

+// strictRead's 4096-byte bound, so every line is scanned.

+func readProcs(f *os.File) ([]int, error) {

+ if _, err := f.Seek(0, 0); err != nil {

return nil, err

}

- // cgroup.procs may exceed strictRead's 4096-byte bound. Scan every line.

- scan := bufio.NewScanner(c.state.procs)

- var members []int

+ scan := bufio.NewScanner(f)

+ var pids []int

for scan.Scan() {

raw := strings.TrimSpace(scan.Text())

pid, err := strconv.Atoi(raw)

if err != nil || pid <= 0 {

return nil, fmt.Errorf("%w: invalid cgroup member", ErrStrictRefused)

}

- members = append(members, pid)

+ pids = append(pids, pid)

}

- if err := scan.Err(); err != nil {

- return nil, err

+ return pids, scan.Err()

+}

+

+// cleanupReadBatch is how many directory entries one read takes, so the

+// group budget is checked as the listing is read, not after all of it.

+const cleanupReadBatch = 64

+

+// walkMembers appends the members of every cgroup below dir. A child that

+// vanishes between the listing and its open was removed, so it is empty.

+func walkMembers(dir *os.File, depth int, groups *int, members *[]int) error {

+ names, err := childGroups(dir, depth, groups)

+ if err != nil {

+ return err

}

- if err := p.identityOK(); err != nil {

+ for _, name := range names {

+ child, err := strictOpen(dir, name, unix.O_RDONLY|unix.O_DIRECTORY)

+ if errors.Is(err, unix.ENOENT) {

+ continue

+ }

+ if err != nil {

+ return err

+ }

+ procs, err := strictOpen(child, "cgroup.procs", unix.O_RDONLY)

+ if errors.Is(err, unix.ENOENT) {

+ child.Close()

+ continue

+ }

+ if err != nil {

+ child.Close()

+ return err

+ }

+ pids, err := readProcs(procs)

+ procs.Close()

+ if err == nil {

+ *members = append(*members, pids...)

+ err = walkMembers(child, depth+1, groups, members)

+ }

+ child.Close()

+ if err != nil {

+ return err

+ }

+ return nil

+}

+

+// childGroups lists the child cgroups of dir in bounded batches, counting

+// each against the walk's budget as it is read. A tree past the depth or

+// group bound is refused before any more of it is read, so a tenant's many

+// empty cgroups cost at most the budget.

+func childGroups(dir *os.File, depth int, groups *int) ([]string, error) {

+ fd, err := unix.Openat(int(dir.Fd()), ".", unix.O_RDONLY|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)

+ if err != nil {

return nil, err

}

- return members, nil

+ list := os.NewFile(uintptr(fd), dir.Name())

+ defer list.Close()

+ var names []string

+ for {

+ entries, err := list.ReadDir(cleanupReadBatch)

+ for _, e := range entries {

+ if !e.IsDir() {

+ continue

+ }

+ if depth > maxCleanupDepth {

+ return nil, fmt.Errorf("%w: cgroup tree deeper than %d", ErrStrictRefused, maxCleanupDepth)

+ }

+ if *groups++; *groups > maxCleanupGroups {

+ return nil, fmt.Errorf("%w: more than %d cgroups below one host", ErrStrictRefused, maxCleanupGroups)

+ }

+ names = append(names, e.Name())

+ }

+ if errors.Is(err, io.EOF) {

+ return names, nil

+ }

+ if err != nil {

+ return nil, err

+ }

}

func (c *Cleanup) KillAndEmpty(ctx context.Context) error {

@@ -227,7 +353,7 @@ func (c *Cleanup) Close() error {

}

p.closed = true

var errs []error

- for _, f := range []*os.File{c.state.procs, p.events, p.kill, p.dir, p.root} {

+ for _, f := range []*os.File{c.state.procs, p.events, p.kill, p.dir, p.entryParent, p.root} {

if f != nil {

errs = append(errs, f.Close())

}

diff --git a/internal/cgroup/cleanup_linux_test.go b/internal/cgroup/cleanup_linux_test.go

index 8e774dcc8..b3c00601d 100644

--- a/internal/cgroup/cleanup_linux_test.go

+++ b/internal/cgroup/cleanup_linux_test.go

@@ -8,6 +8,7 @@ import (

"os"

"path/filepath"

"reflect"

+ "strconv"

"testing"

"time"

)

@@ -186,3 +187,129 @@ func TestCleanupRejectsMissingControlsAndSymlinks(t *testing.T) {

})

}

+

+func mkCgroupFixture(t *testing.T, dir, procs, events string) {

+ t.Helper()

+ if err := os.MkdirAll(dir, 0o700); err != nil {

+ t.Fatal(err)

+ }

+ for name, body := range map[string]string{"cgroup.events": events, "cgroup.procs": procs, "cgroup.kill": ""} {

+ if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o600); err != nil {

+ t.Fatal(err)

+ }

+}

+

+func fixtureCleanupAt(t *testing.T, root, host, child string) *Cleanup {

+ t.Helper()

+ c, err := (&Manager{root: root}).openCleanupAt(host, child, func(*os.File) error { return nil })

+ if err != nil {

+ t.Fatal(err)

+ }

+ t.Cleanup(func() { _ = c.Close() })

+ return c

+}

+

+// Stop evidence is taken on the parent (spec §7.3). A generation parent's

+// own cgroup.procs is always empty; its generations' are not.

+func TestCleanupMembersReachEveryGeneration(t *testing.T) {

+ host, root := "gens.example", t.TempDir()

+ parent := filepath.Join(root, ExecName(host))

+ mkCgroupFixture(t, parent, "", "populated 1\n")

+ mkCgroupFixture(t, filepath.Join(parent, "g1"), "11\n", "populated 1\n")

+ mkCgroupFixture(t, filepath.Join(parent, "g2"), "22\n", "populated 1\n")

+ // Review focus 3: a flat tenant runs as the agent's uid and can make

+ // cgroups of its own under its generation.

+ mkCgroupFixture(t, filepath.Join(parent, "g2", "tenant"), "33\n", "populated 1\n")

+ c := fixtureCleanupAt(t, root, host, "")

+ members, err := c.Members()

+ if err != nil || !reflect.DeepEqual(members, []int{11, 22, 33}) {

+ t.Fatalf("members = %v, %v; want every generation's", members, err)

+ }

+}

+

+func TestCleanupMembersRefuseATreeTooDeep(t *testing.T) {

+ host, root := "deep.example", t.TempDir()

+ dir := filepath.Join(root, ExecName(host))

+ mkCgroupFixture(t, dir, "", "populated 1\n")

+ for i := range maxCleanupDepth + 1 {

+ dir = filepath.Join(dir, "d"+strconv.Itoa(i))

+ mkCgroupFixture(t, dir, "", "populated 0\n")

+ }

+ c := fixtureCleanupAt(t, root, host, "")

+ if _, err := c.Members(); !errors.Is(err, ErrStrictRefused) {

+ t.Fatalf("a tree deeper than %d: %v, want ErrStrictRefused", maxCleanupDepth, err)

+ }

+}

+

+// The group budget is a width bound too, and it is enforced while the

+// directory is read: a tree one group too wide is refused before any child

+// is opened. Every child here is unreadable (mode 0), so a walk that

+// opened one before counting them all would fail with EACCES instead.

+func TestCleanupMembersRefuseATreeTooWide(t *testing.T) {

+ if os.Geteuid() == 0 {

+ t.Skip("root opens a mode-0 directory; the probe needs an ordinary user")

+ }

+ host, root := "wide.example", t.TempDir()

+ dir := filepath.Join(root, ExecName(host))

+ mkCgroupFixture(t, dir, "", "populated 1\n")

+ for i := range maxCleanupGroups + 1 {

+ if err := os.Mkdir(filepath.Join(dir, "w"+strconv.Itoa(i)), 0); err != nil {

+ t.Fatal(err)

+ }

+ c := fixtureCleanupAt(t, root, host, "")

+ if _, err := c.Members(); !errors.Is(err, ErrStrictRefused) {

+ t.Fatalf("%d groups at one level: %v, want ErrStrictRefused before any child is opened", maxCleanupGroups+1, err)

+ }

+}

+

+func TestCleanupMembersAcceptATreeExactlyAtTheBound(t *testing.T) {

+ host, root := "bound.example", t.TempDir()

+ dir := filepath.Join(root, ExecName(host))

+ mkCgroupFixture(t, dir, "", "populated 1\n")

+ for i := range maxCleanupGroups {

+ if err := os.Mkdir(filepath.Join(dir, "w"+strconv.Itoa(i)), 0o700); err != nil {

+ t.Fatal(err)

+ }

+ mkCgroupFixture(t, filepath.Join(dir, "w7"), "77\n", "populated 1\n")

+ c := fixtureCleanupAt(t, root, host, "")

+ if members, err := c.Members(); err != nil || !reflect.DeepEqual(members, []int{77}) {

+ t.Fatalf("%d groups: %v, %v; want [77]", maxCleanupGroups, members, err)

+ }

+}

+

+// KillAndEmpty on a generation acts only on that generation's child.

+func TestCleanupChildKillsOnlyItsGeneration(t *testing.T) {

+ host, root := "kill.example", t.TempDir()

+ parent := filepath.Join(root, ExecName(host))

+ mkCgroupFixture(t, parent, "", "populated 1\n")

+ mkCgroupFixture(t, filepath.Join(parent, "g1"), "11\n", "populated 1\n")

+ mkCgroupFixture(t, filepath.Join(parent, "g2"), "", "populated 0\n")

+ c := fixtureCleanupAt(t, root, host, "g2")

+ ctx, cancel := context.WithTimeout(t.Context(), time.Second)

+ defer cancel()

+ if err := c.KillAndEmpty(ctx); err != nil {

+ t.Fatal(err)

+ }

+ for dir, want := range map[string]string{filepath.Join(parent, "g2"): "1", filepath.Join(parent, "g1"): "", parent: ""} {

+ if got := readFixture(t, filepath.Join(dir, "cgroup.kill")); got != want {

+ t.Errorf("%s/cgroup.kill = %q, want %q", dir, got, want)

+ }

+}

+

+func TestOpenCleanupChildRefusesForeignNamesAndMissingChildren(t *testing.T) {

+ host, root := "names.example", t.TempDir()

+ mkCgroupFixture(t, filepath.Join(root, ExecName(host)), "", "populated 0\n")

+ m := &Manager{root: root}

+ for _, child := range []string{"child", "../x", "g0"} {

+ if _, err := m.openCleanupAt(host, child, func(*os.File) error { return nil }); !errors.Is(err, ErrStrictRefused) {

+ t.Errorf("child %q: %v, want ErrStrictRefused", child, err)

+ }

+ if _, err := m.openCleanupAt(host, "g9", func(*os.File) error { return nil }); !errors.Is(err, ErrNoCgroup) {

+ t.Errorf("missing g9: %v, want ErrNoCgroup", err)

+ }

+}

diff --git a/internal/cgroup/cleanup_other.go b/internal/cgroup/cleanup_other.go

index 6b7c33abf..c76077b61 100644

--- a/internal/cgroup/cleanup_other.go

+++ b/internal/cgroup/cleanup_other.go

@@ -12,6 +12,14 @@ func (m *Manager) OpenCleanup(string) (*Cleanup, error) {

}

return nil, ErrStrictUnsupported

}

+

+func (m *Manager) OpenCleanupChild(string, string) (*Cleanup, error) {

+ if m == nil || m.root == "" {

+ return nil, ErrNoDelegation

+ }

+ return nil, ErrStrictUnsupported

+}

+

func (*Cleanup) Populated() (bool, error) { return false, ErrStrictUnsupported }

func (*Cleanup) Members() ([]int, error) { return nil, ErrStrictUnsupported }

func (*Cleanup) KillAndEmpty(context.Context) error { return ErrStrictUnsupported }

diff --git a/internal/cgroup/gen.go b/internal/cgroup/gen.go

new file mode 100644

index 000000000..997357278

--- /dev/null

+++ b/internal/cgroup/gen.go

@@ -0,0 +1,200 @@

+package cgroup

+

+import (

+ "errors"

+ "fmt"

+ "os"

+ "os/exec"

+ "path/filepath"

+ "slices"

+ "strconv"

+ "strings"

+

+ "github.com/carlosframework/platform/internal/controlname"

+)

+

+// The generation layout (zero-downtime deploys spec §7.3). A host's

+// carlos-exec@<host> becomes a PARENT that holds no processes. Each

+// instance spawn lands in its own generation child g<n>, which carries the

+// tier's full limits and memory.oom.group=1, so a successor that overruns

+// dies alone and the incumbent is untouched. The legacy co-spawned sidecar

+// and the seed land in role- children with no limit of their own. The

+// parent carries the tier with memory.oom.group=0: a backstop for the

+// instance as a whole, raised to OverlapLimit while two generations run.

+// Nothing here runs unless the exec backend's GenContainment is set

+// (plan 5's CARLOS_SWAP).

+

+// GenChild is generation n's child cgroup: "g<n>".

+func GenChild(n uint64) string { return "g" + strconv.FormatUint(n, 10) }

+

+// RoleChild is the child cgroup a non-generation spawn of a host lands in

+// under the generation layout: "role-<role>", the name PrepareRoleChild

+// pins.

+func RoleChild(role string) string { return "role-" + role }

+

+// isChildName reports whether name is one of the layout's child names:

+// g<n> with n canonical and above zero, or role-<label>. Only these are

+// ever created, removed or swept.

+func isChildName(name string) bool {

+ if rest, ok := strings.CutPrefix(name, "role-"); ok {

+ return controlname.ValidLabel(rest)

+ }

+ rest, ok := strings.CutPrefix(name, "g")

+ if !ok || rest == "" {

+ return false

+ }

+ n, err := strconv.ParseUint(rest, 10, 64)

+ return err == nil && n > 0 && strconv.FormatUint(n, 10) == rest

+}

+

+// isGenName reports whether name is a generation child's name.

+func isGenName(name string) bool { return strings.HasPrefix(name, "g") && isChildName(name) }

+

+// OverlapLimit is the parent's limit while two generations overlap: the two

+// tiers summed. With equal tiers that is the spec's "twice the tier". When

+// a release changes tier it still has room for both (plan 2, D3).

+func OverlapLimit(incumbent, successor Limit) Limit {

+ return Limit{MemoryMax: incumbent.MemoryMax + successor.MemoryMax}

+}

+

+// ValidParentLimit reports whether l is a limit a generation parent may

+// carry: one tier, or the sum of two. Strict containment and the root

+// control service accept nothing else.

+func ValidParentLimit(l Limit) bool {

+ tiers := []int64{TierDefault.MemoryMax, TierChildren.MemoryMax}

+ for _, a := range tiers {

+ if l.MemoryMax == a {

+ return true

+ }

+ for _, b := range tiers {

+ if l.MemoryMax == a+b {

+ return true

+ }

+ return false

+}

+

+// delegatedControllers is the subtree_control write that hands a parent's

+// children every controller of cpu, memory, io and pids that the parent

+// itself has (its cgroup.controllers). A user manager delegates no io, for

+// one, and a write naming a controller the parent lacks fails as a whole.

+func delegatedControllers(have string) string {

+ fields := strings.Fields(have)

+ var want []string

+ for _, c := range []string{"cpu", "memory", "io", "pids"} {

+ if slices.Contains(fields, c) {

+ want = append(want, "+"+c)

+ }

+ return strings.Join(want, " ")

+}

+

+// enableChildren delegates dir's controllers to its children. Memory must

+// be among them, or a generation's limits could not exist.

+func enableChildren(dir string) error {

+ have, err := os.ReadFile(filepath.Join(dir, "cgroup.controllers"))

+ if err != nil {

+ return err

+ }

+ want := delegatedControllers(string(have))

+ if !strings.Contains(want, "+memory") {

+ return errors.New("no memory controller to delegate")

+ }

+ return os.WriteFile(filepath.Join(dir, "cgroup.subtree_control"), []byte(want), 0o644)

+}

+

+// parentPopulated reads dir's cgroup.events. An absent file means nothing

+// has ever run there: not populated. A file that cannot be read counts as

+// populated, so a raised limit is never lowered on a guess.

+func parentPopulated(dir string) bool {

+ b, err := os.ReadFile(filepath.Join(dir, "cgroup.events"))

+ if errors.Is(err, os.ErrNotExist) {

+ return false

+ }

+ if err != nil {

+ return true

+ }

+ return !strings.Contains(string(b), "populated 0")

+}

+

+// ensureParent makes carlos-exec@<host> a generation parent. When nothing

+// is alive under it (a cold wake), it delegates the controllers and writes

+// the tier with memory.oom.group=0. With a live generation under it, the

+// parent is left as it is, because a swap raised it on purpose. Best-effort,

+// like the rest of the accounting path: a failure is one logged line and

+// children that run without their limits, never a refused spawn.

+func (m *Manager) ensureParent(host string, tier Limit) {

+ dir := m.hostDir(host)

+ if err := os.MkdirAll(dir, 0o755); err != nil {

+ m.logf("cgroup: %s: create parent cgroup: %v (children run unaccounted)", host, err)

+ return

+ }

+ if parentPopulated(dir) {

+ return

+ }

+ if err := enableChildren(dir); err != nil {

+ m.logf("cgroup: %s: delegate controllers to generations: %v (children run uncapped)", host, err)

+ return

+ }

+ if tier.MemoryMax > 0 {

+ if err := m.SetParentLimit(host, tier); err != nil {

+ m.logf("cgroup: %s: parent limit: %v", host, err)

+ }

+}

+

+// StartChild is Start for the generation layout: the child lands in

+// carlos-exec@<host>/<child>. own is the child's own limit: the tier for a

+// generation, zero for a role child, which then lives under the parent's

+// limit alone. parent is the tier the parent takes when this spawn finds

+// nothing alive under it. Nil-safe and best-effort like Start.

+func (m *Manager) StartChild(host, child string, parent, own Limit, build func() *exec.Cmd) (*exec.Cmd, error) {

+ if !isChildName(child) {

+ return nil, fmt.Errorf("cgroup: %s: %q is not a child cgroup name", host, child)

+ }

+ if m == nil {

+ cmd := build()

+ return cmd, startCmd(cmd)

+ }

+ m.ensureParent(host, parent)

+ return m.startIn(host, filepath.Join(m.hostDir(host), child), own, build)

+}

+

+// SetParentLimit writes l onto host's parent: memory.max, memory.high

+// (HighOf), memory.swap.max, and memory.oom.group=0 (spec §7.3). The swap

+// raises it to OverlapLimit before the successor spawns and restores the

+// tier once the retiring child is empty. Unlike applyLimit it reports the

+// first failure, because a raise that did not land must stop the overlap.

+// A nil manager has no parent to set and answers nil.

+func (m *Manager) SetParentLimit(host string, l Limit) error {

+ if m == nil {

+ return nil

+ }

+ if l.MemoryMax <= 0 {

+ return fmt.Errorf("cgroup: %s: parent limit %d is not a limit", host, l.MemoryMax)

+ }

+ dir := m.hostDir(host)

+ for _, w := range []struct{ name, value string }{

+ {"memory.max", strconv.FormatInt(l.MemoryMax, 10)},

+ {"memory.high", strconv.FormatInt(HighOf(l.MemoryMax), 10)},

+ {"memory.swap.max", strconv.FormatInt(l.MemoryMax, 10)},

+ {"memory.oom.group", "0"},

+ } {

+ if err := os.WriteFile(filepath.Join(dir, w.name), []byte(w.value), 0o644); err != nil {

+ return fmt.Errorf("cgroup: %s: parent %s: %w", host, w.name, err)

+ }

+ m.logf("cgroup: %s: parent memory.max=%dM", host, l.MemoryMax>>20)

+ return nil

+}

+

+// RemoveChild removes host's child cgroup once it is empty. Nil-safe and

+// rmdir-tolerant like CleanupHost, and it touches only the layout's own

+// child names. The parent is CleanupHost's to remove.

+func (m *Manager) RemoveChild(host, child string) {

+ if m == nil || !isChildName(child) {

+ return

+ }

+ _ = os.Remove(filepath.Join(m.hostDir(host), child))

+}

diff --git a/internal/cgroup/gen_kernel_linux_test.go b/internal/cgroup/gen_kernel_linux_test.go

new file mode 100644

index 000000000..d6fc09c80

--- /dev/null

+++ b/internal/cgroup/gen_kernel_linux_test.go

@@ -0,0 +1,479 @@

+//go:build linux

+

+package cgroup

+

+import (

+ "bufio"

+ "context"

+ "errors"

+ "fmt"

+ "io"

+ "os"

+ "os/exec"

+ "path/filepath"

+ "strconv"

+ "strings"

+ "sync"

+ "syscall"

+ "testing"

+ "time"

+)

+

+// The generation layout against a real kernel (zero-downtime plan 2, Task

+// 4). Opt-in: CARLOS_GEN_KERNEL=1 inside a disposable unit with

+// Delegate=yes (the plan's recipe). Without the variable, or without a

+// delegated cgroup v2 subtree to work in, every test here skips.

+

+func genKernelManager(t *testing.T) *Manager {

+ t.Helper()

+ if os.Getenv("CARLOS_GEN_KERNEL") != "1" {

+ t.Skip("set CARLOS_GEN_KERNEL=1 inside a unit with Delegate=yes (zero-downtime plan 2, Task 4)")

+ }

+ raw, err := os.ReadFile("/proc/self/cgroup")

+ if err != nil {

+ t.Skipf("cgroup v2 delegation unavailable: %v", err)

+ }

+ rel, ok := ParseSelfCgroup(string(raw))

+ if !ok {

+ t.Skip("cgroup v2 delegation unavailable: no unified cgroup path")

+ }

+ root := filepath.Join("/sys/fs/cgroup", rel)

+ if filepath.Base(root) == "supervise" {

+ root = filepath.Dir(root) // an earlier test in this process moved us

+ }

+ sup := filepath.Join(root, "supervise")

+ if err := os.MkdirAll(sup, 0o755); err != nil {

+ t.Skipf("cgroup v2 delegation unavailable: %v", err)

+ }

+ if err := os.WriteFile(filepath.Join(sup, "cgroup.procs"), []byte(strconv.Itoa(os.Getpid())), 0o644); err != nil {

+ t.Skipf("cgroup v2 delegation unavailable: %v", err)

+ }

+ if err := enableChildren(root); err != nil {

+ t.Skipf("cgroup v2 delegation unavailable: %v", err)

+ }

+ return &Manager{root: root, logf: t.Logf}

+}

+

+func genKernelCmd(mode string) *exec.Cmd {

+ cmd := exec.Command(os.Args[0], "-test.run=^TestGenKernelChild$", "-test.count=1")

+ cmd.Env = []string{"CARLOS_GEN_CHILD=" + mode}

+ cmd.Stderr = os.Stderr // a child that fails says why

+ return cmd

+}

+

+// TestGenKernelChild is the synthetic process: "echo" answers each line it

+// reads, standing in for an incumbent serving requests; "hog" touches

+// memory until something kills it.

+func TestGenKernelChild(t *testing.T) {

+ switch os.Getenv("CARLOS_GEN_CHILD") {

+ case "echo":

+ fmt.Println("ready")

+ sc := bufio.NewScanner(os.Stdin)

+ for sc.Scan() {

+ fmt.Println(sc.Text())

+ }

+ os.Exit(0)

+ case "hog":

+ // Wait for the parent's go-ahead, which it gives only after it has

+ // checked that this process is inside its capped cgroup. EOF means

+ // the parent is gone: exit rather than allocate unwatched.

+ if _, err := bufio.NewReader(os.Stdin).ReadString('\n'); err != nil {

+ fmt.Fprintln(os.Stderr, "hog: no go-ahead:", err)

+ os.Exit(4)

+ }

+ // One populating mmap: the kernel faults every page in without

+ // returning to user space, so memory.high's throttle (a sleep on

+ // each return) cannot slow the climb to memory.max. A hog that

+ // touches pages one at a time crawls at the throttle, swaps, and

+ // can sit under the limit for minutes instead of dying.

+ mem, err := syscall.Mmap(-1, 0, 256<<20, syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_ANON|syscall.MAP_PRIVATE|syscall.MAP_POPULATE)

+ if err != nil {

+ fmt.Fprintln(os.Stderr, "hog: mmap:", err)

+ os.Exit(2)

+ }

+ _ = mem

+ // Bounded: a hog that survives 1 GiB in all was not confined.

+ hold := make([][]byte, 0, 192)

+ for len(hold) < cap(hold) {

+ b := make([]byte, 4<<20)

+ for i := 0; i < len(b); i += 4096 {

+ b[i] = 1

+ }

+ hold = append(hold, b)

+ }

+ fmt.Fprintln(os.Stderr, "hog: survived 1 GiB: not confined")

+ os.Exit(3)

+ default:

+ t.Skip("synthetic child only")

+ }

+}

+

+type echoChild struct {

+ in io.WriteCloser

+ out *bufio.Reader

+}

+

+func (e *echoChild) answer(t *testing.T, send, want string) {

+ t.Helper()

+ got := make(chan string, 1)

+ go func() {

+ if send != "" {

+ fmt.Fprintln(e.in, send)

+ }

+ line, _ := e.out.ReadString('\n')

+ got <- line

+ }()

+ select {

+ case line := <-got:

+ if line != want+"\n" {

+ t.Fatalf("incumbent answered %q, want %q", line, want)

+ }

+ case <-time.After(5 * time.Second):

+ t.Fatal("incumbent stopped answering")

+ }

+}

+

+func (e *echoChild) ping(t *testing.T) { t.Helper(); e.answer(t, "ping", "ping") }

+

+func echoPipes(t *testing.T, cmd *exec.Cmd) *echoChild {

+ t.Helper()

+ in, err := cmd.StdinPipe()

+ if err != nil {

+ t.Fatal(err)

+ }

+ out, err := cmd.StdoutPipe()

+ if err != nil {

+ t.Fatal(err)

+ }

+ return &echoChild{in: in, out: bufio.NewReader(out)}

+}

+

+func expectFile(t *testing.T, path, want string) {

+ t.Helper()

+ b, err := os.ReadFile(path)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if got := strings.TrimSpace(string(b)); got != want {

+ t.Fatalf("%s = %q, want %q", path, got, want)

+ }

+}

+

+func oomKills(t *testing.T, dir string) int {

+ t.Helper()

+ raw, err := os.ReadFile(filepath.Join(dir, "memory.events"))

+ if err != nil {

+ t.Fatal(err)

+ }

+ for _, line := range strings.Split(string(raw), "\n") {

+ if f := strings.Fields(line); len(f) == 2 && f[0] == "oom_kill" {

+ n, _ := strconv.Atoi(f[1])

+ return n

+ }

+ t.Fatalf("%s/memory.events has no oom_kill line", dir)

+ return 0

+}

+

+// cgroupOf returns pid's unified cgroup. It reports rather than fails, so

+// a caller holding a process that must not run on can kill it first.

+func cgroupOf(pid int) (string, error) {

+ raw, err := os.ReadFile(fmt.Sprintf("/proc/%d/cgroup", pid))

+ if err != nil {

+ return "", err

+ }

+ rel, ok := ParseSelfCgroup(string(raw))

+ if !ok {

+ return "", fmt.Errorf("pid %d has no unified cgroup", pid)

+ }

+ return rel, nil

+}

+

+// kernelKid is a synthetic child the test started. Its one waiter runs at

+// most once, so the test body and the cleanup can both wait on it.

+type kernelKid struct {

+ cmd *exec.Cmd

+ once sync.Once

+ done chan struct{}

+ err error

+}

+

+func (k *kernelKid) reaped() <-chan struct{} {

+ k.once.Do(func() {

+ go func() { k.err = k.cmd.Wait(); close(k.done) }()

+ })

+ return k.done

+}

+

+func (k *kernelKid) wait() error { <-k.reaped(); return k.err }

+

+// reapOnCleanup is registered right after a spawn, before anything that

+// can fail: whatever the test's outcome, the child ends. It tries the

+// cgroup's own kill first, on a fresh budget, then kills the process

+// directly, closes the pipes the test holds, and reaps on a bounded wait.

+// A child the spawn never started is left alone.

+func reapOnCleanup(t *testing.T, m *Manager, host, child string, cmd *exec.Cmd, pipes ...io.Closer) *kernelKid {

+ t.Helper()

+ k := &kernelKid{cmd: cmd, done: make(chan struct{})}

+ t.Cleanup(func() {

+ closePipes := func() {

+ for _, p := range pipes {

+ if p != nil {

+ _ = p.Close()

+ }

+ if cmd == nil || cmd.Process == nil {

+ closePipes()

+ return

+ }

+ if c, err := m.OpenCleanupChild(host, child); err == nil {

+ ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)

+ if err := c.KillAndEmpty(ctx); err != nil {

+ t.Logf("cleanup: %s/%s: cgroup kill: %v (killing pid %d directly)", host, child, err, cmd.Process.Pid)

+ }

+ cancel()

+ _ = c.Close()

+ }

+ _ = cmd.Process.Kill()

+ closePipes()

+ select {

+ case <-k.reaped():

+ case <-time.After(5 * time.Second):

+ t.Errorf("cleanup: pid %d was not reaped within 5s", cmd.Process.Pid)

+ }

+ })

+ return k

+}

+

+// Spec §12, Containment: "a memory-hog successor that overruns its child

+// limit and is OOM-killed while the incumbent keeps serving." A small

+// limit, not a tier: the accounting path takes any limit, and 64M makes

+// the hog die in seconds rather than after a gigabyte of swap.

+func TestGenOOMKernel(t *testing.T) {

+ m := genKernelManager(t)

+ const host = "oom.example"

+ small := Limit{MemoryMax: 64 << 20}

+ parent := m.hostDir(host)

+ // Registered first, so it runs last: after every child is reaped.

+ t.Cleanup(func() {

+ m.RemoveChild(host, GenChild(1))

+ m.RemoveChild(host, GenChild(2))

+ m.CleanupHost(host)

+ })

+

+ var echo *echoChild

+ incumbentCmd, err := m.StartChild(host, GenChild(1), small, small, func() *exec.Cmd {

+ cmd := genKernelCmd("echo")

+ echo = echoPipes(t, cmd)

+ return cmd

+ })

+ var echoIn io.Closer

+ if echo != nil {

+ echoIn = echo.in

+ }

+ incumbent := reapOnCleanup(t, m, host, GenChild(1), incumbentCmd, echoIn)

+ if err != nil {

+ t.Fatal(err)

+ }

+ echo.answer(t, "", "ready")

+ expectFile(t, filepath.Join(parent, "memory.max"), "67108864")

+ expectFile(t, filepath.Join(parent, "memory.oom.group"), "0")

+ expectFile(t, filepath.Join(parent, GenChild(1), "memory.oom.group"), "1")

+ got1, err := cgroupOf(incumbent.cmd.Process.Pid)

+ if want := filepath.Join(strings.TrimPrefix(m.Root(), "/sys/fs/cgroup/"), ExecName(host), GenChild(1)); err != nil || got1 != want {

+ t.Fatalf("incumbent runs in %s (%v), want %s", got1, err, want)

+ }

+

+ // The raise happens before the successor spawns.

+ if err := m.SetParentLimit(host, OverlapLimit(small, small)); err != nil {

+ t.Fatal(err)

+ }

+ expectFile(t, filepath.Join(parent, "memory.max"), "134217728")

+

+ var hogIn io.WriteCloser

+ var hogPipeErr error

+ hogCmd, err := m.StartChild(host, GenChild(2), small, small, func() *exec.Cmd {

+ cmd := genKernelCmd("hog")

+ hogIn, hogPipeErr = cmd.StdinPipe()

+ return cmd

+ })

+ hog := reapOnCleanup(t, m, host, GenChild(2), hogCmd, hogIn)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if hogPipeErr != nil {

+ t.Fatalf("hog stdin: %v", hogPipeErr)

+ }

+ // StartChild is best-effort: a refused limit write starts the child

+ // uncapped. The hog waits on stdin, so prove it is confined first. Any

+ // failure from here kills it: the cleanup registered above runs.

+ g2 := filepath.Join(parent, GenChild(2))

+ wantG2 := filepath.Join(strings.TrimPrefix(m.Root(), "/sys/fs/cgroup/"), ExecName(host), GenChild(2))

+ if got, err := os.ReadFile(filepath.Join(g2, "memory.max")); err != nil || strings.TrimSpace(string(got)) != "67108864" {

+ t.Fatalf("g2 memory.max = %q, %v; want 67108864 before the hog runs", got, err)

+ }

+ if gotCg, err := cgroupOf(hog.cmd.Process.Pid); err != nil || gotCg != wantG2 {

+ t.Fatalf("hog runs in %s (%v), want %s", gotCg, err, wantG2)

+ }

+ if _, err := fmt.Fprintln(hogIn, "go"); err != nil {

+ t.Fatalf("hog go-ahead: %v", err)

+ }

+ select {

+ case <-hog.reaped():

+ err = hog.err

+ case <-time.After(60 * time.Second):

+ t.Fatal("the hog was not OOM-killed within 60s")

+ }

+ var exitErr *exec.ExitError

+ if !errors.As(err, &exitErr) {

+ t.Fatalf("hog exit: %v", err)

+ }

+ if ws, ok := exitErr.Sys().(syscall.WaitStatus); !ok || !ws.Signaled() || ws.Signal() != syscall.SIGKILL {

+ t.Fatalf("hog ended with %v, want SIGKILL from the OOM killer", err)

+ }

+ if n := oomKills(t, filepath.Join(parent, GenChild(2))); n < 1 {

+ t.Fatalf("g2 memory.events oom_kill = %d, want at least 1", n)

+ }

+ if n := oomKills(t, filepath.Join(parent, GenChild(1))); n != 0 {

+ t.Fatalf("the incumbent's child saw %d OOM kills", n)

+ }

+ echo.ping(t) // the incumbent keeps serving

+

+ // The restore happens after the retiring child is empty.

+ m.RemoveChild(host, GenChild(2))

+ if _, err := os.Stat(filepath.Join(parent, GenChild(2))); !os.IsNotExist(err) {

+ t.Fatalf("empty g2 not removed: %v", err)

+ }

+ if err := m.SetParentLimit(host, small); err != nil {

+ t.Fatal(err)

+ }

+ expectFile(t, filepath.Join(parent, "memory.max"), "67108864")

+ echo.ping(t)

+}

+

+// Strict containment against the kernel: per-generation reservation, the

+// parent limit through one API, retirement of one generation, and the

+// parent's kill taking every generation.

+func TestGenStrictKernel(t *testing.T) {

+ m := genKernelManager(t)

+ const host = "strictgen.example"

+ dir := m.hostDir(host)

+ parent, err := m.PrepareStrictParent(host, TierDefault)

+ if err != nil {

+ t.Fatal(err)

+ }

+ // Registered first, so it runs last: after every child is reaped. The

+ // handles close before the parent's, then the directories go.

+ var handles []*Prepared

+ t.Cleanup(func() {

+ for i := len(handles) - 1; i >= 0; i-- {

+ _ = handles[i].Close()

+ }

+ _ = parent.Close()

+ for n := uint64(1); n <= 3; n++ {

+ m.RemoveChild(host, GenChild(n))

+ }

+ m.CleanupHost(host)

+ })

+ expectFile(t, filepath.Join(dir, "memory.oom.group"), "0")

+ expectFile(t, filepath.Join(dir, "memory.max"), "536870912")

+ if b, err := os.ReadFile(filepath.Join(dir, "cgroup.subtree_control")); err != nil || !strings.Contains(string(b), "memory") {

+ t.Fatalf("parent subtree_control = %q, %v", b, err)

+ }

+ refused := genKernelCmd("echo")

+ err = parent.Start(refused)

+ reapOnCleanup(t, m, host, "", refused)

+ if !errors.Is(err, ErrStrictState) {

+ t.Fatalf("a generation parent accepted a process: %v", err)

+ }

+

+ prepareGen := func(n uint64) *Prepared {

+ t.Helper()

+ g, err := parent.PrepareGen(n, TierDefault)

+ if err != nil {

+ t.Fatal(err)

+ }

+ handles = append(handles, g)

+ return g

+ }

+ start := func(g *Prepared, n uint64) (*kernelKid, *echoChild) {

+ t.Helper()

+ cmd := genKernelCmd("echo")

+ e := echoPipes(t, cmd)

+ err := g.Start(cmd)

+ k := reapOnCleanup(t, m, host, GenChild(n), cmd, e.in)

+ if err != nil {

+ t.Fatal(err)

+ }

+ e.answer(t, "", "ready")

+ return k, e

+ }

+ g1 := prepareGen(1)

+ kid1, echo1 := start(g1, 1)

+ expectFile(t, filepath.Join(dir, GenChild(1), "memory.oom.group"), "1")

+ if err := parent.SetParentLimit(OverlapLimit(TierDefault, TierDefault)); err != nil {

+ t.Fatal(err)

+ }

+ expectFile(t, filepath.Join(dir, "memory.max"), "1073741824")

+ g2 := prepareGen(2)

+ kid2, echo2 := start(g2, 2)

+ base := filepath.Join(strings.TrimPrefix(m.Root(), "/sys/fs/cgroup/"), ExecName(host))

+ if got, err := cgroupOf(kid1.cmd.Process.Pid); err != nil || got != filepath.Join(base, GenChild(1)) {

+ t.Fatalf("g1 process in %s (%v)", got, err)

+ }

+ if got, err := cgroupOf(kid2.cmd.Process.Pid); err != nil || got != filepath.Join(base, GenChild(2)) {

+ t.Fatalf("g2 process in %s (%v)", got, err)

+ }

+ echo1.ping(t)

+

+ // Retire g1 alone.

+ ctx1, cancel1 := context.WithTimeout(t.Context(), 3*time.Second)

+ defer cancel1()

+ if err := g1.KillAndEmpty(ctx1); err != nil {

+ t.Fatal(err)

+ }

+ _ = kid1.wait()

+ echo2.ping(t)

+ if err := g1.CloseRemove(); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := os.Stat(filepath.Join(dir, GenChild(1))); !os.IsNotExist(err) {

+ t.Fatalf("g1 directory survived CloseRemove: %v", err)

+ }

+ if err := parent.SetParentLimit(TierDefault); err != nil {

+ t.Fatal(err)

+ }

+ expectFile(t, filepath.Join(dir, "memory.max"), "536870912")

+

+ // A limit this API did not set stops the next spawn.

+ if err := os.WriteFile(filepath.Join(dir, "memory.max"), []byte("268435456"), 0o644); err != nil {

+ t.Fatal(err)

+ }

+ g3 := prepareGen(3)

+ refused3 := genKernelCmd("echo")

+ err = g3.Start(refused3)

+ reapOnCleanup(t, m, host, GenChild(3), refused3)

+ if !errors.Is(err, ErrStrictRefused) {

+ t.Fatalf("spawn under a foreign parent limit: %v", err)

+ }

+ if err := g3.CloseRemove(); err != nil {

+ t.Fatal(err)

+ }

+

+ // A terminal stop: the parent's kill takes every generation. Its own

+ // budget, not the one g1's retirement spent.

+ ctx2, cancel2 := context.WithTimeout(t.Context(), 3*time.Second)

+ defer cancel2()

+ if err := parent.KillAndEmpty(ctx2); err != nil {

+ t.Fatal(err)

+ }

+ _ = kid2.wait()

+ if err := g2.CloseRemove(); err != nil {

+ t.Fatal(err)

+ }

+ if err := parent.Close(); err != nil {

+ t.Fatal(err)

+ }

+ m.CleanupHost(host)

+}

diff --git a/internal/cgroup/gen_test.go b/internal/cgroup/gen_test.go

new file mode 100644

index 000000000..367938eab

--- /dev/null

+++ b/internal/cgroup/gen_test.go

@@ -0,0 +1,277 @@

+package cgroup

+

+import (

+ "os"

+ "os/exec"

+ "path/filepath"

+ "strconv"

+ "testing"

+)

+

+// quietOOM points the oom_score_adj write at a temp dir, so a fake start

+// that reports our own PID never re-scores the test process.

+func quietOOM(t *testing.T) {

+ t.Helper()

+ dir := t.TempDir()

+ orig := oomScoreAdjPath

+ oomScoreAdjPath = func(pid int) string { return filepath.Join(dir, strconv.Itoa(pid)) }

+ t.Cleanup(func() { oomScoreAdjPath = orig })

+}

+

+func writeFixture(t *testing.T, path, body string) {

+ t.Helper()

+ if err := os.WriteFile(path, []byte(body), 0o644); err != nil {

+ t.Fatal(err)

+ }

+}

+

+func readFixture(t *testing.T, path string) string {

+ t.Helper()

+ b, err := os.ReadFile(path)

+ if err != nil {

+ t.Fatalf("read %s: %v", path, err)

+ }

+ return string(b)

+}

+

+// genManager is a delegated fixture root whose host dir already lists the

+// four controllers, as a real child of the delegated root would.

+func genManager(t *testing.T, host string) (*Manager, string) {

+ t.Helper()

+ quietOOM(t)

+ m, err := SetupAt(newDelegatedRoot(t), t.Logf)

+ if err != nil {

+ t.Fatal(err)

+ }

+ parent := m.hostDir(host)

+ if err := os.MkdirAll(parent, 0o755); err != nil {

+ t.Fatal(err)

+ }

+ writeFixture(t, filepath.Join(parent, "cgroup.controllers"), "cpu io memory pids\n")

+ return m, parent

+}

+

+func TestGenAndRoleChildNames(t *testing.T) {

+ if GenChild(7) != "g7" || RoleChild("sidecar") != "role-sidecar" {

+ t.Fatalf("GenChild(7) = %q, RoleChild(sidecar) = %q", GenChild(7), RoleChild("sidecar"))

+ }

+ for name, want := range map[string]bool{

+ "g7": true, "g18446744073709551615": true, "role-seed": true, "role-sidecar": true,

+ "g0": false, "g07": false, "g": false, "gx": false, "g-1": false,

+ "role-": false, "role-a/b": false, "role-..": false, "child": false,

+ "carlos-exec@h.example.com": false, "../g7": false,

+ } {

+ if got := isChildName(name); got != want {

+ t.Errorf("isChildName(%q) = %v, want %v", name, got, want)

+ }

+}

+

+func TestOverlapAndValidParentLimits(t *testing.T) {

+ if got := OverlapLimit(TierDefault, TierChildren); got.MemoryMax != 1536<<20 {

+ t.Errorf("OverlapLimit(512M, 1G) = %d", got.MemoryMax)

+ }

+ for max, want := range map[int64]bool{

+ 512 << 20: true, 1 << 30: true, 1536 << 20: true, 2 << 30: true,

+ 0: false, 64 << 20: false, 3 << 30: false,

+ } {

+ if got := ValidParentLimit(Limit{MemoryMax: max}); got != want {

+ t.Errorf("ValidParentLimit(%d) = %v, want %v", max, got, want)

+ }

+}

+

+func TestStartChildPutsTheTierOnTheGenerationAndTheParent(t *testing.T) {

+ const host = "h.example.com"

+ m, parent := genManager(t, host)

+ fakeStart(t, func(*exec.Cmd) error { return nil })

+ if _, err := m.StartChild(host, GenChild(1), TierDefault, TierDefault, buildTrue); err != nil {

+ t.Fatal(err)

+ }

+ high := strconv.FormatInt(HighOf(TierDefault.MemoryMax), 10)

+ child := filepath.Join(parent, "g1")

+ for file, want := range map[string]string{

+ filepath.Join(child, "memory.max"): "536870912",

+ filepath.Join(child, "memory.high"): high,

+ filepath.Join(child, "memory.swap.max"): "536870912",

+ filepath.Join(child, "memory.oom.group"): "1",

+ filepath.Join(parent, "memory.max"): "536870912",

+ filepath.Join(parent, "memory.high"): high,

+ filepath.Join(parent, "memory.swap.max"): "536870912",

+ filepath.Join(parent, "memory.oom.group"): "0",

+ filepath.Join(parent, "cgroup.subtree_control"): "+cpu +memory +io +pids",

+ } {

+ if got := readFixture(t, file); got != want {

+ t.Errorf("%s = %q, want %q", file, got, want)

+ }

+ if _, err := os.Stat(filepath.Join(parent, "cgroup.procs")); !os.IsNotExist(err) {

+ t.Errorf("the parent must hold no process, but its cgroup.procs was written (%v)", err)

+ }

+}

+

+func TestStartChildRoleChildCarriesNoLimitOfItsOwn(t *testing.T) {

+ const host = "h.example.com"

+ m, parent := genManager(t, host)

+ fakeStart(t, func(*exec.Cmd) error { return nil })

+ if _, err := m.StartChild(host, RoleChild("sidecar"), TierDefault, Limit{}, buildTrue); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := os.Stat(filepath.Join(parent, "role-sidecar", "memory.max")); !os.IsNotExist(err) {

+ t.Errorf("a role child got a limit of its own: %v", err)

+ }

+ if got := readFixture(t, filepath.Join(parent, "memory.max")); got != "536870912" {

+ t.Errorf("parent memory.max = %q, want the tier", got)

+ }

+}

+

+// A successor spawns while the incumbent lives: the swap has raised the

+// parent on purpose, and the spawn must not lower it.

+func TestStartChildLeavesALiveParentAlone(t *testing.T) {

+ const host = "h.example.com"

+ m, parent := genManager(t, host)

+ fakeStart(t, func(*exec.Cmd) error { return nil })

+ writeFixture(t, filepath.Join(parent, "cgroup.events"), "populated 1\nfrozen 0\n")

+ writeFixture(t, filepath.Join(parent, "memory.max"), "1073741824")

+ if _, err := m.StartChild(host, GenChild(2), TierDefault, TierDefault, buildTrue); err != nil {

+ t.Fatal(err)

+ }

+ if got := readFixture(t, filepath.Join(parent, "memory.max")); got != "1073741824" {

+ t.Errorf("a successor's spawn changed the raised parent to %q", got)

+ }

+}

+

+// Review focus 1: an agent killed between the raise and the restore leaves

+// the parent at two tiers. The next cold wake puts it back.

+func TestStartChildResetsAStaleRaisedParentOnAColdWake(t *testing.T) {

+ const host = "h.example.com"

+ m, parent := genManager(t, host)

+ fakeStart(t, func(*exec.Cmd) error { return nil })

+ writeFixture(t, filepath.Join(parent, "cgroup.events"), "populated 0\nfrozen 0\n")

+ writeFixture(t, filepath.Join(parent, "memory.max"), "1073741824")

+ if _, err := m.StartChild(host, GenChild(3), TierDefault, TierDefault, buildTrue); err != nil {

+ t.Fatal(err)

+ }

+ if got := readFixture(t, filepath.Join(parent, "memory.max")); got != "536870912" {

+ t.Errorf("cold wake left the parent at %q, want the tier", got)

+ }

+}

+

+func TestStartChildRefusesAForeignChildName(t *testing.T) {

+ m, _ := genManager(t, "h.example.com")

+ calls := fakeStart(t, func(*exec.Cmd) error { return nil })

+ if _, err := m.StartChild("h.example.com", "../carlos-exec@other.example.com", TierDefault, TierDefault, buildTrue); err == nil {

+ t.Fatal("a child name outside the layout was accepted")

+ }

+ if len(*calls) != 0 {

+ t.Fatalf("spawned %d times for a refused child name", len(*calls))

+ }

+}

+

+func TestSetParentLimitWritesTheBackstopAndReportsFailure(t *testing.T) {

+ const host = "h.example.com"

+ m, parent := genManager(t, host)

+ if err := m.SetParentLimit(host, OverlapLimit(TierDefault, TierDefault)); err != nil {

+ t.Fatal(err)

+ }

+ for file, want := range map[string]string{

+ "memory.max": "1073741824",

+ "memory.high": strconv.FormatInt(HighOf(1<<30), 10),

+ "memory.swap.max": "1073741824",

+ "memory.oom.group": "0",

+ } {

+ if got := readFixture(t, filepath.Join(parent, file)); got != want {

+ t.Errorf("%s = %q, want %q", file, got, want)

+ }

+ // A raise that did not land must say so: the swap stops on it.

+ if err := os.Remove(filepath.Join(parent, "memory.max")); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.Mkdir(filepath.Join(parent, "memory.max"), 0o755); err != nil {

+ t.Fatal(err)

+ }

+ if err := m.SetParentLimit(host, TierDefault); err == nil {

+ t.Fatal("a limit write that failed reported success")

+ }

+ var nilM *Manager

+ if err := nilM.SetParentLimit(host, TierDefault); err != nil {

+ t.Fatalf("nil manager: %v, want nil (no delegation, nothing to set)", err)

+ }

+}

+

+// Review focus 2: rmdir refuses a cgroup with child cgroups, empty or not.

+func TestSweepRemovesEmptyGenerationChildrenThenTheParent(t *testing.T) {

+ root := newDelegatedRoot(t)

+ stale := filepath.Join(root, ExecName("old.example.com"))

+ for _, d := range []string{"g3", "role-seed"} {

+ if err := os.MkdirAll(filepath.Join(stale, d), 0o755); err != nil {

+ t.Fatal(err)

+ }

+ live := filepath.Join(root, ExecName("live.example.com"))

+ if err := os.MkdirAll(filepath.Join(live, "g4"), 0o755); err != nil {

+ t.Fatal(err)

+ }

+ // A file stands in for a member: rmdir refuses the directory.

+ writeFixture(t, filepath.Join(live, "g4", "cgroup.procs"), "123\n")

+ if _, err := SetupAt(root, t.Logf); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := os.Stat(stale); !os.IsNotExist(err) {

+ t.Errorf("stale generation parent survived the sweep: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(live, "g4")); err != nil {

+ t.Errorf("a populated generation child must be left alone: %v", err)

+ }

+}

+

+func TestRemoveChildRemovesOnlyItsOwnNames(t *testing.T) {

+ const host = "h.example.com"

+ m, parent := genManager(t, host)

+ for _, d := range []string{"g5", "keep"} {

+ if err := os.Mkdir(filepath.Join(parent, d), 0o755); err != nil {

+ t.Fatal(err)

+ }

+ m.RemoveChild(host, "g5")

+ m.RemoveChild(host, "keep")

+ m.RemoveChild(host, "../"+ExecName("other.example.com"))

+ if _, err := os.Stat(filepath.Join(parent, "g5")); !os.IsNotExist(err) {

+ t.Errorf("empty g5 survived: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(parent, "keep")); err != nil {

+ t.Errorf("a foreign child was removed: %v", err)

+ }

+ var nilM *Manager

+ nilM.RemoveChild(host, "g5") // must not panic

+}

+

+// Flag off: Start keeps today's single-level layout.

+func TestStartKeepsTheSingleLevelLayout(t *testing.T) {

+ quietOOM(t)

+ m, err := SetupAt(newDelegatedRoot(t), t.Logf)

+ if err != nil {

+ t.Fatal(err)

+ }

+ fakeStart(t, func(*exec.Cmd) error { return nil })

+ if _, err := m.Start("h.example.com", TierDefault, buildTrue); err != nil {

+ t.Fatal(err)

+ }

+ dir := m.hostDir("h.example.com")

+ if got := readFixture(t, filepath.Join(dir, "memory.oom.group")); got != "1" {

+ t.Errorf("host cgroup memory.oom.group = %q, want 1", got)

+ }

+ if _, err := os.Stat(filepath.Join(dir, "cgroup.subtree_control")); !os.IsNotExist(err) {

+ t.Errorf("flag-off Start delegated controllers: %v", err)

+ }

+ entries, err := os.ReadDir(dir)

+ if err != nil {

+ t.Fatal(err)

+ }

+ for _, e := range entries {

+ if e.IsDir() {

+ t.Errorf("flag-off Start made a child cgroup %s", e.Name())

+ }

+}

diff --git a/internal/cgroup/role_linux.go b/internal/cgroup/role_linux.go

index ff5ded117..c6e5c5c51 100644

--- a/internal/cgroup/role_linux.go

+++ b/internal/cgroup/role_linux.go

@@ -4,8 +4,6 @@ package cgroup

import (

"errors"

- "os"

- "os/exec"

"github.com/carlosframework/platform/internal/controlname"

"golang.org/x/sys/unix"

@@ -54,61 +52,22 @@ func (p *Prepared) prepareRoleChild(role string, intent func(ContainmentIdentity

return nil, err

}

name := "role-" + role

- if intent != nil {

- var st unix.Stat_t

- if e := unix.Fstatat(int(p.dir.Fd()), name, &st, unix.AT_SYMLINK_NOFOLLOW); !errors.Is(e, unix.ENOENT) {

- return nil, ErrStrictRefused

+ beforeMkdir := func() error {

+ if intent != nil {

+ var st unix.Stat_t

+ if e := unix.Fstatat(int(p.dir.Fd()), name, &st, unix.AT_SYMLINK_NOFOLLOW); !errors.Is(e, unix.ENOENT) {

+ return ErrStrictRefused

+ }

+ if e := intent(ContainmentIdentity{Device: p.identity.device, Inode: p.identity.inode}); e != nil {

+ return e

+ }

}

- if err = intent(ContainmentIdentity{Device: p.identity.device, Inode: p.identity.inode}); err != nil {

- return nil, err

- }

- if err = unix.Mkdirat(int(p.dir.Fd()), name, 0755); err != nil && (intent != nil || !errors.Is(err, unix.EEXIST)) {

- return nil, ErrStrictRefused

- }

- child := &Prepared{strictPreparedState: &strictPreparedState{parent: p, host: p.host, limits: map[string]*os.File{}, expected: map[string]string{}, entryName: name, start: func(cmd *exec.Cmd) error { return cmd.Start() }}}

- defer func() {

- if err != nil {

- child.closeFiles()

- }

- }()

- fd, e := unix.FcntlInt(p.dir.Fd(), unix.F_DUPFD_CLOEXEC, 0)

- if e != nil {

- return nil, ErrStrictUnsupported

- }

- child.entryParent = os.NewFile(uintptr(fd), name+"-parent")

- child.dir, e = strictOpen(p.dir, name, unix.O_RDONLY|unix.O_DIRECTORY)

- if e != nil {

- return nil, ErrStrictRefused

- }

- child.identity, e = strictID(child.dir)

- if e != nil {

- return nil, ErrStrictRefused

- }

- strictReservations.Lock()

- if strictReservations.held[child.identity] {

- strictReservations.Unlock()

- return nil, ErrStrictState

- }

- strictReservations.held[child.identity] = true

- strictReservations.Unlock()

- defer func() {

- if err != nil {

- strictReservations.Lock()

- delete(strictReservations.held, child.identity)

- strictReservations.Unlock()

- }

- }()

- child.events, e = strictOpen(child.dir, "cgroup.events", unix.O_RDONLY)

- if e != nil {

- return nil, ErrStrictUnsupported

- }

- child.kill, e = strictOpen(child.dir, "cgroup.kill", unix.O_WRONLY)

- if e != nil {

- return nil, ErrStrictUnsupported

+ return nil

}

- if populated, e := strictPopulation(child.events); e != nil || populated {

- return nil, ErrStrictRefused

+ allowExisting := intent == nil

+ child, err := pinChild(p, name, beforeMkdir, allowExisting)

+ if err != nil {

+ return nil, err

}

return child, nil

}

diff --git a/internal/cgroup/role_linux_test.go b/internal/cgroup/role_linux_test.go

index 6cb75d1bb..0cde0ec74 100644

--- a/internal/cgroup/role_linux_test.go

+++ b/internal/cgroup/role_linux_test.go

@@ -169,3 +169,36 @@ func TestRoleChildRefusesAfterCleanupAcquisitionCancellation(t *testing.T) {

t.Fatalf("new role after cancelled cleanup: %v", err)

}

+

+// TestPrepareRoleChildWithIntentRefusesExistingDirectory verifies that the intent

+// path refuses with ErrStrictRefused when the child directory appears between the

+// intent callback and mkdir (i.e., the intent callback creates the directory).

+func TestPrepareRoleChildWithIntentRefusesExistingDirectory(t *testing.T) {

+ p := strictFixture(t)

+ intentCalled := false

+ intent := func(id ContainmentIdentity) error {

+ intentCalled = true

+ // Simulate another actor creating the directory after our intent checks for it

+ // but before mkdir is called

+ if err := os.Mkdir(filepath.Join(p.dir.Name(), "role-newchild"), 0700); err != nil {

+ t.Fatal(err)

+ }

+ // Also create the required cgroup files

+ for n, v := range map[string]string{"cgroup.events": "populated 0\n", "cgroup.kill": ""} {

+ if err := os.WriteFile(filepath.Join(p.dir.Name(), "role-newchild", n), []byte(v), 0600); err != nil {

+ t.Fatal(err)

+ }

+ return nil

+ }

+ child, err := p.PrepareRoleChildWithIntent("newchild", intent)

+ if err != nil && !intentCalled {

+ t.Fatalf("intent was not called before error: %v", err)

+ }

+ if !intentCalled {

+ t.Fatal("intent callback was not called")

+ }

+ if !errors.Is(err, ErrStrictRefused) || child != nil {

+ t.Fatalf("intent path did not refuse pre-existing directory: err=%v child=%v", err, child)

+ }

+}

diff --git a/internal/cgroup/strict_gen_linux.go b/internal/cgroup/strict_gen_linux.go

new file mode 100644

index 000000000..923532316

--- /dev/null

+++ b/internal/cgroup/strict_gen_linux.go

@@ -0,0 +1,265 @@

+//go:build linux

+

+package cgroup

+

+import (

+ "errors"

+ "fmt"

+ "os"

+ "os/exec"

+ "slices"

+ "strconv"

+ "strings"

+

+ "golang.org/x/sys/unix"

+)

+

+// tierExpected is the four limit files for l, as the kernel reads them

+// back: memory.high rounded down to the page.

+func tierExpected(l Limit, oomGroup string) map[string]string {

+ page := int64(os.Getpagesize())

+ return map[string]string{

+ "memory.max": strconv.FormatInt(l.MemoryMax, 10),

+ "memory.high": strconv.FormatInt(HighOf(l.MemoryMax)/page*page, 10),

+ "memory.swap.max": strconv.FormatInt(l.MemoryMax, 10),

+ "memory.oom.group": oomGroup,

+ }

+}

+

+// pinChild pins a child cgroup under parent: beforeMkdir callback (if provided),

+// mkdir, create child struct, FcntlInt, strictOpen (dir, events, kill), strictID,

+// reservation, and population check.

+// beforeMkdir is called before mkdir (for role intent persistence; may be nil).

+// allowExisting controls whether mkdir's EEXIST is accepted: true allows reusing

+// an existing directory (generation children, role children without intent),

+// false refuses it (role children with intent require a new name).

+// On success, returns a fully pinned, reserved child.

+// On error, all cleanup is done internally; caller needs only to set up a defer

+// for cleanup of any subsequent operations (e.g. tier writes for generation children).

+func pinChild(parent *Prepared, name string, beforeMkdir func() error, allowExisting bool) (*Prepared, error) {

+ if beforeMkdir != nil {

+ if err := beforeMkdir(); err != nil {

+ return nil, err

+ }

+ if err := unix.Mkdirat(int(parent.dir.Fd()), name, 0755); err != nil && (!allowExisting || !errors.Is(err, unix.EEXIST)) {

+ return nil, ErrStrictRefused

+ }

+ child := &Prepared{strictPreparedState: &strictPreparedState{parent: parent, host: parent.host, limits: map[string]*os.File{}, expected: map[string]string{}, entryName: name, start: func(cmd *exec.Cmd) error { return cmd.Start() }}}

+

+ fd, e := unix.FcntlInt(parent.dir.Fd(), unix.F_DUPFD_CLOEXEC, 0)

+ if e != nil {

+ return nil, ErrStrictUnsupported

+ }

+ child.entryParent = os.NewFile(uintptr(fd), name+"-parent")

+

+ var err error

+ child.dir, err = strictOpen(parent.dir, name, unix.O_RDONLY|unix.O_DIRECTORY)

+ if err != nil {

+ child.closeFiles()

+ return nil, ErrStrictRefused

+ }

+

+ child.identity, err = strictID(child.dir)

+ if err != nil {

+ child.closeFiles()

+ return nil, ErrStrictRefused

+ }

+

+ strictReservations.Lock()

+ if strictReservations.held[child.identity] {

+ strictReservations.Unlock()

+ child.closeFiles()

+ return nil, ErrStrictState

+ }

+ strictReservations.held[child.identity] = true

+ strictReservations.Unlock()

+

+ child.events, err = strictOpen(child.dir, "cgroup.events", unix.O_RDONLY)

+ if err != nil {

+ child.closeFiles()

+ strictReservations.Lock()

+ delete(strictReservations.held, child.identity)

+ strictReservations.Unlock()

+ return nil, ErrStrictUnsupported

+ }

+

+ child.kill, err = strictOpen(child.dir, "cgroup.kill", unix.O_WRONLY)

+ if err != nil {

+ child.closeFiles()

+ strictReservations.Lock()

+ delete(strictReservations.held, child.identity)

+ strictReservations.Unlock()

+ return nil, ErrStrictUnsupported

+ }

+

+ populated, err := strictPopulation(child.events)

+ if err != nil || populated {

+ child.closeFiles()

+ strictReservations.Lock()

+ delete(strictReservations.held, child.identity)

+ strictReservations.Unlock()

+ return nil, ErrStrictRefused

+ }

+

+ return child, nil

+}

+

+// delegateStrict delegates a generation parent's controllers to its

+// children and proves memory is among them: a generation child without it

+// could carry no limit.

+func delegateStrict(dir *os.File) error {

+ ctl, e := strictOpen(dir, "cgroup.controllers", unix.O_RDONLY)

+ if e != nil {

+ return ErrStrictUnsupported

+ }

+ have, e := strictRead(ctl)

+ ctl.Close()

+ if e != nil {

+ return ErrStrictRefused

+ }

+ want := delegatedControllers(have)

+ if !strings.Contains(want, "+memory") {

+ return ErrStrictUnsupported

+ }

+ sub, e := strictOpen(dir, "cgroup.subtree_control", unix.O_RDWR)

+ if e != nil {

+ return ErrStrictUnsupported

+ }

+ defer sub.Close()

+ if e = strictWrite(sub, want); e != nil {

+ return ErrStrictRefused

+ }

+ got, e := strictRead(sub)

+ if e != nil || !slices.Contains(strings.Fields(got), "memory") {

+ return ErrStrictRefused

+ }

+ return nil

+}

+

+// PrepareStrictParent pins host's cgroup as a generation parent for the life

+// of one activator entry (zero-downtime deploys spec §7.3): empty, the tier

+// written with memory.oom.group=0, its controllers delegated to its

+// children. It holds no process: Start refuses it. Each spawn pins and

+// reserves its own child with PrepareGen. The swap moves the parent's limit

+// with SetParentLimit, which keeps the expected-limit record in step.

+func (m *Manager) PrepareStrictParent(host string, tier Limit) (*Prepared, error) {

+ return m.prepareStrict(host, tier, true)

+}

+

+// PrepareGen pins generation n's child under a generation parent: newly

+// empty, reserved for this generation alone, with the tier's full limits

+// and memory.oom.group=1 (spec §7.3). tier is this generation's own and may

+// differ from the incumbent's when a release changes tier.

+func (p *Prepared) PrepareGen(n uint64, tier Limit) (out *Prepared, err error) {

+ if p == nil || p.strictPreparedState == nil || n == 0 || (tier != TierDefault && tier != TierChildren) {

+ return nil, ErrStrictRefused

+ }

+ p.mu.Lock()

+ defer p.mu.Unlock()

+ if !p.genParent || p.closed || p.empty || p.draining.Load() {

+ return nil, ErrStrictState

+ }

+ if err = p.identityOK(); err != nil {

+ return nil, err

+ }

+ name := GenChild(n)

+ child, err := pinChild(p, name, nil, true)

+ if err != nil {

+ return nil, err

+ }

+ defer func() {

+ if err != nil {

+ child.closeFiles()

+ strictReservations.Lock()

+ delete(strictReservations.held, child.identity)

+ strictReservations.Unlock()

+ }

+ }()

+ child.tier = tier

+ want := tierExpected(tier, "1")

+ var e error

+ for name := range want {

+ if child.limits[name], e = strictOpen(child.dir, name, unix.O_RDWR); e != nil {

+ return nil, ErrStrictUnsupported

+ }

+ for _, name := range []string{"memory.max", "memory.high", "memory.swap.max", "memory.oom.group"} {

+ if e = strictWrite(child.limits[name], want[name]); e != nil {

+ return nil, ErrStrictRefused

+ }

+ child.expected[name] = want[name]

+ }

+ if e = child.limitsOK(); e != nil {

+ return nil, e

+ }

+ if e = child.identityOK(); e != nil {

+ return nil, e

+ }

+ return child, nil

+}

+

+// SetParentLimit writes l onto a generation parent and records it as the

+// expected value in the same step (spec §7.3). A child's Start checks the

+// parent with limitsOK and must never see a limit this API did not set.

+// Each file's record moves only after its own write lands, so a partial

+// failure leaves the record matching the kernel. A limit someone else set

+// is refused, not overwritten.

+func (p *Prepared) SetParentLimit(l Limit) error {

+ if p == nil || p.strictPreparedState == nil || !ValidParentLimit(l) {

+ return ErrStrictRefused

+ }

+ p.mu.Lock()

+ defer p.mu.Unlock()

+ if !p.genParent || p.closed || p.empty || p.draining.Load() {

+ return ErrStrictState

+ }

+ if e := p.identityOK(); e != nil {

+ return e

+ }

+ if e := p.limitsOK(); e != nil {

+ return e

+ }

+ want := tierExpected(l, "0")

+ for _, name := range []string{"memory.max", "memory.high", "memory.swap.max"} {

+ if e := strictWrite(p.limits[name], want[name]); e != nil {

+ return ErrStrictRefused

+ }

+ p.expected[name] = want[name]

+ }

+ return p.limitsOK()

+}

+

+// CloseRemove is Close for a generation child, then removes the child's

+// directory through the parent descriptor pinned at PrepareGen. Generation

+// numbers are never reused, so a child left behind would stay until the

+// agent restarts. Only an empty child is closed. The handle is closed and

+// its reservation released even when the rmdir fails; that error is

+// returned, and the next agent start's sweep removes the directory.

+func (p *Prepared) CloseRemove() error {

+ if p == nil || p.strictPreparedState == nil {

+ return ErrStrictState

+ }

+ p.mu.Lock()

+ defer p.mu.Unlock()

+ p.draining.Store(true)

+ if p.closed || p.parent == nil || p.entryParent == nil || !isGenName(p.entryName) {

+ return ErrStrictState

+ }

+ if e := p.identityOK(); e != nil {

+ return e

+ }

+ if populated, e := strictPopulation(p.events); e != nil || populated {

+ return ErrStrictRefused

+ }

+ rmErr := unix.Unlinkat(int(p.entryParent.Fd()), p.entryName, unix.AT_REMOVEDIR)

+ p.closed = true

+ p.closeFiles()

+ strictReservations.Lock()

+ delete(strictReservations.held, p.identity)

+ strictReservations.Unlock()

+ if rmErr != nil {

+ return fmt.Errorf("cgroup: %s: remove %s: %w", p.host, p.entryName, rmErr)

+ }

+ return nil

+}

diff --git a/internal/cgroup/strict_gen_linux_test.go b/internal/cgroup/strict_gen_linux_test.go

new file mode 100644

index 000000000..dc6f873ae

--- /dev/null

+++ b/internal/cgroup/strict_gen_linux_test.go

@@ -0,0 +1,213 @@

+//go:build linux

+

+package cgroup

+

+import (

+ "context"

+ "errors"

+ "os"

+ "os/exec"

+ "path/filepath"

+ "strconv"

+ "syscall"

+ "testing"

+ "time"

+)

+

+// genParentFixture is strictFixture turned into a generation parent: the

+// tier on its files, memory.oom.group=0, the record matching.

+func genParentFixture(t *testing.T) *Prepared {

+ t.Helper()

+ p := strictFixture(t)

+ p.genParent, p.tier = true, TierDefault

+ for name, value := range tierExpected(TierDefault, "0") {

+ if err := strictWrite(p.limits[name], value); err != nil {

+ t.Fatal(err)

+ }

+ p.expected[name] = value

+ }

+ return p

+}

+

+func makeGenFixture(t *testing.T, p *Prepared, n uint64, events string) string {

+ t.Helper()

+ d := filepath.Join(p.dir.Name(), GenChild(n))

+ if err := os.Mkdir(d, 0o700); err != nil {

+ t.Fatal(err)

+ }

+ for name, v := range map[string]string{"cgroup.events": events, "cgroup.kill": "", "memory.max": "", "memory.high": "", "memory.swap.max": "", "memory.oom.group": ""} {

+ if err := os.WriteFile(filepath.Join(d, name), []byte(v), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ return d

+}

+

+func TestPrepareGenWritesTheFullTier(t *testing.T) {

+ p := genParentFixture(t)

+ d := makeGenFixture(t, p, 1, "populated 0\n")

+ child, err := p.PrepareGen(1, TierDefault)

+ if err != nil {

+ t.Fatal(err)

+ }

+ defer child.Close()

+ want := tierExpected(TierDefault, "1")

+ for name, value := range want {

+ if got := readFixture(t, filepath.Join(d, name)); got != value {

+ t.Errorf("g1/%s = %q, want %q", name, got, value)

+ }

+ if want["memory.max"] != "536870912" || want["memory.oom.group"] != "1" {

+ t.Fatalf("tierExpected = %v", want)

+ }

+}

+

+func TestPrepareGenReservesPerChild(t *testing.T) {

+ p := genParentFixture(t)

+ makeGenFixture(t, p, 1, "populated 0\n")

+ makeGenFixture(t, p, 2, "populated 0\n")

+ g1, err := p.PrepareGen(1, TierDefault)

+ if err != nil {

+ t.Fatal(err)

+ }

+ defer g1.Close()

+ if again, err := p.PrepareGen(1, TierDefault); !errors.Is(err, ErrStrictState) || again != nil {

+ t.Fatalf("second reservation of g1: %v", err)

+ }

+ g2, err := p.PrepareGen(2, TierDefault)

+ if err != nil {

+ t.Fatalf("g2 beside a live g1: %v", err)

+ }

+ defer g2.Close()

+ if _, err := p.PrepareGen(0, TierDefault); !errors.Is(err, ErrStrictRefused) {

+ t.Errorf("generation 0: %v", err)

+ }

+ if _, err := p.PrepareGen(3, Limit{MemoryMax: 64 << 20}); !errors.Is(err, ErrStrictRefused) {

+ t.Errorf("a limit that is not a tier: %v", err)

+ }

+}

+

+func TestPrepareGenAndSetParentLimitRefuseAPlainHostGroup(t *testing.T) {

+ p := strictFixture(t)

+ if _, err := p.PrepareGen(1, TierDefault); !errors.Is(err, ErrStrictState) {

+ t.Errorf("PrepareGen on a leaf host group: %v", err)

+ }

+ if err := p.SetParentLimit(TierDefault); !errors.Is(err, ErrStrictState) {

+ t.Errorf("SetParentLimit on a leaf host group: %v", err)

+ }

+}

+

+func TestGenerationParentRefusesAProcess(t *testing.T) {

+ p := genParentFixture(t)

+ if err := p.Start(exec.Command("/bin/true")); !errors.Is(err, ErrStrictState) {

+ t.Fatalf("a generation parent accepted a process: %v", err)

+ }

+}

+

+// The parent's record moves with the limit it raises, so a child's Start

+// passes limitsOK after a raise; a limit set by anyone else fails it.

+func TestSetParentLimitRecordsWhatItWrites(t *testing.T) {

+ p := genParentFixture(t)

+ makeGenFixture(t, p, 1, "populated 0\n")

+ makeGenFixture(t, p, 2, "populated 0\n")

+ child, err := p.PrepareGen(1, TierDefault)

+ if err != nil {

+ t.Fatal(err)

+ }

+ defer child.Close()

+ if err := p.SetParentLimit(OverlapLimit(TierDefault, TierDefault)); err != nil {

+ t.Fatal(err)

+ }

+ if got := readFixture(t, filepath.Join(p.dir.Name(), "memory.max")); got != "1073741824" {

+ t.Fatalf("parent memory.max = %q", got)

+ }

+ // The fixture's start answers ENOSYS: reaching it means every check passed.

+ child.start = func(*exec.Cmd) error { return syscall.ENOSYS }

+ if err := child.Start(exec.Command("/bin/true")); !errors.Is(err, ErrStrictUnsupported) {

+ t.Fatalf("child start after an API raise: %v, want it to reach the spawn", err)

+ }

+ // strictWrite does not truncate a fixture file: "9" over "1073741824"

+ // leaves "9073741824", a limit this API never set.

+ if err := strictWrite(p.limits["memory.max"], "9"); err != nil {

+ t.Fatal(err)

+ }

+ other, err := p.PrepareGen(2, TierDefault)

+ if err != nil {

+ t.Fatal(err)

+ }

+ defer other.Close()

+ if err := other.Start(exec.Command("/bin/true")); !errors.Is(err, ErrStrictRefused) {

+ t.Fatalf("child start under a limit the API did not set: %v", err)

+ }

+ if err := p.SetParentLimit(TierDefault); !errors.Is(err, ErrStrictRefused) {

+ t.Fatalf("SetParentLimit papered over a foreign limit: %v", err)

+ }

+}

+

+// Review focus 4: a release that changes tier.

+func TestSetParentLimitAcceptsATierChange(t *testing.T) {

+ p := genParentFixture(t)

+ if err := p.SetParentLimit(OverlapLimit(TierDefault, TierChildren)); err != nil {

+ t.Fatalf("512M + 1G: %v", err)

+ }

+ if got := readFixture(t, filepath.Join(p.dir.Name(), "memory.max")); got != strconv.FormatInt(1536<<20, 10) {

+ t.Fatalf("parent memory.max = %q", got)

+ }

+ if err := p.SetParentLimit(Limit{MemoryMax: 64 << 20}); !errors.Is(err, ErrStrictRefused) {

+ t.Fatalf("a limit that is no tier sum: %v", err)

+ }

+}

+

+func TestGenerationKillAndEmptyTouchesOnlyItsChild(t *testing.T) {

+ p := genParentFixture(t)

+ d1 := makeGenFixture(t, p, 1, "populated 1\n")

+ d2 := makeGenFixture(t, p, 2, "populated 0\n")

+ g1, err := p.PrepareGen(1, TierDefault)

+ if err == nil {

+ t.Fatal("PrepareGen pinned a populated group")

+ }

+ _ = g1

+ if err := os.WriteFile(filepath.Join(d1, "cgroup.events"), []byte("populated 0\n"), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ if g1, err = p.PrepareGen(1, TierDefault); err != nil {

+ t.Fatal(err)

+ }

+ defer g1.Close()

+ g2, err := p.PrepareGen(2, TierDefault)

+ if err != nil {

+ t.Fatal(err)

+ }

+ ctx, cancel := context.WithTimeout(t.Context(), time.Second)

+ defer cancel()

+ if err := g2.KillAndEmpty(ctx); err != nil {

+ t.Fatal(err)

+ }

+ for dir, want := range map[string]string{d2: "1", d1: "", p.dir.Name(): ""} {

+ if got := readFixture(t, filepath.Join(dir, "cgroup.kill")); got != want {

+ t.Errorf("%s/cgroup.kill = %q, want %q", dir, got, want)

+ }

+ // CloseRemove: refused while populated; once empty it closes and frees

+ // the reservation even when rmdir fails (a fixture dir holds files).

+ if err := os.WriteFile(filepath.Join(d2, "cgroup.events"), []byte("populated 1\n"), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ if err := g2.CloseRemove(); !errors.Is(err, ErrStrictRefused) {

+ t.Fatalf("CloseRemove of a populated child: %v", err)

+ }

+ if err := os.WriteFile(filepath.Join(d2, "cgroup.events"), []byte("populated 0\n"), 0o600); err != nil {

+ t.Fatal(err)

+ }

+ if err := g2.CloseRemove(); err == nil {

+ t.Fatal("rmdir of a fixture dir holding files reported success")

+ }

+ if _, err := g2.Populated(); !errors.Is(err, ErrStrictState) {

+ t.Fatalf("handle still open after CloseRemove: %v", err)

+ }

+ again, err := p.PrepareGen(2, TierDefault)

+ if err != nil {

+ t.Fatalf("reservation not released by CloseRemove: %v", err)

+ }

+ again.Close()

+}

diff --git a/internal/cgroup/strict_linux.go b/internal/cgroup/strict_linux.go

index 37a057902..581e22466 100644

--- a/internal/cgroup/strict_linux.go

+++ b/internal/cgroup/strict_linux.go

@@ -9,7 +9,6 @@ import (

"os"

"os/exec"

"path/filepath"

- "strconv"

"strings"

"sync"

"sync/atomic"

@@ -44,6 +43,12 @@ type strictPreparedState struct {

host string

attempted, empty, closed bool

start func(*exec.Cmd) error

+ // genParent marks a generation parent (PrepareStrictParent): it holds

+ // no process, pins PrepareGen children, and moves its limit only

+ // through SetParentLimit. tier is the tier it was prepared with, or a

+ // generation child's own.

+ genParent bool

+ tier Limit

}

func strictOpen(dir *os.File, name string, flags int) (*os.File, error) {

@@ -196,7 +201,14 @@ func (m *Manager) StrictPreflight() error {

// PrepareStrict never uses accounting's fallback. Only the two existing tiers

// are accepted; a zero or missing limit cannot create an uncapped command.

-func (m *Manager) PrepareStrict(host string, limit Limit) (out *Prepared, err error) {

+func (m *Manager) PrepareStrict(host string, limit Limit) (*Prepared, error) {

+ return m.prepareStrict(host, limit, false)

+}

+

+// prepareStrict is PrepareStrict and PrepareStrictParent: with genParent the

+// group is pinned as a generation parent (memory.oom.group=0, controllers

+// delegated to its children, Start refused) instead of a leaf.

+func (m *Manager) prepareStrict(host string, limit Limit, genParent bool) (out *Prepared, err error) {

if m == nil || m.root == "" {

return nil, ErrStrictUnsupported

}

@@ -249,8 +261,11 @@ func (m *Manager) PrepareStrict(host string, limit Limit) (out *Prepared, err er

if populated, e := strictPopulation(p.events); e != nil || populated {

return nil, ErrStrictRefused

}

- high := HighOf(limit.MemoryMax) / int64(os.Getpagesize()) * int64(os.Getpagesize())

- p.expected = map[string]string{"memory.max": strconv.FormatInt(limit.MemoryMax, 10), "memory.high": strconv.FormatInt(high, 10), "memory.swap.max": strconv.FormatInt(limit.MemoryMax, 10), "memory.oom.group": "1"}

+ oomGroup := "1"

+ if genParent {

+ oomGroup = "0"

+ }

+ p.expected = tierExpected(limit, oomGroup)

for name := range p.expected {

p.limits[name], e = strictOpen(p.dir, name, unix.O_RDWR)

if e != nil {

@@ -265,6 +280,12 @@ func (m *Manager) PrepareStrict(host string, limit Limit) (out *Prepared, err er

if e = p.limitsOK(); e != nil {

return nil, e

}

+ if genParent {

+ if e = delegateStrict(p.dir); e != nil {

+ return nil, e

+ }

+ p.genParent, p.tier = true, limit

+ }

if e = p.identityOK(); e != nil {

return nil, e

}

@@ -292,6 +313,10 @@ func (p *Prepared) Start(cmd *exec.Cmd) error {

}

p.mu.Lock()

defer p.mu.Unlock()

+ if p.genParent {

+ // A generation parent holds no process (zero-downtime spec §7.3).

+ return ErrStrictState

+ }

if p.closed || p.attempted || p.empty || p.draining.Load() {

return ErrStrictState

}

diff --git a/internal/cgroup/strict_other.go b/internal/cgroup/strict_other.go

index 34ddee7cb..9e9b73c50 100644

--- a/internal/cgroup/strict_other.go

+++ b/internal/cgroup/strict_other.go

@@ -17,3 +17,9 @@ func (p *Prepared) Start(cmd *exec.Cmd) error { return ErrStrictUns

func (p *Prepared) Populated() (bool, error) { return false, ErrStrictUnsupported }

func (p *Prepared) KillAndEmpty(ctx context.Context) error { return ErrStrictUnsupported }

func (p *Prepared) Close() error { return ErrStrictUnsupported }

+func (m *Manager) PrepareStrictParent(string, Limit) (*Prepared, error) {

+ return nil, ErrStrictUnsupported

+}

+func (p *Prepared) PrepareGen(uint64, Limit) (*Prepared, error) { return nil, ErrStrictUnsupported }

+func (p *Prepared) SetParentLimit(Limit) error { return ErrStrictUnsupported }

+func (p *Prepared) CloseRemove() error { return ErrStrictUnsupported }

diff --git a/internal/host/claimunit.go b/internal/host/claimunit.go

index 5d84859ff..50bd00e9e 100644

--- a/internal/host/claimunit.go

+++ b/internal/host/claimunit.go

@@ -96,12 +96,12 @@ func claimUnitContent(name string, content []byte, cfg claimUnitSettings) []byte

}

return strconv.Quote(strings.ReplaceAll(p, "%", "%%"))

}

- roots := []string{"/run/carlos-control", "/run/carlos", cfg.data, cfg.opt, "/etc/systemd/system", "/sys/fs/cgroup"}

+ roots := []string{"/run/carlos-control", "/run/carlos", cfg.data, cfg.opt, "/etc/systemd/system", "/sys/fs/cgroup", "/run/systemd/system"}

for i, p := range roots {

roots[i] = quote(p)

}

content = bytes.Replace(content, []byte("[Service]\n"), []byte("[Service]\n"+claimControlEnvironment), 1)

- content = bytes.Replace(content, []byte("ReadWritePaths=/run/carlos-control\n"), []byte("ReadWritePaths="+strings.Join(roots, " ")+"\n"), 1)

+ content = bytes.Replace(content, []byte("ReadWritePaths=/run/carlos-control /run/systemd/system\n"), []byte("ReadWritePaths="+strings.Join(roots, " ")+"\n"), 1)

return bytes.Replace(content, []byte("RestrictAddressFamilies=AF_UNIX\n"), []byte("RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6\n"), 1)

}

diff --git a/internal/host/claimunit_test.go b/internal/host/claimunit_test.go

index e86d71f47..9d3169f0e 100644

--- a/internal/host/claimunit_test.go

+++ b/internal/host/claimunit_test.go

@@ -77,7 +77,7 @@ func TestClaimControlUnitInstallers(t *testing.T) {

}

for key, want := range map[string][]string{

"EnvironmentFile": {"-/etc/carlos/host.env"},

- "ReadWritePaths": {"/run/carlos-control /run/carlos /data /opt/carlos /etc/systemd/system /sys/fs/cgroup"},

+ "ReadWritePaths": {"/run/carlos-control /run/carlos /data /opt/carlos /etc/systemd/system /sys/fs/cgroup /run/systemd/system"},

"RestrictAddressFamilies": {"AF_UNIX AF_INET AF_INET6"},

"ProtectSystem": {"strict"},

"ProtectHome": {"yes"},

diff --git a/internal/host/host_test.go b/internal/host/host_test.go

index 4bd257c09..2313b98e3 100644

--- a/internal/host/host_test.go

+++ b/internal/host/host_test.go

@@ -41,6 +41,7 @@ var expectedUnits = []string{

"carlos-hibernate-verify.timer",

"carlos-instance-control.service",

"carlos-instance-control.socket",

+ "carlos-instance-gen@.service",

"carlos-instance-probe.service",

"carlos-instance-seed@.service",

"carlos-instance-sidecar@.service",

diff --git a/internal/host/metrics_probes.go b/internal/host/metrics_probes.go

index 255e33d9c..6ebe91951 100644

--- a/internal/host/metrics_probes.go

+++ b/internal/host/metrics_probes.go

@@ -265,10 +265,23 @@ func listExecCgroups(root string) []unitCgroup {

}

dir := filepath.Join(root, entry.Name())

procs, err := os.ReadFile(filepath.Join(dir, "cgroup.procs"))

- if err != nil || len(strings.TrimSpace(string(procs))) == 0 {

+ if err != nil {

+ continue

+ }

+ // A generation parent (zero-downtime spec §7.3) holds no process of

+ // its own; its generations do. cgroup.events "populated 1" is the

+ // kernel's word that something under it is alive.

+ if len(strings.TrimSpace(string(procs))) == 0 && !subtreePopulated(dir) {

continue

}

units = append(units, unitCgroup{name: entry.Name(), dir: dir})

}

return units

}

+

+// subtreePopulated reports whether dir's cgroup.events says a process is

+// alive anywhere under it. Unreadable is no: absence is the data.

+func subtreePopulated(dir string) bool {

+ b, err := os.ReadFile(filepath.Join(dir, "cgroup.events"))

+ return err == nil && strings.Contains(string(b), "populated 1")

+}

diff --git a/internal/host/metrics_probes_test.go b/internal/host/metrics_probes_test.go

index 65100a32a..b155c8cee 100644

--- a/internal/host/metrics_probes_test.go

+++ b/internal/host/metrics_probes_test.go

@@ -263,3 +263,38 @@ func TestDBFileBytes(t *testing.T) {

t.Fatalf("a directory = %d, want nil", *got)

}

+

+// A generation parent (zero-downtime spec §7.3) holds no process of its

+// own, so its cgroup.procs is empty while its generations run. It must

+// still be sampled, under its own name: its memory, cpu, io and pids

+// files already count the whole subtree.

+func TestListExecCgroupsCountsAGenerationParent(t *testing.T) {

+ root := t.TempDir()

+ mk := func(rel, file, body string) {

+ t.Helper()

+ dir := filepath.Join(root, rel)

+ if err := os.MkdirAll(dir, 0o755); err != nil {

+ t.Fatal(err)

+ }

+ if file != "" {

+ if err := os.WriteFile(filepath.Join(dir, file), []byte(body), 0o644); err != nil {

+ t.Fatal(err)

+ }

+ mk("carlos-exec@flat.example.com", "cgroup.procs", "1234\n")

+ mk("carlos-exec@gens.example.com", "cgroup.procs", "")

+ mk("carlos-exec@gens.example.com", "cgroup.events", "populated 1\nfrozen 0\n")

+ mk("carlos-exec@gens.example.com/g7", "cgroup.procs", "5678\n")

+ mk("carlos-exec@empty.example.com", "cgroup.procs", "")

+ mk("carlos-exec@empty.example.com", "cgroup.events", "populated 0\nfrozen 0\n")

+

+ got := listExecCgroups(root)

+ names := map[string]bool{}

+ for _, u := range got {

+ names[u.name] = true

+ }

+ if len(got) != 2 || !names["carlos-exec@flat.example.com"] || !names["carlos-exec@gens.example.com"] {

+ t.Fatalf("sampled %+v, want the flat host and the populated generation parent", got)

+ }

+}

diff --git a/internal/host/units/carlos-instance-control.service b/internal/host/units/carlos-instance-control.service

index b3eaab0ac..59b2d2b55 100644

--- a/internal/host/units/carlos-instance-control.service

+++ b/internal/host/units/carlos-instance-control.service

@@ -30,7 +30,9 @@ After=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount

[Service]

ExecStart=/opt/carlos/carlos instance-control

ProtectSystem=strict

-ReadWritePaths=/run/carlos-control

+# /run/systemd/system: the per-generation runtime drop-in (zero-downtime

+# deploys, plan 2 D1); nothing else is written there.

+ReadWritePaths=/run/carlos-control /run/systemd/system

ProtectHome=yes

PrivateTmp=yes

PrivateDevices=yes

diff --git a/internal/host/units/carlos-instance-gen@.service b/internal/host/units/carlos-instance-gen@.service

new file mode 100644

index 000000000..cf30f3330

--- /dev/null

+++ b/internal/host/units/carlos-instance-gen@.service

@@ -0,0 +1,149 @@

+# One generation of an isolated exec-backed instance (zero-downtime

+# deploys design §7.6). The instance string is "<instance-id>:<generation>"

+# (%I), and the trampoline and the exit writer split it. systemd has no

+# specifier for part of an instance string, and ':' is BindPaths='s field

+# separator, so everything keyed by the host ID alone (the slice, the data

+# mounts) or by the generation (the executable mount, the log socket)

+# comes from a runtime drop-in the control service writes before each

+# start. Without that drop-in this unit has an empty /data and its

+# trampoline refuses, and the control service will not start it unless

+# systemd reports the drop-in's slice. carlos-instance@.service stays for

+# every box and binary that still runs it.

+[Unit]

+Description=carlos exec-backed instance generation %I

+# Stop-only coupling, as carlos-instance@.service (codex R1-7).

+BindsTo=carlos-edge.service

+Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount

+After=carlos-edge.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount

+

+[Service]

+Type=simple

+ExecStart=/opt/carlos/carlos instance-exec run %I

+# The tenant runs in its own data directory (#256 D). %I here is

+# <id>:<n>, not the host's id, so the real directory comes from the

+# generation's runtime drop-in, which overrides this line. This one only

+# applies if the drop-in did not, and then the start is already refused

+# (the control service checks the drop-in's slice first), and /data is an

+# empty tmpfs, so the chdir fails closed.

+WorkingDirectory=/data/instances

+ExecStopPost=/opt/carlos/carlos instance-exit run %I

+User=carlos

+Group=carlos

+# The activator owns recycling; systemd must not race it.

+Restart=no

+KillMode=mixed

+TimeoutStopSec=20

+# The main process is the trampoline, which stays as the namespace's init

+# and runs the app as its child (#126): it forwards the SIGTERM and exits

+# 128+n when the app dies by signal n. These are SIGHUP, SIGINT, SIGPIPE

+# and SIGTERM, the deaths systemd counts as clean for a main process, so a

+# stop still leaves the unit inactive rather than failed.

+SuccessExitStatus=129 130 141 143

+StandardError=inherit

+

+# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) ---

+ProtectSystem=strict

+TemporaryFileSystem=/data:ro

+TemporaryFileSystem=/run/carlos:ro

+# Every other tree a carlos-uid unit writes on the box is hidden the same

+# way (codex R9-2): the restore drills' cache under /var/cache, and

+# /var/lib as a class. An instance needs nothing from either.

+TemporaryFileSystem=/var/cache:ro

+TemporaryFileSystem=/var/lib:ro

+# The version store holds every app's build on the box, world-readable so

+# a unit can exec its own (#214 item 2). An instance sees none of it: an

+# empty /opt/carlos with the platform binary that ExecStart= and

+# ExecStopPost= run put back, and, outside this block because it is per

+# instance, its own build at /opt/carlos/live/<host>.

+TemporaryFileSystem=/opt/carlos:ro

+BindReadOnlyPaths=/opt/carlos/carlos

+# Root-written credentials a carlos-GROUP process may read live here:

+# /etc/carlos/relay-masters is root:carlos 0640 for carlos-relay.service,

+# and an instance's Group=carlos would read it too (codex R10-1). An

+# instance gets its configuration through its launch file and needs

+# nothing from /etc/carlos. /run/carlos-control is the root control

+# service's socket directory (design §5.2): hiding it here too means a

+# connect from inside any instance's namespace is impossible outright,

+# not merely refused by the peer check on the far end (final review,

+# plan 2, item 3).

+InaccessiblePaths=/etc/carlos /run/carlos-control

+ProtectHome=yes

+PrivateTmp=yes

+PrivateDevices=yes

+# PrivateDevices= bind-mounts the HOST's /dev/shm into the private /dev,

+# and every instance is uid carlos, so a sibling's shared memory would be

+# one open away. Each unit gets its own, charged to its own cgroup.

+TemporaryFileSystem=/dev/shm:mode=1777,nosuid,nodev

+# POSIX message queues and SysV IPC belong to an IPC namespace, and every

+# instance is uid carlos, so a sibling's queue would be one mq_open away

+# (#178). PrivateIPC= gives each unit its own namespace, and mq_open()

+# goes through that namespace's own internal mount. PrivateDevices=

+# carries the host's /dev/mqueue into the private /dev the way it carries

+# /dev/shm, and that mount would still list and open the host namespace's

+# queues, so it is replaced the same way.

+PrivateIPC=yes

+TemporaryFileSystem=/dev/mqueue:ro

+# /sys/fs/cgroup read-only: the edge's delegated subtree is carlos-owned,

+# and its cgroup.kill / cgroup.freeze would otherwise be one write away

+# for a same-uid tenant (codex R9-1).

+ProtectControlGroups=yes

+ProtectProc=invisible

+ProcSubset=pid

+PrivatePIDs=yes

+RestrictNamespaces=yes

+NoNewPrivileges=yes

+CapabilityBoundingSet=

+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

+# Abstract unix sockets live in the network namespace, which instances

+# share with the box because they need outbound TCP, and the line above

+# cannot tell them from path sockets (#178). The trampoline scopes them

+# with Landlock before it execs the app (instanceexec.ScopeAbstractUnix):

+# an instance can connect only to abstract sockets its own processes

+# bound. A sibling can still bind a name first; that is a refusal of

+# service, never a connection.

+#

+# Egress (#214 item 3): the internet, and nothing on the box or private.

+# Loopback is where the console and other tenants' addr upstreams listen;

+# RFC 1918, ULA, link-local and CGNAT are the VPC, other boxes and the

+# cloud's internal services. IMDS stays named so the property is

+# greppable. A cgroup filter: it drops in both directions for every

+# socket the unit opens. The one thing an instance needs from these

+# ranges is the box's DNS resolver, so the control service re-opens

+# exactly the nameservers the instance sees with IPAddressAllow= on

+# carlos-instances.slice (an allow wins, and a slice's lists reach every

+# unit below it). The probe checks both halves.

+IPAddressDeny=localhost multicast 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 169.254.0.0/16 169.254.169.254/32 172.16.0.0/12 192.0.0.0/24 192.168.0.0/16 198.18.0.0/15 240.0.0.0/4 64:ff9b::/96 fc00::/7 fd00:ec2::254/128 fe80::/10 fec0::/10

+# An allow opens an address both ways and on every port, so a loopback

+# resolver can never be opened: a tenant could bind systemd-resolved's

+# 127.0.0.53 itself and reach every listener bound to all addresses

+# through it. Where resolved runs, instances resolve through its

+# upstreams instead, and the control service opens those. The "-" makes

+# this a no-op on a box without resolved.

+BindReadOnlyPaths=-/run/systemd/resolve/resolv.conf:/etc/resolv.conf

+ProtectKernelTunables=yes

+ProtectKernelModules=yes

+ProtectKernelLogs=yes

+ProtectClock=yes

+ProtectHostname=yes

+RemoveIPC=yes

+LockPersonality=yes

+RestrictRealtime=yes

+# No RestrictSUIDSGID=: systemd cannot read the mode inside openat2's

+# open_how, so it refuses openat2 outright (ENOSYS), and every instancefs

+# open in here is an openat2 (#126, first roll; systemd 257). What it

+# guarded is covered: with no capabilities an instance can only make a

+# set-id file owned by carlos, which gains nothing for another carlos

+# process, NoNewPrivileges= voids it in here, and no other uid can reach

+# one (the tenant trees are nosuid, PrivateTmp's sit under a root 0700

+# directory).

+MemoryDenyWriteExecute=yes

+SystemCallArchitectures=native

+

+# --- per-service resources; the tier lives on the host slice (§4.3) ---

+TasksMax=256

+# The kernel picks a tenant before it picks anything else in carlos.slice.

+OOMScoreAdjust=500

+# One OOM kill takes every process of THIS service (systemd sets

+# memory.oom.group on the service cgroup for OOMPolicy=kill). The sidecar

+# is a sibling service with its own family rule (§4.3).

+OOMPolicy=kill

diff --git a/internal/host/units_hardening_test.go b/internal/host/units_hardening_test.go

index 0cd6fc864..e07da5c74 100644

--- a/internal/host/units_hardening_test.go

+++ b/internal/host/units_hardening_test.go

@@ -39,10 +39,10 @@ const instanceDeny = "IPAddressDeny=localhost multicast " +

// Every directive the hardening adds to the shared namespace block (#214

// items 2 and 3, #178), pinned exactly: the block is byte-identical across

-// the three templates and the probe (TestInstanceTemplatesShareOneNamespace),

-// so pinning it on carlos-instance@ pins it on all four.

+// the four templates and the probe (TestInstanceTemplatesShareOneNamespace),

+// so pinning it on carlos-instance@ pins it on all five.

func TestNamespaceBlockCarriesTheHardening(t *testing.T) {

- for _, name := range []string{"carlos-instance@.service", "carlos-instance-seed@.service", "carlos-instance-sidecar@.service", "carlos-instance-probe.service"} {

+ for _, name := range []string{"carlos-instance@.service", "carlos-instance-gen@.service", "carlos-instance-seed@.service", "carlos-instance-sidecar@.service", "carlos-instance-probe.service"} {

u := readUnit(t, name)

for _, want := range []string{

// #214 item 2: no other app's build is visible.

diff --git a/internal/host/units_isolation_test.go b/internal/host/units_isolation_test.go

index f7a939f57..b2d3336ee 100644

--- a/internal/host/units_isolation_test.go

+++ b/internal/host/units_isolation_test.go

@@ -239,7 +239,7 @@ func TestInstanceTemplatesShareOneNamespace(t *testing.T) {

return u[start:end]

}

ref := block("carlos-instance@.service")

- for _, name := range []string{"carlos-instance-seed@.service", "carlos-instance-sidecar@.service", "carlos-instance-probe.service"} {

+ for _, name := range []string{"carlos-instance-gen@.service", "carlos-instance-seed@.service", "carlos-instance-sidecar@.service", "carlos-instance-probe.service"} {

if block(name) != ref {

t.Errorf("%s namespace block differs from carlos-instance@.service", name)

}

@@ -275,6 +275,10 @@ func TestInstanceTemplatesShareOneNamespace(t *testing.T) {

"Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount", "OOMScoreAdjust=500", "OOMPolicy=kill", "TasksMax=256", "Type=simple",

// The trampoline's init exits 128+n for a death by signal n (#126).

"SuccessExitStatus=129 130 141 143"},

+ "carlos-instance-gen@.service": {"ExecStart=/opt/carlos/carlos instance-exec run %I", "ExecStopPost=/opt/carlos/carlos instance-exit run %I",

+ "Restart=no", "KillMode=mixed", "TimeoutStopSec=20", "BindsTo=carlos-edge.service",

+ "Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount", "OOMScoreAdjust=500", "OOMPolicy=kill", "TasksMax=256", "Type=simple",

+ "SuccessExitStatus=129 130 141 143"},

"carlos-instance-seed@.service": {"Type=oneshot", "RemainAfterExit=yes", "TimeoutStartSec=120", "ExecStart=/opt/carlos/carlos instance-exec seed %I",

"ExecStopPost=/opt/carlos/carlos instance-exit seed %I", "StandardOutput=file:/run/carlos/logs/%I.seed.sock",

"KillMode=mixed", "TimeoutStopSec=20", "WorkingDirectory=/data/instances/%I"},

@@ -318,7 +322,7 @@ func TestControlUnitsShape(t *testing.T) {

}

u := readUnit(t, "carlos-instance-control.service")

for _, want := range []string{"ExecStart=/opt/carlos/carlos instance-control", "Requires=carlos-instance-control.socket",

- "data-instances.mount", "ProtectSystem=strict", "ReadWritePaths=/run/carlos-control", "RestrictAddressFamilies=AF_UNIX", "MemoryMax=64M"} {

+ "data-instances.mount", "ProtectSystem=strict", "ReadWritePaths=/run/carlos-control /run/systemd/system", "RestrictAddressFamilies=AF_UNIX", "MemoryMax=64M"} {

if !strings.Contains(u, want) {

t.Errorf("control service lacks %q", want)

}

@@ -335,7 +339,7 @@ func TestControlUnitsShape(t *testing.T) {

if !have["carlos-instance-control.socket"] {

t.Error("enabledUnits lacks the control socket")

}

- if have["carlos-instance-control.service"] || have["carlos-instance-probe.service"] || have["carlos-instance@.service"] {

+ if have["carlos-instance-control.service"] || have["carlos-instance-probe.service"] || have["carlos-instance@.service"] || have["carlos-instance-gen@.service"] {

t.Error("a template, the probe or the socket-activated service must not be enabled")

}

for _, want := range []string{"d /data/.instances/_probe 0700 carlos carlos -", "d /run/carlos/.instances/_probe 2750 carlos carlos-sockets -"} {

@@ -405,3 +409,28 @@ func TestSameUIDAuditIsCoveredByTheTemplates(t *testing.T) {

}

+

+// The generation template's instance string is "<iid>:<n>" (zero-downtime

+// spec §7.6, plan 2 D1). No specifier can take the instance ID out of it,

+// so the template keys NOTHING by %i or %I but the trampoline's argument.

+// The slice, the data and executable mounts and the log socket come from

+// the control service's per-generation drop-in (instancectl.GenDropIn).

+func TestGenerationTemplateKeysNothingByItsInstanceString(t *testing.T) {

+ u := readUnit(t, "carlos-instance-gen@.service")

+ for _, d := range directives(u) {

+ for _, key := range []string{"Slice=", "BindPaths=", "ReadWritePaths=", "StandardOutput="} {

+ if strings.HasPrefix(d, key) {

+ t.Errorf("generation template carries %q; it belongs in the drop-in", d)

+ }

+ if strings.HasPrefix(d, "BindReadOnlyPaths=") && strings.Contains(d, "/run/carlos-control/exe") {

+ t.Errorf("generation template binds an executable itself: %q", d)

+ }

+ if strings.Contains(d, "/opt/carlos/versions") {

+ t.Errorf("generation template names the version store: %q", d)

+ }

+ if strings.Contains(d, "%i") || (strings.Contains(d, "%I") && !strings.HasPrefix(d, "ExecStart=") && !strings.HasPrefix(d, "ExecStopPost=") && !strings.HasPrefix(d, "Description=")) {

+ t.Errorf("generation template keys %q by its instance string", d)

+ }

+}

diff --git a/internal/instancectl/client.go b/internal/instancectl/client.go

index cae0ca4cb..9d495efd5 100644

--- a/internal/instancectl/client.go

+++ b/internal/instancectl/client.go

@@ -170,6 +170,53 @@ func (c *Client) Reset(ctx context.Context, host string, role Role) error {

return err

}

+// genValues is the query of a generation verb: the run role and gen=<n>.

+func genValues(host string, n uint64) url.Values {

+ q := hostRole(host, RoleRun)

+ q.Set("gen", strconv.FormatUint(n, 10))

+ return q

+}

+

+// StartGen starts generation n's unit (zero-downtime spec §7.6) with its

+// tier and returns the invocation id the server read after the start. A

+// failure whose cleanup could not prove the unit gone wraps ErrGenUnproven.

+func (c *Client) StartGen(ctx context.Context, host string, n uint64, memoryMax int64) (string, error) {

+ q := genValues(host, n)

+ q.Set("memory_max", strconv.FormatInt(memoryMax, 10))

+ var out StartResponse

+ if status, _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/start", q, &out); err != nil {

+ if status == statusGenUnproven {

+ return "", fmt.Errorf("%w: %w", ErrGenUnproven, err)

+ }

+ return "", err

+ }

+ return out.InvocationID, nil

+}

+

+// StopGen stops generation n's unit only. kill kills it outright and

+// returns once its cgroup is empty.

+func (c *Client) StopGen(ctx context.Context, host string, n uint64, kill bool) error {

+ q := genValues(host, n)

+ if kill {

+ q.Set("kill", "1")

+ }

+ _, _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/stop", q, nil)

+ return err

+}

+

+func (c *Client) ResetGen(ctx context.Context, host string, n uint64) error {

+ _, _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/reset", genValues(host, n), nil)

+ return err

+}

+

+// Parent sets host's slice memory, the generation parent's backstop (spec

+// §7.3): one tier, or two summed during an overlap.

+func (c *Client) Parent(ctx context.Context, host string, memoryMax int64) error {

+ q := url.Values{"host": {host}, "memory_max": {strconv.FormatInt(memoryMax, 10)}}

+ _, _, err := c.do(ctx, DeadlineControl, http.MethodPost, "/parent", q, nil)

+ return err

+}

+

// Sweep stops every instance unit on the box (design §6.7). A 502 carries

// the units that were stopped and the ones that would not; both are

// returned with the error.

diff --git a/internal/instancectl/gen.go b/internal/instancectl/gen.go

new file mode 100644

index 000000000..0bdcd5315

--- /dev/null

+++ b/internal/instancectl/gen.go

@@ -0,0 +1,325 @@

+package instancectl

+

+import (

+ "context"

+ "errors"

+ "net/http"

+ "os"

+ "path/filepath"

+ "strconv"

+ "strings"

+ "time"

+

+ "github.com/carlosframework/platform/internal/cgroup"

+ "github.com/carlosframework/platform/internal/provision"

+)

+

+// Generation units (zero-downtime deploys spec §7.6, plan 2 D1). The

+// template carlos-instance-gen@<iid>:<n>.service cannot key its slice and

+// data mounts by itself: systemd has no specifier for part of an instance

+// string. So before each start this service writes a runtime drop-in with

+// everything the template leaves out, reloads, checks that systemd took

+// it, and only then starts the unit.

+

+// ErrGenUnproven is StartGen's answer when the start failed after systemd

+// may have started the unit and the stop that followed did not prove the

+// unit's cgroup empty. The generation may still be running: its files are

+// kept, and the caller must not release anything it owns for it.

+var ErrGenUnproven = errors.New("instance control: generation termination unproven")

+

+// statusGenUnproven is the status that carries ErrGenUnproven. No other

+// verb answers it, so the client can tell it from an ordinary 502.

+const statusGenUnproven = http.StatusInternalServerError

+

+// DropInName is the drop-in's file name inside <unit>.d.

+const DropInName = "50-carlos-generation.conf"

+

+// GenDropIn is the drop-in's text. The paths are the templates' literal

+// ones (/data/instances, /run/carlos/instances, /run/carlos-control/exe).

+// The memory lines are the generation's own tier (spec §7.3). The

+// template's OOMPolicy=kill gives the unit's cgroup memory.oom.group=1.

+func GenDropIn(host string, n uint64, tier cgroup.Limit) string {

+ iid := provision.InstanceID(host)

+ key := GenKey(host, n)

+ max := strconv.FormatInt(tier.MemoryMax, 10)

+ return "# Written by carlos instance-control for one generation; removed when it stops (zero-downtime plan 2).\n" +

+ "[Service]\n" +

+ "Slice=" + SliceName(host) + "\n" +

+ "BindPaths=/data/instances/" + iid + " /run/carlos/instances/" + iid + "\n" +

+ "ReadWritePaths=/data/instances/" + iid + " /run/carlos/instances/" + iid + "\n" +

+ "WorkingDirectory=/data/instances/" + iid + "\n" +

+ "BindReadOnlyPaths=/run/carlos-control/exe/" + key + "/app:/opt/carlos/app /run/carlos-control/exe/" + key + "/live:/opt/carlos/live\n" +

+ "StandardOutput=file:" + GenLogSocket(host, n) + "\n" +

+ "MemoryMax=" + max + "\n" +

+ "MemoryHigh=" + strconv.FormatInt(cgroup.HighOf(tier.MemoryMax), 10) + "\n" +

+ "MemorySwapMax=" + max + "\n"

+}

+

+// writeGenDropIn writes the drop-in atomically: a temp file systemd never

+// reads (only *.conf is read), renamed into place.

+func (s *Server) writeGenDropIn(host string, n uint64, tier cgroup.Limit) error {

+ dir := filepath.Join(s.DropInDir, GenUnitName(host, n)+".d")

+ if err := os.MkdirAll(dir, 0o755); err != nil {

+ return err

+ }

+ tmp := filepath.Join(dir, "."+DropInName+".tmp")

+ if err := os.WriteFile(tmp, []byte(GenDropIn(host, n, tier)), 0o644); err != nil {

+ return err

+ }

+ return os.Rename(tmp, filepath.Join(dir, DropInName))

+}

+

+// ensureGenExe lays out generation n's view of its build under ExeRoot,

+// keyed by GenKey, as ensureHostExe does for the original template. Its

+// target is the host's live link, as today. Plan 3 binds the attempt's

+// staged path instead (spec §7.5 step 3).

+func (s *Server) ensureGenExe(host string, n uint64) error {

+ return ensureExe(s.ExeRoot, GenKey(host, n), filepath.Join(LiveDir, host), host)

+}

+

+// removeGenFiles removes a generation unit's drop-in and executable layout,

+// by the unit's name as systemd reports it. Best-effort: a leftover is a

+// few bytes in /run, never a running unit.

+func (s *Server) removeGenFiles(unit string) {

+ esc, n, ok := ParseGenUnitName(unit)

+ if !ok {

+ return

+ }

+ if s.DropInDir != "" {

+ _ = os.RemoveAll(filepath.Join(s.DropInDir, unit+".d"))

+ }

+ iid, ok := Unescape(esc)

+ if !ok || iid == "" || strings.ContainsAny(iid, "/\x00") || iid == "." || iid == ".." {

+ return

+ }

+ if s.ExeRoot != "" {

+ _ = os.RemoveAll(filepath.Join(s.ExeRoot, GenKeyOf(iid, n)))

+ }

+}

+

+// slicePopulated reports whether anything runs in host's slice. Absent is

+// no. Unreadable is yes, so a raised limit is never lowered on a guess.

+func (s *Server) slicePopulated(host string) bool {

+ b, err := os.ReadFile(filepath.Join(s.CgroupRoot, parentSlices, SliceName(host), "cgroup.events"))

+ if errors.Is(err, os.ErrNotExist) {

+ return false

+ }

+ if err != nil {

+ return true

+ }

+ return !strings.Contains(string(b), "populated 0")

+}

+

+// startGen is /start with a generation. The host slice is the parent (spec

+// §7.3): a cold start sets it to the tier. A start that finds the slice

+// populated leaves it alone, because a swap raised it with /parent before

+// the successor's start.

+func (s *Server) startGen(w http.ResponseWriter, r *http.Request, req request) {

+ ctx := r.Context()

+ unit := GenUnitName(req.host, req.gen)

+ if s.DropInDir == "" {

+ http.Error(w, "not starting "+unit+": generation units are not configured on this service", http.StatusNotImplemented)

+ return

+ }

+ s.markGenStarting(unit, true)

+ defer s.markGenStarting(unit, false)

+ // A refusal for the disk writes nothing. An exe layout that fails

+ // half-way may have made its directory, so that failure cleans up too.

+ if s.refuseStart(w, req.host, unit, func() error {

+ err := s.ensureGenExe(req.host, req.gen)

+ if err != nil {

+ s.removeGenFiles(unit)

+ }

+ return err

+ }) {

+ return

+ }

+ fail := func(status int, msg string) {

+ s.removeGenFiles(unit)

+ http.Error(w, msg, status)

+ }

+ if err := s.writeGenDropIn(req.host, req.gen, req.tier); err != nil {

+ fail(http.StatusBadGateway, "drop-in: "+err.Error())

+ return

+ }

+ if out, err := s.Systemctl(ctx, "daemon-reload"); err != nil {

+ fail(http.StatusBadGateway, "daemon-reload: "+strings.TrimSpace(out+" "+err.Error()))

+ return

+ }

+ if got := s.showValue(ctx, "Slice", unit); got != SliceName(req.host) {

+ fail(http.StatusBadGateway, "not starting "+unit+": its drop-in did not apply (slice "+strconv.Quote(got)+")")

+ return

+ }

+ if !s.slicePopulated(req.host) {

+ if err := s.setTier(ctx, req.host, req.tier); err != nil {

+ fail(http.StatusBadGateway, "set-property: "+err.Error())

+ return

+ }

+ // From the start on, systemd may have started the unit, whatever the

+ // answer says. A failure past this point kills it and proves it empty

+ // before its files go; one that cannot prove that keeps the files and

+ // says so (statusGenUnproven).

+ if out, err := s.Systemctl(ctx, "start", unit); err != nil {

+ s.abandonGen(ctx, w, req.host, unit, "start: "+strings.TrimSpace(out+" "+err.Error()))

+ return

+ }

+ inv, err := s.Systemctl(ctx, "show", "-p", "InvocationID", "--value", unit)

+ if err != nil {

+ s.abandonGen(ctx, w, req.host, unit, "show: "+err.Error())

+ return

+ }

+ s.logf("instance-control: %s: started %s memory.max=%d", req.host, unit, req.tier.MemoryMax)

+ writeJSON(w, http.StatusOK, StartResponse{InvocationID: strings.TrimSpace(inv)})

+}

+

+// abandonGen answers a generation start that failed after systemctl start

+// was issued. The unit may be running, so it is killed and waited on, on a

+// context the client's hang-up cannot cancel. Only a stop that proved the

+// unit's cgroup empty removes its files; otherwise they stay, and the

+// answer is statusGenUnproven so the client knows the generation may live.

+func (s *Server) abandonGen(ctx context.Context, w http.ResponseWriter, host, unit, msg string) {

+ dctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), DeadlineControl+s.waitBudget())

+ defer cancel()

+ if err := s.killGen(dctx, host, unit); err != nil {

+ s.logf("instance-control: %s: abandon %s: termination unproven, files kept: %v", host, unit, err)

+ http.Error(w, msg+"; termination unproven: "+err.Error(), statusGenUnproven)

+ return

+ }

+ s.removeGenFiles(unit)

+ http.Error(w, msg, http.StatusBadGateway)

+}

+

+// killGen kills a generation unit outright, stops it, and waits until its

+// cgroup is empty (spec §8.2 step 1). nil is the proof that nothing of it

+// runs. A kill that finds nothing to kill is not a failure: the stop and

+// the wait are the proof.

+func (s *Server) killGen(ctx context.Context, host, unit string) error {

+ if out, err := s.Systemctl(ctx, "kill", "--signal=SIGKILL", unit); err != nil && !strings.Contains(out+err.Error(), "not loaded") {

+ s.logf("instance-control: %s: kill %s: %v", host, unit, err)

+ }

+ if out, err := s.Systemctl(ctx, "stop", unit); err != nil {

+ return errors.New("stop: " + strings.TrimSpace(out+" "+err.Error()))

+ }

+ wctx, cancel := context.WithTimeout(ctx, s.waitBudget())

+ defer cancel()

+ if err := WaitCgroupEmpty(wctx, s.CgroupRoot, SliceName(host), unit, 50*time.Millisecond); err != nil {

+ return errors.New("stop: " + err.Error())

+ }

+ return nil

+}

+

+// stopGen is /stop with a generation. With kill=1 the generation is killed

+// outright (spec §8.2 step 1) and waited on until its cgroup is empty.

+// Either way the stop acts on that generation's unit only, and its files

+// go once it has stopped.

+func (s *Server) stopGen(w http.ResponseWriter, r *http.Request, req request) {

+ ctx := r.Context()

+ unit := GenUnitName(req.host, req.gen)

+ if req.kill {

+ if err := s.killGen(ctx, req.host, unit); err != nil {

+ http.Error(w, err.Error(), http.StatusBadGateway)

+ return

+ }

+ } else if out, err := s.Systemctl(ctx, "stop", unit); err != nil {

+ http.Error(w, "stop: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)

+ return

+ }

+ s.removeGenFiles(unit)

+ w.WriteHeader(http.StatusNoContent)

+}

+

+func (s *Server) markGenStarting(unit string, on bool) {

+ s.genMu.Lock()

+ defer s.genMu.Unlock()

+ if !on {

+ delete(s.genStarting, unit)

+ return

+ }

+ if s.genStarting == nil {

+ s.genStarting = map[string]bool{}

+ }

+ s.genStarting[unit] = true

+}

+

+// sweepGenLeftovers is the sweep's recovery for generation units that are

+// gone but kept their files: their stop's answer was lost, or the agent

+// that would have finalised them restarted. Such a unit is inactive, so

+// the sweep had nothing to stop. Each drop-in directory under DropInDir is

+// checked: a unit the sweep could not stop is skipped, as is one a start

+// is preparing. The rest lose their files only when systemd reports them

+// inactive or failed AND their cgroup is empty. No daemon-reload follows:

+// stopGen removes files the same way, and a generation's name is never

+// started again.

+func (s *Server) sweepGenLeftovers(ctx context.Context, failed map[string]string) {

+ if s.DropInDir == "" {

+ return

+ }

+ entries, err := os.ReadDir(s.DropInDir)

+ if err != nil {

+ if !errors.Is(err, os.ErrNotExist) {

+ s.logf("instance-control: sweep: %s: %v", s.DropInDir, err)

+ }

+ return

+ }

+ for _, e := range entries {

+ unit, ok := strings.CutSuffix(e.Name(), ".d")

+ if !ok || !e.IsDir() {

+ continue

+ }

+ if _, _, ok := ParseGenUnitName(unit); !ok {

+ continue

+ }

+ if _, stuck := failed[unit]; stuck {

+ continue

+ }

+ s.sweepGenLeftover(ctx, unit)

+ }

+}

+

+func (s *Server) sweepGenLeftover(ctx context.Context, unit string) {

+ // Held across the check and the removal, so a start of the same unit

+ // cannot write fresh files in between.

+ s.genMu.Lock()

+ defer s.genMu.Unlock()

+ if s.genStarting[unit] {

+ return

+ }

+ if state := s.showValue(ctx, "ActiveState", unit); state != "inactive" && state != "failed" {

+ return

+ }

+ _, esc, _ := ParseUnitName(unit)

+ // One look at the cgroup, no wait: a populated one is not swept.

+ wctx, cancel := context.WithTimeout(ctx, time.Millisecond)

+ defer cancel()

+ if err := WaitCgroupEmpty(wctx, s.CgroupRoot, "carlos-instances-"+esc+".slice", unit, time.Millisecond); err != nil {

+ return

+ }

+ s.removeGenFiles(unit)

+ s.logf("instance-control: sweep: removed the files of %s", unit)

+}

+

+// handleParent sets the host slice's memory: the generation parent's

+// backstop (spec §7.3). The swap raises it before the successor starts and

+// restores it once the retiring generation has stopped. CPU and IO shares

+// stay at one tier (plan 2 D6).

+func (s *Server) handleParent(w http.ResponseWriter, r *http.Request) {

+ q := r.URL.Query()

+ host := q.Get("host")

+ if !provision.ValidHostShape(host) {

+ http.Error(w, "host "+strconv.Quote(host)+" is not a host", http.StatusBadRequest)

+ return

+ }

+ n, err := strconv.ParseInt(q.Get("memory_max"), 10, 64)

+ if err != nil || !cgroup.ValidParentLimit(cgroup.Limit{MemoryMax: n}) {

+ http.Error(w, "memory_max "+strconv.Quote(q.Get("memory_max"))+" is not one tier or two", http.StatusBadRequest)

+ return

+ }

+ max := strconv.FormatInt(n, 10)

+ if out, err := s.Systemctl(r.Context(), "set-property", "--runtime", SliceName(host),

+ "MemoryMax="+max, "MemoryHigh="+strconv.FormatInt(cgroup.HighOf(n), 10), "MemorySwapMax="+max); err != nil {

+ http.Error(w, "set-property: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)

+ return

+ }

+ w.WriteHeader(http.StatusNoContent)

+}

diff --git a/internal/instancectl/hardening.go b/internal/instancectl/hardening.go

index 2653c9902..e141cdada 100644

--- a/internal/instancectl/hardening.go

+++ b/internal/instancectl/hardening.go

@@ -314,9 +314,10 @@ func (s *Server) diskRoom() error {

return nil

}

-// refuseStart answers a start the disk or the exe layout cannot take. It

-// reports whether it answered.

-func (s *Server) refuseStart(w http.ResponseWriter, host, unit string) bool {

+// refuseStart answers a start the disk or the exe layout cannot take. exe

+// lays out the unit's view of its build (ensureHostExe, or ensureGenExe

+// for a generation). It reports whether it answered.

+func (s *Server) refuseStart(w http.ResponseWriter, host, unit string, exe func() error) bool {

if err := s.diskRoom(); err != nil {

s.logf("instance-control: %s: not starting %s: %v", host, unit, err)

status := http.StatusBadGateway

@@ -326,8 +327,8 @@ func (s *Server) refuseStart(w http.ResponseWriter, host, unit string) bool {

http.Error(w, "not starting "+unit+": "+err.Error(), status)

return true

}

- if s.ExeRoot != "" {

- if err := s.ensureHostExe(host); err != nil {

+ if s.ExeRoot != "" && exe != nil {

+ if err := exe(); err != nil {

http.Error(w, "exe: "+err.Error(), http.StatusBadGateway)

return true

}

diff --git a/internal/instancectl/names.go b/internal/instancectl/names.go

index f07a94624..323407d88 100644

--- a/internal/instancectl/names.go

+++ b/internal/instancectl/names.go

@@ -14,6 +14,8 @@ package instancectl

import (

"fmt"

+ "path/filepath"

+ "strconv"

"strings"

"time"

@@ -93,6 +95,95 @@ func SliceName(host string) string {

return "carlos-instances-" + Escape(provision.InstanceID(host)) + ".slice"

}

+// UnitPrefixGen is the generation template (zero-downtime deploys spec

+// §7.6): one instance per spawn of a host's run role. The three templates

+// above stay for every box and binary still running them.

+const UnitPrefixGen = "carlos-instance-gen@"

+

+// GenInstance is the generation template's unescaped instance string,

+// "<iid>:<n>" (%I inside the unit). No instance ID holds ':'.

+func GenInstance(host string, n uint64) string {

+ return provision.InstanceID(host) + ":" + strconv.FormatUint(n, 10)

+}

+

+// ParseGenInstance splits "<iid>:<n>", the trampoline's and the exit

+// writer's half of §7.6. n is canonical decimal and never 0. iid is

+// non-empty and holds no ':', '/' or NUL, and is neither "." nor "..".

+func ParseGenInstance(s string) (iid string, n uint64, ok bool) {

+ i := strings.LastIndexByte(s, ':')

+ if i <= 0 {

+ return "", 0, false

+ }

+ iid, num := s[:i], s[i+1:]

+ n, err := strconv.ParseUint(num, 10, 64)

+ if err != nil || n == 0 || strconv.FormatUint(n, 10) != num {

+ return "", 0, false

+ }

+ if strings.ContainsAny(iid, ":/\x00") || iid == "." || iid == ".." {

+ return "", 0, false

+ }

+ return iid, n, true

+}

+

+// GenUnitName is carlos-instance-gen@<Escape(GenInstance(host, n))>.service.

+func GenUnitName(host string, n uint64) string {

+ return UnitPrefixGen + Escape(GenInstance(host, n)) + ".service"

+}

+

+// ParseGenUnitName recognises a generation unit and returns its ESCAPED

+// instance ID (systemd reports names escaped) and its generation.

+func ParseGenUnitName(unit string) (esc string, n uint64, ok bool) {

+ body, found := strings.CutSuffix(unit, ".service")

+ if !found {

+ return "", 0, false

+ }

+ inst, found := strings.CutPrefix(body, UnitPrefixGen)

+ if !found {

+ return "", 0, false

+ }

+ return ParseGenInstance(inst)

+}

+

+// GenKeyOf and GenKey key a generation's files: its executable mount

+// directory and its log socket (spec §7.6). It is "<iid>@<n>", not the

+// unit's "<iid>:<n>", because ':' is BindPaths='s field separator. No host

+// and no instance ID holds '@', so a key never meets a legacy name.

+func GenKeyOf(iid string, n uint64) string { return iid + "@" + strconv.FormatUint(n, 10) }

+

+func GenKey(host string, n uint64) string { return GenKeyOf(provision.InstanceID(host), n) }

+

+// GenLogSocket is the generation's run log socket:

+// /run/carlos/logs/<GenKey>.run.sock.

+func GenLogSocket(host string, n uint64) string {

+ return filepath.Join(provision.SocketDir, "logs", GenKey(host, n)+".run.sock")

+}

+

+// Unescape is Escape's inverse for the names this platform produces:

+// "\xHH" becomes the byte and "-" becomes "/". It reports false on a

+// malformed escape.

+func Unescape(s string) (string, bool) {

+ var b strings.Builder

+ for i := 0; i < len(s); i++ {

+ switch c := s[i]; {

+ case c == '-':

+ b.WriteByte('/')

+ case c == '\\':

+ if i+3 >= len(s) || s[i+1] != 'x' {

+ return "", false

+ }

+ v, err := strconv.ParseUint(s[i+2:i+4], 16, 8)

+ if err != nil {

+ return "", false

+ }

+ b.WriteByte(byte(v))

+ i += 3

+ default:

+ b.WriteByte(c)

+ }

+ return b.String(), true

+}

+

// ParseUnitName recognises the three templates' instances and returns the

// role and the ESCAPED identifier (systemd reports names escaped). Anything

// else, including a template with no instance, is not ours.

@@ -100,6 +191,10 @@ func ParseUnitName(unit string) (Role, string, bool) {

if !strings.HasSuffix(unit, ".service") {

return "", "", false

}

+ if esc, _, ok := ParseGenUnitName(unit); ok {

+ // A generation unit runs in its host's slice, as the run role.

+ return RoleRun, esc, true

+ }

body := strings.TrimSuffix(unit, ".service")

for _, p := range []struct {

prefix string

@@ -133,6 +228,10 @@ type Launch struct {

// LaunchFileName is "launch.<role>.json" inside the instance directory.

func LaunchFileName(role Role) string { return "launch." + string(role) + ".json" }

+// GenLaunchFileName is generation n's launch file in the instance

+// directory: "launch.gen.<n>.json".

+func GenLaunchFileName(n uint64) string { return "launch.gen." + strconv.FormatUint(n, 10) + ".json" }

+

// ExitRecord is the advisory record the unit's ExecStopPost leaves in the

// instance's run directory (design §6.5). It decides the wording of the

// member-facing event line and nothing else.

@@ -149,6 +248,15 @@ func ExitRecordName(role Role, invocation string) string {

return "exit." + string(role) + "." + invocation + ".json"

}

+// GenExitRecordName is generation n's exit record: "exit.gen.<n>.<inv>.json".

+// It is never under "exit.run.", which prepare sweeps for the run role.

+func GenExitRecordName(n uint64, invocation string) string {

+ return GenExitPrefix(n) + invocation + ".json"

+}

+

+// GenExitPrefix is every exit record of generation n.

+func GenExitPrefix(n uint64) string { return "exit.gen." + strconv.FormatUint(n, 10) + "." }

+

// ProbeDirName is the instance directory the readiness probe runs in. It

// starts with an underscore, which no valid host does, so it can never

// collide with a real instance's directory.

diff --git a/internal/instancectl/names_gen_test.go b/internal/instancectl/names_gen_test.go

new file mode 100644

index 000000000..e968a1f36

--- /dev/null

+++ b/internal/instancectl/names_gen_test.go

@@ -0,0 +1,123 @@

+package instancectl

+

+import (

+ "math"

+ "strings"

+ "testing"

+

+ "github.com/carlosframework/platform/internal/provision"

+)

+

+func TestGenerationUnitNames(t *testing.T) {

+ const host = "bill-2.example.com"

+ if got := GenInstance(host, 7); got != "bill-2.example.com:7" {

+ t.Errorf("GenInstance = %q", got)

+ }

+ if got := GenUnitName(host, 7); got != `carlos-instance-gen@bill\x2d2.example.com:7.service` {

+ t.Errorf("GenUnitName = %q", got)

+ }

+ esc, n, ok := ParseGenUnitName(GenUnitName(host, 7))

+ if !ok || esc != `bill\x2d2.example.com` || n != 7 {

+ t.Errorf("ParseGenUnitName = %q %d %v", esc, n, ok)

+ }

+ // The sweep's parser knows the generation template, as a run unit in

+ // the host's slice.

+ role, esc, ok := ParseUnitName(GenUnitName(host, 7))

+ if !ok || role != RoleRun || esc != `bill\x2d2.example.com` {

+ t.Errorf("ParseUnitName(gen) = %q %q %v", role, esc, ok)

+ }

+ // Flag off: the original names are unchanged.

+ if got := UnitName(RoleRun, host); got != `carlos-instance@bill\x2d2.example.com.service` {

+ t.Errorf("UnitName(run) = %q", got)

+ }

+ if role, esc, ok := ParseUnitName(`carlos-instance@bill\x2d2.example.com.service`); !ok || role != RoleRun || esc != `bill\x2d2.example.com` {

+ t.Errorf("ParseUnitName(run) = %q %q %v", role, esc, ok)

+ }

+ for _, bad := range []string{"carlos-instance-gen@.service", `carlos-instance-gen@bill.example.com.service`, "carlos-instance-gen@bill.example.com:0.service"} {

+ if _, _, ok := ParseUnitName(bad); ok {

+ t.Errorf("ParseUnitName(%q) accepted", bad)

+ }

+}

+

+func TestParseGenInstanceIsStrict(t *testing.T) {

+ if iid, n, ok := ParseGenInstance("_abc-123:42"); !ok || iid != "_abc-123" || n != 42 {

+ t.Errorf("ParseGenInstance = %q %d %v", iid, n, ok)

+ }

+ for _, bad := range []string{"x", "x:", ":7", "x:0", "x:07", "x:7a", "a/b:7", "..:7", "x:18446744073709551616", "a:b:7"} {

+ if _, _, ok := ParseGenInstance(bad); ok {

+ t.Errorf("ParseGenInstance(%q) accepted", bad)

+ }

+}

+

+func TestGenerationFileNames(t *testing.T) {

+ const host = "bill-2.example.com"

+ for got, want := range map[string]string{

+ GenKey(host, 7): "bill-2.example.com@7",

+ GenLogSocket(host, 7): "/run/carlos/logs/bill-2.example.com@7.run.sock",

+ GenLaunchFileName(7): "launch.gen.7.json",

+ GenExitRecordName(7, "abc"): "exit.gen.7.abc.json",

+ GenExitPrefix(7): "exit.gen.7.",

+ } {

+ if got != want {

+ t.Errorf("got %q, want %q", got, want)

+ }

+ // Never under a legacy prefix: prepare's stale sweep for the run role

+ // removes "exit.run.*" and must not touch a live generation's record.

+ if strings.HasPrefix(GenExitRecordName(7, "abc"), "exit.run.") || GenLaunchFileName(7) == LaunchFileName(RoleRun) {

+ t.Error("a generation file shares a legacy name")

+ }

+}

+

+// D2: no instance ID holds '@', so a generation key never equals a legacy

+// instance ID, whatever the host.

+func TestGenKeyNeverMeetsALegacyName(t *testing.T) {

+ for _, host := range []string{"shop.example.com", "shop.example.com.g7", strings.Repeat("x", 60) + ".example.com"} {

+ if iid := provision.InstanceID(host); strings.Contains(iid, "@") {

+ t.Fatalf("InstanceID(%q) = %q holds '@'", host, iid)

+ }

+ if strings.Count(GenKey(host, 7), "@") != 1 {

+ t.Errorf("GenKey(%q) = %q", host, GenKey(host, 7))

+ }

+ if GenLogSocket("shop.example.com", 7) == provision.LogSocket("shop.example.com.g7", "run") {

+ t.Error("a generation log socket met a legacy one")

+ }

+}

+

+// Review focus: the longest host and the largest generation stay inside

+// sockaddr_un and systemd's unit-name limit (UNIT_NAME_MAX 256 with NUL).

+func TestGenerationNamesStayBoundedForTheLongestHost(t *testing.T) {

+ host := strings.Repeat("a-b.", 60) + "example.com"

+ n := uint64(math.MaxUint64)

+ if got := GenLogSocket(host, n); len(got) > provision.MaxUnixPath {

+ t.Errorf("GenLogSocket is %d bytes: %q", len(got), got)

+ }

+ unit := GenUnitName(host, n)

+ if len(unit) > 255 {

+ t.Errorf("GenUnitName is %d bytes: %q", len(unit), unit)

+ }

+ if !strings.HasPrefix(GenInstance(host, n), "_") {

+ t.Errorf("GenInstance(long) = %q, want the hashed instance ID", GenInstance(host, n))

+ }

+ esc, m, ok := ParseGenUnitName(unit)

+ iid, uok := Unescape(esc)

+ if !ok || m != n || !uok || iid != provision.InstanceID(host) {

+ t.Errorf("round trip: %q %d %v -> %q %v", esc, m, ok, iid, uok)

+ }

+}

+

+func TestUnescapeInvertsEscape(t *testing.T) {

+ for _, s := range []string{"bill-2.example.com", "_abcdefghijklmnopqrstuv-0123456789abcdef", "a.b"} {

+ if got, ok := Unescape(Escape(s)); !ok || got != s {

+ t.Errorf("Unescape(Escape(%q)) = %q %v", s, got, ok)

+ }

+ for _, bad := range []string{`a\x2`, `a\y2d`, `a\xzz`} {

+ if _, ok := Unescape(bad); ok {

+ t.Errorf("Unescape(%q) accepted", bad)

+ }

+}

diff --git a/internal/instancectl/server.go b/internal/instancectl/server.go

index 64b96d55c..9d91bffcf 100644

--- a/internal/instancectl/server.go

+++ b/internal/instancectl/server.go

@@ -57,6 +57,10 @@ type Server struct {

// ExeRoot is where ensureExe lays out each instance's view of its own

// build (#214 item 2); empty skips it, for tests of other verbs.

ExeRoot string

+ // DropInDir is where generation units' runtime drop-ins go

+ // (/run/systemd/system). Empty refuses every generation start: the

+ // template alone runs nothing (zero-downtime plan 2 D1).

+ DropInDir string

// ResolvConf is the box's resolv.conf and ResolvedConf resolved's

// upstream list; whichever an instance sees (resolverFile) has its

// nameservers re-opened through the egress fence before every probe

@@ -95,6 +99,11 @@ type Server struct {

// cgroupWaitBudget bounds a wait for an empty cgroup; tests shorten it.

cgroupWaitBudget time.Duration

+ // genMu guards genStarting, the generation units a start is preparing

+ // now. The sweep never removes the files of one of them.

+ genMu sync.Mutex

+ genStarting map[string]bool

+

mu sync.Mutex

report ProbeReport

probed bool

@@ -138,6 +147,7 @@ func runWithClaim(args []string, claim http.Handler) {

ProbeEvery: 5 * time.Minute,

Logf: log.Printf,

ExeRoot: filepath.Join(controlsock.Dir, "exe"),

+ DropInDir: "/run/systemd/system",

ResolvConf: "/etc/resolv.conf",

ResolvedConf: ResolvedConf,

LocalAddrs: func() ([]netip.Prefix, error) { return localPrefixes(fibTriePath, ifInet6Path) },

@@ -258,6 +268,7 @@ func (s *Server) Handler() http.Handler {

mux.HandleFunc("POST /stop", s.claimAwareLegacy(s.handleStop))

mux.HandleFunc("POST /reset", s.claimAwareLegacy(s.handleReset))

mux.HandleFunc("POST /sweep", s.claimAwareLegacy(s.handleSweep))

+ mux.HandleFunc("POST /parent", s.claimAwareLegacy(s.handleParent))

mux.HandleFunc("GET /ready", s.handleReady)

return mux

}

@@ -268,22 +279,32 @@ type request struct {

host string

role Role

tier cgroup.Limit

+ gen uint64 // a generation of the run role (zero-downtime spec §7.6); 0 is none

+ kill bool

+}

+

+// verbShape is what a verb's query may carry.

+type verbShape struct {

+ roles []Role

+ tier bool

+ gen bool

+ kill bool

}

-func parseRequest(r *http.Request, roles []Role, wantTier bool) (request, error) {

+func parseRequest(r *http.Request, v verbShape) (request, error) {

q := r.URL.Query()

host := q.Get("host")

if !provision.ValidHostShape(host) {

return request{}, fmt.Errorf("host %q is not a host", host)

}

req := request{host: host}

- if len(roles) > 0 {

+ if len(v.roles) > 0 {

role, ok := ParseRole(q.Get("role"))

if !ok {

return request{}, fmt.Errorf("role %q is not run, seed or sidecar", q.Get("role"))

}

allowed := false

- for _, r := range roles {

+ for _, r := range v.roles {

allowed = allowed || r == role

}

if !allowed {

@@ -291,7 +312,7 @@ func parseRequest(r *http.Request, roles []Role, wantTier bool) (request, error)

}

req.role = role

}

- if wantTier {

+ if v.tier {

n, err := strconv.ParseInt(q.Get("memory_max"), 10, 64)

if err != nil {

return request{}, fmt.Errorf("memory_max %q is not a number", q.Get("memory_max"))

@@ -303,6 +324,26 @@ func parseRequest(r *http.Request, roles []Role, wantTier bool) (request, error)

return request{}, fmt.Errorf("memory_max %d is not a platform tier", n)

}

+ if q.Has("gen") {

+ raw := q.Get("gen")

+ if !v.gen {

+ return request{}, errors.New("this verb takes no generation")

+ }

+ n, err := strconv.ParseUint(raw, 10, 64)

+ if err != nil || n == 0 || strconv.FormatUint(n, 10) != raw {

+ return request{}, fmt.Errorf("gen %q is not a generation", raw)

+ }

+ if req.role != RoleRun {

+ return request{}, fmt.Errorf("role %s has no generations", req.role)

+ }

+ req.gen = n

+ }

+ if q.Has("kill") {

+ if !v.kill || req.gen == 0 || q.Get("kill") != "1" {

+ return request{}, errors.New("kill=1 belongs to a generation stop only")

+ }

+ req.kill = true

+ }

return req, nil

}

@@ -327,14 +368,18 @@ func writeJSON(w http.ResponseWriter, status int, v any) {

}

func (s *Server) handleStart(w http.ResponseWriter, r *http.Request) {

- req, err := parseRequest(r, []Role{RoleRun, RoleSidecar}, true)

+ req, err := parseRequest(r, verbShape{roles: []Role{RoleRun, RoleSidecar}, tier: true, gen: true})

if err != nil {

http.Error(w, err.Error(), http.StatusBadRequest)

return

}

+ if req.gen != 0 {

+ s.startGen(w, r, req)

+ return

+ }

ctx := r.Context()

unit := UnitName(req.role, req.host)

- if s.refuseStart(w, req.host, unit) {

+ if s.refuseStart(w, req.host, unit, func() error { return s.ensureHostExe(req.host) }) {

return

}

if err := s.setTier(ctx, req.host, req.tier); err != nil {

@@ -388,14 +433,14 @@ func parseShow(out string) map[string]string {

// to empty: a populated cgroup after the wait is still reported as a 502

// naming the unit.

func (s *Server) handleSeed(w http.ResponseWriter, r *http.Request) {

- req, err := parseRequest(r, nil, true)

+ req, err := parseRequest(r, verbShape{tier: true})

if err != nil {

http.Error(w, err.Error(), http.StatusBadRequest)

return

}

ctx := r.Context()

unit := UnitName(RoleSeed, req.host)

- if s.refuseStart(w, req.host, unit) {

+ if s.refuseStart(w, req.host, unit, func() error { return s.ensureHostExe(req.host) }) {

return

}

if err := s.setTier(ctx, req.host, req.tier); err != nil {

@@ -471,11 +516,15 @@ func (s *Server) stopUnit(ctx context.Context, unit string) {

}

func (s *Server) handleStop(w http.ResponseWriter, r *http.Request) {

- req, err := parseRequest(r, []Role{RoleRun, RoleSeed, RoleSidecar}, false)

+ req, err := parseRequest(r, verbShape{roles: []Role{RoleRun, RoleSeed, RoleSidecar}, gen: true, kill: true})

if err != nil {

http.Error(w, err.Error(), http.StatusBadRequest)

return

}

+ if req.gen != 0 {

+ s.stopGen(w, r, req)

+ return

+ }

if out, err := s.Systemctl(r.Context(), "stop", UnitName(req.role, req.host)); err != nil {

http.Error(w, "stop: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)

return

@@ -484,12 +533,16 @@ func (s *Server) handleStop(w http.ResponseWriter, r *http.Request) {

}

func (s *Server) handleReset(w http.ResponseWriter, r *http.Request) {

- req, err := parseRequest(r, []Role{RoleRun, RoleSeed, RoleSidecar}, false)

+ req, err := parseRequest(r, verbShape{roles: []Role{RoleRun, RoleSeed, RoleSidecar}, gen: true})

if err != nil {

http.Error(w, err.Error(), http.StatusBadRequest)

return

}

- if out, err := s.Systemctl(r.Context(), "reset-failed", UnitName(req.role, req.host)); err != nil && !strings.Contains(out+err.Error(), "not loaded") {

+ unit := UnitName(req.role, req.host)

+ if req.gen != 0 {

+ unit = GenUnitName(req.host, req.gen)

+ }

+ if out, err := s.Systemctl(r.Context(), "reset-failed", unit); err != nil && !strings.Contains(out+err.Error(), "not loaded") {

http.Error(w, "reset-failed: "+strings.TrimSpace(out+" "+err.Error()), http.StatusBadGateway)

return

}

@@ -504,7 +557,7 @@ func (s *Server) handleSweep(w http.ResponseWriter, r *http.Request) {

ctx := r.Context()

targets := map[string]bool{}

out, err := s.Systemctl(ctx, "list-units", "--all", "--plain", "--no-legend", "--no-pager",

- UnitPrefixRun+"*.service", UnitPrefixSeed+"*.service", UnitPrefixSidecar+"*.service")

+ UnitPrefixRun+"*.service", UnitPrefixSeed+"*.service", UnitPrefixSidecar+"*.service", UnitPrefixGen+"*.service")

if err != nil {

http.Error(w, "list-units: "+err.Error(), http.StatusBadGateway)

return

@@ -565,7 +618,9 @@ func (s *Server) handleSweep(w http.ResponseWriter, r *http.Request) {

continue

}

stopped = append(stopped, unit)

+ s.removeGenFiles(unit)

}

+ s.sweepGenLeftovers(ctx, failed)

if len(failed) > 0 {

writeJSON(w, http.StatusBadGateway, SweepResponse{Units: stopped, Failed: failed})

return

diff --git a/internal/instancectl/server_gen_test.go b/internal/instancectl/server_gen_test.go

new file mode 100644

index 000000000..b0fd390cb

--- /dev/null

+++ b/internal/instancectl/server_gen_test.go

@@ -0,0 +1,453 @@

+package instancectl

+

+import (

+ "context"

+ "encoding/json"

+ "errors"

+ "os"

+ "path/filepath"

+ "strings"

+ "testing"

+ "time"

+

+ "github.com/carlosframework/platform/internal/cgroup"

+)

+

+const genHost = "bill-2.example.com"

+

+var genUnit = GenUnitName(genHost, 7)

+

+func genServer(t *testing.T, ctl *fakeCtl) (*Server, string, string) {

+ t.Helper()

+ dropins, exe := t.TempDir(), t.TempDir()

+ s, sock := serveOnSocket(t, ctl, func(s *Server) { s.DropInDir = dropins; s.ExeRoot = exe })

+ return s, sock, dropins

+}

+

+func genCtl() *fakeCtl {

+ return &fakeCtl{answers: map[string]string{

+ "show -p Slice --value": SliceName(genHost) + "\n",

+ "show -p InvocationID --value": "abc123\n",

+ "list-units": "",

+ "list-jobs": "",

+ "show -p ActiveState --value": "inactive\n",

+ }}

+}

+

+func TestGenDropInText(t *testing.T) {

+ want := `# Written by carlos instance-control for one generation; removed when it stops (zero-downtime plan 2).

+[Service]

+Slice=carlos-instances-bill\x2d2.example.com.slice

+BindPaths=/data/instances/bill-2.example.com /run/carlos/instances/bill-2.example.com

+ReadWritePaths=/data/instances/bill-2.example.com /run/carlos/instances/bill-2.example.com

+WorkingDirectory=/data/instances/bill-2.example.com

+BindReadOnlyPaths=/run/carlos-control/exe/bill-2.example.com@7/app:/opt/carlos/app /run/carlos-control/exe/bill-2.example.com@7/live:/opt/carlos/live

+StandardOutput=file:/run/carlos/logs/bill-2.example.com@7.run.sock

+MemoryMax=536870912

+MemoryHigh=483183819

+MemorySwapMax=536870912

+`

+ if got := GenDropIn(genHost, 7, cgroup.TierDefault); got != want {

+ t.Errorf("GenDropIn =\n%s\nwant\n%s", got, want)

+ }

+}

+

+func TestGenStartWritesTheDropInReloadsChecksTheSliceThenStarts(t *testing.T) {

+ ctl := genCtl()

+ s, sock, dropins := genServer(t, ctl)

+ inv, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax)

+ if err != nil || inv != "abc123" {

+ t.Fatalf("StartGen = %q, %v", inv, err)

+ }

+ raw, err := os.ReadFile(filepath.Join(dropins, genUnit+".d", DropInName))

+ if err != nil || string(raw) != GenDropIn(genHost, 7, cgroup.TierDefault) {

+ t.Fatalf("drop-in = %q, %v", raw, err)

+ }

+ if target, err := os.Readlink(filepath.Join(s.ExeRoot, GenKey(genHost, 7), "app")); err != nil || target != "/opt/carlos/live/"+genHost {

+ t.Fatalf("exe app link = %q, %v", target, err)

+ }

+ want := strings.Join([]string{

+ "daemon-reload",

+ "show -p Slice --value " + genUnit,

+ `set-property --runtime carlos-instances-bill\x2d2.example.com.slice MemoryMax=536870912 MemoryHigh=483183819 MemorySwapMax=536870912 CPUWeight=100 CPUQuota=100% IOWeight=100`,

+ "start " + genUnit,

+ "show -p InvocationID --value " + genUnit,

+ }, "\n")

+ if got := ctl.joined(); got != want {

+ t.Errorf("calls =\n%s\nwant\n%s", got, want)

+ }

+}

+

+// A successor's start during an overlap: the slice holds the incumbent,

+// and its raised limit is the swap's. The start leaves it alone.

+func TestGenStartLeavesAPopulatedSliceAlone(t *testing.T) {

+ ctl := genCtl()

+ s, sock, _ := genServer(t, ctl)

+ events := filepath.Join(s.CgroupRoot, parentSlices, SliceName(genHost), "cgroup.events")

+ if err := os.MkdirAll(filepath.Dir(events), 0o755); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.WriteFile(events, []byte("populated 1\nfrozen 0\n"), 0o644); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax); err != nil {

+ t.Fatal(err)

+ }

+ if strings.Contains(ctl.joined(), "set-property") {

+ t.Errorf("a successor's start touched the live slice:\n%s", ctl.joined())

+ }

+}

+

+// Review focus 1, nested: an empty slice is reset to the tier.

+func TestGenStartResetsAnEmptySlice(t *testing.T) {

+ ctl := genCtl()

+ s, sock, _ := genServer(t, ctl)

+ events := filepath.Join(s.CgroupRoot, parentSlices, SliceName(genHost), "cgroup.events")

+ if err := os.MkdirAll(filepath.Dir(events), 0o755); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.WriteFile(events, []byte("populated 0\nfrozen 0\n"), 0o644); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax); err != nil {

+ t.Fatal(err)

+ }

+ if !strings.Contains(ctl.joined(), "set-property --runtime "+SliceName(genHost)+" MemoryMax=536870912") {

+ t.Errorf("a cold start left the slice as it was:\n%s", ctl.joined())

+ }

+}

+

+// Review focus 5: a drop-in systemd did not apply means the default slice

+// and an empty /data. Refuse, and leave nothing behind.

+func TestGenStartRefusesWhenTheDropInDidNotApply(t *testing.T) {

+ ctl := genCtl()

+ ctl.answers["show -p Slice --value"] = `system-carlos\x2dinstance\x2dgen.slice` + "\n"

+ s, sock, dropins := genServer(t, ctl)

+ if _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax); err == nil {

+ t.Fatal("started a generation whose drop-in did not apply")

+ }

+ if strings.Contains(ctl.joined(), "start "+genUnit) {

+ t.Errorf("the unit was started:\n%s", ctl.joined())

+ }

+ if _, err := os.Stat(filepath.Join(dropins, genUnit+".d")); !os.IsNotExist(err) {

+ t.Errorf("drop-in left behind: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(s.ExeRoot, GenKey(genHost, 7))); !os.IsNotExist(err) {

+ t.Errorf("exe dir left behind: %v", err)

+ }

+}

+

+func TestGenStartRefusedWithoutADropInDir(t *testing.T) {

+ ctl := genCtl()

+ _, sock := serveOnSocket(t, ctl)

+ if _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax); err == nil || !strings.Contains(err.Error(), "501") {

+ t.Fatalf("StartGen without DropInDir: %v, want a 501", err)

+ }

+ if ctl.joined() != "" {

+ t.Errorf("systemctl was called:\n%s", ctl.joined())

+ }

+}

+

+func TestGenParametersRefusedWhereTheyDoNotBelong(t *testing.T) {

+ ctl := genCtl()

+ _, c := newServer(t, ctl, func(s *Server) { s.DropInDir = t.TempDir() })

+ for _, path := range []string{

+ "/start?host=" + genHost + "&role=sidecar&memory_max=536870912&gen=7",

+ "/start?host=" + genHost + "&role=run&memory_max=536870912&gen=0",

+ "/start?host=" + genHost + "&role=run&memory_max=536870912&gen=07",

+ "/seed?host=" + genHost + "&memory_max=536870912&gen=7",

+ "/stop?host=" + genHost + "&role=run&kill=1",

+ "/start?host=" + genHost + "&role=run&memory_max=536870912&gen=7&kill=1",

+ } {

+ if resp := post(t, c, path); resp.StatusCode != 400 {

+ t.Errorf("%s: status %d, want 400", path, resp.StatusCode)

+ }

+ if ctl.joined() != "" {

+ t.Errorf("systemctl was called:\n%s", ctl.joined())

+ }

+}

+

+func TestGenStopWithKillKillsStopsWaitsAndRemovesFiles(t *testing.T) {

+ ctl := genCtl()

+ s, sock, dropins := genServer(t, ctl)

+ if err := s.writeGenDropIn(genHost, 7, cgroup.TierDefault); err != nil {

+ t.Fatal(err)

+ }

+ if err := s.ensureGenExe(genHost, 7); err != nil {

+ t.Fatal(err)

+ }

+ if err := NewClient(sock).StopGen(context.Background(), genHost, 7, true); err != nil {

+ t.Fatal(err)

+ }

+ if want := "kill --signal=SIGKILL " + genUnit + "\nstop " + genUnit; ctl.joined() != want {

+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)

+ }

+ if _, err := os.Stat(filepath.Join(dropins, genUnit+".d")); !os.IsNotExist(err) {

+ t.Errorf("drop-in survived the stop: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(s.ExeRoot, GenKey(genHost, 7))); !os.IsNotExist(err) {

+ t.Errorf("exe dir survived the stop: %v", err)

+ }

+}

+

+func TestParentSetsOnlyTheSliceMemory(t *testing.T) {

+ ctl := genCtl()

+ _, sock := serveOnSocket(t, ctl)

+ c := NewClient(sock)

+ if err := c.Parent(context.Background(), genHost, cgroup.OverlapLimit(cgroup.TierDefault, cgroup.TierDefault).MemoryMax); err != nil {

+ t.Fatal(err)

+ }

+ want := `set-property --runtime carlos-instances-bill\x2d2.example.com.slice MemoryMax=1073741824 MemoryHigh=966367638 MemorySwapMax=1073741824`

+ if ctl.joined() != want {

+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)

+ }

+ if err := c.Parent(context.Background(), genHost, 64<<20); err == nil {

+ t.Error("a parent limit that is no tier sum was accepted")

+ }

+}

+

+func TestSweepStopsGenerationUnitsAndRemovesTheirFiles(t *testing.T) {

+ ctl := genCtl()

+ ctl.answers["list-units"] = genUnit + " loaded active running carlos exec-backed instance generation\n"

+ s, sock, dropins := genServer(t, ctl)

+ if err := s.writeGenDropIn(genHost, 7, cgroup.TierDefault); err != nil {

+ t.Fatal(err)

+ }

+ if err := s.ensureGenExe(genHost, 7); err != nil {

+ t.Fatal(err)

+ }

+ out, err := NewClient(sock).Sweep(context.Background())

+ if err != nil || len(out.Units) != 1 || out.Units[0] != genUnit {

+ raw, _ := json.Marshal(out)

+ t.Fatalf("sweep = %s, %v", raw, err)

+ }

+ if !strings.Contains(ctl.joined(), UnitPrefixGen+"*.service") {

+ t.Errorf("list-units did not ask for generation units:\n%s", ctl.joined())

+ }

+ if _, err := os.Stat(filepath.Join(dropins, genUnit+".d")); !os.IsNotExist(err) {

+ t.Errorf("drop-in survived the sweep: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(s.ExeRoot, GenKey(genHost, 7))); !os.IsNotExist(err) {

+ t.Errorf("exe dir survived the sweep: %v", err)

+ }

+}

+

+// A start that failed leaves nothing behind: the unit is stopped and the

+// drop-in and exe layout are removed.

+func TestGenStartFailureStopsTheUnitAndRemovesTheFiles(t *testing.T) {

+ ctl := genCtl()

+ ctl.fail = map[string]error{"start ": errors.New("boom")}

+ s, sock, dropins := genServer(t, ctl)

+ if _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax); err == nil {

+ t.Fatal("a failed start was reported as started")

+ }

+ if !strings.Contains(ctl.joined(), "stop "+genUnit) {

+ t.Errorf("the failed unit was not stopped:\n%s", ctl.joined())

+ }

+ if _, err := os.Stat(filepath.Join(dropins, genUnit+".d")); !os.IsNotExist(err) {

+ t.Errorf("drop-in left behind: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(s.ExeRoot, GenKey(genHost, 7))); !os.IsNotExist(err) {

+ t.Errorf("exe dir left behind: %v", err)

+ }

+}

+

+func TestGenResetResetsTheGenerationUnitOnly(t *testing.T) {

+ ctl := genCtl()

+ _, sock := serveOnSocket(t, ctl)

+ if err := NewClient(sock).ResetGen(context.Background(), genHost, 7); err != nil {

+ t.Fatal(err)

+ }

+ if want := "reset-failed " + genUnit; ctl.joined() != want {

+ t.Errorf("calls =\n%s\nwant\n%s", ctl.joined(), want)

+ }

+}

+

+func TestGenAndParentInputsAreValidated(t *testing.T) {

+ ctl := genCtl()

+ _, c := newServer(t, ctl, func(s *Server) { s.DropInDir = t.TempDir() })

+ for _, path := range []string{

+ "/start?host=" + genHost + "&role=run&memory_max=12345&gen=7",

+ "/start?host=" + genHost + "&role=run&memory_max=536870912&gen=",

+ "/start?host=" + genHost + "&role=run&memory_max=536870912&gen=-1",

+ "/start?host=" + genHost + "&role=run&memory_max=536870912&gen=18446744073709551616",

+ "/stop?host=" + genHost + "&role=run&gen=7&kill=true",

+ "/stop?host=" + genHost + "&role=seed&gen=7",

+ "/reset?host=" + genHost + "&role=run&gen=0",

+ "/reset?host=" + genHost + "&role=sidecar&gen=7",

+ "/parent?host=not_a_host&memory_max=536870912",

+ "/parent?host=" + genHost,

+ "/parent?host=" + genHost + "&memory_max=-536870912",

+ "/parent?host=" + genHost + "&memory_max=0",

+ } {

+ if resp := post(t, c, path); resp.StatusCode != 400 {

+ t.Errorf("%s: status %d, want 400", path, resp.StatusCode)

+ }

+ if ctl.joined() != "" {

+ t.Errorf("systemctl was called:\n%s", ctl.joined())

+ }

+}

+

+// genFilesPresent reports whether the generation's drop-in directory and

+// its executable layout are on disk.

+func genFilesPresent(t *testing.T, s *Server, dropins string) (dropIn, exe bool) {

+ t.Helper()

+ _, err1 := os.Stat(filepath.Join(dropins, genUnit+".d"))

+ _, err2 := os.Stat(filepath.Join(s.ExeRoot, GenKey(genHost, 7)))

+ return err1 == nil, err2 == nil

+}

+

+// populateGenCgroup makes the generation unit's cgroup report processes,

+// so no stop can prove it empty.

+func populateGenCgroup(t *testing.T, s *Server) {

+ t.Helper()

+ events := filepath.Join(s.CgroupRoot, parentSlices, SliceName(genHost), genUnit, "cgroup.events")

+ if err := os.MkdirAll(filepath.Dir(events), 0o755); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.WriteFile(events, []byte("populated 1\nfrozen 0\n"), 0o644); err != nil {

+ t.Fatal(err)

+ }

+}

+

+// systemctl start succeeded, then the invocation lookup failed: the unit

+// may be running. It is killed and proven empty before its files go, and

+// the answer is an ordinary start failure.

+func TestGenStartWhoseInvocationLookupFailsIsKilledBeforeItsFilesGo(t *testing.T) {

+ ctl := genCtl()

+ ctl.fail = map[string]error{"show -p InvocationID": errors.New("bus timeout")}

+ s, sock, dropins := genServer(t, ctl)

+ _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax)

+ if err == nil || errors.Is(err, ErrGenUnproven) {

+ t.Fatalf("StartGen = %v, want an ordinary start failure", err)

+ }

+ if want := "start " + genUnit + "\nshow -p InvocationID --value " + genUnit + "\nkill --signal=SIGKILL " + genUnit + "\nstop " + genUnit; !strings.HasSuffix(ctl.joined(), want) {

+ t.Errorf("calls =\n%s\nwant them to end\n%s", ctl.joined(), want)

+ }

+ if dropIn, exe := genFilesPresent(t, s, dropins); dropIn || exe {

+ t.Errorf("files left after a proven stop: drop-in %v exe %v", dropIn, exe)

+ }

+}

+

+// The same, but the generation's cgroup never empties: termination is

+// unproven, so the files stay and the answer says so.

+func TestGenStartWhoseCleanupIsUnprovenKeepsTheFiles(t *testing.T) {

+ ctl := genCtl()

+ ctl.fail = map[string]error{"show -p InvocationID": errors.New("bus timeout")}

+ s, sock, dropins := genServer(t, ctl)

+ s.cgroupWaitBudget = 100 * time.Millisecond

+ populateGenCgroup(t, s)

+ _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax)

+ if !errors.Is(err, ErrGenUnproven) {

+ t.Fatalf("StartGen = %v, want ErrGenUnproven", err)

+ }

+ if dropIn, exe := genFilesPresent(t, s, dropins); !dropIn || !exe {

+ t.Errorf("files removed while termination is unproven: drop-in %v exe %v", dropIn, exe)

+ }

+}

+

+// A failed systemctl start whose cleanup stop also fails: nothing proves

+// the unit gone, so the files stay.

+func TestGenStartFailureWhoseStopFailsKeepsTheFiles(t *testing.T) {

+ ctl := genCtl()

+ ctl.fail = map[string]error{"start ": errors.New("boom"), "stop ": errors.New("stop timed out")}

+ s, sock, dropins := genServer(t, ctl)

+ _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax)

+ if !errors.Is(err, ErrGenUnproven) {

+ t.Fatalf("StartGen = %v, want ErrGenUnproven", err)

+ }

+ if dropIn, exe := genFilesPresent(t, s, dropins); !dropIn || !exe {

+ t.Errorf("files removed after a failed stop: drop-in %v exe %v", dropIn, exe)

+ }

+}

+

+// Nothing was started when the drop-in cannot be written: the exe layout

+// already made goes, and systemctl is never asked to start.

+func TestGenStartDropInWriteFailureRemovesTheFiles(t *testing.T) {

+ ctl := genCtl()

+ s, sock, dropins := genServer(t, ctl)

+ // A file where the unit's drop-in directory goes: MkdirAll fails.

+ if err := os.WriteFile(filepath.Join(dropins, genUnit+".d"), nil, 0o644); err != nil {

+ t.Fatal(err)

+ }

+ _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax)

+ if err == nil || errors.Is(err, ErrGenUnproven) {

+ t.Fatalf("StartGen = %v, want an ordinary failure", err)

+ }

+ if ctl.joined() != "" {

+ t.Errorf("systemctl was called:\n%s", ctl.joined())

+ }

+ if _, err := os.Stat(filepath.Join(s.ExeRoot, GenKey(genHost, 7))); !os.IsNotExist(err) {

+ t.Errorf("exe dir left behind: %v", err)

+ }

+}

+

+func TestGenStartDaemonReloadFailureRemovesTheFiles(t *testing.T) {

+ ctl := genCtl()

+ ctl.fail = map[string]error{"daemon-reload": errors.New("bus gone")}

+ s, sock, dropins := genServer(t, ctl)

+ _, err := NewClient(sock).StartGen(context.Background(), genHost, 7, cgroup.TierDefault.MemoryMax)

+ if err == nil || errors.Is(err, ErrGenUnproven) {

+ t.Fatalf("StartGen = %v, want an ordinary failure", err)

+ }

+ if strings.Contains(ctl.joined(), "start "+genUnit) {

+ t.Errorf("the unit was started:\n%s", ctl.joined())

+ }

+ if dropIn, exe := genFilesPresent(t, s, dropins); dropIn || exe {

+ t.Errorf("files left behind: drop-in %v exe %v", dropIn, exe)

+ }

+}

+

+// A generation that stopped while nobody removed its files (its stop's

+// answer was lost, the agent restarted) is inactive, so the sweep has no

+// unit to stop. Its leftovers still go once systemd reports it inactive

+// and its cgroup is empty.

+func TestSweepRemovesTheFilesOfAnInactiveGeneration(t *testing.T) {

+ ctl := genCtl()

+ s, sock, dropins := genServer(t, ctl)

+ if err := s.writeGenDropIn(genHost, 7, cgroup.TierDefault); err != nil {

+ t.Fatal(err)

+ }

+ if err := s.ensureGenExe(genHost, 7); err != nil {

+ t.Fatal(err)

+ }

+ out, err := NewClient(sock).Sweep(context.Background())

+ if err != nil || len(out.Units) != 0 {

+ t.Fatalf("sweep = %+v, %v", out, err)

+ }

+ if !strings.Contains(ctl.joined(), "show -p ActiveState --value "+genUnit) {

+ t.Errorf("the leftover unit's state was not checked:\n%s", ctl.joined())

+ }

+ if dropIn, exe := genFilesPresent(t, s, dropins); dropIn || exe {

+ t.Errorf("leftovers of an inactive generation survived the sweep: drop-in %v exe %v", dropIn, exe)

+ }

+}

+

+// Leftovers whose unit is not proven gone stay: one systemd reports

+// active, and one whose cgroup still holds processes.

+func TestSweepKeepsTheFilesOfAGenerationNotProvenGone(t *testing.T) {

+ for name, setup := range map[string]func(*fakeCtl, *Server){

+ "active": func(c *fakeCtl, _ *Server) { c.answers["show -p ActiveState --value"] = "active\n" },

+ "populated": func(_ *fakeCtl, s *Server) { populateGenCgroup(t, s) },

+ } {

+ t.Run(name, func(t *testing.T) {

+ ctl := genCtl()

+ s, sock, dropins := genServer(t, ctl)

+ setup(ctl, s)

+ if err := s.writeGenDropIn(genHost, 7, cgroup.TierDefault); err != nil {

+ t.Fatal(err)

+ }

+ if err := s.ensureGenExe(genHost, 7); err != nil {

+ t.Fatal(err)

+ }

+ if _, err := NewClient(sock).Sweep(context.Background()); err != nil {

+ t.Fatal(err)

+ }

+ if dropIn, exe := genFilesPresent(t, s, dropins); !dropIn || !exe {

+ t.Errorf("files of a generation not proven gone were removed: drop-in %v exe %v", dropIn, exe)

+ }

+ })

+ }

+}

diff --git a/internal/instanceexec/exec.go b/internal/instanceexec/exec.go

index 9b2adfb93..a52dbe6db 100644

--- a/internal/instanceexec/exec.go

+++ b/internal/instanceexec/exec.go

@@ -69,9 +69,13 @@ func RunExec(args []string) {

// in the file. The file's host must also match the instance the unit was

// started for, so a rewritten file can execute only what the tenant could

// execute anyway: its own binary with an argv and env it chose.

-func execLaunch(dataDir string, role instancectl.Role, iid string, execFn func(argv0 string, argv, env []string) error) error {

+func execLaunch(dataDir string, role instancectl.Role, inst string, execFn func(argv0 string, argv, env []string) error) error {

+ iid, launchName, err := launchTarget(role, inst)

+ if err != nil {

+ return err

+ }

dir := filepath.Join(dataDir, "instances", iid)

- raw, err := instancefs.ReadBeneath(dir, instancectl.LaunchFileName(role))

+ raw, err := instancefs.ReadBeneath(dir, launchName)

if err != nil {

if errors.Is(err, instancefs.ErrSymlink) || errors.Is(err, instancefs.ErrEscapes) {

return fmt.Errorf("%w: %w", errSymlinkOrEscape, err)

@@ -108,3 +112,41 @@ func execLaunch(dataDir string, role instancectl.Role, iid string, execFn func(a

}

return execFn(argv0, l.Argv, l.Env)

}

+

+// launchTarget is the trampoline's split of the unit's instance string

+// (zero-downtime spec §7.6): "<iid>" for the three original templates,

+// "<iid>:<n>" for the generation template, whose launch file is keyed by

+// generation. Only a run unit has generations.

+func launchTarget(role instancectl.Role, inst string) (iid, name string, err error) {

+ if !strings.Contains(inst, ":") {

+ return inst, instancectl.LaunchFileName(role), nil

+ }

+ iid, n, err := splitGeneration(role, inst)

+ if err != nil {

+ return "", "", err

+ }

+ return iid, instancectl.GenLaunchFileName(n), nil

+}

+

+// recordTarget is launchTarget for the exit record.

+func recordTarget(role instancectl.Role, inst, invocation string) (iid, name string, err error) {

+ if !strings.Contains(inst, ":") {

+ return inst, instancectl.ExitRecordName(role, invocation), nil

+ }

+ iid, n, err := splitGeneration(role, inst)

+ if err != nil {

+ return "", "", err

+ }

+ return iid, instancectl.GenExitRecordName(n, invocation), nil

+}

+

+func splitGeneration(role instancectl.Role, inst string) (string, uint64, error) {

+ if role != instancectl.RoleRun {

+ return "", 0, fmt.Errorf("instance %q: only a run unit has generations", inst)

+ }

+ iid, n, ok := instancectl.ParseGenInstance(inst)

+ if !ok {

+ return "", 0, fmt.Errorf("instance %q is not <instance-id>:<generation>", inst)

+ }

+ return iid, n, nil

+}

diff --git a/internal/instanceexec/exec_test.go b/internal/instanceexec/exec_test.go

index 75250e002..fc1165991 100644

--- a/internal/instanceexec/exec_test.go

+++ b/internal/instanceexec/exec_test.go

@@ -156,3 +156,90 @@ func TestExecLaunchRefusesAHostThatIsNotThisInstance(t *testing.T) {

t.Errorf("host match (long): calls = %d; want 1", c2.calls)

}

+

+func TestExecLaunchSplitsAGenerationInstance(t *testing.T) {

+ data := t.TempDir()

+ host := "bill.example.com"

+ dir := filepath.Join(data, "instances", host)

+ // The legacy launch file is a decoy: a generation must never read it.

+ writeLaunch(t, dir, instancectl.RoleRun, instancectl.Launch{Schema: 1, Host: host, Argv: []string{"/opt/carlos/live/" + host, "legacy"}})

+ raw, err := json.Marshal(instancectl.Launch{Schema: 1, Host: host, Argv: []string{"/opt/carlos/live/" + host, "--socket", "/run/carlos/instances/" + host + "/app.7.sock"}})

+ if err != nil {

+ t.Fatal(err)

+ }

+ if err := os.WriteFile(filepath.Join(dir, instancectl.GenLaunchFileName(7)), raw, 0o600); err != nil {

+ t.Fatal(err)

+ }

+ var c captured

+ if err := execLaunch(data, instancectl.RoleRun, instancectl.GenInstance(host, 7), c.exec); err != nil {

+ t.Fatalf("execLaunch: %v", err)

+ }

+ if c.calls != 1 || len(c.argv) != 3 || c.argv[2] != "/run/carlos/instances/"+host+"/app.7.sock" {

+ t.Errorf("exec = %+v", c)

+ }

+}

+

+// Each malformed instance string has an otherwise valid, host-matching

+// launch file planted wherever a lenient parser could look, so only the

+// parser can refuse it: the refusal must be launchTarget's own.

+func TestExecLaunchRefusesMalformedOrMisplacedGenerations(t *testing.T) {

+ data := t.TempDir()

+ host := "bill.example.com"

+ dir := filepath.Join(data, "instances", host)

+ plant := func(dir string, n uint64, l instancectl.Launch) {

+ t.Helper()

+ if err := os.MkdirAll(dir, 0o700); err != nil {

+ t.Fatal(err)

+ }

+ raw, err := json.Marshal(l)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if err := os.WriteFile(filepath.Join(dir, instancectl.GenLaunchFileName(n)), raw, 0o600); err != nil {

+ t.Fatal(err)

+ }

+ valid := instancectl.Launch{Schema: 1, Host: host, Argv: []string{"/opt/carlos/live/" + host}}

+ for _, n := range []uint64{0, 1, 7} {

+ plant(dir, n, valid)

+ }

+ // Where a parser that let "", ".." or "a:b" through would look.

+ plant(filepath.Join(data, "instances"), 7, valid)

+ plant(filepath.Join(data, "x"), 1, valid)

+ plant(filepath.Join(data, "instances", "a:b"), 1, valid)

+ plant(filepath.Join(data, "instances", "a"), 1, valid)

+

+ // The planted file is otherwise valid: the well-formed instance runs it.

+ var ok captured

+ if err := execLaunch(data, instancectl.RoleRun, host+":7", ok.exec); err != nil || ok.calls != 1 {

+ t.Fatalf("the planted file was refused for a valid instance: %v, %d execs", err, ok.calls)

+ }

+

+ for _, tc := range []struct {

+ role instancectl.Role

+ inst string

+ }{

+ {instancectl.RoleRun, host + ":0"}, {instancectl.RoleRun, host + ":07"},

+ {instancectl.RoleRun, ":7"}, {instancectl.RoleRun, host + ":"},

+ {instancectl.RoleRun, "../x:1"}, {instancectl.RoleRun, "a:b:1"},

+ {instancectl.RoleSidecar, host + ":7"}, {instancectl.RoleSeed, host + ":7"},

+ } {

+ _, _, perr := launchTarget(tc.role, tc.inst)

+ if perr == nil {

+ t.Errorf("launchTarget(%s, %q) accepted it", tc.role, tc.inst)

+ continue

+ }

+ var c captured

+ err := execLaunch(data, tc.role, tc.inst, c.exec)

+ if err == nil || c.calls != 0 || err.Error() != perr.Error() {

+ t.Errorf("execLaunch(%s, %q) = %v after %d execs; want the parser's refusal %q and no exec", tc.role, tc.inst, err, c.calls, perr)

+ }

+

+ // Misplaced: a well-formed generation whose file names another host.

+ plant(dir, 9, instancectl.Launch{Schema: 1, Host: "other.example.com", Argv: []string{"/opt/carlos/live/other.example.com"}})

+ var c captured

+ if err := execLaunch(data, instancectl.RoleRun, host+":9", c.exec); err == nil || c.calls != 0 {

+ t.Errorf("a generation file naming another host: %v after %d execs; want a refusal", err, c.calls)

+ }

+}

diff --git a/internal/instanceexec/exit.go b/internal/instanceexec/exit.go

index cf32a4a04..7b25fb7e7 100644

--- a/internal/instanceexec/exit.go

+++ b/internal/instanceexec/exit.go

@@ -51,7 +51,7 @@ func RunExit(args []string) {

// writeExitRecord writes exit.<role>.<invocation>.json with O_EXCL into

// <runDir>/instances/<iid>. A missing invocation id is a no-op (the

// variables are systemd's to provide); a malformed one is an error.

-func writeExitRecord(runDir string, role instancectl.Role, iid string, env func(string) string, now time.Time) error {

+func writeExitRecord(runDir string, role instancectl.Role, inst string, env func(string) string, now time.Time) error {

inv := env("INVOCATION_ID")

if inv == "" {

return nil

@@ -59,6 +59,10 @@ func writeExitRecord(runDir string, role instancectl.Role, iid string, env func(

if !invocationShape.MatchString(inv) {

return fmt.Errorf("INVOCATION_ID %q is not 32 hex digits", inv)

}

+ iid, name, err := recordTarget(role, inst, inv)

+ if err != nil {

+ return err

+ }

rec := instancectl.ExitRecord{

Invocation: inv,

Result: env("SERVICE_RESULT"),

@@ -76,7 +80,7 @@ func writeExitRecord(runDir string, role instancectl.Role, iid string, env func(

// systemd, which is diagnostic detail about the instance's own process,

// not something every reader of the instance directory needs (final

// review, plan 2, item 6).

- f, err := instancefs.OpenBeneath(dir, instancectl.ExitRecordName(role, inv), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)

+ f, err := instancefs.OpenBeneath(dir, name, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)

if err != nil {

if errors.Is(err, os.ErrExist) {

return fmt.Errorf("record for invocation %s already exists", inv)

diff --git a/internal/instanceexec/exit_test.go b/internal/instanceexec/exit_test.go

index de580d64d..217f62bfe 100644

--- a/internal/instanceexec/exit_test.go

+++ b/internal/instanceexec/exit_test.go

@@ -82,3 +82,25 @@ func TestWriteExitRecordRefusesAnInvocationThatIsNotAName(t *testing.T) {

t.Error("an invocation id with a path separator was accepted")

}

+

+func TestWriteExitRecordKeysAGenerationRecordByGeneration(t *testing.T) {

+ run := t.TempDir()

+ iid := "h.example.com"

+ if err := os.MkdirAll(filepath.Join(run, "instances", iid), 0o700); err != nil {

+ t.Fatal(err)

+ }

+ now := time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC)

+ env := envOf(map[string]string{"INVOCATION_ID": "deadbeefdeadbeefdeadbeefdeadbeef", "SERVICE_RESULT": "success", "EXIT_CODE": "exited", "EXIT_STATUS": "0"})

+ if err := writeExitRecord(run, instancectl.RoleRun, instancectl.GenInstance(iid, 7), env, now); err != nil {

+ t.Fatalf("writeExitRecord: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(run, "instances", iid, "exit.gen.7.deadbeefdeadbeefdeadbeefdeadbeef.json")); err != nil {

+ t.Errorf("generation record missing: %v", err)

+ }

+ if _, err := os.Stat(filepath.Join(run, "instances", iid, "exit.run.deadbeefdeadbeefdeadbeefdeadbeef.json")); !os.IsNotExist(err) {

+ t.Errorf("a generation wrote a legacy record: %v", err)

+ }

+ if err := writeExitRecord(run, instancectl.RoleSidecar, iid+":7", env, now); err == nil {

+ t.Error("a sidecar generation was accepted")

+ }

+}

diff --git a/internal/stopevidence/evidence.go b/internal/stopevidence/evidence.go

index 4c26a13c8..ede38012e 100644

--- a/internal/stopevidence/evidence.go

+++ b/internal/stopevidence/evidence.go

@@ -25,7 +25,11 @@ type Inputs struct {

Host string

Generation uint64

Fence *activator.StopEvidence

- Cgroup interface {

+ // Cgroup is the host's PARENT handle (cgroup.Manager.OpenCleanup), never

+ // one generation's child: under the generation layout the parent is

+ // empty only when every generation is (zero-downtime spec §7.3), and its

+ // Members walks every cgroup below it.

+ Cgroup interface {

Populated() (bool, error)

Members() ([]int, error)

}

carlos / platform Public

zdd-2-containment