amadan

carlos / platform Public

zdd-1-generations

Squashed from zdd-1-generations into main.
claude 8e066e29ff08d987ab63b3e3b71436d16787f630 parent 1621d27
27 files changed, +3107 −66
  • docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-1-R2.md +15 −0
  • docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-1.md +60 −0
  • internal/activator/activator.go +54 −2
  • internal/activator/backend_exec.go +110 −25
  • internal/activator/backend_unit.go +23 −0
  • internal/activator/canonlink.go +79 −0
  • internal/activator/canonlink_test.go +248 −0
  • internal/activator/generations.go +120 −0
  • internal/activator/generations_backend_test.go +259 −0
  • internal/activator/generations_test.go +95 −0
  • internal/activator/genservice.go +115 −0
  • internal/activator/genservice_test.go +236 −0
  • internal/activator/prepare.go +12 −4
  • internal/edge/answered.go +21 −0
  • internal/edge/cache.go +49 −0
  • internal/edge/edge.go +67 −28
  • internal/edge/gencache_test.go +187 −0
  • internal/edge/genedge_test.go +98 −0
  • internal/edge/genproxy.go +389 −0
  • internal/edge/genproxy_test.go +508 −0
  • internal/edge/gensource.go +39 −0
  • internal/host/agent.go +14 −3
  • internal/host/generations_test.go +168 −0
  • internal/host/probes.go +24 −0
  • internal/host/schedule_wiring.go +40 −3
  • internal/host/socket_advice.go +31 −1
  • internal/host/socket_advice_gen_test.go +46 −0

CI passed — run details

diff --git a/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-1-R2.md b/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-1-R2.md
new file mode 100644
index 000000000..855a4cd6e
--- /dev/null
+++ b/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-1-R2.md
@@ -0,0 +1,15 @@
+**Overall verdict: ACCEPT** — reviewed `c8dbe3de..eb942159` against spec §7.2.
+
+**Finding 1: ADDRESSED.** [internal/edge/genproxy.go:158](../../../internal/edge/genproxy.go:158) validates the sampled generation while holding `GenPools.mu`, before returning or creating a pool. `Retire` takes that same lock at line 179. Retirement therefore either removes the published pool or precedes validation that refuses to recreate it.
+
+**New findings: none.**
+
+- **Lock order and deadlock:** `ServingGen` takes only the activator lock. Retirement releases that lock before invoking hooks ([activator.go:2423](../../../internal/activator/activator.go:2423)); hook dispatch also releases `retiredMu` first. No reverse lock order found.
+- **Bounded reselection:** [genproxy.go:269](../../../internal/edge/genproxy.go:269) permits three reselections and exits on the fourth refusal. These do not consume the separate two dial retries. Exhaustion returns the current generation in `genMovedError`, or `errNoGeneration` when absent. Earlier disappearance enters the existing context-aware, bounded wait.
+- **Body release:** the new exhaustion path releases the held body before either error return. Reselection itself neither reads nor closes it; the existing no-generation timeout also releases it.
+- **Flag-off behavior:** unchanged. Both edge and scheduled delivery still gate generation proxies on `GenerationsOn`; the fix changes neither gate.
+- **Test determinism:** [genproxy_test.go:446](../../../internal/edge/genproxy_test.go:446) forces each retirement synchronously during the initial sample. `sync.Once` prevents retirement from firing again inside the pool lock. The flapping source deterministically changes its number on every call; its test verifies termination, zero retained pools and one body close.
+
+`git diff --check` and formatting checks passed. I reviewed the supplied RED/GREEN evidence but did not rerun builds or tests, preserving the read-only scope. No files changed.
+
+**Out of scope:** no additional observations.
\ No newline at end of file
diff --git a/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-1.md b/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-1.md
new file mode 100644
index 000000000..158c8f654
--- /dev/null
+++ b/docs/superpowers/plans/CODEX-FINDINGS-ZDD-IMPL-1.md
@@ -0,0 +1,60 @@
+**ACCEPT WITH FIXES**
+
+Reviewed `4f26f15c..c8dbe3de`. No flag-off traffic regression found for supported routes with canonical real sockets. Production never calls `activator.SetGenerations`; the required cross-plan interfaces are present.
+
+**Findings affecting the dormant branch**
+
+None found.
+
+**Findings latent until generations are enabled**
+
+1. **Medium — retired generation pools can accumulate without bound.**
+ [internal/edge/genproxy.go:145](../../../internal/edge/genproxy.go:145), also the selection-to-pool interval at line 233.
+
+ Sequence: a request samples generation N → execution pauses → the activator switches generations and `Retire` deletes N’s pool → the request resumes and `pool` recreates N’s entry → `dialGen` rejects the obsolete generation and the request retries successfully. Nothing removes the recreated entry. Repeating this across generations retains arbitrarily many transports in both edge and delivery pool maps. `IdleConnTimeout` does not remove map entries.
+
+ **Smallest fix:** while holding `GenPools.mu`, revalidate the sampled generation against `ServingGen` before finding or creating its pool. If it changed or disappeared, return to generation selection. Because retirement takes the same pool lock, it then either removes an already-published pool or precedes a validation that refuses publication. Add a deterministic repeated-retirement test asserting pool count remains bounded.
+
+ **Fix before merge.** The ledger’s description as a “bounded leak” is incorrect.
+
+`git diff --check` and formatting checks on changed Go files passed. I did not execute builds or tests because they write files; runtime results remain those reported in the ledger.
+
+**Plan’s spec questions**
+
+1. **Foreign layouts: confirm.** Rev 17 §2.9 explicitly requires canonical sockets and `hold`.
+2. **Canonical links: confirm.** Publishing/removing links and skipping the advisory’s checked dial are consistent. Flag-off removal of a leftover symlink is the intentional rollback exception; regular files and real sockets remain untouched.
+3. **Cache while asleep: confirm.** Retaining the last serving generation preserves cache-before-wake.
+4. **Request generation: confirm.** Attribution to the selected connection’s generation handles first wakes correctly.
+5. **Version headers: confirm.** Successful managed responses use the serving build; dial failures have no serving-build header.
+6. **Claims-on wakes: confirm as staged work.** Canonical handles are acceptable here while dormant; generation-aware claim spawning remains a plan-3 prerequisite.
+7. **Version probes: confirm unchanged here.** There are no production callers requiring migration on this branch. Future generation health checks must use handles.
+8. **Workers: confirm as staged work.** Socketless generation identity remains later-plan work, not a completed guarantee of plan 1.
+
+**Ledger deferred-minor triage**
+
+| Item | Disposition |
+|---|---|
+| Task 1: global-switch tripwire | Defer. No demonstrated defect. |
+| Task 2: missing socket-sandbox guards, forwarding/refusal tests | Defer. These are coverage and test-portability gaps. |
+| Task 2: stale comments | Nonblocking. |
+| Task 3: duplicate guards | Nonblocking. |
+| Task 3: unconditional `noteCurrent` store | Confirm current implementation. Calls are serialized at publication; generation allocation order is not itself publication order. Later switches must maintain that serialization. |
+| Task 3: claim-path and empty-socket coverage | Defer; claim integration needs coverage in plan 3. |
+| Task 4: advisory race and cleanup-helper coverage | Defer. The described advisory race produces diagnostic noise, not a false stray warning. |
+| Task 5: recreated retired pool | **Must fix before merge: finding 1.** |
+| Task 5: delayed retirement closes a replacement’s shared-path pool | Defer. It causes reconnection without interrupting active requests or misattributing responses. |
+| Task 5: in-flight connection retained for 90 seconds after retirement | Reject as a general defect. Go’s `CloseIdleConnections` sets `closeIdle`, closing connections when they subsequently become idle. |
+| Task 5: dead branch, body mutation, retry/timeout coverage, unused assignments | Nonblocking. The proxy supplies its outgoing request; the listed coverage gaps do not establish another defect. |
+| Task 6: hook test only counts registration | Defer. Production wiring directly registers `genPools.Retire`. |
+| Task 7: incomplete integration coverage | Defer. No flag-off tagging or unmanaged-route defect found. |
+| Task 7: stale cache entries occupy variant slots | Defer. Existing cache limits bound retention; generation checks prevent stale delivery. |
+| Task 8: status, exclusions and delivery-retirement coverage | Defer. No additional implementation defect found. |
+
+**Each controller “Ruling:”**
+
+1. **Worktree location — confirm.** It does not change the branch’s behavior or reviewability.
+2. **Go cache and socket-test scratch — partially overrule.** An existing on-disk cache fallback is reasonable when the default is unavailable. `/tmp/k` conflicts with the supplied machine rules; future scratch must use the permitted on-disk location. The consequence is not merely build speed.
+3. **Model allocation — partially overrule.** Mechanical implementation assignments fit the repository policy. Sonnet reviews alone do not satisfy its production-path review requirement; this explicitly requested whole-branch Astra review provides the final assessment.
+4. **Explicit listener closure in the Task-5 fixture — confirm.** Closing the listener synchronously addresses the `Serve` registration race and prevents its later close from unlinking a replacement socket.
+5. **Controller performs Task-9 verification — confirm.** Delegation adds no necessary independence to inventory and gate execution. This ruling does not establish that the complete gate has passed; the supplied ledger does not record its completion.
+6. **`LaunchEnvironment` remains canonical — confirm.** It describes launch composition rather than dialing a process. More decisively, `moves.LayoutVars` classifies `CARLOS_CONTROL_SOCKET` as a presence-only comparison: different generation paths do not produce the hypothesized value-mismatch failure.
\ No newline at end of file
diff --git a/internal/activator/activator.go b/internal/activator/activator.go
index f71e93bfd..5758b5e6e 100644
--- a/internal/activator/activator.go
+++ b/internal/activator/activator.go
@@ -382,6 +382,12 @@ type actEntry struct {
spawnVersion string
staleSince time.Time
+ // gen is the generation this entry's process is (zero-downtime spec
+ // §7.1): its number, socket, control socket and build. Set under a.mu at
+ // the awake transition and never changed after it in this plan (plan 3's
+ // switch replaces it). Zero for a worker and before the wake resolves.
+ gen Gen
+
// Sidecar co-spawn: the sidecar exists exactly while its instance is
// awake. All sidecar fields are written under a.mu in the awake state only
// (wake's attach, the sweeper's restart); a stop flips the state under the
@@ -661,6 +667,18 @@ type Service struct {
sweepStop chan struct{}
sweepDone chan struct{}
+ // gens is the generations switch (SetGenerations) as read at New. Off:
+ // every spawn binds the canonical socket and no retire hook fires.
+ gens bool
+ // cacheGens is host -> *atomic.Uint64, the newest generation that has
+ // been current for that host (CacheGen). A sync.Map, never under a.mu:
+ // the edge cache reads it while holding the cache's own lock.
+ cacheGens sync.Map
+ // retiredMu guards retiredHooks (OnGenRetired). A leaf lock: nothing is
+ // called while it is held.
+ retiredMu sync.Mutex
+ retiredHooks []func(host, socket string)
+
// OnWake, when set, is called with the ROUTE after EVERY successful wake
// — fleet boxes use it to publish a binding object to the deployment
// bucket (customer-fleets plan, Task 6) naming this box as the current
@@ -691,6 +709,7 @@ func New(cfg Config, backend Backend, due DueStore) *Service {
due: due,
now: time.Now,
entries: make(map[string]*actEntry),
+ gens: GenerationsEnabled(),
}
// The service lifetime, cancelled by Shutdown. Only work that a shutdown
// should ABANDON hangs off it — today that is exactly the supervised
@@ -993,6 +1012,24 @@ func (a *Service) wake(e *actEntry) {
// cycle. Fresh wake, so it can't be stale yet.
e.spawnVersion = w.spawnVersion
e.staleSince = time.Time{}
+ // The generation is current from here (spec §7.1). A wake the
+ // claim path ran (doClaimWake) spawned through its provider, not
+ // startGen, so it gets the canonical handle and a number of its own.
+ e.gen = w.gen
+ if e.gen.N == 0 && !e.worker {
+ e.gen = canonicalGen(e.route, nextGen(), w.spawnVersion)
+ }
+ if e.gen.N != 0 {
+ a.noteCurrent(e.host, e.gen.N)
+ }
+ // Outside consumers of the canonical path reach the generation
+ // that just became current (spec §7.1). A failed link costs them
+ // the canonical path, not the wake: internal traffic never uses it.
+ if a.gens && linksCanonical(e.route, e.gen) {
+ if err := swapCanonicalLink(e.route.Socket, e.gen.Socket); err != nil {
+ log.Printf("activator: %s: link %s to %s: %v (processes dialling the canonical path will not reach it)", e.host, e.route.Socket, e.gen.Socket, err)
+ }
+ }
if a.fences[e.host] != 0 {
a.beginStopLocked(e)
} else if e.bounceAfterWake {
@@ -1075,6 +1112,7 @@ type wokenInstance struct {
sidecar Proc
sidecarWanted bool
spawnVersion string // live/<host> target read at exec — for build convergence
+ gen Gen // the spawned generation's handle (instances only; zero for a worker)
// spawnTook is how long Backend.Start took: exec to the socket accepting,
// which is the instance's OWN boot and nothing of the platform's. wakeTook
// is the whole wake around it, filled in by wake() rather than here
@@ -1279,7 +1317,9 @@ func (a *Service) doWake(ctx context.Context, e *actEntry) (*wokenInstance, erro
}
proc, spawnVer, err = a.backend.StartWorker(ctx, rt, workerEnv(a.agentSocket(), host, token))
} else {
- proc, spawnVer, err = a.backend.Start(ctx, rt)
+ var g Gen
+ proc, g, err = a.startGen(ctx, rt)
+ spawnVer, w.gen = g.Build, g
}
spawnTook := a.now().Sub(spawnStart)
w.proc = proc
@@ -2214,7 +2254,7 @@ func (a *Service) stop(e *actEntry) {
// sidecar's ONLY way back up, since nothing can dial one.
if !e.worker {
dueCtx, cancelDue := context.WithTimeout(context.Background(), 5*time.Second)
- due, err := a.backend.NextDue(dueCtx, e.route)
+ due, err := a.nextDue(dueCtx, e)
cancelDue()
if err != nil {
log.Printf("activator: next-due probe %s: %v — recording none (it sleeps until a request)", e.host, err)
@@ -2371,7 +2411,19 @@ func (a *Service) finishStop(e *actEntry, outcome *stopObservation) {
e.stopOutcome = outcome
a.rememberStopLocked(e.host, outcome)
delete(a.entries, e.host)
+ retired := e.gen
+ // The canonical link goes with the last generation (spec §7.1). Under
+ // a.mu, the lock the awake transition creates links under, so a new
+ // entry's link cannot be removed by this one's stop.
+ if a.gens && linksCanonical(e.route, retired) {
+ if err := removeCanonicalLink(e.route.Socket, retired.Socket); err != nil {
+ log.Printf("activator: %s: remove link %s: %v", e.host, e.route.Socket, err)
+ }
+ }
a.mu.Unlock()
+ // The generation stopped being current when its entry went (spec §7.2:
+ // its idle connections close now, not at the transport's idle timeout).
+ a.genRetired(e.host, retired)
close(e.stopped)
log.Printf("activator: %s hibernated", e.host)
}
diff --git a/internal/activator/backend_exec.go b/internal/activator/backend_exec.go
index 638c1d616..c2724e40f 100644
--- a/internal/activator/backend_exec.go
+++ b/internal/activator/backend_exec.go
@@ -1063,6 +1063,14 @@ func (b *ExecBackend) waitSeedCgroupEmpty(ctx context.Context, rt registry.Route
// same env file the spawn path hands the child (the activator holds it either
// way).
func (b *ExecBackend) NextDue(ctx context.Context, rt registry.Route) (int64, error) {
+ return b.NextDueVia(ctx, rt, rt.Socket)
+}
+
+// NextDueVia is NextDue dialling socket instead of rt.Socket: the activator
+// passes the instance's current generation socket (AppSocket, zero-downtime
+// spec §7.1), because with generations on nothing listens on the canonical
+// path. With them off the activator passes rt.Socket, which is NextDue.
+func (b *ExecBackend) NextDueVia(ctx context.Context, rt registry.Route, socket string) (int64, error) {
paths := b.pathsFor(rt)
// A nested host's env file lives on tenant-writable ground (design
// §3.2), the same as instanceEnv's read of it: go through instancefs
@@ -1087,7 +1095,7 @@ func (b *ExecBackend) NextDue(ctx context.Context, rt registry.Route) (int64, er
return 0, err
}
req.Header.Set("Authorization", "Bearer "+token)
- resp, err := socketClient(rt.Socket).Do(req)
+ resp, err := socketClient(socket).Do(req)
if err != nil {
return 0, err
}
@@ -1131,17 +1139,94 @@ func (b *ExecBackend) Start(ctx context.Context, rt registry.Route) (Proc, strin
p := b.pathsFor(rt)
switch p.layout {
case LayoutNested:
- return b.startUnit(ctx, rt)
+ return b.startUnitAt(ctx, rt, rt.Socket)
case LayoutFenced, LayoutConflict:
return nil, "", fmt.Errorf("%s: refusing to wake: %s", rt.Host, p.reason)
}
- return b.startChild(ctx, rt)
+ return b.startChildAt(ctx, rt, rt.Socket)
+}
+
+// StartGen is Start for generation n (zero-downtime spec §7.1). perGen false
+// is Start exactly: the canonical socket, the canonical control socket, the
+// same argv. perGen true binds the generation's own socket: GenSocketPath
+// for a flat row, app.<n>.sock in the instance run dir for a nested one. A
+// foreign row keeps its canonical socket either way: it is outside the two
+// layouts the spec gives generation paths to, and keeping it is the
+// conservative reading (plan 1, "Spec questions").
+//
+// The stale-file rule (spec §7.1): before the app binds, the file at THIS
+// generation's own path is removed (left there by an earlier agent process,
+// whose counter started at 1 like this one's). A sibling generation's path
+// is never touched.
+func (b *ExecBackend) StartGen(ctx context.Context, rt registry.Route, n uint64, perGen bool) (Proc, Gen, error) {
+ p := b.pathsFor(rt)
+ sock := rt.Socket
+ if perGen {
+ switch p.layout {
+ case LayoutFlat:
+ sock = b.flatGenSocket(rt.Host, n)
+ case LayoutNested:
+ sock = nestedGenSocketIn(p.runDir, n)
+ }
+ }
+ var (
+ proc Proc
+ ver string
+ err error
+ )
+ switch p.layout {
+ case LayoutNested:
+ // prepareAt removes the stale file at sock for the nested layout.
+ proc, ver, err = b.startUnitAt(ctx, rt, sock)
+ case LayoutFenced, LayoutConflict:
+ return nil, Gen{}, fmt.Errorf("%s: refusing to wake: %s", rt.Host, p.reason)
+ default:
+ if !perGen {
+ // A link left at the canonical path by an agent that ran with
+ // generations on would make the app's bind fail.
+ if err := clearLeftoverCanonicalLink(rt.Socket); err != nil {
+ return nil, Gen{}, fmt.Errorf("%s: leftover canonical link: %w", rt.Host, err)
+ }
+ }
+ if sock != rt.Socket {
+ if err := clearOwnGenSocket(sock); err != nil {
+ return nil, Gen{}, fmt.Errorf("%s: generation socket: %w", rt.Host, err)
+ }
+ }
+ proc, ver, err = b.startChildAt(ctx, rt, sock)
+ }
+ if err != nil {
+ return proc, Gen{}, err
+ }
+ return proc, Gen{N: n, Socket: sock, Control: sock + ".control", Build: ver}, nil
+}
+
+// flatGenSocket is GenSocketPath, under RunDirOverride/g in a test.
+func (b *ExecBackend) flatGenSocket(host string, n uint64) string {
+ if b.RunDirOverride != "" {
+ return genSocketPathIn(filepath.Join(b.RunDirOverride, "g"), host, n)
+ }
+ return GenSocketPath(host, n)
+}
+
+// clearOwnGenSocket makes the flat generation directory and removes a stale
+// file at sock, and only at sock. The directory inherits /run/carlos's
+// setgid carlos-sockets group, as the nested run dirs do.
+func clearOwnGenSocket(sock string) error {
+ if err := os.MkdirAll(filepath.Dir(sock), 0o750|os.ModeSetgid); err != nil {
+ return err
+ }
+ if err := os.Remove(sock); err != nil && !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ return nil
}
-// startChild is Start's exec-child body (flat and foreign layouts):
+// startChildAt is Start's exec-child body (flat and foreign layouts):
// exec of the versioned binary directly, exactly as before nested layouts
-// existed.
-func (b *ExecBackend) startChild(ctx context.Context, rt registry.Route) (Proc, string, error) {
+// existed. sock is the app socket this spawn binds: rt.Socket, or a
+// generation socket (StartGen).
+func (b *ExecBackend) startChildAt(ctx context.Context, rt registry.Route, sock string) (Proc, string, error) {
host := rt.Host
paths := b.pathsFor(rt)
dbPath := paths.dbPath
@@ -1176,7 +1261,7 @@ func (b *ExecBackend) startChild(ctx context.Context, rt registry.Route) (Proc,
// passing --control-socket would refuse to start every app that hasn't
// also shipped support for it, a fleet-wide outage on agent upgrade. An
// environment variable a binary doesn't know about is simply ignored.
- controlSocket := rt.Socket + ".control"
+ controlSocket := sock + ".control"
// Bind the control channel BEFORE the spawn, so CARLOS_CONTROL_SOCKET can
// follow the OUTCOME rather than the intention. It used to be set from
// "b.Cache != nil" and the listener started afterwards; now that the
@@ -1240,7 +1325,7 @@ func (b *ExecBackend) startChild(ctx context.Context, rt registry.Route) (Proc,
// the rollout ordering harmless. It does not replace the socket channel;
// both are live for exec-backed instances, and converging them is deferred
// (spec §10.3).
- childVars := b.agentOwnedVars(rt, fileEnv, ctl != nil)
+ childVars := b.agentOwnedVars(rt, controlSocket, fileEnv, ctl != nil)
// Record the build we're about to exec — read the symlink immediately before
// the start so it names the version the kernel will resolve. Reading it here,
// rather than back in wake() after the health wait, closes a real window: a
@@ -1258,7 +1343,7 @@ func (b *ExecBackend) startChild(ctx context.Context, rt registry.Route) (Proc,
// twice — the manager's clone3-refusal fallback — so everything about
// the Cmd is configured inside it.
cmd, settle, err := b.spawnInCgroup(rt, func() *exec.Cmd {
- cmd := exec.Command(bin, "--socket", rt.Socket, "--db", dbPath)
+ cmd := exec.Command(bin, "--socket", sock, "--db", dbPath)
cmd.Env = childEnv(childVars)
// The child logs through the agent's own stderr/stdout (journald
// under systemd); there's no per-instance unit to own a log stream
@@ -1308,13 +1393,13 @@ func (b *ExecBackend) startChild(ctx context.Context, rt registry.Route) (Proc,
close(p.done)
}()
- if err := waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout); err != nil {
+ if err := waitHealthyCtx(ctx, sock, spawnHealthTimeout); err != nil {
stopCtx, cancel := context.WithTimeout(context.Background(), actStopTimeout)
defer cancel()
_ = p.Stop(stopCtx)
return nil, "", err
}
- b.normalizeSocketMode(rt, p)
+ b.normalizeSocketMode(rt, sock, p)
return p, spawnVer, nil
}
@@ -1325,10 +1410,10 @@ func (b *ExecBackend) startChild(ctx context.Context, rt registry.Route) (Proc,
// (design §7.2): haveControl is whether this call's control listener
// actually bound. childVars must follow the OUTCOME of the bind, not the
// intention, per the long comment this replaced, still true here.
-func (b *ExecBackend) agentOwnedVars(rt registry.Route, fileEnv map[string]string, haveControl bool) map[string]string {
+func (b *ExecBackend) agentOwnedVars(rt registry.Route, control string, fileEnv map[string]string, haveControl bool) map[string]string {
childVars := fileEnv
if haveControl {
- childVars = withVar(childVars, "CARLOS_CONTROL_SOCKET", rt.Socket+".control")
+ childVars = withVar(childVars, "CARLOS_CONTROL_SOCKET", control)
} else {
childVars = withoutVars(childVars, "CARLOS_CONTROL_SOCKET")
}
@@ -1384,30 +1469,30 @@ func seedVars(fileEnv map[string]string, host string) map[string]string {
// anything. All three branches log and never fail the wake: a box with
// no consumers must not lose a healthy wake to a normalization it
// doesn't need.
-func (b *ExecBackend) normalizeSocketMode(rt registry.Route, proc Proc) {
+func (b *ExecBackend) normalizeSocketMode(rt registry.Route, sock string, proc Proc) {
host := rt.Host
if proc.Exited() {
log.Printf("activator: %s: child exited during the health wait (external process serving its socket?) — not normalizing the socket mode", host)
- } else if fi, statErr := os.Lstat(rt.Socket); statErr != nil || fi.Mode()&os.ModeSocket == 0 {
- log.Printf("activator: %s: %s is not a socket we can normalize (stat: %v)", host, rt.Socket, statErr)
- } else if err := os.Chmod(rt.Socket, 0o660); err != nil {
+ } else if fi, statErr := os.Lstat(sock); statErr != nil || fi.Mode()&os.ModeSocket == 0 {
+ log.Printf("activator: %s: %s is not a socket we can normalize (stat: %v)", host, sock, statErr)
+ } else if err := os.Chmod(sock, 0o660); err != nil {
log.Printf("activator: normalize socket mode for %s: %v (enrolled consumers would get permission denied)", host, err)
}
}
-// startUnit is Start's nested body (design §6): the same admin token, env
+// startUnitAt is Start's nested body (design §6): the same admin token, env
// composition and health gate as startChild, but the process itself is
// started by the root instance-control channel from a launch file rather
// than exec'd directly. The agent has no privilege to enter the tenant's
// namespace, only the control service does.
-func (b *ExecBackend) startUnit(ctx context.Context, rt registry.Route) (Proc, string, error) {
+func (b *ExecBackend) startUnitAt(ctx context.Context, rt registry.Route, sock string) (Proc, string, error) {
if err := b.nestedGate(rt); err != nil {
return nil, "", err
}
if err := b.ensureLiveLink(ctx, rt.Host); err != nil {
return nil, "", err
}
- p, ctl, logs, fileEnv, err := b.prepare(rt, instancectl.RoleRun)
+ p, ctl, logs, fileEnv, err := b.prepareAt(rt, instancectl.RoleRun, sock)
if err != nil {
return nil, "", err
}
@@ -1418,8 +1503,8 @@ func (b *ExecBackend) startUnit(ctx context.Context, rt registry.Route) (Proc, s
logs.Close()
return nil, "", err
}
- childVars := b.agentOwnedVars(rt, fileEnv, ctl != nil)
- argv := []string{b.liveLink(rt.Host), "--socket", rt.Socket, "--db", p.dbPath}
+ childVars := b.agentOwnedVars(rt, sock+".control", fileEnv, ctl != nil)
+ argv := []string{b.liveLink(rt.Host), "--socket", sock, "--db", p.dbPath}
if err := b.writeLaunch(p, instancectl.RoleRun, rt.Host, argv, childEnv(childVars)); err != nil {
return fail(err)
}
@@ -1446,7 +1531,7 @@ func (b *ExecBackend) startUnit(ctx context.Context, rt registry.Route) (Proc, s
log.Printf("activator: purged %d cached response(s) for %s on spawn", n, rt.Host)
}
}
- if err := waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout); err != nil {
+ if err := waitHealthyCtx(ctx, sock, spawnHealthTimeout); err != nil {
// Set before the Stop call below, closing the same race Stop's own
// internal set leaves open by a few instructions: observe runs
// concurrently from the moment newUnitProc started it, and setting
@@ -1467,7 +1552,7 @@ func (b *ExecBackend) startUnit(ctx context.Context, rt registry.Route) (Proc, s
}
return nil, "", err
}
- b.normalizeSocketMode(rt, up)
+ b.normalizeSocketMode(rt, sock, up)
return up, spawnVer, nil
}
@@ -1567,7 +1652,7 @@ func (b *ExecBackend) LaunchEnvironment(rt registry.Route) (map[string]string, [
sort.Strings(shadowed)
fileEnv := mergeEnv(cfgEnv, ownEnv)
out := map[string]string{}
- for _, kv := range childEnv(b.agentOwnedVars(rt, fileEnv, true)) {
+ for _, kv := range childEnv(b.agentOwnedVars(rt, rt.Socket+".control", fileEnv, true)) {
if i := strings.IndexByte(kv, '='); i > 0 {
out[kv[:i]] = kv[i+1:]
}
diff --git a/internal/activator/backend_unit.go b/internal/activator/backend_unit.go
index 137f63de0..9beabd4a5 100644
--- a/internal/activator/backend_unit.go
+++ b/internal/activator/backend_unit.go
@@ -322,6 +322,29 @@ func (b *SplitBackend) NextDue(ctx context.Context, rt registry.Route) (int64, e
}
func (b *SplitBackend) HasSidecar(rt registry.Route) bool { return b.pick(rt).HasSidecar(rt) }
+// StartGen dispatches like Start. A side without generation support (the
+// unit side: systemd binds a unit tenant's canonical socket) is started
+// with Start and gets the canonical handle.
+func (b *SplitBackend) StartGen(ctx context.Context, rt registry.Route, n uint64, perGen bool) (Proc, Gen, error) {
+ if g, ok := b.pick(rt).(GenBackend); ok {
+ return g.StartGen(ctx, rt, n, perGen)
+ }
+ proc, ver, err := b.pick(rt).Start(ctx, rt)
+ if err != nil {
+ return proc, Gen{}, err
+ }
+ return proc, canonicalGen(rt, n, ver), nil
+}
+
+// NextDueVia dispatches like NextDue; a side without generation support
+// answers on rt's own socket.
+func (b *SplitBackend) NextDueVia(ctx context.Context, rt registry.Route, socket string) (int64, error) {
+ if g, ok := b.pick(rt).(GenBackend); ok {
+ return g.NextDueVia(ctx, rt, socket)
+ }
+ return b.pick(rt).NextDue(ctx, rt)
+}
+
// Layout implements Classifier for the composed backend: a unit-backed row
// is always today's flat unit shape (the exec-only layouts do not apply to
// it), and an exec row defers to the exec side's own classification.
diff --git a/internal/activator/canonlink.go b/internal/activator/canonlink.go
new file mode 100644
index 000000000..d545d567b
--- /dev/null
+++ b/internal/activator/canonlink.go
@@ -0,0 +1,79 @@
+package activator
+
+import (
+ "errors"
+ "os"
+
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// canonlink.go keeps a route's canonical socket path (rt.Socket) pointing at
+// its current generation, for processes outside the edge and the agent that
+// dial it, such as members of the carlos-sockets group (#103; zero-downtime
+// spec §7.1, rev 15). Unix connect() follows the symlink. Internal dials
+// never use it: they go through AppSocket or a handle, and unixdial.Checked
+// would refuse the link anyway (the listener's bound path is the generation
+// path, not the dialled one).
+//
+// Plan 1 creates the link when the first generation becomes current and
+// removes it when the entry stops. Plan 3 calls swapCanonicalLink at every
+// switch.
+
+// linksCanonical reports whether g's process is reached through a symlink at
+// rt's canonical path: only when the generation bound a path of its own. A
+// generation on the canonical socket itself (generations off, a unit tenant,
+// a foreign-layout row) is never linked: the link would replace the real
+// socket.
+func linksCanonical(rt registry.Route, g Gen) bool {
+ return rt.Socket != "" && g.Socket != "" && g.Socket != rt.Socket
+}
+
+// swapCanonicalLink points canonical at target atomically: a temporary link
+// beside it, then rename over it. A dialler sees the old target or the new
+// one, never no path. Whatever was at canonical (an earlier link, or a stale
+// socket file from before generations) is replaced.
+func swapCanonicalLink(canonical, target string) error {
+ tmp := canonical + ".lnk"
+ if err := os.Remove(tmp); err != nil && !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ if err := os.Symlink(target, tmp); err != nil {
+ return err
+ }
+ if err := os.Rename(tmp, canonical); err != nil {
+ _ = os.Remove(tmp)
+ return err
+ }
+ return nil
+}
+
+// removeCanonicalLink removes canonical only while it is a symlink to
+// target: a link a later generation (or a later entry) has already moved,
+// and a real socket, are left alone.
+func removeCanonicalLink(canonical, target string) error {
+ got, err := os.Readlink(canonical)
+ if err != nil || got != target {
+ return nil
+ }
+ if err := os.Remove(canonical); err != nil && !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ return nil
+}
+
+// clearLeftoverCanonicalLink removes a symlink at canonical, whatever it
+// names, and nothing else. An agent with generations on that died without
+// stopping its entries leaves one; an app told to bind canonical with the
+// switch off would otherwise fail with "address in use". A canonical path
+// that is not a symlink (every box that never ran generations) is not
+// touched.
+func clearLeftoverCanonicalLink(canonical string) error {
+ fi, err := os.Lstat(canonical)
+ if err != nil || fi.Mode()&os.ModeSymlink == 0 {
+ return nil
+ }
+ if err := os.Remove(canonical); err != nil && !errors.Is(err, os.ErrNotExist) {
+ return err
+ }
+ return nil
+}
diff --git a/internal/activator/canonlink_test.go b/internal/activator/canonlink_test.go
new file mode 100644
index 000000000..0149776c9
--- /dev/null
+++ b/internal/activator/canonlink_test.go
@@ -0,0 +1,248 @@
+package activator
+
+import (
+ "context"
+ "fmt"
+ "io"
+ "net"
+ "net/http"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// serveOn binds an HTTP server at sock that answers body, the way a
+// generation's process would. It returns the server so a test can stop it.
+func serveOn(t *testing.T, sock, body string) *http.Server {
+ t.Helper()
+ ln, err := net.Listen("unix", sock)
+ if err != nil {
+ if strings.Contains(err.Error(), "operation not permitted") {
+ t.Skipf("sandbox forbids unix sockets: %v", err)
+ }
+ t.Fatalf("listen %s: %v", sock, err)
+ }
+ srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { io.WriteString(w, body) })}
+ go srv.Serve(ln)
+ t.Cleanup(func() { srv.Close() })
+ return srv
+}
+
+// dialAsOutsider GETs / over a plain unix dial of path, the way a member of
+// the carlos-sockets group dials an instance: no unixdial checks, so the
+// kernel follows a symlink at path.
+func dialAsOutsider(t *testing.T, path string) string {
+ t.Helper()
+ c := &http.Client{Timeout: 2 * time.Second, Transport: &http.Transport{
+ DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+ return (&net.Dialer{}).DialContext(ctx, "unix", path)
+ },
+ }}
+ resp, err := c.Get("http://carlos/")
+ if err != nil {
+ t.Fatalf("outside dial of %s: %v", path, err)
+ }
+ defer resp.Body.Close()
+ b, _ := io.ReadAll(resp.Body)
+ return string(b)
+}
+
+// servingProc is a fake child whose Stop also stops its server, so the
+// generation's socket goes away with it.
+type servingProc struct {
+ Proc
+ srv *http.Server
+}
+
+func (p *servingProc) Stop(ctx context.Context) error {
+ p.srv.Close()
+ return p.Proc.Stop(ctx)
+}
+
+// sockGenBackend is genFakeBackend whose generations really serve: perGen
+// binds <dir>/g<n>.sock, otherwise rt.Socket, each answering "gen <n>".
+type sockGenBackend struct {
+ *fakeBackend
+ t *testing.T
+ dir string
+}
+
+func (b *sockGenBackend) StartGen(ctx context.Context, rt registry.Route, n uint64, perGen bool) (Proc, Gen, error) {
+ proc, ver, err := b.fakeBackend.Start(ctx, rt)
+ if err != nil {
+ return proc, Gen{}, err
+ }
+ sock := rt.Socket
+ if perGen {
+ sock = filepath.Join(b.dir, fmt.Sprintf("g%d.sock", n))
+ }
+ srv := serveOn(b.t, sock, fmt.Sprintf("gen %d", n))
+ return &servingProc{Proc: proc, srv: srv}, Gen{N: n, Socket: sock, Control: sock + ".control", Build: ver}, nil
+}
+
+func (b *sockGenBackend) NextDueVia(ctx context.Context, rt registry.Route, _ string) (int64, error) {
+ return b.fakeBackend.NextDue(ctx, rt)
+}
+
+// sockGenActivator is a Service over sockGenBackend, its sockets in a short
+// directory (run with TMPDIR=/tmp/k), and a route whose canonical socket is
+// <dir>/c.sock.
+func sockGenActivator(t *testing.T, on bool) (*Service, registry.Route) {
+ t.Helper()
+ withGenerations(t, on)
+ dir := shortRunDir(t)
+ reg, err := registry.Open(filepath.Join(dir, "r.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ a := New(Config{Bucket: "test-bucket", Idle: time.Hour}, &sockGenBackend{fakeBackend: &fakeBackend{}, t: t, dir: dir}, reg)
+ rt := testRoute("link.example.com")
+ rt.Socket = filepath.Join(dir, "c.sock")
+ return a, rt
+}
+
+func isSymlink(path string) bool {
+ fi, err := os.Lstat(path)
+ return err == nil && fi.Mode()&os.ModeSymlink != 0
+}
+
+// The helper plan 3 calls at every switch: it replaces whatever is at the
+// canonical path, and leaves no temporary link behind.
+func TestSwapCanonicalLinkReplacesAtomically(t *testing.T) {
+ dir := t.TempDir()
+ canonical := filepath.Join(dir, "c.sock")
+ if err := os.WriteFile(canonical, []byte("stale socket file"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ for _, target := range []string{"/run/carlos/g/aaaaaaaaaaaa.1.sock", "/run/carlos/g/aaaaaaaaaaaa.2.sock"} {
+ if err := swapCanonicalLink(canonical, target); err != nil {
+ t.Fatalf("swapCanonicalLink(%s): %v", target, err)
+ }
+ if got, err := os.Readlink(canonical); err != nil || got != target {
+ t.Fatalf("canonical names %q (%v), want %q", got, err, target)
+ }
+ }
+ if _, err := os.Lstat(canonical + ".lnk"); !os.IsNotExist(err) {
+ t.Fatalf("temporary link left behind (lstat: %v)", err)
+ }
+}
+
+// A process dialling the canonical path follows the link to whichever
+// generation it names, before and after a switch.
+func TestSwapCanonicalLinkMovesOutsideDiallers(t *testing.T) {
+ dir := shortRunDir(t)
+ g1, g2 := filepath.Join(dir, "g1.sock"), filepath.Join(dir, "g2.sock")
+ serveOn(t, g1, "one")
+ serveOn(t, g2, "two")
+ canonical := filepath.Join(dir, "c.sock")
+ if err := swapCanonicalLink(canonical, g1); err != nil {
+ t.Fatal(err)
+ }
+ if got := dialAsOutsider(t, canonical); got != "one" {
+ t.Fatalf("before the switch: %q, want one", got)
+ }
+ if err := swapCanonicalLink(canonical, g2); err != nil {
+ t.Fatal(err)
+ }
+ if got := dialAsOutsider(t, canonical); got != "two" {
+ t.Fatalf("after the switch: %q, want two", got)
+ }
+}
+
+func TestRemoveCanonicalLinkRemovesOnlyItsOwnLink(t *testing.T) {
+ dir := t.TempDir()
+ canonical := filepath.Join(dir, "c.sock")
+ if err := swapCanonicalLink(canonical, "/a"); err != nil {
+ t.Fatal(err)
+ }
+ if err := removeCanonicalLink(canonical, "/b"); err != nil || !isSymlink(canonical) {
+ t.Fatalf("a link to another generation was removed (err %v)", err)
+ }
+ if err := removeCanonicalLink(canonical, "/a"); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := os.Lstat(canonical); !os.IsNotExist(err) {
+ t.Fatalf("own link not removed (lstat: %v)", err)
+ }
+ if err := os.WriteFile(canonical, nil, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if err := removeCanonicalLink(canonical, "/a"); err != nil || !isRegular(canonical) {
+ t.Fatalf("a real file at the canonical path was removed (err %v)", err)
+ }
+}
+
+// Spec §7.1 rev 15, end to end through the Service: the link appears when
+// the generation becomes current, an outside dialler reaches that
+// generation through it, and it goes when the entry stops.
+func TestCanonicalPathLinksToTheCurrentGeneration(t *testing.T) {
+ requireUnixSockets(t)
+ a, rt := sockGenActivator(t, true)
+ arrangeAwake(t, a, rt)
+ g, _ := a.ServingGen(rt.Host)
+ if target, err := os.Readlink(rt.Socket); err != nil || target != g.Socket {
+ t.Fatalf("canonical path names %q (%v), want the current generation %q", target, err, g.Socket)
+ }
+ if got, want := dialAsOutsider(t, rt.Socket), fmt.Sprintf("gen %d", g.N); got != want {
+ t.Fatalf("outside dial of the canonical path got %q, want %q", got, want)
+ }
+ stopAndWait(t, a, rt.Host)
+ if _, err := os.Lstat(rt.Socket); !os.IsNotExist(err) {
+ t.Fatalf("canonical link outlived its entry (lstat: %v)", err)
+ }
+}
+
+// Flag off: the canonical path is the app's own socket, never a link.
+func TestCanonicalPathIsARealSocketWithGenerationsOff(t *testing.T) {
+ requireUnixSockets(t)
+ a, rt := sockGenActivator(t, false)
+ arrangeAwake(t, a, rt)
+ if isSymlink(rt.Socket) || !isSocket(rt.Socket) {
+ t.Fatalf("%s is not a real socket with generations off", rt.Socket)
+ }
+ g, _ := a.ServingGen(rt.Host)
+ if got, want := dialAsOutsider(t, rt.Socket), fmt.Sprintf("gen %d", g.N); got != want {
+ t.Fatalf("outside dial got %q, want %q", got, want)
+ }
+}
+
+// A generation that IS on the canonical socket (a unit tenant, a backend
+// without generations) is never linked, even with the switch on: the link
+// would replace the real socket.
+func TestNoCanonicalLinkWhenTheGenerationOwnsTheCanonicalSocket(t *testing.T) {
+ withGenerations(t, true)
+ a, _, _, _ := testActivator(t, time.Hour)
+ rt := genTestRoute(t, "plain-link.example.com")
+ arrangeAwake(t, a, rt)
+ if _, err := os.Lstat(rt.Socket); !os.IsNotExist(err) {
+ t.Fatalf("something was created at the canonical path (lstat: %v)", err)
+ }
+}
+
+// Flag off, after an agent that ran with generations on died without
+// stopping its entries: the leftover link is cleared so the app can bind.
+func TestStartGenWithGenerationsOffClearsALeftoverCanonicalLink(t *testing.T) {
+ requireUnixSockets(t)
+ b := testExecBackend(t)
+ b.RunDirOverride = shortRunDir(t)
+ host := "leftover.example.com"
+ stageServingLive(t, b, host)
+ rt := flatGenRoute(b, host)
+ if err := os.Symlink(filepath.Join(b.RunDirOverride, "g", "gone.1.sock"), rt.Socket); err != nil {
+ t.Fatal(err)
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+ defer cancel()
+ proc, _, err := b.StartGen(ctx, rt, 1, false)
+ if err != nil {
+ t.Fatalf("StartGen over a leftover link: %v", err)
+ }
+ defer proc.Stop(context.Background())
+ if !isSocket(rt.Socket) {
+ t.Fatalf("%s is not the app's socket", rt.Socket)
+ }
+}
diff --git a/internal/activator/generations.go b/internal/activator/generations.go
new file mode 100644
index 000000000..2de681242
--- /dev/null
+++ b/internal/activator/generations.go
@@ -0,0 +1,120 @@
+// generations.go is the generation identity of zero-downtime deploys
+// (docs/superpowers/specs/2026-10-07-zero-downtime-deploys-design.md §7.1):
+// the process-wide counter, the bounded generation socket paths, the Gen
+// handle, and the dormant switch that keeps all of it off by default.
+package activator
+
+import (
+ "context"
+ "crypto/sha256"
+ "encoding/hex"
+ "path/filepath"
+ "strconv"
+ "sync/atomic"
+
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// generationsOn is the dormant switch (spec §15). Off, every spawn binds the
+// route's canonical socket exactly as before, and nothing outside this
+// package changes behaviour. Plan 5 turns it on from CARLOS_SWAP=overlap by
+// calling SetGenerations(true) in internal/host's runAgent BEFORE
+// activator.New: a Service reads the switch once, at construction, so one
+// process never mixes the two socket schemes.
+var generationsOn atomic.Bool
+
+// SetGenerations sets the dormant switch for Services built after the call.
+func SetGenerations(on bool) { generationsOn.Store(on) }
+
+// GenerationsEnabled reports the switch as it stands now.
+func GenerationsEnabled() bool { return generationsOn.Load() }
+
+// genCounter is the ONE generation counter for the whole process (spec
+// §7.1). It is package-level, never per Service and never per entry, and it
+// is never reset: an entry deleted at a stop and recreated at the next wake
+// cannot reuse a number an earlier entry had, so a cached (route, n) pair in
+// the edge (same process) can never match a later generation.
+var genCounter atomic.Uint64
+
+// nextGen takes the next generation number. The first is 1; 0 means "no
+// generation" everywhere a Gen is optional.
+func nextGen() uint64 { return genCounter.Add(1) }
+
+// Gen is one spawn of one instance (spec §3 "Generation"). It is the
+// generation HANDLE (spec §7.1): returned by the spawn and held by whoever
+// started it. Health checks, settle observation and retirement use the
+// handle, never AppSocket. Plan 1 fills N, Socket, Control and Build;
+// StagedPath, Digest and Demand stay empty until plans 3 and 4 exec by
+// staged path.
+type Gen struct {
+ N uint64 // process-wide, never reused (spec §7.1)
+ Socket string // generation app socket (bounded path, spec §7.1)
+ Control string // Socket + ".control"
+ Build string // version name the generation runs
+ StagedPath string // immutable staged executable it exec'd (spec §7.5 step 3)
+ Digest string // sha256 of StagedPath at exec, hex
+ Demand string // the release's own `swap` demand verified for this artifact: "hold" or "" (spec §4.2, §8.5)
+}
+
+// GenSocketDir is where flat generation sockets live: /run/carlos/g.
+const GenSocketDir = provision.SocketDir + "/g"
+
+// GenSocketPath is the bounded flat generation path (spec §7.1):
+// /run/carlos/g/<first 12 hex of sha256(host)>.<n>.sock. It is at most 52
+// bytes (60 with ".control") whatever the hostname, because the hostname
+// never appears in it.
+func GenSocketPath(host string, n uint64) string {
+ return genSocketPathIn(GenSocketDir, host, n)
+}
+
+// genSocketPathIn is GenSocketPath under another directory (a test's
+// RunDirOverride).
+func genSocketPathIn(dir, host string, n uint64) string {
+ sum := sha256.Sum256([]byte(host))
+ return filepath.Join(dir, hex.EncodeToString(sum[:])[:12]+"."+strconv.FormatUint(n, 10)+".sock")
+}
+
+// NestedGenSocket is the nested layout's generation socket:
+// /run/carlos/instances/<InstanceID>/app.<n>.sock (spec §7.1). The
+// identifier is bounded at 40 bytes, so this is at most 92 bytes (100 with
+// ".control").
+func NestedGenSocket(host string, n uint64) string {
+ return nestedGenSocketIn(provision.InstanceRunDir(host), n)
+}
+
+// nestedGenSocketIn is NestedGenSocket under an already-resolved run dir.
+func nestedGenSocketIn(runDir string, n uint64) string {
+ return filepath.Join(runDir, "app."+strconv.FormatUint(n, 10)+".sock")
+}
+
+// canonicalGen is the generation handle of a spawn that bound the route's
+// canonical socket: every spawn with the switch off, a backend with no
+// generation support (unit-backed tenants, test doubles), and a foreign
+// layout row.
+func canonicalGen(rt registry.Route, n uint64, build string) Gen {
+ g := Gen{N: n, Socket: rt.Socket, Build: build}
+ if rt.Socket != "" {
+ g.Control = rt.Socket + ".control"
+ }
+ return g
+}
+
+// GenBackend is the optional Backend capability behind generations (spec
+// §7.1). ExecBackend and SplitBackend implement it. A backend that does not
+// (UnitBackend, every test double that predates this plan) is spawned with
+// Start and gets canonicalGen: one generation per entry, on the canonical
+// socket.
+type GenBackend interface {
+ // StartGen spawns generation n of rt's instance and returns its handle.
+ // perGen false must behave exactly as Start; true binds the generation's
+ // own socket where the layout has one.
+ StartGen(ctx context.Context, rt registry.Route, n uint64, perGen bool) (Proc, Gen, error)
+ // NextDueVia is NextDue dialling socket rather than rt.Socket.
+ NextDueVia(ctx context.Context, rt registry.Route, socket string) (int64, error)
+}
+
+var (
+ _ GenBackend = (*ExecBackend)(nil)
+ _ GenBackend = (*SplitBackend)(nil)
+)
diff --git a/internal/activator/generations_backend_test.go b/internal/activator/generations_backend_test.go
new file mode 100644
index 000000000..d1cec5b28
--- /dev/null
+++ b/internal/activator/generations_backend_test.go
@@ -0,0 +1,259 @@
+package activator
+
+import (
+ "context"
+ "encoding/json"
+ "net"
+ "net/http"
+ "os"
+ "path/filepath"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/instancectl"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// shortRunDir is a RunDirOverride short enough for sockets under it: the
+// test name is not part of the path (t.TempDir's would be), so a long test
+// name cannot push a socket past sockaddr_un's limit. Run socket tests with
+// TMPDIR=/tmp/k.
+func shortRunDir(t *testing.T) string {
+ t.Helper()
+ dir, err := os.MkdirTemp("", "g")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { os.RemoveAll(dir) })
+ return dir
+}
+
+// flatGenRoute is a route that classifies as FLAT against b: the default
+// db path and <RunDirOverride>/<host>.sock.
+func flatGenRoute(b *ExecBackend, host string) registry.Route {
+ return registry.Route{Host: host, Kind: "instance", Hibernate: true,
+ DBPath: provision.DefaultDBPath(b.DataDir, host),
+ Socket: filepath.Join(b.RunDirOverride, host+".sock")}
+}
+
+func isSocket(path string) bool {
+ fi, err := os.Lstat(path)
+ return err == nil && fi.Mode()&os.ModeSocket != 0
+}
+
+func isRegular(path string) bool {
+ fi, err := os.Lstat(path)
+ return err == nil && fi.Mode().IsRegular()
+}
+
+// Flag off: StartGen(perGen=false) is Start, byte for byte. The child gets
+// the canonical socket in argv and the canonical control socket in env.
+func TestStartGenWithGenerationsOffIsStart(t *testing.T) {
+ b := testExecBackend(t)
+ host := "gen-off.example.com"
+ argvPath, envPath := stageRecordingLive(t, b, host, "v1")
+ rt := registry.Route{Host: host, Kind: "instance", Socket: filepath.Join(shortRunDir(t), "instance.sock")}
+ ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+ defer cancel()
+ done := make(chan struct{})
+ var proc Proc
+ go func() {
+ proc, _, _ = b.StartGen(ctx, rt, 5, false) // the recorder never binds; a health error is expected
+ close(done)
+ }()
+ gotArgv := strings.Fields(waitForFile(t, argvPath, time.Second))
+ gotEnv := strings.TrimSpace(waitForFile(t, envPath, time.Second))
+ <-done
+ if proc != nil {
+ defer proc.Stop(context.Background())
+ }
+ wantArgv := []string{"--socket", rt.Socket, "--db", filepath.Join(b.DataDir, host+".db")}
+ if !slices.Equal(gotArgv, wantArgv) {
+ t.Fatalf("argv = %v, want %v (today's argv, unchanged)", gotArgv, wantArgv)
+ }
+ if gotEnv != rt.Socket+".control" {
+ t.Fatalf("CARLOS_CONTROL_SOCKET = %q, want %q", gotEnv, rt.Socket+".control")
+ }
+}
+
+// Flag off, healthy: the handle names the canonical socket the app bound.
+func TestStartGenWithGenerationsOffBindsTheCanonicalSocket(t *testing.T) {
+ requireUnixSockets(t)
+ b := testExecBackend(t)
+ b.RunDirOverride = shortRunDir(t)
+ host := "canon.example.com"
+ stageServingLive(t, b, host)
+ rt := flatGenRoute(b, host)
+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+ defer cancel()
+ proc, g, err := b.StartGen(ctx, rt, 11, false)
+ if err != nil {
+ t.Fatalf("StartGen: %v", err)
+ }
+ defer proc.Stop(context.Background())
+ if g.N != 11 || g.Socket != rt.Socket || g.Control != rt.Socket+".control" {
+ t.Fatalf("handle = %+v, want the canonical socket and control", g)
+ }
+ if !isSocket(rt.Socket) {
+ t.Fatalf("%s is not a bound socket", rt.Socket)
+ }
+ if _, err := os.Stat(filepath.Join(b.RunDirOverride, "g")); !os.IsNotExist(err) {
+ t.Fatalf("the generation directory exists with generations off (stat: %v)", err)
+ }
+}
+
+// Flag on, flat: a 120-byte host binds the bounded generation socket and
+// its control socket, and only the generation's OWN stale file is removed.
+// The canonical path for this host would not even fit a socket address.
+func TestStartGenFlatBindsItsOwnGenerationSocket(t *testing.T) {
+ requireUnixSockets(t)
+ b := testExecBackend(t)
+ b.RunDirOverride = shortRunDir(t)
+ host := strings.Repeat("b", 113) + ".bab.io" // 120 bytes
+ stageServingLive(t, b, host)
+ rt := flatGenRoute(b, host)
+ if l, _ := b.Layout(rt); l != LayoutFlat {
+ t.Fatalf("fixture classifies %q, want flat", l)
+ }
+ own := genSocketPathIn(filepath.Join(b.RunDirOverride, "g"), host, 9)
+ sibling := genSocketPathIn(filepath.Join(b.RunDirOverride, "g"), host, 8)
+ if err := os.MkdirAll(filepath.Dir(own), 0o750); err != nil {
+ t.Fatal(err)
+ }
+ for _, p := range []string{own, sibling} {
+ if err := os.WriteFile(p, []byte("stale"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ }
+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+ defer cancel()
+ proc, g, err := b.StartGen(ctx, rt, 9, true)
+ if err != nil {
+ t.Fatalf("StartGen: %v", err)
+ }
+ defer proc.Stop(context.Background())
+ if g.N != 9 || g.Socket != own || g.Control != own+".control" {
+ t.Fatalf("handle = %+v, want socket %s", g, own)
+ }
+ if !isSocket(own) {
+ t.Fatalf("own generation path %s is not a bound socket: the stale file was not removed", own)
+ }
+ if !isSocket(g.Control) {
+ t.Fatalf("control socket %s was not bound", g.Control)
+ }
+ if !isRegular(sibling) {
+ t.Fatalf("sibling generation's file %s was touched; only the own path may be removed", sibling)
+ }
+ if _, err := os.Lstat(rt.Socket); !os.IsNotExist(err) {
+ t.Fatalf("canonical path %s exists with generations on (lstat: %v)", rt.Socket, err)
+ }
+}
+
+// Flag on, nested: the launch file names app.<n>.sock, the control env
+// follows it, and prepare removes only that generation's stale file.
+func TestStartGenNestedUsesAppNSock(t *testing.T) {
+ ctl := &fakeControl{inv: "inv1"}
+ unit := &fakeUnit{}
+ b, rt := nestedBackend(t, ctl, unit)
+ runDir := filepath.Dir(rt.Socket)
+ own := filepath.Join(runDir, "app.9.sock")
+ sibling := filepath.Join(runDir, "app.8.sock")
+ if err := os.MkdirAll(runDir, 0o750); err != nil {
+ t.Fatal(err)
+ }
+ for _, p := range []string{rt.Socket, sibling, own} {
+ if err := os.WriteFile(p, []byte("stale"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ }
+ var stopServing func()
+ ctl.onStart = func() { stopServing = serveOK(t, own) }
+ defer func() {
+ if stopServing != nil {
+ stopServing()
+ }
+ }()
+ unit.set("active", "inv1", true, true)
+ p, g, err := b.StartGen(context.Background(), rt, 9, true)
+ if err != nil {
+ t.Fatalf("StartGen: %v", err)
+ }
+ if g.Socket != own || g.Control != own+".control" {
+ t.Fatalf("handle = %+v, want %s", g, own)
+ }
+ raw, err := os.ReadFile(filepath.Join(provision.InstanceDir(b.DataDir, rt.Host), "launch.run.json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var l instancectl.Launch
+ if err := json.Unmarshal(raw, &l); err != nil {
+ t.Fatal(err)
+ }
+ if len(l.Argv) != 5 || l.Argv[1] != "--socket" || l.Argv[2] != own {
+ t.Fatalf("launch argv = %v, want --socket %s", l.Argv, own)
+ }
+ if !slices.Contains(l.Env, "CARLOS_CONTROL_SOCKET="+own+".control") {
+ t.Fatalf("launch env lacks CARLOS_CONTROL_SOCKET=%s.control: %v", own, l.Env)
+ }
+ if !isRegular(rt.Socket) || !isRegular(sibling) {
+ t.Fatal("prepare removed a file that is not this generation's own socket")
+ }
+ unit.set("inactive", "inv1", false, false)
+ waitFor(t, "the unit to exit", func() bool { return p.Exited() })
+ ctx, cancel := context.WithTimeout(context.Background(), time.Second)
+ defer cancel()
+ _ = p.Stop(ctx)
+}
+
+// NextDueVia dials the socket it is given, not rt.Socket.
+func TestNextDueViaDialsTheGivenSocket(t *testing.T) {
+ requireUnixSockets(t)
+ b := testExecBackend(t)
+ host := "due.example.com"
+ if err := os.WriteFile(filepath.Join(b.DataDir, host+".env"), []byte("CARLOS_ADMIN_TOKEN=tok\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ sock := tmpSock(t)
+ ln, err := net.Listen("unix", sock)
+ if err != nil {
+ t.Fatal(err)
+ }
+ srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if r.URL.Path != "/api/next-due" || r.Header.Get("Authorization") != "Bearer tok" {
+ http.Error(w, "no", http.StatusForbidden)
+ return
+ }
+ w.Write([]byte(`{"due":42}`))
+ })}
+ go srv.Serve(ln)
+ defer srv.Close()
+ rt := registry.Route{Host: host, Kind: "instance", Socket: "/nonexistent/canonical.sock"}
+ due, err := b.NextDueVia(context.Background(), rt, sock)
+ if err != nil || due != 42 {
+ t.Fatalf("NextDueVia = %d, %v; want 42 from the generation socket", due, err)
+ }
+ if _, err := b.NextDue(context.Background(), rt); err == nil {
+ t.Fatal("NextDue reached something at the canonical path; the fixture has nothing there")
+ }
+}
+
+// A unit-backed route has no generation sockets: SplitBackend starts it
+// with Start and hands back the canonical handle, whatever perGen says.
+func TestSplitBackendStartGenKeepsUnitTenantsCanonical(t *testing.T) {
+ unitSide := &fakeBackend{execVerFn: func() string { return "v3" }}
+ sb := &SplitBackend{Exec: &fakeBackend{}, Unit: unitSide}
+ rt := registry.Route{Host: "u.example.com", Kind: "instance", Backing: registry.BackingUnit, Socket: "/run/carlos/u.example.com.sock"}
+ _, g, err := sb.StartGen(context.Background(), rt, 4, true)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if g != canonicalGen(rt, 4, "v3") {
+ t.Fatalf("handle = %+v, want the canonical one", g)
+ }
+ if unitSide.starts != 1 {
+ t.Fatalf("unit side Start calls = %d, want 1", unitSide.starts)
+ }
+}
diff --git a/internal/activator/generations_test.go b/internal/activator/generations_test.go
new file mode 100644
index 000000000..bb26d138d
--- /dev/null
+++ b/internal/activator/generations_test.go
@@ -0,0 +1,95 @@
+package activator
+
+import (
+ "math"
+ "strings"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// withGenerations sets the dormant switch for one test and restores it after.
+// Every test that builds a Service with the switch on goes through here: the
+// switch is package state, so a test that forgot to restore it would turn
+// generations on for every test after it.
+func withGenerations(t *testing.T, on bool) {
+ t.Helper()
+ prev := GenerationsEnabled()
+ SetGenerations(on)
+ t.Cleanup(func() { SetGenerations(prev) })
+}
+
+// The cross-plan review item this plan owns: a 120-byte hostname gets
+// generation and control sockets under the 108-byte sockaddr_un limit, in
+// both layouts, at the largest generation number a uint64 can hold.
+func TestGenerationSocketsFitSockaddrUnForLongHosts(t *testing.T) {
+ for _, host := range []string{
+ "h",
+ strings.Repeat("a", 110) + ".bab.io", // 117 bytes
+ strings.Repeat("b", 113) + ".bab.io", // 120 bytes
+ strings.Repeat("c", 253),
+ } {
+ for _, n := range []uint64{1, 42, math.MaxUint64} {
+ flat := GenSocketPath(host, n)
+ if got := len(flat + ".control"); got > 60 {
+ t.Errorf("GenSocketPath(%d-byte host, %d)+.control = %d bytes, want at most 60 (spec §7.1)", len(host), n, got)
+ }
+ nested := NestedGenSocket(host, n)
+ for _, p := range []string{flat, flat + ".control", nested, nested + ".control"} {
+ if len(p) > provision.MaxUnixPath {
+ t.Errorf("%q is %d bytes, over the %d a unix socket address holds", p, len(p), provision.MaxUnixPath)
+ }
+ }
+ }
+ }
+}
+
+func TestGenSocketPathShape(t *testing.T) {
+ got := GenSocketPath("bill.go.tito.io", 7)
+ if !strings.HasPrefix(got, "/run/carlos/g/") || !strings.HasSuffix(got, ".7.sock") {
+ t.Fatalf("GenSocketPath = %q, want /run/carlos/g/<h12>.7.sock", got)
+ }
+ h12 := strings.TrimSuffix(strings.TrimPrefix(got, "/run/carlos/g/"), ".7.sock")
+ if len(h12) != 12 || strings.Trim(h12, "0123456789abcdef") != "" {
+ t.Fatalf("hash part %q is not 12 lower-case hex digits", h12)
+ }
+ if strings.Contains(got, "bill") {
+ t.Fatalf("GenSocketPath %q embeds the hostname; the bound depends on it never doing so", got)
+ }
+ if GenSocketPath("bill.go.tito.io", 8) == got || GenSocketPath("other.go.tito.io", 7) == got {
+ t.Fatal("generation paths collide across generations or hosts")
+ }
+ if got, want := NestedGenSocket("bill.go.tito.io", 7), "/run/carlos/instances/bill.go.tito.io/app.7.sock"; got != want {
+ t.Fatalf("NestedGenSocket = %q, want %q", got, want)
+ }
+}
+
+func TestGenCounterIsProcessWideAndNeverRepeats(t *testing.T) {
+ seen := map[uint64]bool{}
+ var last uint64
+ for i := 0; i < 1000; i++ {
+ n := nextGen()
+ if n == 0 || seen[n] || n <= last {
+ t.Fatalf("nextGen returned %d after %d (seen=%v); want strictly increasing, never 0", n, last, seen[n])
+ }
+ seen[n], last = true, n
+ }
+}
+
+func TestCanonicalGenKeepsTheRouteSocket(t *testing.T) {
+ rt := registry.Route{Host: "h.example.com", Socket: "/run/carlos/h.example.com.sock"}
+ g := canonicalGen(rt, 3, "v1")
+ if g.N != 3 || g.Socket != rt.Socket || g.Control != rt.Socket+".control" || g.Build != "v1" {
+ t.Fatalf("canonicalGen = %+v", g)
+ }
+ if g := canonicalGen(registry.Route{Host: "w"}, 4, ""); g.Control != "" {
+ t.Fatalf("a socketless route got control %q", g.Control)
+ }
+}
+
+func TestGenerationsSwitchDefaultsOff(t *testing.T) {
+ if GenerationsEnabled() {
+ t.Fatal("generations are on by default; CARLOS_SWAP unset must leave them off (spec §15)")
+ }
+}
diff --git a/internal/activator/genservice.go b/internal/activator/genservice.go
new file mode 100644
index 000000000..5b6cbeac6
--- /dev/null
+++ b/internal/activator/genservice.go
@@ -0,0 +1,115 @@
+package activator
+
+import (
+ "context"
+ "sync/atomic"
+
+ "github.com/carlosframework/platform/internal/edge"
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// The Service's generation surface (zero-downtime spec §7.1). AppSocket is
+// for traffic; the handle (Gen) is for whoever started the generation.
+var _ edge.GenSource = (*Service)(nil)
+
+// GenerationsOn is the dormant switch as this Service read it at New.
+func (a *Service) GenerationsOn() bool { return a.gens }
+
+// startGen spawns the next generation of rt's instance: the process-wide
+// counter's next number, and the backend's generation spawn when it has
+// one. A backend without one is spawned with Start and gets the canonical
+// handle.
+func (a *Service) startGen(ctx context.Context, rt registry.Route) (Proc, Gen, error) {
+ n := nextGen()
+ if g, ok := a.backend.(GenBackend); ok {
+ return g.StartGen(ctx, rt, n, a.gens)
+ }
+ proc, ver, err := a.backend.Start(ctx, rt)
+ if err != nil {
+ return proc, Gen{}, err
+ }
+ return proc, canonicalGen(rt, n, ver), nil
+}
+
+// AppSocket names host's CURRENT generation's socket, for traffic: the edge
+// proxy, scheduled delivery, the next-due probe at a stop, and the status
+// probe (spec §7.1). It answers for an awake or stopping entry (a stopping
+// generation is still the one in-flight requests reach). Asleep, waking or
+// a worker: false.
+func (a *Service) AppSocket(host string) (string, bool) {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ e := a.entries[host]
+ if e == nil || e.state == actWaking || e.gen.Socket == "" {
+ return "", false
+ }
+ return e.gen.Socket, true
+}
+
+// ServingGen is AppSocket with the generation's number and build: the edge
+// picks a connection pool, stamps X-Carlos-Version and tags cache entries
+// from it (spec §7.2).
+func (a *Service) ServingGen(host string) (edge.GenInfo, bool) {
+ a.mu.Lock()
+ defer a.mu.Unlock()
+ e := a.entries[host]
+ if e == nil || e.state == actWaking || e.gen.Socket == "" {
+ return edge.GenInfo{}, false
+ }
+ return edge.GenInfo{Socket: e.gen.Socket, N: e.gen.N, Build: e.gen.Build}, true
+}
+
+// CacheGen is the newest generation that has been current for host in this
+// process. Lock-free on purpose: the edge cache calls it under its own lock,
+// and plan 3's switch purges the cache while holding a.mu.
+func (a *Service) CacheGen(host string) (uint64, bool) {
+ v, ok := a.cacheGens.Load(host)
+ if !ok {
+ return 0, false
+ }
+ return v.(*atomic.Uint64).Load(), true
+}
+
+// noteCurrent records that generation n of host has become current.
+func (a *Service) noteCurrent(host string, n uint64) {
+ v, _ := a.cacheGens.LoadOrStore(host, new(atomic.Uint64))
+ v.(*atomic.Uint64).Store(n)
+}
+
+// OnGenRetired registers fn for every generation that stops being current
+// (edge.GenSource). Hooks fire only with generations on.
+func (a *Service) OnGenRetired(fn func(host, socket string)) {
+ a.retiredMu.Lock()
+ defer a.retiredMu.Unlock()
+ a.retiredHooks = append(a.retiredHooks, fn)
+}
+
+// genRetired tells every hook that g has stopped being host's current
+// generation. The caller holds no lock. A no-op with generations off: the
+// canonical path is shared by every generation of a host then, so closing
+// "its" idle connections could close the next process's.
+func (a *Service) genRetired(host string, g Gen) {
+ if !a.gens || g.Socket == "" {
+ return
+ }
+ a.retiredMu.Lock()
+ hooks := append([]func(string, string){}, a.retiredHooks...)
+ a.retiredMu.Unlock()
+ for _, fn := range hooks {
+ fn(host, g.Socket)
+ }
+}
+
+// nextDue is the sleep-time next-due probe, dialled at the instance's
+// current generation (spec §7.1, `backend_exec.go`'s NextDue). Hibernation
+// joins any successor first (plan 3), so at a stop the current generation
+// is the only one. With generations off AppSocket is rt.Socket, and this is
+// the old call.
+func (a *Service) nextDue(ctx context.Context, e *actEntry) (int64, error) {
+ if g, ok := a.backend.(GenBackend); ok {
+ if sock, ok := a.AppSocket(e.host); ok {
+ return g.NextDueVia(ctx, e.route, sock)
+ }
+ }
+ return a.backend.NextDue(ctx, e.route)
+}
diff --git a/internal/activator/genservice_test.go b/internal/activator/genservice_test.go
new file mode 100644
index 000000000..51d678540
--- /dev/null
+++ b/internal/activator/genservice_test.go
@@ -0,0 +1,236 @@
+package activator
+
+import (
+ "context"
+ "fmt"
+ "path/filepath"
+ "slices"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// genFakeBackend is fakeBackend with the generation capability. A spawn
+// with perGen "binds" /gen/<n>.sock (nothing is bound: the fake proc stands
+// in for the process) and NextDueVia records the socket it was asked to
+// dial, so a test can see which generation a probe reached.
+type genFakeBackend struct {
+ *fakeBackend
+ gmu sync.Mutex
+ perGens []bool
+ dueSocks []string
+}
+
+func (b *genFakeBackend) StartGen(ctx context.Context, rt registry.Route, n uint64, perGen bool) (Proc, Gen, error) {
+ b.gmu.Lock()
+ b.perGens = append(b.perGens, perGen)
+ b.gmu.Unlock()
+ proc, ver, err := b.fakeBackend.Start(ctx, rt)
+ if err != nil {
+ return proc, Gen{}, err
+ }
+ sock := rt.Socket
+ if perGen {
+ sock = fmt.Sprintf("/gen/%d.sock", n)
+ }
+ return proc, Gen{N: n, Socket: sock, Control: sock + ".control", Build: ver}, nil
+}
+
+func (b *genFakeBackend) NextDueVia(ctx context.Context, rt registry.Route, socket string) (int64, error) {
+ b.gmu.Lock()
+ b.dueSocks = append(b.dueSocks, socket)
+ b.gmu.Unlock()
+ return b.fakeBackend.NextDue(ctx, rt)
+}
+
+func (b *genFakeBackend) seen() (perGens []bool, dueSocks []string) {
+ b.gmu.Lock()
+ defer b.gmu.Unlock()
+ return slices.Clone(b.perGens), slices.Clone(b.dueSocks)
+}
+
+// genActivator is testActivator over genFakeBackend, with the switch set
+// for the test's duration BEFORE New reads it.
+func genActivator(t *testing.T, on bool) (*Service, *genFakeBackend) {
+ t.Helper()
+ withGenerations(t, on)
+ gb := &genFakeBackend{fakeBackend: &fakeBackend{execVerFn: func() string { return "v7" }}}
+ reg, err := registry.Open(filepath.Join(t.TempDir(), "registry.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return New(Config{Bucket: "test-bucket", Idle: time.Hour}, gb, reg), gb
+}
+
+// genTestRoute is testRoute with its canonical socket in the test's own
+// temp dir. With generations on the activator links a route's canonical
+// path to its current generation (Task 4), and testRoute's shared
+// /tmp/nope.sock must never become a real link.
+func genTestRoute(t *testing.T, host string) registry.Route {
+ t.Helper()
+ rt := testRoute(host)
+ rt.Socket = filepath.Join(t.TempDir(), "c.sock")
+ return rt
+}
+
+// stopAndWait bounces host's awake instance and waits for the stop to finish.
+func stopAndWait(t *testing.T, a *Service, host string) {
+ t.Helper()
+ if !a.Bounce(host) {
+ t.Fatalf("Bounce(%s) started no stop", host)
+ }
+ waitFor(t, host+" to stop", func() bool { return !a.Owns(host) })
+}
+
+func TestAppSocketNamesTheCurrentGeneration(t *testing.T) {
+ a, gb := genActivator(t, true)
+ rt := genTestRoute(t, "cur.example.com")
+ if _, ok := a.AppSocket(rt.Host); ok {
+ t.Fatal("AppSocket answered for an instance that never woke")
+ }
+ arrangeAwake(t, a, rt)
+ g, ok := a.ServingGen(rt.Host)
+ if !ok || g.N == 0 {
+ t.Fatalf("ServingGen = %+v, %v; want the awake generation", g, ok)
+ }
+ sock, ok := a.AppSocket(rt.Host)
+ if !ok || sock != fmt.Sprintf("/gen/%d.sock", g.N) || g.Socket != sock || g.Build != "v7" {
+ t.Fatalf("AppSocket = %q (%v), ServingGen = %+v; want /gen/%d.sock and build v7", sock, ok, g, g.N)
+ }
+ if n, ok := a.CacheGen(rt.Host); !ok || n != g.N {
+ t.Fatalf("CacheGen = %d, %v; want %d", n, ok, g.N)
+ }
+ if perGens, _ := gb.seen(); !slices.Equal(perGens, []bool{true}) {
+ t.Fatalf("StartGen perGen = %v, want [true]", perGens)
+ }
+}
+
+// Flag off: one generation per entry, on the canonical socket.
+func TestAppSocketIsTheCanonicalSocketWithGenerationsOff(t *testing.T) {
+ a, gb := genActivator(t, false)
+ rt := genTestRoute(t, "off.example.com")
+ arrangeAwake(t, a, rt)
+ if sock, ok := a.AppSocket(rt.Host); !ok || sock != rt.Socket {
+ t.Fatalf("AppSocket = %q, %v; want the canonical %q", sock, ok, rt.Socket)
+ }
+ if perGens, _ := gb.seen(); !slices.Equal(perGens, []bool{false}) {
+ t.Fatalf("StartGen perGen = %v, want [false]", perGens)
+ }
+}
+
+// A backend without the capability (UnitBackend, older doubles) still gets a
+// numbered, canonical handle.
+func TestPlainBackendGetsTheCanonicalHandle(t *testing.T) {
+ withGenerations(t, true)
+ a, _, _, _ := testActivator(t, time.Hour)
+ rt := genTestRoute(t, "plain.example.com")
+ arrangeAwake(t, a, rt)
+ g, ok := a.ServingGen(rt.Host)
+ if !ok || g.Socket != rt.Socket || g.N == 0 {
+ t.Fatalf("ServingGen = %+v, %v; want the canonical socket and a number", g, ok)
+ }
+}
+
+// Spec §12 "Generation identity": a stop, then a crash, then a wake never
+// reuses a generation number. CacheGen keeps the last number while the
+// instance sleeps.
+func TestGenerationNumbersNeverRepeatAcrossStopCrashAndWake(t *testing.T) {
+ a, gb := genActivator(t, true)
+ rt := genTestRoute(t, "ident.example.com")
+ arrangeAwake(t, a, rt)
+ g1, _ := a.ServingGen(rt.Host)
+
+ stopAndWait(t, a, rt.Host)
+ if _, ok := a.AppSocket(rt.Host); ok {
+ t.Fatal("AppSocket answered for a sleeping instance")
+ }
+ if n, ok := a.CacheGen(rt.Host); !ok || n != g1.N {
+ t.Fatalf("CacheGen while asleep = %d, %v; want the last current generation %d", n, ok, g1.N)
+ }
+
+ arrangeAwake(t, a, rt)
+ g2, _ := a.ServingGen(rt.Host)
+ if g2.N <= g1.N {
+ t.Fatalf("wake after a stop took generation %d after %d", g2.N, g1.N)
+ }
+
+ gb.fakeBackend.mu.Lock()
+ crashed := gb.fakeBackend.procs[len(gb.fakeBackend.procs)-1]
+ gb.fakeBackend.mu.Unlock()
+ crashed.exit()
+ arrangeAwake(t, a, rt) // finds the dead child, stops it, wakes a fresh one
+ g3, _ := a.ServingGen(rt.Host)
+ if g3.N <= g2.N {
+ t.Fatalf("wake after a crash took generation %d after %d", g3.N, g2.N)
+ }
+ if n, _ := a.CacheGen(rt.Host); n != g3.N {
+ t.Fatalf("CacheGen = %d, want %d", n, g3.N)
+ }
+}
+
+// The sleep-time next-due probe dials AppSocket: the generation socket with
+// the switch on, the canonical one with it off.
+func TestNextDueAtStopDialsTheCurrentGeneration(t *testing.T) {
+ for _, on := range []bool{true, false} {
+ t.Run(fmt.Sprintf("generations=%v", on), func(t *testing.T) {
+ a, gb := genActivator(t, on)
+ rt := genTestRoute(t, "due.example.com")
+ arrangeAwake(t, a, rt)
+ g, _ := a.ServingGen(rt.Host)
+ stopAndWait(t, a, rt.Host)
+ want := rt.Socket
+ if on {
+ want = fmt.Sprintf("/gen/%d.sock", g.N)
+ }
+ if _, dueSocks := gb.seen(); !slices.Equal(dueSocks, []string{want}) {
+ t.Fatalf("next-due probe dialled %v, want [%s]", dueSocks, want)
+ }
+ })
+ }
+}
+
+// A retire hook hears about a generation that stopped being current, with
+// the switch on, and never with it off (the canonical path is shared by every
+// generation then).
+func TestGenRetiredFiresAtStopOnlyWithGenerationsOn(t *testing.T) {
+ for _, on := range []bool{true, false} {
+ t.Run(fmt.Sprintf("generations=%v", on), func(t *testing.T) {
+ a, _ := genActivator(t, on)
+ var mu sync.Mutex
+ var got []string
+ a.OnGenRetired(func(host, socket string) {
+ mu.Lock()
+ got = append(got, host+" "+socket)
+ mu.Unlock()
+ })
+ rt := genTestRoute(t, "ret.example.com")
+ arrangeAwake(t, a, rt)
+ g, _ := a.ServingGen(rt.Host)
+ stopAndWait(t, a, rt.Host)
+ mu.Lock()
+ defer mu.Unlock()
+ var want []string
+ if on {
+ want = []string{rt.Host + " " + g.Socket}
+ }
+ if !slices.Equal(got, want) {
+ t.Fatalf("retire hooks = %v, want %v", got, want)
+ }
+ })
+ }
+}
+
+// A worker binds nothing, so it has no app socket for traffic.
+func TestAppSocketIsFalseForAWorker(t *testing.T) {
+ a, _ := genActivator(t, true)
+ rt := testWorkerRoute("bot.x.internal")
+ if err := a.WakeWorker(rt); err != nil {
+ t.Fatalf("WakeWorker: %v", err)
+ }
+ waitFor(t, "the worker to wake", func() bool { return a.Owns(rt.Host) })
+ if sock, ok := a.AppSocket(rt.Host); ok {
+ t.Fatalf("AppSocket(worker) = %q, want false", sock)
+ }
+}
diff --git a/internal/activator/prepare.go b/internal/activator/prepare.go
index e3a9f4205..f8ac05fcb 100644
--- a/internal/activator/prepare.go
+++ b/internal/activator/prepare.go
@@ -31,6 +31,14 @@ import (
// and stale records cleared. It returns the listeners so the caller can
// hand them to the proc or close them on a failed start.
func (b *ExecBackend) prepare(rt registry.Route, role instancectl.Role) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) {
+ return b.prepareAt(rt, role, rt.Socket)
+}
+
+// prepareAt is prepare for a run role whose app socket is sock: rt.Socket,
+// or a generation socket (StartGen, zero-downtime spec §7.1). Only the
+// file at sock is removed, never a sibling generation's, and the control
+// listener binds sock + ".control". The seed and sidecar roles ignore sock.
+func (b *ExecBackend) prepareAt(rt registry.Route, role instancectl.Role, sock string) (instancePaths, *edge.ControlListener, *logListener, map[string]string, error) {
p := b.pathsFor(rt)
if err := os.MkdirAll(p.dataDir, 0o700); err != nil {
return p, nil, nil, nil, err
@@ -88,8 +96,8 @@ func (b *ExecBackend) prepare(rt registry.Route, role instancectl.Role) (instanc
// no member-facing diagnosis at all, unlike the flat path's #129
// handling, which at least turns the bind failure into a named
// error (memberSafeWakeReason).
- _ = os.Remove(rt.Socket)
- ctl = b.listenControl(rt, p.envFile)
+ _ = os.Remove(sock)
+ ctl = b.listenControl(rt, sock+".control", p.envFile)
}
logs, err := listenLogs(b.logSocket(rt.Host, role), b.logEmitter(rt))
if err != nil {
@@ -106,7 +114,7 @@ func (b *ExecBackend) prepare(rt registry.Route, role instancectl.Role) (instanc
// reading beneath the instance directory rather than following a plain
// path, design §3.2); a bind failure is logged and the instance runs
// without its control verbs, as today.
-func (b *ExecBackend) listenControl(rt registry.Route, envPath string) *edge.ControlListener {
+func (b *ExecBackend) listenControl(rt registry.Route, control, envPath string) *edge.ControlListener {
tokenReader := func() (string, error) {
env, err := readEnvFileBeneath(filepath.Dir(envPath), filepath.Base(envPath))
if err != nil {
@@ -118,7 +126,7 @@ func (b *ExecBackend) listenControl(rt registry.Route, envPath string) *edge.Con
if b.Timers != nil {
mount = func(mux *http.ServeMux) { schedule.Mount(mux, rt.Host, b.Timers, tokenReader, time.Now) }
}
- ctl, err := edge.ListenControl(rt.Socket+".control", rt.Host, b.Cache, mount)
+ ctl, err := edge.ListenControl(control, rt.Host, b.Cache, mount)
if err != nil {
log.Printf("activator: control listener for %s: %v (continuing without it)", rt.Host, err)
return nil
diff --git a/internal/edge/answered.go b/internal/edge/answered.go
index 2df6e81d9..9699a7a13 100644
--- a/internal/edge/answered.go
+++ b/internal/edge/answered.go
@@ -30,6 +30,11 @@ type outcome struct {
answered bool
cause string
start time.Time // when the edge began serving, so a proxy failure knows how long the client waited
+ // genHost and gen name the generation whose connection pool carried the
+ // request (genproxy.go), set only by an activator-managed route with
+ // generations on. The cache tags what it stores with them (spec §7.2).
+ genHost string
+ gen uint64
}
// causeClientGone is the cause noted when a proxy fails because the CLIENT
@@ -83,3 +88,19 @@ func noteCause(r *http.Request, cause string) {
o.cause = cause
}
}
+
+// noteServedGen records on r's outcome that generation n of host carried
+// the request; a no-op when r carries no outcome, like MarkAnswered.
+func noteServedGen(r *http.Request, host string, n uint64) {
+ if o, ok := r.Context().Value(answeredKey{}).(*outcome); ok {
+ o.genHost, o.gen = host, n
+ }
+}
+
+// servedGen is what noteServedGen recorded: ok false when nothing was.
+func servedGen(r *http.Request) (host string, n uint64, ok bool) {
+ if o, isOutcome := r.Context().Value(answeredKey{}).(*outcome); isOutcome && o.gen != 0 {
+ return o.genHost, o.gen, true
+ }
+ return "", 0, false
+}
diff --git a/internal/edge/cache.go b/internal/edge/cache.go
index d1c08cb16..0bebc2c0d 100644
--- a/internal/edge/cache.go
+++ b/internal/edge/cache.go
@@ -47,6 +47,15 @@ type CachedResponse struct {
Body []byte
StoredAt time.Time
TTL time.Duration
+ // Route and Gen tag an entry with the RESOLVED serving route (the
+ // instance host, whatever name the request used: an alias, a wildcard,
+ // a path-routed front door) and the generation of it that produced the
+ // response (zero-downtime spec §7.2). Empty Route: untagged, which is
+ // every entry with generations off and every route the activator does
+ // not manage. A tagged entry is live only while Gen is still the
+ // route's newest generation (SetGenerations).
+ Route string
+ Gen uint64
}
// Cache is an in-memory, per-host response cache for cacheable GET/HEAD
@@ -64,6 +73,10 @@ type Cache struct {
byURL map[cacheURLKey]map[cacheFullKey]*list.Element // urlKey -> its Vary variants
byPath map[cachePathKey]map[cacheFullKey]*list.Element // pathKey -> every entry under that path
byHost map[string]map[cacheFullKey]*list.Element // host -> every entry for that host
+ // genOf answers a tagged entry's route's newest generation
+ // (SetGenerations); nil with generations off. Called under mu, so it
+ // must take no lock its own side holds while calling into the cache.
+ genOf func(route string) (uint64, bool)
}
type cacheURLKey struct {
@@ -141,6 +154,30 @@ func NewCache(maxEntries, maxBodyBytes int) *Cache {
}
}
+// SetGenerations arms the generation guard (spec §7.2): a tagged entry is
+// refused at Put, and treated as a miss at lookup, unless its Gen is
+// genOf(Route). The edge calls it once, from EnableCache, only with
+// generations on.
+func (c *Cache) SetGenerations(genOf func(route string) (uint64, bool)) {
+ c.mu.Lock()
+ defer c.mu.Unlock()
+ c.genOf = genOf
+}
+
+// genCurrentLocked reports whether resp may be served or stored: untagged,
+// or tagged with its route's newest generation. A tagged entry with no
+// guard armed is not current: it cannot be vouched for. Caller holds c.mu.
+func (c *Cache) genCurrentLocked(resp *CachedResponse) bool {
+ if resp.Route == "" {
+ return true
+ }
+ if c.genOf == nil {
+ return false
+ }
+ n, ok := c.genOf(resp.Route)
+ return ok && n == resp.Gen
+}
+
func (k CacheKey) urlKey() cacheURLKey {
return cacheURLKey{Host: k.Host, Method: k.Method, Path: k.Path, RawQuery: k.RawQuery}
}
@@ -246,6 +283,12 @@ func (c *Cache) selectLocked(uk cacheURLKey, want func(*cacheEntry) bool) (*Cach
if !want(e) {
continue
}
+ // A generation that is no longer its route's newest is a miss for
+ // every name the route answers to (spec §7.2). Left in place: the
+ // switch's purge, the TTL or the LRU reclaims it.
+ if !c.genCurrentLocked(e.resp) {
+ continue
+ }
if bestEntry == nil || better(e, bestEntry) {
best, bestEntry = el, e
}
@@ -319,6 +362,12 @@ func (c *Cache) Put(key CacheKey, vary string, resp *CachedResponse) {
}
c.mu.Lock()
defer c.mu.Unlock()
+ // A response whose generation stopped being current while it was in
+ // flight (a late answer from a draining process) never fills the cache
+ // after the switch (spec §7.2).
+ if !c.genCurrentLocked(resp) {
+ return
+ }
fk := key.fullKey(canonicalVary(vary))
uk := fk.cacheURLKey
if el, ok := c.entries[fk]; ok {
diff --git a/internal/edge/edge.go b/internal/edge/edge.go
index 4f95941a8..80772e355 100644
--- a/internal/edge/edge.go
+++ b/internal/edge/edge.go
@@ -59,7 +59,6 @@ import (
"github.com/carlosframework/platform/internal/provision"
"github.com/carlosframework/platform/internal/registry"
"github.com/carlosframework/platform/internal/release"
- "github.com/carlosframework/platform/internal/unixdial"
)
// Default cache sizing for EnableCache: 4096 entries is comfortably above a
@@ -276,6 +275,14 @@ type Edge struct {
doorAddr string
doorKeys *doorKeyMirror
closeDoor func()
+ // gens is the activator's generation view (zero-downtime spec §7.2),
+ // set in New only when the activator implements GenSource AND its
+ // switch is on; genPools is the edge's per-generation-socket idle
+ // pools, retired through gens.OnGenRetired. Both nil with the switch off
+ // (CARLOS_SWAP unset), and then nothing below reads them: managed
+ // routes get unixProxy(t.Socket) and the row's version, as before.
+ gens GenSource
+ genPools *GenPools
}
// RunOption configures the Edge that Run/RunWithActivator* builds, before it
@@ -349,6 +356,10 @@ type upstream struct {
// captured here would be a stale copy of the target's entry the moment
// the target's route changed within the same map build.
paths map[string]string
+ // genManaged marks an activator-managed instance proxied through
+ // appSocketProxy (generations on). Its cacheable responses are stored
+ // only when tagged with the generation that served them (cache.go).
+ genManaged bool
}
// New builds an Edge over reg and performs the startup reload. act may be nil
@@ -357,6 +368,12 @@ func New(reg *registry.Registry, act Activator) (*Edge, error) {
e := &Edge{reg: reg, act: act, gate: gate.New(), probers: newProberSet(), gzipOn: gzipEnabled(), steeringHost: steeringProbeHostFromEnv(), appsDomain: appsDomainFromEnv()}
e.geo = geo.NewSwappable()
e.geoLookup = e.geo.Lookup
+ // Generations (zero-downtime spec §7.2), before the startup Reload so
+ // its first table build already proxies through them.
+ if gs, ok := act.(GenSource); ok && gs.GenerationsOn() {
+ e.gens, e.genPools = gs, NewGenPools()
+ gs.OnGenRetired(e.genPools.Retire)
+ }
if err := e.Reload(); err != nil {
return nil, err
}
@@ -388,6 +405,11 @@ func (e *Edge) EnableStatic(store release.Store, pub ed25519.PublicKey, cacheDir
// existing nil-means-disabled convention for Activator.
func (e *Edge) EnableCache(cache *Cache) {
e.cache = cache
+ // The generation guard (spec §7.2), only with generations on: e.gens is
+ // nil otherwise and the cache stays untagged, exactly as before.
+ if cache != nil && e.gens != nil {
+ cache.SetGenerations(e.gens.CacheGen)
+ }
}
// EnableTunnels wires the reverse-channel hub in: the edge answers hub.Host()
@@ -622,6 +644,9 @@ func (e *Edge) Reload() error {
// viaTunnel: this host's bytes leave the box over a reverse channel,
// which is what disqualifies it from hibernation below.
viaTunnel := false
+ // genManaged: an activator-managed instance reached through its
+ // current generation (appSocketProxy), generations on only.
+ genManaged := false
fleet, isFleetPlaced := registry.FleetPlacement(t.Region)
switch {
// Both reverse proxies get the Carlos-Purge response hook
@@ -682,7 +707,17 @@ func (e *Edge) Reload() error {
proxy = p
viaTunnel = true
case t.Socket != "":
- p := unixProxy(t.Socket)
+ // An activator-managed instance's process binds a generation
+ // socket, not t.Socket, once generations are on (spec §7.1):
+ // every new connection asks the activator for the current one.
+ // t.Kind/t.Hibernate are the activator-managed test below
+ // (hib); viaTunnel is false in this arm by construction.
+ var p *httputil.ReverseProxy
+ if e.gens != nil && t.Kind == "instance" && t.Hibernate {
+ p, genManaged = appSocketProxy(t.Host, e.gens, e.genPools), true
+ } else {
+ p = unixProxy(t.Socket)
+ }
p.ModifyResponse = e.responseModifier(t)
proxy = p
// TCP upstreams. The gate is AddrList (registry), not t.Addr, so a
@@ -772,7 +807,11 @@ func (e *Edge) Reload() error {
// wrapping those would put a version on a response that is serving
// none, which is exactly the false "verified" the header exists to
// prevent.
- if t.Version != "" && t.Kind != "static" && t.Kind != "status" {
+ //
+ // A genManaged route is excluded too: its header comes from the
+ // generation that served the request (genRoundTripper, spec §7.2),
+ // and the registry's Version names only what is installed.
+ if t.Version != "" && t.Kind != "static" && t.Kind != "status" && !genManaged {
proxy = versionHeader(t.Version, proxy)
}
// Hibernation is per-host: only an instance whose route carries the
@@ -833,7 +872,7 @@ func (e *Edge) Reload() error {
"route %s is a hibernating instance but this box has no activator, so nothing here will create %s or start its process. If this box runs `carlos agent`, set CARLOS_HIBERNATE_BUCKET in /etc/carlos/host.env and restart carlos-edge; `carlos edge` on its own never runs instances.",
t.Host, t.Socket))
}
- m[t.Host] = upstream{proxy: proxy, route: t, hibernating: hib, viaTunnel: viaTunnel}
+ m[t.Host] = upstream{proxy: proxy, route: t, hibernating: hib, viaTunnel: viaTunnel, genManaged: genManaged}
}
// Path routes, folded BEFORE aliases so an alias of a prefix-routed
// hostname inherits its map along with everything else it copies — an
@@ -1709,7 +1748,7 @@ func (e *Edge) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// The response is always recorded (even when not cacheable, in which
// case maybeStore is a no-op on rec.cacheable) so this branch stays a
// single shape regardless of method.
- rec := &recordingResponseWriter{ResponseWriter: w, request: r, cacheable: cacheable, maxBody: e.cacheMaxBody()}
+ rec := &recordingResponseWriter{ResponseWriter: w, request: r, cacheable: cacheable, genManaged: up.genManaged, maxBody: e.cacheMaxBody()}
if e.act.ServeHTTP(rec, r, up.route, up.proxy) {
e.maybeStore(key, rec)
return
@@ -1825,13 +1864,25 @@ func (e *Edge) maybeStore(key CacheKey, rec *recordingResponseWriter) {
// request — and so a response that varies by different headers, or doesn't
// vary at all, becomes a variant beside this one rather than a replacement
// for it.
- e.cache.Put(key, vary, &CachedResponse{
+ resp := &CachedResponse{
StatusCode: rec.status,
Header: rec.Header().Clone(),
Body: rec.body,
StoredAt: time.Now(),
TTL: ttl,
- })
+ }
+ // Tagged with the resolved serving route and its generation (spec
+ // §7.2). A managed response that no generation is recorded for (it
+ // reached the process before any generation was current) is not stored:
+ // nothing could later prove it current.
+ if rec.genManaged {
+ route, n, ok := servedGen(rec.request)
+ if !ok {
+ return
+ }
+ resp.Route, resp.Gen = route, n
+ }
+ e.cache.Put(key, vary, resp)
}
// hasTrailerSetCookie reports whether h carries a Set-Cookie TRAILER. An
@@ -2045,11 +2096,14 @@ type recordingResponseWriter struct {
http.ResponseWriter
request *http.Request
cacheable bool
- maxBody int
- status int
- body []byte
- truncated bool
- wroteHdr bool
+ // genManaged: the response came through appSocketProxy, so maybeStore
+ // stores it only tagged with the generation that served it.
+ genManaged bool
+ maxBody int
+ status int
+ body []byte
+ truncated bool
+ wroteHdr bool
}
func (r *recordingResponseWriter) WriteHeader(status int) {
@@ -2967,22 +3021,7 @@ func backhaulProxy(host, eip, box, auth string, tr http.RoundTripper, logThrottl
// in-flight request is never 502'd mid-deploy; a sustained-down backend hangs
// this long before 502 (rare: systemd Restart=always makes "down" transient).
func dialUnixWithRetry(ctx context.Context, d *net.Dialer, socketPath string) (net.Conn, error) {
- const window = 13 * time.Second
- deadline := time.Now().Add(window)
- for {
- conn, err := unixdial.Checked(ctx, d, socketPath)
- if err == nil {
- return conn, nil
- }
- if ctx.Err() != nil || !isRetryableDialErr(err) || time.Now().After(deadline) {
- return nil, err
- }
- select {
- case <-ctx.Done():
- return nil, ctx.Err()
- case <-time.After(100 * time.Millisecond):
- }
- }
+ return dialUnixRetrying(ctx, d, func() (string, bool) { return socketPath, true })
}
func isRetryableDialErr(err error) bool {
diff --git a/internal/edge/gencache_test.go b/internal/edge/gencache_test.go
new file mode 100644
index 000000000..b71d03a4c
--- /dev/null
+++ b/internal/edge/gencache_test.go
@@ -0,0 +1,187 @@
+package edge
+
+import (
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// The guard at the cache's own level: a tagged entry is live only while its
+// generation is its route's newest, and a stale one is refused at Put.
+func TestCacheGenerationGuard(t *testing.T) {
+ c := NewCache(DefaultCacheMaxEntries, DefaultCacheMaxBodyBytes)
+ cur := uint64(1)
+ c.SetGenerations(func(route string) (uint64, bool) { return cur, route == "inst.example.com" })
+ key := CacheKey{Host: "www.customer.example", Method: "GET", Path: "/"}
+ tagged := func(n uint64) *CachedResponse {
+ return &CachedResponse{StatusCode: 200, Body: []byte("x"), StoredAt: time.Now(), TTL: time.Minute, Route: "inst.example.com", Gen: n}
+ }
+ c.Put(key, "", tagged(1))
+ if _, ok := c.get(key); !ok {
+ t.Fatal("an entry tagged with the current generation missed")
+ }
+ cur = 2
+ if _, ok := c.get(key); ok {
+ t.Fatal("an entry from generation 1 hit after generation 2 became current")
+ }
+ other := CacheKey{Host: "inst.example.com", Method: "GET", Path: "/late"}
+ c.Put(other, "", tagged(1))
+ if len(c.byURL[other.urlKey()]) != 0 {
+ t.Fatal("Put stored a response from a generation that is no longer current")
+ }
+ c.Put(other, "", &CachedResponse{StatusCode: 200, Body: []byte("y"), StoredAt: time.Now(), TTL: time.Minute})
+ if _, ok := c.get(other); !ok {
+ t.Fatal("an untagged entry was refused; the guard must leave unmanaged routes alone")
+ }
+ unarmed := NewCache(DefaultCacheMaxEntries, DefaultCacheMaxBodyBytes)
+ unarmed.Put(key, "", tagged(2))
+ if _, ok := unarmed.get(key); ok {
+ t.Fatal("a tagged entry was served by a cache with no generation guard")
+ }
+}
+
+// genCacheFixture is genFixture's managed instance reached under four names
+// (its own host, a custom-domain alias, a wildcard alias, and a path-routed
+// front door), with the cache on and a second generation's app ready.
+func genCacheFixture(t *testing.T) (e *Edge, act *genActivator, a, b *genApp) {
+ t.Helper()
+ dir := genSockDir(t)
+ a = startGenApp(t, filepath.Join(dir, "a.sock"), body("A", "max-age=60"))
+ b = startGenApp(t, filepath.Join(dir, "b.sock"), body("B", "max-age=60"))
+ front := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { io.WriteString(w, "front") }))
+ t.Cleanup(front.Close)
+ reg := openReg(t)
+ for _, rt := range []registry.Route{
+ {Host: "inst.example.com", Kind: "instance", Hibernate: true, Socket: filepath.Join(dir, "canon.sock"), Status: "ready", Version: "v0"},
+ {Host: "front.example", Kind: "service", Addr: front.Listener.Addr().String(), Status: "ready"},
+ } {
+ if err := reg.Upsert(rt); err != nil {
+ t.Fatal(err)
+ }
+ }
+ upsertAlias(t, reg, "www.customer.example", "inst.example.com")
+ upsertAlias(t, reg, "*.wild.example", "inst.example.com")
+ upsertPath(t, reg, "front.example", "acct", "inst.example.com")
+ act = &genActivator{fakeGens: newFakeGens(true)}
+ act.switchTo("inst.example.com", GenInfo{Socket: a.sock, N: 1, Build: "v1"})
+ var err error
+ e, err = New(reg, act)
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ e.EnableCache(NewCache(DefaultCacheMaxEntries, DefaultCacheMaxBodyBytes))
+ return e, act, a, b
+}
+
+// Spec §12 "Cache aliases": a custom domain, a wildcard and a path-routed
+// request each miss after the switch, without anything listing the aliases.
+func TestCachedAliasesMissAfterTheSwitch(t *testing.T) {
+ e, act, _, b := genCacheFixture(t)
+ urls := []string{
+ "http://inst.example.com/p",
+ "http://www.customer.example/p",
+ "http://x.wild.example/p",
+ "http://front.example/acct/p",
+ }
+ for _, u := range urls {
+ if got := serve(t, e, u).Body.String(); got != "A" {
+ t.Fatalf("GET %s = %q, want A from generation 1", u, got)
+ }
+ }
+ before := act.served.Load()
+ for _, u := range urls {
+ if got := serve(t, e, u).Body.String(); got != "A" {
+ t.Fatalf("repeat GET %s = %q, want the cached A", u, got)
+ }
+ }
+ if act.served.Load() != before {
+ t.Fatal("repeat requests reached the activator; they should have been cache hits")
+ }
+ act.switchTo("inst.example.com", GenInfo{Socket: b.sock, N: 2, Build: "v2"})
+ for _, u := range urls {
+ rec := serve(t, e, u)
+ if rec.Body.String() != "B" || rec.Header().Get("X-Carlos-Version") != "v2" {
+ t.Fatalf("GET %s after the switch = %q (version %q), want B from v2", u, rec.Body.String(), rec.Header().Get("X-Carlos-Version"))
+ }
+ }
+}
+
+// A response that finishes on the old generation after the switch is
+// returned to its client and never stored (spec §7.2 insertion rule).
+func TestLateResponseFromARetiredGenerationIsNotCached(t *testing.T) {
+ dir := genSockDir(t)
+ entered, gate := make(chan struct{}), make(chan struct{})
+ a := startGenApp(t, filepath.Join(dir, "a.sock"), http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ close(entered)
+ <-gate
+ w.Header().Set("Cache-Control", "max-age=60")
+ io.WriteString(w, "A")
+ }))
+ b := startGenApp(t, filepath.Join(dir, "b.sock"), body("B", "max-age=60"))
+ reg := openReg(t)
+ if err := reg.Upsert(registry.Route{Host: "inst.example.com", Kind: "instance", Hibernate: true,
+ Socket: filepath.Join(dir, "canon.sock"), Status: "ready"}); err != nil {
+ t.Fatal(err)
+ }
+ act := &genActivator{fakeGens: newFakeGens(true)}
+ act.switchTo("inst.example.com", GenInfo{Socket: a.sock, N: 1, Build: "v1"})
+ e, err := New(reg, act)
+ if err != nil {
+ t.Fatal(err)
+ }
+ e.EnableCache(NewCache(DefaultCacheMaxEntries, DefaultCacheMaxBodyBytes))
+
+ done := make(chan string)
+ go func() { done <- serve(t, e, "http://inst.example.com/slow").Body.String() }()
+ <-entered
+ act.switchTo("inst.example.com", GenInfo{Socket: b.sock, N: 2, Build: "v2"})
+ close(gate)
+ if got := <-done; got != "A" {
+ t.Fatalf("in-flight request got %q, want its own A", got)
+ }
+ if n := len(e.cache.byURL[cacheURLKey{Host: "inst.example.com", Method: "GET", Path: "/slow"}]); n != 0 {
+ t.Fatalf("%d entries stored for a response that finished after the switch, want 0", n)
+ }
+ if got := serve(t, e, "http://inst.example.com/slow").Body.String(); got != "B" {
+ t.Fatalf("next request = %q, want B", got)
+ }
+}
+
+// Cache-before-wake survives generations: a sleeping instance's cached page
+// still answers without a wake, because the entry's generation is still the
+// route's newest.
+func TestSleepingInstanceStillAnswersFromTheCache(t *testing.T) {
+ e, act, a, _ := genCacheFixture(t)
+ if got := serve(t, e, "http://inst.example.com/p").Body.String(); got != "A" {
+ t.Fatalf("first GET = %q, want A", got)
+ }
+ act.sleep("inst.example.com")
+ a.srv.Close()
+ before := act.served.Load()
+ if got := serve(t, e, "http://inst.example.com/p").Body.String(); got != "A" {
+ t.Fatalf("GET while asleep = %q, want the cached A", got)
+ }
+ if act.served.Load() != before {
+ t.Fatal("a cached page woke the instance")
+ }
+}
+
+// Flag off: entries are untagged and nothing about generations touches them.
+func TestGenerationsOffCachesUntagged(t *testing.T) {
+ e, act, _ := genFixture(t, false, true)
+ e.EnableCache(NewCache(DefaultCacheMaxEntries, DefaultCacheMaxBodyBytes))
+ e.cache.Put(CacheKey{Host: "inst.example.com", Method: "GET", Path: "/p"}, "", &CachedResponse{
+ StatusCode: 200, Body: []byte("cached"), StoredAt: time.Now(), TTL: time.Minute})
+ act.switchTo("inst.example.com", GenInfo{Socket: "/nonexistent.sock", N: 9, Build: "v9"})
+ if got := serve(t, e, "http://inst.example.com/p").Body.String(); got != "cached" {
+ t.Fatalf("GET = %q, want the untagged cached body", got)
+ }
+ if e.cache.genOf != nil {
+ t.Fatal("the generation guard was armed with the switch off")
+ }
+}
diff --git a/internal/edge/genedge_test.go b/internal/edge/genedge_test.go
new file mode 100644
index 000000000..4f28d8b70
--- /dev/null
+++ b/internal/edge/genedge_test.go
@@ -0,0 +1,98 @@
+package edge
+
+import (
+ "net/http"
+ "path/filepath"
+ "sync/atomic"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// genActivator is an Activator that is also a GenSource: an instant wake
+// that proxies straight through, over a hand-driven fakeGens. served counts
+// the requests that reached it, so a test can tell a cache hit (none) from
+// a proxied request.
+type genActivator struct {
+ *fakeGens
+ served atomic.Int64
+}
+
+func (a *genActivator) ServeHTTP(w http.ResponseWriter, r *http.Request, _ registry.Route, proxy http.Handler) bool {
+ a.served.Add(1)
+ proxy.ServeHTTP(w, r)
+ return true
+}
+
+func (a *genActivator) RetireExcept(map[string]bool) {}
+
+// genFixture is a registry with one activator-managed instance whose
+// CANONICAL socket is served by an app answering "canonical", and a
+// generation-1 app answering "A". Which one a request reaches says which
+// path the edge took.
+func genFixture(t *testing.T, on bool, hibernate bool) (*Edge, *genActivator, *genApp) {
+ t.Helper()
+ dir := genSockDir(t)
+ startGenApp(t, filepath.Join(dir, "canon.sock"), body("canonical", ""))
+ a := startGenApp(t, filepath.Join(dir, "a.sock"), body("A", ""))
+ reg := openReg(t)
+ if err := reg.Upsert(registry.Route{Host: "inst.example.com", Kind: "instance", Hibernate: hibernate,
+ Socket: filepath.Join(dir, "canon.sock"), Status: "ready", Version: "v0"}); err != nil {
+ t.Fatal(err)
+ }
+ act := &genActivator{fakeGens: newFakeGens(on)}
+ act.switchTo("inst.example.com", GenInfo{Socket: a.sock, N: 1, Build: "v1"})
+ e, err := New(reg, act)
+ if err != nil {
+ t.Fatalf("New: %v", err)
+ }
+ return e, act, a
+}
+
+func TestManagedRouteDialsTheCurrentGeneration(t *testing.T) {
+ e, act, _ := genFixture(t, true, true)
+ rec := serve(t, e, "http://inst.example.com/")
+ if rec.Body.String() != "A" {
+ t.Fatalf("body %q, want A: the edge dialled the canonical socket, not the current generation", rec.Body.String())
+ }
+ if got := rec.Header().Get("X-Carlos-Version"); got != "v1" {
+ t.Fatalf("X-Carlos-Version = %q, want v1, the serving generation's build, not the row's v0", got)
+ }
+ if !e.current()["inst.example.com"].genManaged {
+ t.Fatal("the managed route's upstream is not marked genManaged")
+ }
+ if len(act.retired) != 1 {
+ t.Fatalf("New registered %d retire hooks, want 1 (the edge's pools)", len(act.retired))
+ }
+}
+
+// Flag off: byte for byte the old edge. unixProxy on the canonical socket,
+// the row's version, no retire hook, no pools.
+func TestGenerationsOffKeepsTheCanonicalProxyAndTheRowVersion(t *testing.T) {
+ e, act, _ := genFixture(t, false, true)
+ rec := serve(t, e, "http://inst.example.com/")
+ if rec.Body.String() != "canonical" || rec.Header().Get("X-Carlos-Version") != "v0" {
+ t.Fatalf("got %q version %q, want canonical v0", rec.Body.String(), rec.Header().Get("X-Carlos-Version"))
+ }
+ if e.gens != nil || e.genPools != nil || len(act.retired) != 0 {
+ t.Fatal("generations were wired with the switch off")
+ }
+ up := e.current()["inst.example.com"]
+ if up.genManaged {
+ t.Fatal("upstream marked genManaged with the switch off")
+ }
+}
+
+// An unflagged socket route is an externally supervised process: canonical,
+// whatever the switch says.
+func TestUnmanagedSocketRouteStaysCanonicalWithGenerationsOn(t *testing.T) {
+ e, _, _ := genFixture(t, true, false)
+ rec := serve(t, e, "http://inst.example.com/")
+ if rec.Body.String() != "canonical" || rec.Header().Get("X-Carlos-Version") != "v0" {
+ t.Fatalf("got %q version %q, want canonical v0", rec.Body.String(), rec.Header().Get("X-Carlos-Version"))
+ }
+ up := e.current()["inst.example.com"]
+ if up.genManaged {
+ t.Fatal("an unflagged route was marked genManaged")
+ }
+}
diff --git a/internal/edge/genproxy.go b/internal/edge/genproxy.go
new file mode 100644
index 000000000..f876386b2
--- /dev/null
+++ b/internal/edge/genproxy.go
@@ -0,0 +1,389 @@
+package edge
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "io"
+ "net"
+ "net/http"
+ "net/http/httputil"
+ "sync"
+ "syscall"
+ "time"
+
+ "github.com/carlosframework/platform/internal/unixdial"
+)
+
+// genproxy.go is the edge half of generations (zero-downtime spec §7.2): a
+// reverse proxy for an activator-managed route whose app socket moves from
+// one generation to the next. Every request asks the activator for the
+// current generation and uses that generation's pool, whose connections all
+// reach that generation's socket (dialGen), so the response is attributed to
+// the generation that served it. A connection to generation n is never
+// reused for n+1, and a generation's idle connections are closed when it
+// stops being current.
+//
+// Used only when the activator's GenerationsOn is true. With it false the
+// edge builds unixProxy(t.Socket) exactly as before.
+
+// errNoGeneration is a dial's answer while the route has no current
+// generation (between a stop and the next wake). It wraps ENOENT on purpose:
+// to dialUnixRetrying it is the same "socket briefly absent" gap a restart
+// always had, retried inside the same window, and each retry asks again.
+var errNoGeneration = fmt.Errorf("edge: instance has no current generation: %w", syscall.ENOENT)
+
+// dialUnixRetrying is dialUnixWithRetry with the socket resolved per attempt:
+// resolve is asked before the first dial and again before every retry, so a
+// dial that races a switch lands on the generation that is current by then.
+func dialUnixRetrying(ctx context.Context, d *net.Dialer, resolve func() (string, bool)) (net.Conn, error) {
+ const window = 13 * time.Second
+ deadline := time.Now().Add(window)
+ for {
+ var (
+ conn net.Conn
+ err error
+ )
+ if sock, ok := resolve(); ok {
+ conn, err = unixdial.Checked(ctx, d, sock)
+ } else {
+ err = errNoGeneration
+ }
+ if err == nil {
+ return conn, nil
+ }
+ if ctx.Err() != nil || !isRetryableDialErr(err) || time.Now().After(deadline) {
+ return nil, err
+ }
+ select {
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ case <-time.After(100 * time.Millisecond):
+ }
+ }
+}
+
+// genMovedError is a generation pool's dial giving up: its own socket is
+// gone and another generation is current. Nothing was sent, so the request
+// can go again on the current generation's pool.
+type genMovedError struct{ to GenInfo }
+
+func (e *genMovedError) Error() string {
+ return fmt.Sprintf("edge: generation moved to %d while dialling the one before it", e.to.N)
+}
+
+// dialGen is a generation pool's dial: own's socket and nothing else, and
+// a connection proven to be own's (a shared foreign path is checked
+// against the current generation once connected), so every connection in
+// the pool is own's and a response through it is attributed to own exactly
+// (spec §7.2). The retiring generation still
+// accepts until its SIGTERM, which comes after the switch, so a request
+// that chose own's pool an instant before the switch is served there. A
+// retryable failure is retried inside the usual window while own is still
+// current or no generation is; once another one is current, own is gone for
+// good and the dial returns genMovedError.
+func dialGen(ctx context.Context, d *net.Dialer, src GenSource, host string, own GenInfo) (net.Conn, error) {
+ const window = 13 * time.Second
+ deadline := time.Now().Add(window)
+ for {
+ conn, err := unixdial.Checked(ctx, d, own.Socket)
+ if err == nil {
+ // Spec §7.2 (rev 17): if the generation stops being current
+ // before the dial completes, the request is retried against the
+ // new current generation. So every new connection is checked
+ // once connected: own must still be current. This also covers a
+ // foreign row, whose generations share one canonical path, so a
+ // connection on it could have reached the replacement. A
+ // connection already made (an idle one in own's pool) is not
+ // re-checked: the request that chose own's pool sampled own.
+ cur, ok := src.ServingGen(host)
+ if ok && cur.N == own.N {
+ return conn, nil
+ }
+ conn.Close()
+ if ok {
+ return nil, &genMovedError{to: cur}
+ }
+ err = errNoGeneration
+ }
+ if ctx.Err() != nil || !isRetryableDialErr(err) || time.Now().After(deadline) {
+ return nil, err
+ }
+ if cur, ok := src.ServingGen(host); ok && (cur.N != own.N || cur.Socket != own.Socket) {
+ return nil, &genMovedError{to: cur}
+ }
+ select {
+ case <-ctx.Done():
+ return nil, ctx.Err()
+ case <-time.After(100 * time.Millisecond):
+ }
+ }
+}
+
+// genPool is one generation's transport and the generation it reaches.
+type genPool struct {
+ tr *http.Transport
+ gen GenInfo
+}
+
+// GenPools is the per-socket-path set of transports, one idle pool per
+// generation socket. One set per consumer (the edge has one, scheduled
+// delivery has its own) and shared across route reloads, because the pool
+// belongs to the generation, not to the table build that first reached it.
+type GenPools struct {
+ mu sync.Mutex
+ m map[string]*genPool
+}
+
+// NewGenPools returns an empty set.
+func NewGenPools() *GenPools { return &GenPools{m: map[string]*genPool{}} }
+
+// pool is generation g's pool, made on first use, or false when g is no
+// longer the host's current generation. Its dial reaches g's own socket only
+// (dialGen). A foreign row reuses its canonical path for every generation
+// (plan 1's reading), so a pool found for the path under another generation
+// number is replaced, and its idle connections closed.
+//
+// g was sampled by the caller some time ago. The check against src happens
+// under p.mu, and Retire takes the same lock, so a pool is published only for
+// a generation that was still current at that instant: if a switch has
+// retired g since, either Retire has already run (and this refuses to publish
+// a pool nothing would ever remove) or it runs after the publish and removes
+// it. The caller samples again on false.
+//
+// Lock order: p.mu, then whatever lock src.ServingGen takes (the activator's
+// own). That is safe because Retire runs from OnGenRetired hooks, which the
+// activator calls outside every lock of its own, and ServingGen never calls
+// into this set. A GenSource whose ServingGen reached Retire would deadlock.
+func (p *GenPools) pool(host string, g GenInfo, src GenSource) (*genPool, bool) {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ if cur, ok := src.ServingGen(host); !ok || cur.N != g.N {
+ return nil, false
+ }
+ if gp, ok := p.m[g.Socket]; ok {
+ if gp.gen.N == g.N {
+ return gp, true
+ }
+ gp.tr.CloseIdleConnections()
+ }
+ gp := &genPool{tr: genTransportFor(host, src, &g), gen: g}
+ p.m[g.Socket] = gp
+ return gp, true
+}
+
+// Retire closes socket's idle connections and forgets its pool. It has the
+// shape of GenSource.OnGenRetired's hook. A request still in flight on a
+// retired pool finishes; its connection is not reused by anyone after.
+func (p *GenPools) Retire(_, socket string) {
+ p.mu.Lock()
+ gp, ok := p.m[socket]
+ delete(p.m, socket)
+ p.mu.Unlock()
+ if ok {
+ gp.tr.CloseIdleConnections()
+ }
+}
+
+// Len is the number of live pools.
+func (p *GenPools) Len() int {
+ p.mu.Lock()
+ defer p.mu.Unlock()
+ return len(p.m)
+}
+
+// genTransportFor builds the transport behind one pool: unixProxy's idle
+// bounds, and a dial that reaches own's socket only (dialGen). RoundTrip
+// always passes own: a request that finds no current generation waits for
+// one (waitServingGen) and then uses that generation's pool. With own nil
+// it is a one-shot transport whose dial resolves the socket per attempt.
+func genTransportFor(host string, src GenSource, own *GenInfo) *http.Transport {
+ dialer := &net.Dialer{}
+ dial := func(ctx context.Context, _, _ string) (net.Conn, error) {
+ return dialUnixRetrying(ctx, dialer, func() (string, bool) { return src.AppSocket(host) })
+ }
+ if own != nil {
+ g := *own
+ dial = func(ctx context.Context, _, _ string) (net.Conn, error) {
+ return dialGen(ctx, dialer, src, host, g)
+ }
+ }
+ return &http.Transport{
+ DialContext: dial,
+ MaxIdleConns: 32,
+ IdleConnTimeout: 90 * time.Second,
+ DisableKeepAlives: own == nil,
+ }
+}
+
+// genRoundTripper picks the pool of host's current generation for each
+// request, stamps the response with the build of the generation whose
+// connection carried it, and records that generation on the request's
+// outcome for the cache (spec §7.2). A pool's connections are all its own
+// generation's (dialGen), so the pool's generation is that generation.
+type genRoundTripper struct {
+ host string
+ src GenSource
+ pools *GenPools
+}
+
+// genRetries bounds how often one request is sent again after its
+// generation moved under it.
+const genRetries = 2
+
+// genStaleRetries bounds how often one request goes back to choosing a
+// generation because the one it chose was no longer current when its pool was
+// asked for. That is not a dial attempt, so it does not use genRetries, but a
+// source that keeps switching must not hold the request forever.
+const genStaleRetries = 3
+
+func (t *genRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+ // The generation is chosen, and any switch retried, before a byte of the
+ // body is consumed (spec §7.2, rev 17). The transport gets the body
+ // behind heldBody: it reads it only once it is writing the request on a
+ // connection dialGen has proven to be the chosen generation's, and a
+ // Close it makes after a failed dial is held. So a dial that finds the
+ // generation moved leaves the body untouched, and the request goes again
+ // on the current generation's pool, a streaming POST with no GetBody
+ // included.
+ var hb *heldBody
+ if req.Body != nil && req.Body != http.NoBody {
+ hb = &heldBody{rc: req.Body}
+ req.Body = hb
+ }
+ for attempt, stale := 0, 0; ; {
+ g, ok := t.src.ServingGen(t.host)
+ if !ok {
+ // No current generation (between a stop and the next wake): wait
+ // for one inside the dial's retry window, then take the normal
+ // path, so the response is attributed to the generation that
+ // serves it and its connection passes dialGen's check (spec §7.2).
+ g, ok = waitServingGen(req.Context(), t.src, t.host)
+ if !ok {
+ if hb != nil {
+ hb.release()
+ }
+ return nil, errNoGeneration
+ }
+ }
+ gp, ok := t.pools.pool(t.host, g, t.src)
+ if !ok {
+ // g was retired between the choice and the pool: choose again.
+ if stale++; stale > genStaleRetries {
+ if hb != nil {
+ hb.release()
+ }
+ if cur, ok := t.src.ServingGen(t.host); ok {
+ return nil, &genMovedError{to: cur}
+ }
+ return nil, errNoGeneration
+ }
+ continue
+ }
+ resp, err := gp.tr.RoundTrip(req)
+ var moved *genMovedError
+ if errors.As(err, &moved) && attempt < genRetries && (hb == nil || !hb.wasRead()) {
+ // The generation this request chose stopped being current before
+ // its dial completed. Nothing was sent and nothing of the body
+ // was read: send it again, on the current one.
+ attempt++
+ continue
+ }
+ if err != nil {
+ if hb != nil {
+ hb.release()
+ }
+ return nil, err
+ }
+ // X-Carlos-Version names the build of the generation that served the
+ // request, not the registry's Version (spec §7.2). Set, not Add: the
+ // platform's voice on this header is the only one.
+ if gp.gen.Build != "" {
+ resp.Header.Set("X-Carlos-Version", gp.gen.Build)
+ }
+ noteServedGen(req, t.host, gp.gen.N)
+ return resp, nil
+ }
+}
+
+// waitServingGen waits, within the dial's 13 s retry window and ctx, for
+// host to have a current generation, asking every 100 ms.
+func waitServingGen(ctx context.Context, src GenSource, host string) (GenInfo, bool) {
+ deadline := time.Now().Add(13 * time.Second)
+ for {
+ if g, ok := src.ServingGen(host); ok {
+ return g, true
+ }
+ if time.Now().After(deadline) {
+ return GenInfo{}, false
+ }
+ select {
+ case <-ctx.Done():
+ return GenInfo{}, false
+ case <-time.After(100 * time.Millisecond):
+ }
+ }
+}
+
+// heldBody is a request body the transport cannot consume or close before
+// the request is on a connection. The transport reads a body only while it
+// writes the request on a dialled connection, and closes it on any error,
+// including a dial that never connected. Until the first Read, that Close
+// is held, so the same body can go again on another generation's pool.
+// After it, Close passes through as usual.
+type heldBody struct {
+ rc io.ReadCloser
+ mu sync.Mutex
+ read bool
+}
+
+func (b *heldBody) Read(p []byte) (int, error) {
+ b.mu.Lock()
+ b.read = true
+ b.mu.Unlock()
+ return b.rc.Read(p)
+}
+
+func (b *heldBody) Close() error {
+ b.mu.Lock()
+ read := b.read
+ b.mu.Unlock()
+ if !read {
+ return nil // a failed dial's close: the body may still be sent
+ }
+ return b.rc.Close()
+}
+
+func (b *heldBody) wasRead() bool {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+ return b.read
+}
+
+// release closes the body for good: the request has finished with an error.
+func (b *heldBody) release() { _ = b.rc.Close() }
+
+// appSocketProxy is unixProxy for an activator-managed route with
+// generations on: the same Rewrite, Host preservation, upgrade pass-through
+// and error handler, over genRoundTripper instead of one fixed socket.
+func appSocketProxy(host string, src GenSource, pools *GenPools) *httputil.ReverseProxy {
+ return &httputil.ReverseProxy{
+ Rewrite: func(pr *httputil.ProxyRequest) {
+ setForwarded(pr)
+ pr.Out.URL.Scheme = "http"
+ pr.Out.URL.Host = "carlos" // placeholder; the unix dialer ignores it
+ pr.Out.Host = pr.In.Host // backend sees the real hostname
+ scrubInboundPurgeHeader(pr)
+ },
+ Transport: &genRoundTripper{host: host, src: src, pools: pools},
+ ErrorHandler: defaultProxyError(proxyCause),
+ }
+}
+
+// AppSocketProxy is appSocketProxy for callers outside this package: the
+// scheduled-work deliverer in internal/host, which must reach the current
+// generation the same way the edge does (spec §7.1) without the edge's
+// cache, compression or usage layers.
+func AppSocketProxy(host string, src GenSource, pools *GenPools) http.Handler {
+ return appSocketProxy(host, src, pools)
+}
diff --git a/internal/edge/genproxy_test.go b/internal/edge/genproxy_test.go
new file mode 100644
index 000000000..35ec93dbf
--- /dev/null
+++ b/internal/edge/genproxy_test.go
@@ -0,0 +1,508 @@
+package edge
+
+import (
+ "context"
+ "errors"
+ "io"
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "syscall"
+ "testing"
+ "time"
+)
+
+// fakeGens is a GenSource a test drives by hand: switchTo is plan 3's
+// switch as the edge sees it, sleep is a stop.
+type fakeGens struct {
+ mu sync.Mutex
+ on bool
+ cur map[string]GenInfo // host -> current generation; absent = no current generation
+ last map[string]uint64 // host -> CacheGen
+ asks int // AppSocket calls
+ retired []func(host, socket string)
+ // onServing, when set, runs after every ServingGen answer, with no lock
+ // held: a test's way to land a switch between the pool choice and the
+ // dial.
+ onServing func()
+}
+
+func newFakeGens(on bool) *fakeGens {
+ return &fakeGens{on: on, cur: map[string]GenInfo{}, last: map[string]uint64{}}
+}
+
+func (f *fakeGens) GenerationsOn() bool { return f.on }
+
+func (f *fakeGens) AppSocket(host string) (string, bool) {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.asks++
+ g, ok := f.cur[host]
+ return g.Socket, ok
+}
+
+func (f *fakeGens) ServingGen(host string) (GenInfo, bool) {
+ f.mu.Lock()
+ g, ok := f.cur[host]
+ hook := f.onServing
+ f.mu.Unlock()
+ if hook != nil {
+ hook()
+ }
+ return g, ok
+}
+
+func (f *fakeGens) CacheGen(host string) (uint64, bool) {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ n, ok := f.last[host]
+ return n, ok
+}
+
+func (f *fakeGens) OnGenRetired(fn func(host, socket string)) {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ f.retired = append(f.retired, fn)
+}
+
+// switchTo makes g host's current generation, then tells the retire hooks
+// about the one it replaced.
+func (f *fakeGens) switchTo(host string, g GenInfo) {
+ f.mu.Lock()
+ old, had := f.cur[host]
+ f.cur[host], f.last[host] = g, g.N
+ hooks := append([]func(string, string){}, f.retired...)
+ f.mu.Unlock()
+ if had {
+ for _, fn := range hooks {
+ fn(host, old.Socket)
+ }
+ }
+}
+
+// sleep takes host's current generation away and keeps its number for the
+// cache, as the activator does at a stop.
+func (f *fakeGens) sleep(host string) {
+ f.mu.Lock()
+ old, had := f.cur[host]
+ delete(f.cur, host)
+ hooks := append([]func(string, string){}, f.retired...)
+ f.mu.Unlock()
+ if had {
+ for _, fn := range hooks {
+ fn(host, old.Socket)
+ }
+ }
+}
+
+func (f *fakeGens) askCount() int {
+ f.mu.Lock()
+ defer f.mu.Unlock()
+ return f.asks
+}
+
+// genSockDir is a short directory for generation sockets. Run with
+// TMPDIR=/tmp/k: a socket path must stay under sockaddr_un's 108 bytes.
+func genSockDir(t *testing.T) string {
+ t.Helper()
+ dir, err := os.MkdirTemp("", "e")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { os.RemoveAll(dir) })
+ return dir
+}
+
+// genApp is one generation's process: an HTTP server on a unix socket that
+// counts the connections it opened and closed.
+type genApp struct {
+ sock string
+ ln net.Listener
+ opened, closed atomic.Int64
+ srv *http.Server
+}
+
+func startGenApp(t *testing.T, sock string, h http.Handler) *genApp {
+ t.Helper()
+ ln, err := net.Listen("unix", sock)
+ if err != nil {
+ if errors.Is(err, syscall.EPERM) || strings.Contains(err.Error(), "operation not permitted") {
+ t.Skipf("unix-socket listeners not permitted in this environment: %v", err)
+ }
+ t.Fatalf("listen %s: %v", sock, err)
+ }
+ a := &genApp{sock: sock, ln: ln}
+ a.srv = &http.Server{Handler: h, ConnState: func(_ net.Conn, st http.ConnState) {
+ switch st {
+ case http.StateNew:
+ a.opened.Add(1)
+ case http.StateClosed, http.StateHijacked:
+ a.closed.Add(1)
+ }
+ }}
+ go a.srv.Serve(ln)
+ t.Cleanup(func() { a.srv.Close() })
+ return a
+}
+
+// stop is the generation's process exiting: the server and its listener close now,
+// so the socket file is unlinked before stop returns (srv.Close alone misses a
+// listener Serve has not registered yet, and Serve's late close would unlink the
+// path by name, possibly a replacement's socket).
+func (a *genApp) stop() { a.srv.Close(); a.ln.Close() }
+
+// body answers b with cache-control cc ("" for none).
+func body(b, cc string) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ if cc != "" {
+ w.Header().Set("Cache-Control", cc)
+ }
+ io.WriteString(w, b)
+ })
+}
+
+func serve(t *testing.T, h http.Handler, url string) *httptest.ResponseRecorder {
+ t.Helper()
+ rec := httptest.NewRecorder()
+ h.ServeHTTP(rec, httptest.NewRequest("GET", url, nil))
+ return rec
+}
+
+// Each retry asks the source again: a dial that races a switch reaches the
+// generation that is current by the time it retries.
+func TestDialUnixRetryingAsksAgainOnEveryRetry(t *testing.T) {
+ dir := genSockDir(t)
+ live := startGenApp(t, filepath.Join(dir, "2.sock"), body("B", ""))
+ var asks atomic.Int64
+ resolve := func() (string, bool) {
+ switch asks.Add(1) {
+ case 1:
+ return "", false // between generations: no current one yet
+ case 2:
+ return filepath.Join(dir, "1.sock"), true // a retired generation: its file is gone
+ default:
+ return live.sock, true
+ }
+ }
+ conn, err := dialUnixRetrying(context.Background(), &net.Dialer{}, resolve)
+ if err != nil {
+ t.Fatalf("dial: %v", err)
+ }
+ conn.Close()
+ if got := asks.Load(); got != 3 {
+ t.Fatalf("resolve asked %d times, want 3 (once per attempt)", got)
+ }
+}
+
+// A connection to generation n is never reused for n+1, and n's idle
+// connections close when it stops being current. The version header names
+// the build of the generation that served each request.
+func TestAppSocketProxyFollowsTheCurrentGeneration(t *testing.T) {
+ dir := genSockDir(t)
+ a := startGenApp(t, filepath.Join(dir, "a.sock"), body("A", ""))
+ b := startGenApp(t, filepath.Join(dir, "b.sock"), body("B", ""))
+ src := newFakeGens(true)
+ pools := NewGenPools()
+ src.OnGenRetired(pools.Retire)
+ const host = "inst.example.com"
+ src.switchTo(host, GenInfo{Socket: a.sock, N: 1, Build: "v1"})
+ p := appSocketProxy(host, src, pools)
+
+ for i := 0; i < 2; i++ {
+ rec := serve(t, p, "http://inst.example.com/")
+ if rec.Body.String() != "A" || rec.Header().Get("X-Carlos-Version") != "v1" {
+ t.Fatalf("request %d: %q version %q, want A from v1", i, rec.Body.String(), rec.Header().Get("X-Carlos-Version"))
+ }
+ }
+ if got := a.opened.Load(); got != 1 {
+ t.Fatalf("generation 1 saw %d connections, want 1 (keep-alive reused)", got)
+ }
+
+ src.switchTo(host, GenInfo{Socket: b.sock, N: 2, Build: "v2"})
+ rec := serve(t, p, "http://inst.example.com/")
+ if rec.Body.String() != "B" || rec.Header().Get("X-Carlos-Version") != "v2" {
+ t.Fatalf("after the switch: %q version %q, want B from v2", rec.Body.String(), rec.Header().Get("X-Carlos-Version"))
+ }
+ if got := a.opened.Load(); got != 1 {
+ t.Fatalf("generation 1 saw %d connections after the switch, want still 1", got)
+ }
+ deadline := time.Now().Add(2 * time.Second)
+ for a.closed.Load() != 1 && time.Now().Before(deadline) {
+ time.Sleep(10 * time.Millisecond)
+ }
+ if got := a.closed.Load(); got != 1 {
+ t.Fatalf("generation 1's idle connection was not closed at retirement (closed=%d)", got)
+ }
+ if got := pools.Len(); got != 1 {
+ t.Fatalf("%d pools live, want 1 (the retired one forgotten)", got)
+ }
+}
+
+// Spec §7.2: the version header and the cache's generation name the
+// generation whose connection carried the request. Here the switch lands
+// between the pool choice and the dial. With a pool whose dial re-resolves
+// AppSocket, the request reached B while it was stamped and tagged as A.
+func TestAppSocketProxyAttributesTheConnectionsGeneration(t *testing.T) {
+ const host = "inst.example.com"
+ // Rev 17: a switch that lands before the dial completes sends the request
+ // to the new current generation, even though the retiring one still
+ // accepts. Its response is attributed to the generation that served it.
+ t.Run("a switch before the dial completes", func(t *testing.T) {
+ dir := genSockDir(t)
+ a := startGenApp(t, filepath.Join(dir, "a.sock"), body("A", ""))
+ b := startGenApp(t, filepath.Join(dir, "b.sock"), body("B", ""))
+ src := newFakeGens(true)
+ src.switchTo(host, GenInfo{Socket: a.sock, N: 1, Build: "v1"})
+ var once sync.Once
+ src.onServing = func() {
+ once.Do(func() { src.switchTo(host, GenInfo{Socket: b.sock, N: 2, Build: "v2"}) })
+ }
+ p := appSocketProxy(host, src, NewGenPools())
+ r, _ := withOutcome(httptest.NewRequest("GET", "http://inst.example.com/", nil))
+ rec := httptest.NewRecorder()
+ p.ServeHTTP(rec, r)
+ _, n, _ := servedGen(r)
+ if rec.Body.String() != "B" || rec.Header().Get("X-Carlos-Version") != "v2" || n != 2 {
+ t.Fatalf("served %q, version %q, generation %d: want B, v2, 2 (retried on the current generation)",
+ rec.Body.String(), rec.Header().Get("X-Carlos-Version"), n)
+ }
+ if got := b.opened.Load(); got < 1 {
+ t.Fatal("generation 2 was never reached")
+ }
+ _ = a
+ })
+ // A proxied POST streams its body: no GetBody, unknown length. A switch
+ // before the dial completes must not turn it into a 502. The generation
+ // is settled before a byte of the body is read, so the body goes, whole,
+ // to the current generation. Retrying only replayable requests failed it.
+ t.Run("a streaming POST across a switch", func(t *testing.T) {
+ dir := genSockDir(t)
+ a := startGenApp(t, filepath.Join(dir, "a.sock"), body("A", ""))
+ echo := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ b, _ := io.ReadAll(r.Body)
+ io.WriteString(w, "B:"+string(b))
+ })
+ b := startGenApp(t, filepath.Join(dir, "b.sock"), echo)
+ src := newFakeGens(true)
+ src.switchTo(host, GenInfo{Socket: a.sock, N: 1, Build: "v1"})
+ var once sync.Once
+ src.onServing = func() {
+ once.Do(func() {
+ a.stop() // generation 1 has gone: its socket file with it
+ src.switchTo(host, GenInfo{Socket: b.sock, N: 2, Build: "v2"})
+ })
+ }
+ pr, pw := io.Pipe()
+ go func() {
+ io.WriteString(pw, "hello, ")
+ io.WriteString(pw, "world")
+ pw.Close()
+ }()
+ req := httptest.NewRequest("POST", "http://inst.example.com/form", pr)
+ req.ContentLength = -1
+ if req.GetBody != nil {
+ t.Fatal("fixture: the request must have no GetBody")
+ }
+ r, _ := withOutcome(req)
+ rec := httptest.NewRecorder()
+ appSocketProxy(host, src, NewGenPools()).ServeHTTP(rec, r)
+ if rec.Code != http.StatusOK || rec.Body.String() != "B:hello, world" || rec.Header().Get("X-Carlos-Version") != "v2" {
+ t.Fatalf("status %d, body %q, version %q: want 200, the whole body at generation 2, v2",
+ rec.Code, rec.Body.String(), rec.Header().Get("X-Carlos-Version"))
+ }
+ _ = b
+ })
+ t.Run("the sampled generation is gone", func(t *testing.T) {
+ dir := genSockDir(t)
+ a := startGenApp(t, filepath.Join(dir, "a.sock"), body("A", ""))
+ b := startGenApp(t, filepath.Join(dir, "b.sock"), body("B", ""))
+ src := newFakeGens(true)
+ src.switchTo(host, GenInfo{Socket: a.sock, N: 1, Build: "v1"})
+ var once sync.Once
+ src.onServing = func() {
+ once.Do(func() {
+ a.stop() // A has exited: its socket file is gone
+ src.switchTo(host, GenInfo{Socket: b.sock, N: 2, Build: "v2"})
+ })
+ }
+ p := appSocketProxy(host, src, NewGenPools())
+ r, _ := withOutcome(httptest.NewRequest("GET", "http://inst.example.com/", nil))
+ rec := httptest.NewRecorder()
+ p.ServeHTTP(rec, r)
+ _, n, _ := servedGen(r)
+ if rec.Body.String() != "B" || rec.Header().Get("X-Carlos-Version") != "v2" || n != 2 {
+ t.Fatalf("served %q, version %q, generation %d: want B, v2, 2 (sent again on the current pool)",
+ rec.Body.String(), rec.Header().Get("X-Carlos-Version"), n)
+ }
+ })
+ // A foreign row's generations share one path. Generation 1's pool, chosen
+ // an instant before the hold, dials that path after generation 2 has
+ // bound it: the connection reaches 2, and a dial that trusts the path
+ // stamps 2's response as 1 and keeps the connection in 1's pool.
+ t.Run("a foreign row's path now belongs to the replacement", func(t *testing.T) {
+ dir := genSockDir(t)
+ path := filepath.Join(dir, "canonical.sock")
+ a := startGenApp(t, path, body("A", ""))
+ src := newFakeGens(true)
+ src.switchTo(host, GenInfo{Socket: path, N: 1, Build: "v1"})
+ var once sync.Once
+ var b *genApp
+ src.onServing = func() {
+ once.Do(func() {
+ a.stop() // the hold stops 1; its socket file goes
+ b = startGenApp(t, path, body("B", ""))
+ src.switchTo(host, GenInfo{Socket: path, N: 2, Build: "v2"})
+ })
+ }
+ p := appSocketProxy(host, src, NewGenPools())
+ r, _ := withOutcome(httptest.NewRequest("GET", "http://inst.example.com/", nil))
+ rec := httptest.NewRecorder()
+ p.ServeHTTP(rec, r)
+ _, n, _ := servedGen(r)
+ if rec.Body.String() != "B" || rec.Header().Get("X-Carlos-Version") != "v2" || n != 2 {
+ t.Fatalf("served %q, version %q, generation %d: want B, v2, 2 (never attributed to the old generation)",
+ rec.Body.String(), rec.Header().Get("X-Carlos-Version"), n)
+ }
+ if b == nil || b.opened.Load() < 1 {
+ t.Fatal("the replacement was never reached")
+ }
+ })
+}
+
+// The generation that carried a request is recorded on its outcome, for the
+// cache to tag with.
+func TestAppSocketProxyNotesTheServedGeneration(t *testing.T) {
+ dir := genSockDir(t)
+ a := startGenApp(t, filepath.Join(dir, "a.sock"), body("A", ""))
+ src := newFakeGens(true)
+ src.switchTo("inst.example.com", GenInfo{Socket: a.sock, N: 7, Build: "v1"})
+ p := appSocketProxy("inst.example.com", src, NewGenPools())
+ r, _ := withOutcome(httptest.NewRequest("GET", "http://inst.example.com/", nil))
+ p.ServeHTTP(httptest.NewRecorder(), r)
+ if host, n, ok := servedGen(r); !ok || host != "inst.example.com" || n != 7 {
+ t.Fatalf("servedGen = %q %d %v, want inst.example.com 7", host, n, ok)
+ }
+}
+
+// With no current generation the request waits inside the dial's window for
+// one, then goes the normal way: through that generation's pool, stamped
+// with its build and tagged with its number (spec §7.2). An unbound one-shot
+// transport returned the response unattributed, with no platform version.
+func TestAppSocketProxyWaitsForAGeneration(t *testing.T) {
+ dir := genSockDir(t)
+ a := startGenApp(t, filepath.Join(dir, "a.sock"), body("A", ""))
+ src := newFakeGens(true)
+ pools := NewGenPools()
+ p := appSocketProxy("inst.example.com", src, pools)
+ go func() {
+ time.Sleep(300 * time.Millisecond)
+ src.switchTo("inst.example.com", GenInfo{Socket: a.sock, N: 3, Build: "v1"})
+ }()
+ r, _ := withOutcome(httptest.NewRequest("GET", "http://inst.example.com/", nil))
+ rec := httptest.NewRecorder()
+ p.ServeHTTP(rec, r)
+ if rec.Body.String() != "A" || rec.Header().Get("X-Carlos-Version") != "v1" {
+ t.Fatalf("body %q, version %q: want A from v1 once the generation appeared", rec.Body.String(), rec.Header().Get("X-Carlos-Version"))
+ }
+ if host, n, ok := servedGen(r); !ok || host != "inst.example.com" || n != 3 {
+ t.Fatalf("servedGen = %q %d %v, want generation 3", host, n, ok)
+ }
+ if pools.Len() != 1 {
+ t.Fatalf("%d pools, want generation 3's", pools.Len())
+ }
+}
+
+// A request that sampled generation n before a switch must not leave n's
+// pool behind when it resumes after Retire removed it: the pool is only
+// published for a generation that is still current. The switch lands at the
+// request's own sampling call, the instant between choosing n and asking for
+// its pool, and repeats over many generations; the set stays at most one pool
+// and every request is served by the generation current when it finished.
+func TestGenPoolsRetiredGenerationIsNotRecreated(t *testing.T) {
+ const (
+ host = "inst.example.com"
+ swaps = 20
+ )
+ dir := genSockDir(t)
+ apps := make([]*genApp, swaps+2)
+ for n := 1; n < len(apps); n++ {
+ apps[n] = startGenApp(t, filepath.Join(dir, strconv.Itoa(n)+".sock"), body("G"+strconv.Itoa(n), ""))
+ }
+ info := func(n int) GenInfo {
+ return GenInfo{Socket: apps[n].sock, N: uint64(n), Build: "v" + strconv.Itoa(n)}
+ }
+ src := newFakeGens(true)
+ pools := NewGenPools()
+ src.OnGenRetired(pools.Retire)
+ src.switchTo(host, info(1))
+ p := appSocketProxy(host, src, pools)
+
+ for n := 1; n <= swaps; n++ {
+ var once sync.Once
+ next := n + 1
+ src.onServing = func() { once.Do(func() { src.switchTo(host, info(next)) }) }
+ r, _ := withOutcome(httptest.NewRequest("GET", "http://inst.example.com/", nil))
+ rec := httptest.NewRecorder()
+ p.ServeHTTP(rec, r)
+ want := "G" + strconv.Itoa(next)
+ if rec.Body.String() != want || rec.Header().Get("X-Carlos-Version") != "v"+strconv.Itoa(next) {
+ t.Fatalf("switch %d: served %q version %q, want %s from v%d",
+ n, rec.Body.String(), rec.Header().Get("X-Carlos-Version"), want, next)
+ }
+ if _, got, _ := servedGen(r); got != uint64(next) {
+ t.Fatalf("switch %d: attributed to generation %d, want %d", n, got, next)
+ }
+ if got := pools.Len(); got > 1 {
+ t.Fatalf("switch %d: %d pools live, want at most 1 (a retired generation's pool was recreated)", n, got)
+ }
+ }
+}
+
+// flappingGens answers a different generation on every ServingGen call, so no
+// sampled generation is ever still current when its pool is asked for.
+type flappingGens struct {
+ *fakeGens
+ sock string
+ n atomic.Uint64
+}
+
+func (f *flappingGens) ServingGen(string) (GenInfo, bool) {
+ return GenInfo{Socket: f.sock, N: f.n.Add(1), Build: "v"}, true
+}
+
+type closeCounter struct {
+ io.Reader
+ closed atomic.Int64
+}
+
+func (c *closeCounter) Close() error { c.closed.Add(1); return nil }
+
+// A request whose sampled generation is refused again and again does not spin
+// and does not leave a pool behind: after a few refusals it gives up with the
+// generation-moved error, as exhausted retries do, and its body is released.
+func TestGenRoundTripperGivesUpWhenEveryGenerationIsStale(t *testing.T) {
+ dir := genSockDir(t)
+ a := startGenApp(t, filepath.Join(dir, "a.sock"), body("A", ""))
+ src := &flappingGens{fakeGens: newFakeGens(true), sock: a.sock}
+ pools := NewGenPools()
+ rt := &genRoundTripper{host: "inst.example.com", src: src, pools: pools}
+ b := &closeCounter{Reader: strings.NewReader("x")}
+ req := httptest.NewRequest("POST", "http://inst.example.com/", b)
+ _, err := rt.RoundTrip(req)
+ var moved *genMovedError
+ if !errors.As(err, &moved) {
+ t.Fatalf("RoundTrip error = %v, want a *genMovedError", err)
+ }
+ if got := pools.Len(); got != 0 {
+ t.Fatalf("%d pools live, want none for a generation that was never current", got)
+ }
+ if got := b.closed.Load(); got != 1 {
+ t.Fatalf("body closed %d times, want 1", got)
+ }
+}
diff --git a/internal/edge/gensource.go b/internal/edge/gensource.go
new file mode 100644
index 000000000..02890878a
--- /dev/null
+++ b/internal/edge/gensource.go
@@ -0,0 +1,39 @@
+package edge
+
+// GenInfo is what the edge needs to know about an instance's CURRENT
+// generation (zero-downtime spec §7.1, §7.2): the socket that takes new
+// connections, the process-wide generation number the cache tags entries
+// with, and the build X-Carlos-Version reports.
+type GenInfo struct {
+ Socket string
+ N uint64
+ Build string
+}
+
+// GenSource is the activator's generation view, as the edge sees it. It is
+// declared here for the reason Activator is: internal/activator imports
+// this package, never the reverse. *activator.Service implements it.
+//
+// The edge uses it only when GenerationsOn reports true. With it false
+// (CARLOS_SWAP unset, the default) every path below is skipped and the edge
+// proxies, caches and stamps versions exactly as before.
+type GenSource interface {
+ // GenerationsOn is the activator's dormant switch, read once when the
+ // activator was built.
+ GenerationsOn() bool
+ // AppSocket names the current generation's socket for traffic. False
+ // when the instance has no current generation (asleep, or waking).
+ AppSocket(host string) (socket string, ok bool)
+ // ServingGen is AppSocket with the generation's number and build.
+ ServingGen(host string) (GenInfo, bool)
+ // CacheGen is the newest generation that has been current for host in
+ // this process, kept after the instance sleeps so a sleeping instance's
+ // cached pages still answer. False when no generation of host has ever
+ // been current here. It must not take any lock the activator holds
+ // while calling into the cache.
+ CacheGen(host string) (uint64, bool)
+ // OnGenRetired registers fn to be called, synchronously and outside
+ // every activator lock, each time a generation stops being current.
+ // fn must not block.
+ OnGenRetired(fn func(host, socket string))
+}
diff --git a/internal/host/agent.go b/internal/host/agent.go
index c713732d2..2049b8b69 100644
--- a/internal/host/agent.go
+++ b/internal/host/agent.go
@@ -558,7 +558,7 @@ func runAgent(args []string, buildClaims ClaimFactory) {
if errSink != nil {
go errshipLoop(errSink, status.Label(readMachineID()))
}
- go statusLoop(reg)
+ go statusLoop(reg, svc)
// The memory early-warning (memory-pressure spec §2.4): per box, gated
// by CARLOS_HOST_METRICS — one log line and nothing else where unset.
go memwatchLoop()
@@ -695,6 +695,17 @@ func runAgent(args []string, buildClaims ClaimFactory) {
stopHousekeeper()
}
+// statusSocketProbe is the kernel-table probe the status loop hands
+// wrapListeningProbe: listeningProbe itself with generations off (or no
+// activator), and pointed at each managed route's current generation with
+// them on (currentSocketProbe).
+func statusSocketProbe(routes []registry.Route, svc *activator.Service) func(string) *bool {
+ if svc == nil || !svc.GenerationsOn() {
+ return listeningProbe
+ }
+ return currentSocketProbe(routes, svc.AppSocket, listeningProbe)
+}
+
func configureWakeClaims(w *ClaimWiring, cfg *activator.Config) error {
if w == nil {
return hostwrite.ErrUnavailable
@@ -3075,7 +3086,7 @@ func upstreamStates() map[string][]status.UpstreamState {
// logging that every 5 minutes forever would bury the log in noise. Any
// OTHER acquire error (a real S3/credential problem) is logged, same as any
// other tick failure.
-func statusLoop(reg *registry.Registry) {
+func statusLoop(reg *registry.Registry, svc *activator.Service) {
ctx := context.Background()
// bucket is read for the lease key alone (lease.Acquire below), which a
// fleet box never reaches: shouldSkipPublisherElection short-circuits the
@@ -3188,7 +3199,7 @@ func statusLoop(reg *registry.Registry) {
Region: boxRegion,
MemFrac: readMemFrac(),
DiskFrac: readDiskFrac(),
- Listening: wrapListeningProbe(unitBackedSockets(routes), listeningProbe),
+ Listening: wrapListeningProbe(unitBackedSockets(routes), statusSocketProbe(routes, svc)),
// One systemctl call per 5-minute tick, asking the only question
// that catches an enabled-but-never-armed adoption timer — the
// flagship's silent no-adopt state, which nothing on any page
diff --git a/internal/host/generations_test.go b/internal/host/generations_test.go
new file mode 100644
index 000000000..0a564f909
--- /dev/null
+++ b/internal/host/generations_test.go
@@ -0,0 +1,168 @@
+package host
+
+import (
+ "context"
+ "fmt"
+ "net/http"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/schedule"
+)
+
+// genWakeBackend is fakeWakeBackend with the generation capability: with
+// perGen it starts the fake instance on <dir>/g<n>.sock, never on rt.Socket,
+// so a tick that reached the canonical path would find nothing.
+type genWakeBackend struct {
+ fakeWakeBackend
+ dir string
+}
+
+func (b *genWakeBackend) StartGen(_ context.Context, rt registry.Route, n uint64, perGen bool) (activator.Proc, activator.Gen, error) {
+ sock := rt.Socket
+ if perGen {
+ sock = filepath.Join(b.dir, fmt.Sprintf("g%d.sock", n))
+ }
+ b.inst.socket = sock
+ if err := b.inst.start(); err != nil {
+ return nil, activator.Gen{}, err
+ }
+ return &fakeInstanceProc{inst: b.inst}, activator.Gen{N: n, Socket: sock, Control: sock + ".control"}, nil
+}
+
+func (b *genWakeBackend) NextDueVia(context.Context, registry.Route, string) (int64, error) {
+ return 0, nil
+}
+
+// genDeliveryFixture is wakeFixture over genWakeBackend, with the
+// generations switch set before the activator reads it.
+func genDeliveryFixture(t *testing.T, on bool) (*activator.Service, *fakeInstance, registry.Route) {
+ t.Helper()
+ prev := activator.GenerationsEnabled()
+ activator.SetGenerations(on)
+ t.Cleanup(func() { activator.SetGenerations(prev) })
+ dd, err := os.MkdirTemp("", "h")
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { os.RemoveAll(dd) })
+ reg, err := registry.Open(filepath.Join(dd, "r.db"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { reg.Close() })
+ inst := &fakeInstance{handler: http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusNoContent) })}
+ rt := registry.Route{Host: "gen.bab.oncarlos.com", Kind: "instance", Account: "bab", App: "app", Status: "ready",
+ Hibernate: true, Socket: filepath.Join(dd, "canonical.sock"), DBPath: filepath.Join(dd, "app.db")}
+ if err := reg.Upsert(rt); err != nil {
+ t.Fatal(err)
+ }
+ svc := activator.New(activator.Config{Bucket: "test-bucket", Idle: time.Hour}, &genWakeBackend{fakeWakeBackend: fakeWakeBackend{inst: inst}, dir: dd}, reg)
+ svc.Start()
+ t.Cleanup(func() {
+ ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+ defer cancel()
+ svc.Shutdown(ctx)
+ })
+ return svc, inst, rt
+}
+
+func deliverTick(t *testing.T, d *activatorDeliverer, rt registry.Route) int {
+ t.Helper()
+ req, err := schedule.BuildRequest(rt, registry.Timer{Name: "sync", Kind: schedule.KindSchedule, Path: "/jobs/sync", NextAt: 1, DueAt: 1}, "tok", "test")
+ if err != nil {
+ t.Fatal(err)
+ }
+ status, err := d.Deliver(context.Background(), rt, req)
+ if err != nil {
+ t.Fatalf("Deliver: %v", err)
+ }
+ return status
+}
+
+// Scheduled delivery dials AppSocket (spec §7.1): the tick wakes the
+// instance and reaches its generation socket, though nothing listens on the
+// route's canonical path.
+func TestScheduledDeliveryReachesTheCurrentGeneration(t *testing.T) {
+ requireUnixSockets(t)
+ svc, inst, rt := genDeliveryFixture(t, true)
+ d := &activatorDeliverer{svc: svc}
+ if got := deliverTick(t, d, rt); got != http.StatusNoContent {
+ t.Fatalf("tick status = %d, want 204 from the instance", got)
+ }
+ cur, ok := svc.AppSocket(rt.Host)
+ if !ok || inst.socket != cur || cur == rt.Socket {
+ t.Fatalf("instance bound %q, AppSocket = %q (%v); want a generation socket, not %q", inst.socket, cur, ok, rt.Socket)
+ }
+ if len(inst.served()) != 1 {
+ t.Fatalf("instance served %d ticks, want 1", len(inst.served()))
+ }
+ if d.genPools == nil || d.genPools.Len() != 1 {
+ t.Fatal("the tick did not go through the leg's generation pools")
+ }
+}
+
+// Flag off: the canonical socket and the cached per-socket proxy, as before.
+func TestScheduledDeliveryWithGenerationsOffUsesTheCanonicalSocket(t *testing.T) {
+ requireUnixSockets(t)
+ svc, inst, rt := genDeliveryFixture(t, false)
+ d := &activatorDeliverer{svc: svc}
+ if got := deliverTick(t, d, rt); got != http.StatusNoContent {
+ t.Fatalf("tick status = %d, want 204", got)
+ }
+ if inst.socket != rt.Socket {
+ t.Fatalf("instance bound %q, want the canonical %q", inst.socket, rt.Socket)
+ }
+ if d.genPools != nil {
+ t.Fatal("generation pools were made with the switch off")
+ }
+ if _, ok := d.proxies[rt.Socket]; !ok {
+ t.Fatal("the canonical per-socket proxy was not used")
+ }
+}
+
+// The status probe asks about each managed route's current generation, and
+// about everything else exactly as before.
+func TestCurrentSocketProbePointsAtTheCurrentGeneration(t *testing.T) {
+ routes := []registry.Route{
+ {Host: "awake.example.com", Kind: "instance", Hibernate: true, Socket: "/run/carlos/awake.example.com.sock"},
+ {Host: "asleep.example.com", Kind: "instance", Hibernate: true, Socket: "/run/carlos/asleep.example.com.sock"},
+ {Host: "svc.example.com", Kind: "service", Socket: "/run/carlos/svc.example.com.sock"},
+ }
+ appSocket := func(host string) (string, bool) {
+ if host == "awake.example.com" {
+ return "/run/carlos/g/0123456789ab.4.sock", true
+ }
+ return "", false
+ }
+ var asked []string
+ probe := currentSocketProbe(routes, appSocket, func(sock string) *bool {
+ asked = append(asked, sock)
+ v := true
+ return &v
+ })
+ for _, r := range routes {
+ probe(r.Socket)
+ }
+ want := []string{"/run/carlos/g/0123456789ab.4.sock", "/run/carlos/asleep.example.com.sock", "/run/carlos/svc.example.com.sock"}
+ if fmt.Sprint(asked) != fmt.Sprint(want) {
+ t.Fatalf("probed %v, want %v", asked, want)
+ }
+}
+
+// Flag off (or no activator): the status loop's probe is listeningProbe.
+func TestStatusSocketProbeIsUnchangedWithGenerationsOff(t *testing.T) {
+ got := statusSocketProbe(nil, nil)
+ if fmt.Sprintf("%p", got) != fmt.Sprintf("%p", listeningProbe) {
+ t.Fatal("with no activator the status probe is not listeningProbe")
+ }
+ svc, _, _ := genDeliveryFixture(t, false)
+ got = statusSocketProbe(nil, svc)
+ if fmt.Sprintf("%p", got) != fmt.Sprintf("%p", listeningProbe) {
+ t.Fatal("with generations off the status probe is not listeningProbe")
+ }
+}
diff --git a/internal/host/probes.go b/internal/host/probes.go
index d9e889387..dc42fd9ac 100644
--- a/internal/host/probes.go
+++ b/internal/host/probes.go
@@ -297,6 +297,30 @@ func wrapListeningProbe(unitBacked map[string]bool, probe func(string) *bool) fu
}
}
+// currentSocketProbe points probe at each activator-managed route's CURRENT
+// generation socket (zero-downtime spec §7.1): with generations on, nothing
+// binds a managed route's canonical path, so the kernel table would call
+// every awake instance asleep. A route the activator reports no current
+// generation for (asleep) is probed at its canonical path, where nothing
+// listens: asleep, which is the truth. Any other socket passes straight
+// through. appSocket is the activator's AppSocket.
+func currentSocketProbe(routes []registry.Route, appSocket func(host string) (string, bool), probe func(string) *bool) func(string) *bool {
+ hostOf := map[string]string{}
+ for _, r := range routes {
+ if r.Kind == "instance" && r.Hibernate && r.Socket != "" && r.Backing != registry.BackingUnit {
+ hostOf[r.Socket] = r.Host
+ }
+ }
+ return func(sock string) *bool {
+ if host, ok := hostOf[sock]; ok {
+ if cur, ok := appSocket(host); ok {
+ return probe(cur)
+ }
+ }
+ return probe(sock)
+ }
+}
+
// parseMeminfoBytes reads /proc/meminfo's MemTotal and MemAvailable (both in
// kB) and returns them converted to bytes. Either field missing or
// unparsable -> nil for both.
diff --git a/internal/host/schedule_wiring.go b/internal/host/schedule_wiring.go
index 03eb5be31..b89e0dcf7 100644
--- a/internal/host/schedule_wiring.go
+++ b/internal/host/schedule_wiring.go
@@ -300,6 +300,31 @@ type activatorDeliverer struct {
// rather than waiting out edge.UnixProxy's 90s IdleConnTimeout.
mu sync.Mutex
proxies map[string]http.Handler
+
+ // genPools is the delivery leg's own per-generation-socket pools, used
+ // instead of proxies for an activator-managed instance once generations
+ // are on (zero-downtime spec §7.1): a tick reaches the CURRENT
+ // generation, asked per connection. Made on first use and retired
+ // through the activator's hook, so a retired generation's pool closes
+ // with it rather than waiting for the daily prune. Nil with generations
+ // off, and nothing below reads it then.
+ genOnce sync.Once
+ genPools *edge.GenPools
+}
+
+// managedByGenerations reports whether rt's ticks go through the current
+// generation: generations on, and an activator-managed instance.
+func (d *activatorDeliverer) managedByGenerations(rt registry.Route) bool {
+ return d.svc != nil && d.svc.GenerationsOn() && rt.Kind == "instance" && rt.Hibernate
+}
+
+// genProxyFor is the AppSocket proxy for host over the leg's own pools.
+func (d *activatorDeliverer) genProxyFor(host string) http.Handler {
+ d.genOnce.Do(func() {
+ d.genPools = edge.NewGenPools()
+ d.svc.OnGenRetired(d.genPools.Retire)
+ })
+ return edge.AppSocketProxy(host, d.svc, d.genPools)
}
// maxProxyCache is that ceiling. Deliberately well above the number of
@@ -381,7 +406,16 @@ func (d *activatorDeliverer) deliverSelected(ctx context.Context, rt registry.Ro
// to an empty path (which would be a connect-refused every 15s).
return 0, errors.New("no socket for route " + rt.Host)
}
- proxy := d.proxyFor(rt.Socket)
+ // Generations on: the proxy asks the activator for the current
+ // generation's socket per connection (spec §7.1). Off: the cached proxy
+ // on rt.Socket, exactly as before.
+ gen := d.managedByGenerations(rt)
+ var proxy http.Handler
+ if gen {
+ proxy = d.genProxyFor(rt.Host)
+ } else {
+ proxy = d.proxyFor(rt.Socket)
+ }
var admissionErr error
if admit != nil {
next := proxy
@@ -416,8 +450,11 @@ func (d *activatorDeliverer) deliverSelected(ctx context.Context, rt registry.Ro
// this socket's pool; the next tick dials fresh. Not the case a
// reader expects here: a socket that will not dial is ReverseProxy's
// own 502 with no error, and it has nothing pooled to retire anyway.
- // Retired ROUTES are reclaimed by RetainSockets, not from here.
- d.dropProxy(rt.Socket)
+ // Retired ROUTES are reclaimed by RetainSockets, not from here. A
+ // generation's pool is retired by the activator's hook instead.
+ if !gen {
+ d.dropProxy(rt.Socket)
+ }
}
if err != nil && ctx.Err() == nil {
// Redacted for the same reason as the token error: this string is
diff --git a/internal/host/socket_advice.go b/internal/host/socket_advice.go
index 7f17602f8..366be32e0 100644
--- a/internal/host/socket_advice.go
+++ b/internal/host/socket_advice.go
@@ -12,6 +12,7 @@ import (
"log"
"math/rand/v2"
"net"
+ "os"
"time"
"github.com/carlosframework/platform/internal/activator"
@@ -70,6 +71,31 @@ func socketServing(socket string) bool {
return true
}
+// advisorySocketServing is the advisory's probe with generations on. A
+// managed instance's canonical path is then the activator's own symlink to
+// its current generation (zero-downtime spec §7.1), which socketServing's
+// checked dial would report as a peer mismatch every hour. A link naming
+// the host's current generation is the activator's and is not a stray;
+// every other path is probed as before. appSocket is the activator's.
+func advisorySocketServing(routes []registry.Route, appSocket func(host string) (string, bool), serving func(string) bool) func(string) bool {
+ hostOf := map[string]string{}
+ for _, r := range routes {
+ if r.Kind == "instance" && r.Hibernate && r.Socket != "" && r.Backing != registry.BackingUnit {
+ hostOf[r.Socket] = r.Host
+ }
+ }
+ return func(sock string) bool {
+ if host, ok := hostOf[sock]; ok {
+ if cur, ok := appSocket(host); ok {
+ if target, err := os.Readlink(sock); err == nil && target == cur {
+ return false
+ }
+ }
+ }
+ return serving(sock)
+ }
+}
+
// socketAdvisoryLoop runs the advisory hourly, first pass ~1min after
// boot, like its sibling loops. svc may be nil (activator unarmed):
// owns is then constantly false and every live socket on an exec route
@@ -87,7 +113,11 @@ func socketAdvisoryLoop(reg *registry.Registry, svc *activator.Service) {
log.Printf("agent: socket advisory: list routes: %v", err)
continue
}
- for _, rt := range warnExternalSockets(routes, socketServing, owns) {
+ serving := socketServing
+ if svc != nil && svc.GenerationsOn() {
+ serving = advisorySocketServing(routes, svc.AppSocket, socketServing)
+ }
+ for _, rt := range warnExternalSockets(routes, serving, owns) {
if rt.Hibernate {
log.Printf("agent: route %s is activator-managed (hibernate=1) but its socket %s is served by a process the activator did not spawn — a stray supervisor is squatting the socket and will fight the wake path; find and retire it (see platform issue #17)", rt.Host, rt.Socket)
} else {
diff --git a/internal/host/socket_advice_gen_test.go b/internal/host/socket_advice_gen_test.go
new file mode 100644
index 000000000..162376c89
--- /dev/null
+++ b/internal/host/socket_advice_gen_test.go
@@ -0,0 +1,46 @@
+package host
+
+import (
+ "fmt"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/carlosframework/platform/internal/registry"
+)
+
+// With generations on, the advisory does not dial the activator's own link
+// at a managed route's canonical path (the checked dial would call it a peer
+// mismatch every hour). Any other link, or a path that is not a link, is
+// probed as before.
+func TestSocketAdvisorySkipsTheActivatorsOwnLink(t *testing.T) {
+ dir := t.TempDir()
+ own := filepath.Join(dir, "own.sock")
+ foreign := filepath.Join(dir, "foreign.sock")
+ plain := filepath.Join(dir, "plain.sock")
+ for path, target := range map[string]string{own: "/run/carlos/g/aaaaaaaaaaaa.3.sock", foreign: "/run/carlos/elsewhere.sock"} {
+ if err := os.Symlink(target, path); err != nil {
+ t.Fatal(err)
+ }
+ }
+ routes := []registry.Route{
+ {Host: "own.example.com", Kind: "instance", Hibernate: true, Socket: own},
+ {Host: "foreign.example.com", Kind: "instance", Hibernate: true, Socket: foreign},
+ {Host: "plain.example.com", Kind: "instance", Hibernate: true, Socket: plain},
+ }
+ appSocket := func(string) (string, bool) { return "/run/carlos/g/aaaaaaaaaaaa.3.sock", true }
+ var probed []string
+ serving := advisorySocketServing(routes, appSocket, func(sock string) bool {
+ probed = append(probed, sock)
+ return true
+ })
+ if serving(own) {
+ t.Fatal("the activator's own link was reported as served by a stray")
+ }
+ if !serving(foreign) || !serving(plain) {
+ t.Fatal("a foreign link or a plain path was not probed")
+ }
+ if fmt.Sprint(probed) != fmt.Sprint([]string{foreign, plain}) {
+ t.Fatalf("probed %v, want the foreign link and the plain path only", probed)
+ }
+}