carlos / platform Public

Isolation probe: no netlink; "resolver is not the box's own address" moves root-side (#126)

This blocks production. After v0.25.0 rolled, every dev box reported ready: false, so nested-by-default never fired. The one failing line was:

resolver-ports fail interface addresses: route ip+net: netlinkrib: address family not supported by protocol

The probe called net.InterfaceAddrs, which uses netlink. carlos-instance-probe.service has RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX, the same as every instance unit, so that call could never succeed.

What changes

• Root side (internal/instancectl): resolverAllow now takes the box's local prefixes and leaves out any nameserver inside one. The rule is enforced where the allow is written. - The control unit is RestrictAddressFamilies=AF_UNIX, so it can't use netlink either (the brief's premise that it isn't restricted is wrong). It reads procfs instead: /proc/net/fib_trie for IPv4 LOCAL routes, including whole local ranges like 127.0.0.0/8, and /proc/net/if_inet6. Reading a file needs no socket. - A refused resolver shows up as a new failing resolver-local line in the probe report, so the box stays not ready, the same outcome as before. - If the addresses can't be read, the allow is reset to empty and resolver-allow fails. An allow that might name the box is never written. A nil LocalAddrs also closes every resolver.
• Probe (internal/instanceexec): localAddrs and interfaceAddrs are gone. resolver-ports keeps only what it can test from inside the unit: an allowed resolver doesn't answer on TCP 22.
• No unit changes. Nothing gains AF_NETLINK: not the instance templates, not the probe unit, not the control unit.
• Spec §egress paragraph updated.

Can the same thing hit instance units at runtime?

No.

• New test TestInstanceCodeOpensOnlyTheUnitsAddressFamilies walks every platform package that instanceexec builds from (probe, trampoline instance-exec, instance-exit, instancectl, and their platform imports), as linux sees them. - It refuses net.Interfaces, net.InterfaceAddrs and net.InterfaceBy*, syscall.NetlinkRIB, and any AF_* outside INET, INET6, UNIX and UNSPEC. - It fails on main with probe.go: net.InterfaceAddrs.
• Third-party code, checked by hand: the only such call is google/uuid's net.Interfaces, reached only for v1/v6 UUIDs, and nothing here makes those.
• The binary is CGO_ENABLED=0, so DNS never goes through glibc's netlink use.
• Go's IPv6 zone cache only touches netlink for zoned addresses. Those are link-local, which the fence denies anyway, and on failure it falls back to the numeric zone.

How tested

• Red first, on main: a probe run with interface listing failing exactly as on the fleet gave resolver-ports fail interface addresses: route ip+net: netlinkrib: .... That seam no longer exists. The lasting red-on-main test is the walk above.
• New instancectl tests: - the box's own address is refused, including inside a local range - RunProbe with a local resolver: the allow is written empty and the report is not ready, with a resolver-local fail line - a good resolver gives an ok line - address listing erroring, or nil: the allow is reset and resolver-allow fails - fibtrie and ifinet6 parsing from an EC2-shaped fixture; a missing ifinet6 means no v6; a fibtrie with no LOCAL route is refused
• Sanity checks on a real Debian 13 host: - localPrefixes against the live procfs matches the interface addresses. - In a transient user unit with RestrictAddressFamilies=AF_UNIX ProtectKernelTunables=yes ProtectSystem=strict PrivateDevices=yes, ip addr fails with "Cannot open netlink socket: Address family not supported by protocol", while both procfs files read fine. - Caveat: that was the user manager, not the system one.
• Gate: gofmt clean, build and vet clean, make test: everything passes except known environment failures: - socket-path length (bind/connect EINVAL) in instanceexec, activator, edge and host - the root package reading ~/.carlos/config

Deliberately not done

• No AF_NETLINK anywhere.
• IPv6 local ranges, such as AnyIP routes, aren't read; only IPv6 interface addresses are. That is the same coverage the probe had.
• The probe's dns line also fails for a refused local resolver, because sendto gets EPERM. That is expected: resolver-local gives the reason.

Needs a release and roll to reach the fleet. A box becomes ready on its next probe after the control service restarts on the new binary.