Remote host runbook (software half)

A CARLOS host is anything that can reach the deployment bucket. A remote host is OUTBOUND-ONLY: its whole network surface is outbound 443 to S3 and the edge tunnel. No inbound port, no SSH path required by the platform.

Prereqs: systemd Debian-family OS, 64-bit (the artifact key is GOOS-GOARCH — linux-arm64 or linux-amd64; a 32-bit OS resolves nothing).

Steps

1. Provision the IAM principal + region data bucket (infra/modules/region-remote-host). Scope: deployment bucket + this host's own region bucket, and NOTHING else — wrong-region-fails-at-IAM holds off-cloud too. A scoped access key is acceptable to start; IAM Roles Anywhere is the long-term, hq-grade alternative (no long-lived key on the box at all).

1. Stage /etc/carlos/credentials, owned by root, group carlos, mode 0640:

[default] awsaccesskeyid=... awssecretaccesskey=...

Static keys only. carlos bootstrap -offcloud refuses to proceed if this file is missing or is not mode 0600 or 0640. It will name this path in the error. The agent will not use a file that holds anything else, such as a credential_process line or a second profile.

1. Stage /etc/carlos/host.env — the usual agent env, plus the credentials pointer and the three tunnel variables. All three tunnel variables are required together; if any is missing the dial-out loop stays inert (no crash, no retry storm — it just never dials):

AWSSHAREDCREDENTIALSFILE=/etc/carlos/credentials CARLOSTUNNELURL=https://<edge's reserved tunnel hostname> CARLOSTUNNELTOKEN=<this host's token> CARLOSTUNNEL_NAME=<this host's channel name>

Do not set AWS_PROFILE or AWS_DEFAULT_PROFILE in host.env. The agent ignores both and logs an alert.

1. Stage the binary at /opt/carlos/carlos (matching GOOS-GOARCH).

1. Run: carlos bootstrap -offcloud

This does everything the normal carlos bootstrap does (units, dirs, service user, timers) and, before any of it, checks the credentials precondition from step 2. A failed check leaves the box untouched. On a live root, bootstrap and every agent start rewrite the credentials file as a new file owned by root, group carlos, mode 0640, keeping only the static keys. The service user can read it but not change it. A file that is already right is left alone.

1. systemctl start carlos-edge.service

1. Verify: - journalctl -u carlos-edge shows "tunnel connected". - The box appears in carlos status -live within one heartbeat (5 minutes + jitter).

Edge side (once per edge box, not per remote host)

Set in the edge box's host.env:

CARLOSTUNNELHOST=<reserved hostname> CARLOSTUNNELCLIENTS=name=token,name2=token2,...

CARLOS_TUNNEL_HOST must be a bare hostname — no port, no scheme. Each name in CARLOS_TUNNEL_CLIENTS must match ^[a-z0-9-]{1,32}$ or the edge refuses to boot.

carlos route repoints an EXISTING route — it does not create one. Add the route first, then back it with the tunnel channel:

carlos add <h> carlos route -host <h> -tunnel <name>

Fleet boxes (customer fleets)

Fleet boxes use the same agent with the fleet env contract specified in the customer-fleets design spec (private platform-infrastructure repo, which also holds the worked Pi example). The carlos add and carlos route -tunnel ceremony documented above applies only to platform remote boxes (tito hq), NOT fleet boxes — fleet instances are placed at app create time and boxes auto-join fleet registries.

Security note

In -dev (HTTP) edge mode the registration bearer token travels in clear text. Tunnel hosts must only ever dial a real-TLS edge in production — never point CARLOS_TUNNEL_URL at an HTTP edge outside a dev sandbox.

Known limitations

• Apps see the edge's IP, not the real client's. A tunnel-backed request crosses two reverse-proxy hops: client → edge (over the internet), then edge → app (over the tunnel, on the remote host's own local edge). The second hop re-stamps X-Forwarded-For/X-Forwarded-Proto from its own immediate peer, so an app on a tunnel-backed host sees the edge's IP as client, and X-Forwarded-Proto: http even though the original request was TLS. This fails safe — client identity is dropped, never spoofed — but any app on a tunnel-backed host that logs or gates on client IP will be wrong today.
• No WebSockets to a tunnel-backed host. The tunnel transport can't carry a Connection: Upgrade handshake; every WebSocket attempt to an app on a tunnel-backed host fails with a 502 (loud, not a hang). Plain HTTP request/ response is unaffected. RFC 8441 extended CONNECT is the likely future fix.

Related

The hq PHYSICAL half (racking, power, network uplink for a region-hq box) lands separately in infra/tito/region-hq.md. For a concrete Pi instantiation of this runbook (values, seeding), see the private platform-infrastructure repo.