| 1 | # Remote host runbook (software half) |
| 2 | |
| 3 | A CARLOS host is anything that can reach the deployment bucket. A remote |
| 4 | host is OUTBOUND-ONLY: its whole network surface is outbound 443 to S3 and |
| 5 | the edge tunnel. No inbound port, no SSH path required by the platform. |
| 6 | |
| 7 | Prereqs: systemd Debian-family OS, 64-bit (the artifact key is |
| 8 | GOOS-GOARCH — linux-arm64 or linux-amd64; a 32-bit OS resolves nothing). |
| 9 | |
| 10 | ## Steps |
| 11 | |
| 12 | 1. Provision the IAM principal + region data bucket |
| 13 | (`infra/modules/region-remote-host`). Scope: deployment bucket + this |
| 14 | host's own region bucket, and NOTHING else — wrong-region-fails-at-IAM |
| 15 | holds off-cloud too. A scoped access key is acceptable to start; IAM |
| 16 | Roles Anywhere is the long-term, hq-grade alternative (no long-lived |
| 17 | key on the box at all). |
| 18 | |
| 19 | 2. Stage `/etc/carlos/credentials`, owned by root, group `carlos`, mode 0640: |
| 20 | |
| 21 | [default] |
| 22 | aws_access_key_id=... |
| 23 | aws_secret_access_key=... |
| 24 | |
| 25 | Static keys only. `carlos bootstrap -offcloud` refuses to proceed if this file is missing or is not mode 0600 or 0640. It will name this path in the error. The agent will not use a file that holds anything else, such as a `credential_process` line or a second profile. |
| 26 | |
| 27 | 3. Stage `/etc/carlos/host.env` — the usual agent env, plus the |
| 28 | credentials pointer and the three tunnel variables. All three tunnel |
| 29 | variables are required together; if any is missing the dial-out loop |
| 30 | stays inert (no crash, no retry storm — it just never dials): |
| 31 | |
| 32 | AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials |
| 33 | CARLOS_TUNNEL_URL=https://<edge's reserved tunnel hostname> |
| 34 | CARLOS_TUNNEL_TOKEN=<this host's token> |
| 35 | CARLOS_TUNNEL_NAME=<this host's channel name> |
| 36 | |
| 37 | Do not set `AWS_PROFILE` or `AWS_DEFAULT_PROFILE` in `host.env`. The agent ignores both and logs an alert. |
| 38 | |
| 39 | 4. Stage the binary at `/opt/carlos/carlos` (matching GOOS-GOARCH). |
| 40 | |
| 41 | 5. Run: `carlos bootstrap -offcloud` |
| 42 | |
| 43 | This does everything the normal `carlos bootstrap` does (units, dirs, service user, timers) and, before any of it, checks the credentials precondition from step 2. A failed check leaves the box untouched. On a live root, bootstrap and every agent start rewrite the credentials file as a new file owned by root, group `carlos`, mode 0640, keeping only the static keys. The service user can read it but not change it. A file that is already right is left alone. |
| 44 | |
| 45 | 6. `systemctl start carlos-edge.service` |
| 46 | |
| 47 | 7. Verify: |
| 48 | - `journalctl -u carlos-edge` shows "tunnel connected". |
| 49 | - The box appears in `carlos status -live` within one heartbeat |
| 50 | (5 minutes + jitter). |
| 51 | |
| 52 | ## Edge side (once per edge box, not per remote host) |
| 53 | |
| 54 | Set in the edge box's `host.env`: |
| 55 | |
| 56 | CARLOS_TUNNEL_HOST=<reserved hostname> |
| 57 | CARLOS_TUNNEL_CLIENTS=name=token,name2=token2,... |
| 58 | |
| 59 | `CARLOS_TUNNEL_HOST` must be a bare hostname — no port, no scheme. Each |
| 60 | name in `CARLOS_TUNNEL_CLIENTS` must match `^[a-z0-9-]{1,32}$` or the edge |
| 61 | refuses to boot. |
| 62 | |
| 63 | `carlos route` repoints an EXISTING route — it does not create one. Add the |
| 64 | route first, then back it with the tunnel channel: |
| 65 | |
| 66 | carlos add <h> |
| 67 | carlos route -host <h> -tunnel <name> |
| 68 | |
| 69 | ## Fleet boxes (customer fleets) |
| 70 | |
| 71 | Fleet boxes use the same agent with the fleet env contract specified in the |
| 72 | customer-fleets design spec (private `platform-infrastructure` repo, which |
| 73 | also holds the worked Pi example). The `carlos add` |
| 74 | and `carlos route -tunnel` ceremony documented above applies only to platform |
| 75 | remote boxes (tito hq), NOT fleet boxes — fleet instances are placed at app |
| 76 | create time and boxes auto-join fleet registries. |
| 77 | |
| 78 | ## Security note |
| 79 | |
| 80 | In `-dev` (HTTP) edge mode the registration bearer token travels in clear |
| 81 | text. Tunnel hosts must only ever dial a real-TLS edge in production — |
| 82 | never point `CARLOS_TUNNEL_URL` at an HTTP edge outside a dev sandbox. |
| 83 | |
| 84 | ## Known limitations |
| 85 | |
| 86 | - **Apps see the edge's IP, not the real client's.** A tunnel-backed request |
| 87 | crosses two reverse-proxy hops: client → edge (over the internet), then |
| 88 | edge → app (over the tunnel, on the remote host's own local edge). The |
| 89 | second hop re-stamps `X-Forwarded-For`/`X-Forwarded-Proto` from its own |
| 90 | immediate peer, so an app on a tunnel-backed host sees the edge's IP as |
| 91 | client, and `X-Forwarded-Proto: http` even though the original request was |
| 92 | TLS. This fails safe — client identity is dropped, never spoofed — but any |
| 93 | app on a tunnel-backed host that logs or gates on client IP will be wrong |
| 94 | today. |
| 95 | - **No WebSockets to a tunnel-backed host.** The tunnel transport can't carry |
| 96 | a `Connection: Upgrade` handshake; every WebSocket attempt to an app on a |
| 97 | tunnel-backed host fails with a 502 (loud, not a hang). Plain HTTP request/ |
| 98 | response is unaffected. RFC 8441 extended CONNECT is the likely future fix. |
| 99 | |
| 100 | ## Related |
| 101 | |
| 102 | The hq PHYSICAL half (racking, power, network uplink for a region-hq box) |
| 103 | lands separately in `infra/tito/region-hq.md`. For a concrete Pi |
| 104 | instantiation of this runbook (values, seeding), see the private |
| 105 | `platform-infrastructure` repo. |
| 106 | |
amadan