amadan

carlos / platform Public

carlos db push/pull: copy an instance's SQLite database out or in through the console

`carlos db pull` copies a hibernating, activator-managed instance's
replicated database to the member's machine; `carlos db push` replaces it
with a local file, after a backup and a confirmation; `status` and
`backups` read what those leave behind. A member needs a login only: the
CLI asks the console, the console writes a request onto the instance
record (an additive `db` field, provision.DBTransfer, with guarded CAS
transitions beside the move ones) and stages bytes in the control bucket,
and the box that owns the route bridges to the data bucket.

Box side (internal/host/dbpass.go, armed and run beside the moves
worker): a pull restores the replica without the lease into scratch,
CheckDBs it and stages it; a push holds the route at status
"maintenance", fences and quiesces the child, checkpoints, backs up the
replica under backups/<id>.db (newest five kept), deletes the replica
prefix, replicates the upload into it under a fresh lease, purges the
warm cache so the next wake restores the pushed file, then releases the
route. Every step is idempotent and resumes from the record's Step; a
failure after the prefix delete keeps the request in flight rather than
releasing a route over an empty replica. The delete-then-replicate
sequence is proven against the vendored litestream in
internal/lease/replace_test.go.

Console doors (internal/console/apiinstancedb.go) are closed to every
service role: the payload is a whole customer database. The class rail
is factored out of moveGate and shared. A move refuses while a transfer
is in flight and vice versa. docs/site/cli.md gains `### carlos db`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Tony Wood pushed by tonywood@keymail.dev b014d57d9fa4b2023d444aa3f6720f3a1f347b57 parent 88abfae
19 files changed, +4343 −20
  • auth.go +40 −0
  • db_cli.go +682 −0
  • db_cli_test.go +339 −0
  • docs/site/cli.md +49 −0
  • internal/console/apiinstancedb.go +362 −0
  • internal/console/apiinstancedb_test.go +410 −0
  • internal/console/apiinstancemove.go +53 −18
  • internal/console/authcli.go +14 −0
  • internal/console/svcroutes_test.go +13 −2
  • internal/host/agent.go +5 −0
  • internal/host/dbpass.go +647 −0
  • internal/host/dbpass_test.go +767 −0
  • internal/lease/check.go +13 −0
  • internal/lease/replace_test.go +169 −0
  • internal/provision/dbstage.go +219 −0
  • internal/provision/dbtransfer.go +242 −0
  • internal/provision/dbtransfer_test.go +299 −0
  • internal/provision/provision.go +15 −0
  • main.go +5 −0

CI passed — run details

diff --git a/auth.go b/auth.go
index 3e282cf12..3c1aaf628 100644
--- a/auth.go
+++ b/auth.go
@@ -535,6 +535,46 @@ func (c *consoleClient) doMultipartMethod(ctx context.Context, method, path stri
return res.StatusCode, body, nil
}
+// doStream runs one request whose body is a STREAM in one direction or the
+// other — `carlos db push` sending a database file up, `carlos db pull`
+// reading one down — and hands the response back unread, so the caller
+// copies the body straight to disk (or the request body straight from it)
+// without either side ever holding the file whole. contentLength is the
+// request body's size when known (-1 otherwise); headers are the caller's
+// extras on top of the bearer.
+//
+// Same client shape as doMultipart, for the same reason: no absolute
+// Timeout (the caller's context is the budget — a 256 MiB transfer over a
+// slow link is minutes, and the JSON client's 30s would abort it live),
+// and no redirects. A non-2xx status is NOT an error here either: the
+// caller reads the body through apiErrorFromBody, which knows the
+// {"error"} shape. The caller closes the response body.
+func (c *consoleClient) doStream(ctx context.Context, method, path string, body io.Reader, contentLength int64, headers map[string]string) (*http.Response, error) {
+ req, err := http.NewRequestWithContext(ctx, method, c.base+path, body)
+ if err != nil {
+ return nil, err
+ }
+ if body != nil && contentLength >= 0 {
+ req.ContentLength = contentLength
+ }
+ for k, v := range headers {
+ req.Header.Set(k, v)
+ }
+ if c.token != "" {
+ req.Header.Set("Authorization", "Bearer "+c.token)
+ }
+ client := &http.Client{Timeout: shipHTTPClientTimeout, CheckRedirect: neverFollow}
+ res, err := client.Do(req)
+ if err != nil {
+ return nil, err
+ }
+ if merr := consoleMoved(c.base, res.StatusCode, res.Header.Get("Location")); merr != nil {
+ res.Body.Close()
+ return nil, offerConsoleMoved(merr)
+ }
+ return res, nil
+}
+
// neverFollow is the one redirect policy every console client uses.
func neverFollow(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
diff --git a/db_cli.go b/db_cli.go
new file mode 100644
index 000000000..4fce345db
--- /dev/null
+++ b/db_cli.go
@@ -0,0 +1,682 @@
+// db_cli.go is `carlos db`: copy an instance's SQLite database out of the
+// platform (pull) or a replacement into it (push), for a member with no box
+// and no bucket access. Two more verbs read what the first two leave
+// behind: status (the last transfer on the record) and backups (what a
+// push saved before it replaced anything).
+//
+// The shape is the console-mediated one every carlos verb has: the CLI asks
+// the console, the console writes a request onto the instance's record and
+// bumps the serial, and the box that owns the route does the work within
+// seconds — a pull restores the replica and stages a snapshot in the control
+// bucket for the console to serve; a push takes the staged upload, backs up
+// the current replica, replaces it, and purges the box's warm cache so the
+// next wake restores the pushed file (internal/host/dbpass.go has the
+// ordering and the crash-point argument). This side polls the record until
+// it is done or failed, then downloads or reports.
+//
+// Scope, v1: hibernating, activator-managed exec instances only — the class
+// whose database the platform itself manages. A pull reflects the REPLICA,
+// which for a sleeping instance is byte-final and for a running one is at
+// most the sync interval behind. A push is downtime: the route is held out
+// of service from the moment the box picks the request up until the swap is
+// complete, and it needs confirmation — --yes, or the host typed at a
+// terminal.
+package main
+
+import (
+ "context"
+ "database/sql"
+ "encoding/json"
+ "errors"
+ "flag"
+ "fmt"
+ "io"
+ "log"
+ "net/http"
+ "net/url"
+ "os"
+ "path/filepath"
+ "sort"
+ "strings"
+ "time"
+
+ _ "modernc.org/sqlite"
+)
+
+// dbTransferMaxBytes mirrors the console's cap (provision.DBTransferMaxBytes):
+// checked here first so an oversized file costs a stat, not an upload.
+const dbTransferMaxBytes int64 = 256 << 20
+
+// dbSQLiteContentType is the media type the push door demands and the
+// download doors serve.
+const dbSQLiteContentType = "application/vnd.sqlite3"
+
+// dbConfirmHeader carries the host the push is confirmed for; the door
+// refuses a push whose header does not name the instance.
+const dbConfirmHeader = "X-Carlos-Confirm-Host"
+
+const dbUsage = `usage: carlos db <verb> --app <a> [--host <h>] [flags]
+
+ pull [--out <path> | <path>] [--backup <id>|latest] [--force]
+ copy the instance's database to this machine (default ./<host>.db)
+ push <path> [--yes]
+ replace the instance's database with a local SQLite file — the
+ instance is held out of service for the swap, the current
+ database is backed up first, and the swap needs confirmation
+ status the last transfer requested for the instance
+ backups the backups earlier pushes saved, newest first
+
+Every verb takes --app, and --host when the app has more than one instance;
+the usual --account / --account-id / --console / --destination apply.
+Hibernating, platform-managed instances only, up to 256 MiB.
+
+`
+
+func runDB(args []string) {
+ if len(args) == 0 {
+ fmt.Fprint(os.Stderr, dbUsage)
+ os.Exit(2)
+ }
+ switch args[0] {
+ case "pull":
+ runDBPull(args[1:])
+ case "push":
+ runDBPush(args[1:])
+ case "status":
+ runDBStatus(args[1:])
+ case "backups":
+ runDBBackups(args[1:])
+ case "-h", "--help", "help":
+ fmt.Fprint(stdout, dbUsage)
+ default:
+ log.Fatalf("carlos db: unknown verb %q\n%s", args[0], dbUsage)
+ }
+}
+
+// requireDBAPISession resolves signed-in mode, refusing direct mode by
+// name: a transfer is a console write the fleet listens for, and an
+// operator with the bucket already has the box.
+func requireDBAPISession(consoleFlag string) *apiSession {
+ sess, err := resolveMode(consoleFlag)
+ if err != nil {
+ log.Fatal(err)
+ }
+ if sess == nil {
+ log.Fatal("carlos db works through a console, and this shell is in direct bucket mode\n" +
+ "(CARLOS_DEPLOYMENT_BUCKET/CARLOS_DEPLOYMENT_DIR is set). A transfer is a member verb with no\n" +
+ "direct-bucket equivalent — the request is a console write the box listens for. Unset the\n" +
+ "store env and run: carlos auth login --console <url>.")
+ }
+ return sess
+}
+
+// dbTarget is what every verb resolves first: the session, the account,
+// the app and the one host the verb acts on.
+type dbTarget struct {
+ sess *apiSession
+ acct apiAcct
+ app string
+ host string
+}
+
+// dbFlags registers the flags every verb shares and returns the resolver.
+func dbFlags(fs *flag.FlagSet, verb string) func() dbTarget {
+ app := appFlag(fs, "app the instance belongs to")
+ host := fs.String("host", "", "the instance (omit when the app has exactly one)")
+ acctRaw, acctIDRaw, _ := accountFlags(fs)
+ consoleFlag := consoleOption(fs, "console to work through (default: the only logged-in one)")
+ return func() dbTarget {
+ sess := requireDBAPISession(*consoleFlag)
+ printUsing(stdout, sess)
+ acct := resolvedAccount(sess, acctRaw, acctIDRaw)
+ name := resolvedApp(sess, acct, strings.TrimSpace(*app), dbUsage)
+ h, err := resolveDBHost(context.Background(), sess, acct, name, strings.TrimSpace(*host))
+ if err != nil {
+ log.Fatalf("carlos db %s: %v", verb, err)
+ }
+ return dbTarget{sess: sess, acct: acct, app: name, host: h}
+ }
+}
+
+// resolveDBHost is --host, or the app's only instance when it has exactly
+// one. Anything else is refused with the hosts the app has: a transfer
+// names one database, and guessing which would be the wrong kind of help.
+func resolveDBHost(ctx context.Context, sess *apiSession, acct apiAcct, app, host string) (string, error) {
+ if host != "" {
+ return host, nil
+ }
+ recs, truncated, _, err := sess.instanceList(ctx, acct, app, "", "")
+ if err != nil {
+ return "", fmt.Errorf("list instances to pick a host: %w", err)
+ }
+ if len(recs) == 1 && !truncated {
+ return recs[0].Host, nil
+ }
+ if len(recs) == 0 {
+ return "", fmt.Errorf("%s has no instances — nothing to transfer", app)
+ }
+ hosts := make([]string, 0, len(recs))
+ for _, r := range recs {
+ hosts = append(hosts, r.Host)
+ }
+ sort.Strings(hosts)
+ return "", fmt.Errorf("--host is required: %s has %d instances: %s", app, len(recs), strings.Join(hosts, ", "))
+}
+
+// ---- the console client half ------------------------------------------------
+
+// apiDBView is the flat view every db door answers with
+// (internal/console/apiinstancedb.go's dbView).
+type apiDBView struct {
+ State string `json:"state"`
+ ID string `json:"id"`
+ Op string `json:"op"`
+ Step string `json:"step"`
+ Error string `json:"error"`
+ Bytes int64 `json:"bytes"`
+ Backup string `json:"backup"`
+ RequestedAt int64 `json:"requested_at"`
+ UpdatedAt int64 `json:"updated_at"`
+}
+
+type apiDBBackup struct {
+ ID string `json:"id"`
+ Bytes int64 `json:"bytes"`
+ CreatedAt time.Time `json:"created_at"`
+ Request string `json:"request"`
+}
+
+func dbPath(a apiAcct, app, host, tail string) string {
+ return apiPath(a, app, "/instances/"+url.PathEscape(host)+"/db"+tail)
+}
+
+func decodeDBView(status int, body []byte, want int) (apiDBView, error) {
+ if status != want {
+ return apiDBView{}, apiErrorFromBody(status, body)
+ }
+ var v apiDBView
+ if err := json.Unmarshal(body, &v); err != nil {
+ return apiDBView{}, fmt.Errorf("db: unreadable response: %w", err)
+ }
+ return v, nil
+}
+
+func (s *apiSession) dbStatus(ctx context.Context, a apiAcct, app, host string) (apiDBView, error) {
+ status, body, err := s.getJSON(ctx, dbPath(a, app, host, ""))
+ if err != nil {
+ return apiDBView{}, err
+ }
+ return decodeDBView(status, body, http.StatusOK)
+}
+
+func (s *apiSession) dbPullRequest(ctx context.Context, a apiAcct, app, host string) (apiDBView, error) {
+ status, body, err := s.postJSON(ctx, dbPath(a, app, host, "/pull"), map[string]string{})
+ if err != nil {
+ return apiDBView{}, err
+ }
+ return decodeDBView(status, body, http.StatusAccepted)
+}
+
+func (s *apiSession) dbClear(ctx context.Context, a apiAcct, app, host string) error {
+ status, body, err := s.deleteJSON(ctx, dbPath(a, app, host, ""))
+ if err != nil {
+ return err
+ }
+ if status != http.StatusOK {
+ return apiErrorFromBody(status, body)
+ }
+ return nil
+}
+
+func (s *apiSession) dbBackups(ctx context.Context, a apiAcct, app, host string) ([]apiDBBackup, error) {
+ status, body, err := s.getJSON(ctx, dbPath(a, app, host, "/backups"))
+ if err != nil {
+ return nil, err
+ }
+ if status != http.StatusOK {
+ return nil, apiErrorFromBody(status, body)
+ }
+ var res struct {
+ Backups []apiDBBackup `json:"backups"`
+ }
+ if err := json.Unmarshal(body, &res); err != nil {
+ return nil, fmt.Errorf("db backups: unreadable response: %w", err)
+ }
+ return res.Backups, nil
+}
+
+// dbPush streams the file up with the confirmation header. 202 is the
+// only success: the console has staged the bytes and written the request.
+func (s *apiSession) dbPush(ctx context.Context, a apiAcct, app, host string, body io.Reader, size int64) (apiDBView, error) {
+ res, err := s.c.doStream(ctx, http.MethodPut, dbPath(a, app, host, "/push"), body, size, map[string]string{
+ "Content-Type": dbSQLiteContentType,
+ dbConfirmHeader: host,
+ })
+ if err != nil {
+ return apiDBView{}, err
+ }
+ defer res.Body.Close()
+ raw, err := io.ReadAll(io.LimitReader(res.Body, 1<<20))
+ if err != nil {
+ return apiDBView{}, err
+ }
+ return decodeDBView(res.StatusCode, raw, http.StatusAccepted)
+}
+
+// dbDownload opens a streamed download of one staged database. The caller
+// closes the body; a non-200 is turned into the console's own sentence.
+func (s *apiSession) dbDownload(ctx context.Context, path string) (io.ReadCloser, int64, error) {
+ res, err := s.c.doStream(ctx, http.MethodGet, path, nil, -1, nil)
+ if err != nil {
+ return nil, 0, err
+ }
+ if res.StatusCode != http.StatusOK {
+ defer res.Body.Close()
+ raw, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20))
+ return nil, 0, apiErrorFromBody(res.StatusCode, raw)
+ }
+ return res.Body, res.ContentLength, nil
+}
+
+// ---- the local half ---------------------------------------------------------
+
+// dbPollInterval and dbPollTimeout bound the wait for the box: seconds
+// between polls (the console's instance budget is 60 a minute, shared with
+// everything else this token does), and ten minutes in all — a restore, a
+// backup and a replicate of a 256 MiB database on a busy box.
+const (
+ dbPollInterval = 2 * time.Second
+ dbPollTimeout = 10 * time.Minute
+)
+
+// pollDB polls the status door until transfer id is done or failed. A
+// failure is returned as an error carrying the box's sentence; a timeout
+// names the verb that keeps watching. progress gets one dot per poll so a
+// terminal shows the wait is alive.
+func pollDB(ctx context.Context, sess *apiSession, t dbTarget, id string, interval, timeout time.Duration, progress io.Writer) (apiDBView, error) {
+ deadline := time.Now().Add(timeout)
+ for {
+ v, err := sess.dbStatus(ctx, t.acct, t.app, t.host)
+ if err != nil {
+ return v, err
+ }
+ if v.ID != id {
+ return v, fmt.Errorf("the transfer was superseded on the console (now %s %s)", v.Op, v.State)
+ }
+ switch v.State {
+ case "done":
+ return v, nil
+ case "failed":
+ return v, fmt.Errorf("the box could not complete the %s: %s", v.Op, v.Error)
+ }
+ if time.Now().After(deadline) {
+ return v, fmt.Errorf("still %s after %s — the box has not finished; carlos db status --app %s --host %s keeps watching",
+ v.State, timeout.Round(time.Second), t.app, t.host)
+ }
+ if progress != nil {
+ fmt.Fprint(progress, ".")
+ }
+ select {
+ case <-ctx.Done():
+ return v, ctx.Err()
+ case <-time.After(interval):
+ }
+ }
+}
+
+// writeDownload copies r to path through a temp file in the same
+// directory and a rename, so a torn download never sits at the final name.
+// An existing file is refused unless force is set: a pull that silently
+// overwrote yesterday's copy is the one accident this verb must not have.
+func writeDownload(path string, r io.Reader, force bool) (int64, error) {
+ if _, err := os.Stat(path); err == nil && !force {
+ return 0, fmt.Errorf("%s already exists — pass --force to replace it, or --out <path>", path)
+ } else if err != nil && !errors.Is(err, os.ErrNotExist) {
+ return 0, err
+ }
+ dir := filepath.Dir(path)
+ tmp, err := os.CreateTemp(dir, "."+filepath.Base(path)+".*.part")
+ if err != nil {
+ return 0, err
+ }
+ tmpName := tmp.Name()
+ n, err := io.Copy(tmp, r)
+ if cerr := tmp.Close(); err == nil {
+ err = cerr
+ }
+ if err != nil {
+ _ = os.Remove(tmpName)
+ return 0, err
+ }
+ if err := os.Chmod(tmpName, 0o600); err != nil {
+ _ = os.Remove(tmpName)
+ return 0, err
+ }
+ if err := os.Rename(tmpName, path); err != nil {
+ _ = os.Remove(tmpName)
+ return 0, err
+ }
+ return n, nil
+}
+
+// prepareDBUpload proves the local file is a sound SQLite database and
+// writes a self-contained copy of it to a temp file: VACUUM INTO folds any
+// -wal content into one file, which is what the wire carries and the box
+// checks. The caller removes the returned path.
+func prepareDBUpload(path string) (tmpPath string, size int64, err error) {
+ fi, err := os.Stat(path)
+ if err != nil {
+ return "", 0, err
+ }
+ if fi.IsDir() {
+ return "", 0, fmt.Errorf("%s is a directory", path)
+ }
+ db, err := sql.Open("sqlite", "file:"+path+"?mode=ro")
+ if err != nil {
+ return "", 0, err
+ }
+ defer db.Close()
+ db.SetMaxOpenConns(1)
+ if _, err := db.Exec("PRAGMA busy_timeout=5000"); err != nil {
+ return "", 0, fmt.Errorf("%s is not a SQLite database: %v", path, err)
+ }
+ var res string
+ if err := db.QueryRow("PRAGMA integrity_check").Scan(&res); err != nil {
+ return "", 0, fmt.Errorf("%s is not a SQLite database: %v", path, err)
+ }
+ if res != "ok" {
+ return "", 0, fmt.Errorf("%s fails integrity_check: %s", path, res)
+ }
+ var tables int
+ if err := db.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table'").Scan(&tables); err != nil {
+ return "", 0, err
+ }
+ if tables == 0 {
+ return "", 0, fmt.Errorf("%s has no tables — refusing to push an empty database", path)
+ }
+ // VACUUM INTO wants a name that does not exist yet; CreateTemp reserves
+ // one in a directory we can write, then gives it back.
+ tmp, err := os.CreateTemp("", "carlos-db-push-*.db")
+ if err != nil {
+ return "", 0, err
+ }
+ tmpPath = tmp.Name()
+ tmp.Close()
+ _ = os.Remove(tmpPath)
+ if _, err := db.Exec("VACUUM INTO ?", tmpPath); err != nil {
+ _ = os.Remove(tmpPath)
+ return "", 0, fmt.Errorf("could not write a self-contained copy of %s: %v", path, err)
+ }
+ if err := os.Chmod(tmpPath, 0o600); err != nil {
+ _ = os.Remove(tmpPath)
+ return "", 0, err
+ }
+ out, err := os.Stat(tmpPath)
+ if err != nil {
+ _ = os.Remove(tmpPath)
+ return "", 0, err
+ }
+ if out.Size() > dbTransferMaxBytes {
+ _ = os.Remove(tmpPath)
+ return "", 0, fmt.Errorf("%s is %d MiB once packed; transfers are capped at %d MiB", path, out.Size()>>20, dbTransferMaxBytes>>20)
+ }
+ return tmpPath, out.Size(), nil
+}
+
+// confirmDBPush is the confirmation: --yes, or the host typed back at a
+// terminal. A non-terminal without --yes is refused — a script that
+// replaces a database says so in its own text.
+func confirmDBPush(host string, yes, interactive bool, out io.Writer) error {
+ if yes {
+ return nil
+ }
+ if !interactive {
+ return errors.New("a push replaces the instance's whole database and needs confirmation: pass --yes, or run it at a terminal and type the host")
+ }
+ warnf(out, "this replaces the database of %s. The instance is held out of service for the swap, and the current database is backed up first.\n", host)
+ typed := promptLine(out, "type the host to confirm: ")
+ if typed != host {
+ return fmt.Errorf("confirmation did not match %s — nothing was pushed", host)
+ }
+ return nil
+}
+
+// dbSize renders a byte count with the DB column's own formatter
+// (humanBytes, cli.go), so `carlos db` and `carlos instances list` agree on
+// what a megabyte looks like.
+func dbSize(n int64) string { return humanBytes(&n) }
+
+// ---- the verbs --------------------------------------------------------------
+
+const dbPullUsage = `usage: carlos db pull --app <a> [--host <h>] [--out <path> | <path>]
+ [--backup <id>|latest] [--force]
+
+Copies the instance's replicated database to this machine — by default to
+./<host>.db. The copy is the replica: byte-final for a sleeping instance, at
+most the sync interval behind a running one. --backup fetches one of the
+backups an earlier push saved instead ("latest" for the newest). An existing
+file is never overwritten without --force.
+
+`
+
+func runDBPull(args []string) {
+ fs := flag.NewFlagSet("db pull", flag.ExitOnError)
+ out := fs.String("out", "", "where to write the file (default ./<host>.db)")
+ backup := fs.String("backup", "", "download this backup instead of a fresh pull (an id from carlos db backups, or latest)")
+ force := fs.Bool("force", false, "replace an existing file at the destination")
+ resolve := dbFlags(fs, "pull")
+ fs.Usage = flagUsage(fs, dbPullUsage)
+ fs.Parse(args)
+ if rest := fs.Args(); len(rest) == 1 && *out == "" {
+ *out = rest[0]
+ } else if len(rest) > 0 {
+ log.Fatalf("carlos db pull: unexpected arguments %v\n%s", rest, dbPullUsage)
+ }
+ t := resolve()
+ ctx := context.Background()
+ dest := *out
+ if dest == "" {
+ dest = t.host + ".db"
+ }
+ if !*force {
+ if _, err := os.Stat(dest); err == nil {
+ log.Fatalf("carlos db pull: %s already exists — pass --force to replace it, or --out <path>", dest)
+ }
+ }
+
+ var body io.ReadCloser
+ var err error
+ if *backup != "" {
+ id := *backup
+ if id == "latest" {
+ list, err := t.sess.dbBackups(ctx, t.acct, t.app, t.host)
+ if err != nil {
+ log.Fatalf("carlos db pull: %v", err)
+ }
+ if len(list) == 0 {
+ log.Fatalf("carlos db pull: %s has no backups yet — a push makes one", t.host)
+ }
+ id = list[0].ID
+ }
+ fmt.Fprintf(stdout, "downloading backup %s of %s\n", id, t.host)
+ body, _, err = t.sess.dbDownload(ctx, dbPath(t.acct, t.app, t.host, "/backups/"+url.PathEscape(id)))
+ if err != nil {
+ log.Fatalf("carlos db pull: %v", err)
+ }
+ } else {
+ v, err := t.sess.dbPullRequest(ctx, t.acct, t.app, t.host)
+ if err != nil {
+ log.Fatalf("carlos db pull: %v", err)
+ }
+ fmt.Fprintf(stdout, "pull of %s requested (%s) — waiting for the box", t.host, v.ID)
+ v, err = pollDB(ctx, t.sess, t, v.ID, dbPollInterval, dbPollTimeout, stdout)
+ fmt.Fprintln(stdout)
+ if err != nil {
+ log.Fatalf("carlos db pull: %v", err)
+ }
+ body, _, err = t.sess.dbDownload(ctx, dbPath(t.acct, t.app, t.host, "/file?id="+url.QueryEscape(v.ID)))
+ if err != nil {
+ log.Fatalf("carlos db pull: %v", err)
+ }
+ }
+ defer body.Close()
+ n, err := writeDownload(dest, body, *force)
+ if err != nil {
+ log.Fatalf("carlos db pull: %v", err)
+ }
+ if *backup == "" {
+ // The staged snapshot has served its purpose. Best-effort: a
+ // leftover object is storage the next pull clears anyway.
+ if err := t.sess.dbClear(ctx, t.acct, t.app, t.host); err != nil {
+ mutedf(stdout, " (could not clear the staged snapshot on the console: %v)\n", err)
+ }
+ }
+ fmt.Fprintf(stdout, "%s %s → %s (%s)\n", colorFor(stdout).ok("pulled"), t.host, dest, dbSize(n))
+}
+
+const dbPushUsage = `usage: carlos db push --app <a> [--host <h>] <path> [--yes]
+
+Replaces the instance's database with the SQLite file at <path>. The file is
+checked and packed into a self-contained copy here, uploaded through the
+console, and the instance's box then holds the route out of service, backs up
+the current database, replaces the replica with the upload, and drops its
+local copy so the next wake restores the pushed file. The backup id is
+printed at the end; carlos db pull --backup <id> fetches it.
+
+A push needs confirmation: --yes, or the host typed at a terminal.
+
+`
+
+func runDBPush(args []string) {
+ fs := flag.NewFlagSet("db push", flag.ExitOnError)
+ yes := fs.Bool("yes", false, "confirm the replacement without a prompt")
+ resolve := dbFlags(fs, "push")
+ fs.Usage = flagUsage(fs, dbPushUsage)
+ fs.Parse(args)
+ rest := fs.Args()
+ if len(rest) != 1 {
+ log.Fatalf("carlos db push: name exactly one file to push\n%s", dbPushUsage)
+ }
+ src := rest[0]
+ // The local checks come BEFORE the console is asked anything: a file
+ // that is not a database costs nothing on the wire.
+ tmp, size, err := prepareDBUpload(src)
+ if err != nil {
+ log.Fatalf("carlos db push: %v", err)
+ }
+ defer os.Remove(tmp)
+ t := resolve()
+ if err := confirmDBPush(t.host, *yes, interactiveTerminal(), stdout); err != nil {
+ log.Fatalf("carlos db push: %v", err)
+ }
+ f, err := os.Open(tmp)
+ if err != nil {
+ log.Fatalf("carlos db push: %v", err)
+ }
+ ctx := context.Background()
+ fmt.Fprintf(stdout, "uploading %s (%s) to %s\n", src, dbSize(size), t.host)
+ v, err := t.sess.dbPush(ctx, t.acct, t.app, t.host, f, size)
+ f.Close()
+ if err != nil {
+ log.Fatalf("carlos db push: %v", err)
+ }
+ fmt.Fprintf(stdout, "push requested (%s) — the box is swapping the database", v.ID)
+ v, err = pollDB(ctx, t.sess, t, v.ID, dbPollInterval, dbPollTimeout, stdout)
+ fmt.Fprintln(stdout)
+ if err != nil {
+ if v.Backup != "" {
+ mutedf(stdout, " the backup taken before the attempt is %s: carlos db pull --app %s --host %s --backup %s\n", v.Backup, t.app, t.host, v.Backup)
+ }
+ log.Fatalf("carlos db push: %v", err)
+ }
+ fmt.Fprintf(stdout, "%s %s — the next wake of %s serves the pushed database\n", colorFor(stdout).ok("pushed"), src, t.host)
+ if v.Backup != "" {
+ fmt.Fprintf(stdout, " backup of the previous database: %s\n", v.Backup)
+ mutedf(stdout, " restore it with: carlos db pull --app %s --host %s --backup %s\n", t.app, t.host, v.Backup)
+ } else {
+ mutedf(stdout, " the instance had no replicated database before this push, so there is no backup\n")
+ }
+}
+
+const dbStatusUsage = `usage: carlos db status --app <a> [--host <h>]
+
+Prints the last transfer requested for the instance: its verb, state, step,
+size, backup id and error, if any.
+
+`
+
+func runDBStatus(args []string) {
+ fs := flag.NewFlagSet("db status", flag.ExitOnError)
+ resolve := dbFlags(fs, "status")
+ fs.Usage = flagUsage(fs, dbStatusUsage)
+ fs.Parse(args)
+ t := resolve()
+ v, err := t.sess.dbStatus(context.Background(), t.acct, t.app, t.host)
+ if err != nil {
+ log.Fatalf("carlos db status: %v", err)
+ }
+ fmt.Fprint(stdout, formatDBStatus(t.host, v))
+}
+
+func formatDBStatus(host string, v apiDBView) string {
+ if v.State == "none" {
+ return fmt.Sprintf("%s: no database transfer on record\n", host)
+ }
+ var b strings.Builder
+ fmt.Fprintf(&b, "%s: %s %s (%s)\n", host, v.Op, v.State, v.ID)
+ if v.Step != "" {
+ fmt.Fprintf(&b, " step: %s\n", v.Step)
+ }
+ if v.Bytes > 0 {
+ fmt.Fprintf(&b, " size: %s\n", dbSize(v.Bytes))
+ }
+ if v.Backup != "" {
+ fmt.Fprintf(&b, " backup: %s\n", v.Backup)
+ }
+ if v.Error != "" {
+ fmt.Fprintf(&b, " error: %s\n", v.Error)
+ }
+ if v.RequestedAt > 0 {
+ fmt.Fprintf(&b, " requested: %s\n", time.Unix(v.RequestedAt, 0).UTC().Format(time.RFC3339))
+ }
+ if v.UpdatedAt > 0 {
+ fmt.Fprintf(&b, " updated: %s\n", time.Unix(v.UpdatedAt, 0).UTC().Format(time.RFC3339))
+ }
+ return b.String()
+}
+
+const dbBackupsUsage = `usage: carlos db backups --app <a> [--host <h>]
+
+Lists the backups earlier pushes saved for the instance, newest first. The
+box keeps the newest five. carlos db pull --backup <id> downloads one.
+
+`
+
+func runDBBackups(args []string) {
+ fs := flag.NewFlagSet("db backups", flag.ExitOnError)
+ resolve := dbFlags(fs, "backups")
+ fs.Usage = flagUsage(fs, dbBackupsUsage)
+ fs.Parse(args)
+ t := resolve()
+ list, err := t.sess.dbBackups(context.Background(), t.acct, t.app, t.host)
+ if err != nil {
+ log.Fatalf("carlos db backups: %v", err)
+ }
+ fmt.Fprint(stdout, formatDBBackups(t.host, list))
+}
+
+func formatDBBackups(host string, list []apiDBBackup) string {
+ if len(list) == 0 {
+ return fmt.Sprintf("%s: no backups yet — a push makes one\n", host)
+ }
+ var b strings.Builder
+ fmt.Fprintf(&b, "%-26s %10s %s\n", "BACKUP", "SIZE", "TAKEN")
+ for _, bk := range list {
+ when := "-"
+ if !bk.CreatedAt.IsZero() {
+ when = bk.CreatedAt.UTC().Format(time.RFC3339)
+ }
+ fmt.Fprintf(&b, "%-26s %10s %s\n", bk.ID, dbSize(bk.Bytes), when)
+ }
+ return b.String()
+}
diff --git a/db_cli_test.go b/db_cli_test.go
new file mode 100644
index 000000000..2036a98f3
--- /dev/null
+++ b/db_cli_test.go
@@ -0,0 +1,339 @@
+package main
+
+import (
+ "bytes"
+ "context"
+ "database/sql"
+ "encoding/json"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+)
+
+// dbTestServer is a console that answers the instances list and the db
+// doors from a script: the list body, a queue of status views, and a
+// record of what the push door received.
+type dbTestServer struct {
+ instances string
+ statuses []apiDBView
+ pushed []byte
+ pushHdr http.Header
+ cleared int
+ pullCalls int
+}
+
+func (d *dbTestServer) handler(t *testing.T) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch {
+ case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/instances"):
+ w.Header().Set("Content-Type", "application/json")
+ io.WriteString(w, d.instances)
+ case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/db"):
+ v := d.statuses[0]
+ if len(d.statuses) > 1 {
+ d.statuses = d.statuses[1:]
+ }
+ w.Header().Set("Content-Type", "application/json")
+ json.NewEncoder(w).Encode(v)
+ case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/db/pull"):
+ d.pullCalls++
+ w.WriteHeader(http.StatusAccepted)
+ json.NewEncoder(w).Encode(apiDBView{State: "requested", ID: "0123456789abcdef", Op: "pull"})
+ case r.Method == http.MethodPut && strings.HasSuffix(r.URL.Path, "/db/push"):
+ d.pushHdr = r.Header.Clone()
+ body, _ := io.ReadAll(r.Body)
+ d.pushed = body
+ w.WriteHeader(http.StatusAccepted)
+ json.NewEncoder(w).Encode(apiDBView{State: "requested", ID: "fedcba9876543210", Op: "push", Bytes: int64(len(body))})
+ case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/db/file"):
+ w.Header().Set("Content-Type", dbSQLiteContentType)
+ w.Write([]byte("snapshot-bytes"))
+ case r.Method == http.MethodDelete && strings.HasSuffix(r.URL.Path, "/db"):
+ d.cleared++
+ json.NewEncoder(w).Encode(apiDBView{State: "none"})
+ default:
+ t.Errorf("unexpected request %s %s", r.Method, r.URL)
+ w.WriteHeader(http.StatusTeapot)
+ }
+ })
+}
+
+// --host may be omitted only when the app has exactly one instance;
+// otherwise the refusal names the hosts.
+func TestDBResolveHost(t *testing.T) {
+ cases := []struct {
+ name, list, host, want, wantErr string
+ }{
+ {"given", `{"instances":[]}`, "a.go.example.com", "a.go.example.com", ""},
+ {"one", `{"instances":[{"host":"only.go.example.com"}]}`, "", "only.go.example.com", ""},
+ {"none", `{"instances":[]}`, "", "", "no instances"},
+ {"many", `{"instances":[{"host":"b.go.example.com"},{"host":"a.go.example.com"}]}`, "", "", "--host is required: app has 2 instances: a.go.example.com, b.go.example.com"},
+ {"truncated", `{"instances":[{"host":"x.go.example.com"}],"truncated":true}`, "", "", "--host is required"},
+ }
+ for _, c := range cases {
+ t.Run(c.name, func(t *testing.T) {
+ d := &dbTestServer{instances: c.list}
+ ts := httptest.NewServer(d.handler(t))
+ defer ts.Close()
+ got, err := resolveDBHost(context.Background(), newTestSession(ts.URL), apiAcct{Seg: "bab"}, "app", c.host)
+ if c.wantErr != "" {
+ if err == nil || !strings.Contains(err.Error(), c.wantErr) {
+ t.Fatalf("err = %v, want %q", err, c.wantErr)
+ }
+ return
+ }
+ if err != nil || got != c.want {
+ t.Fatalf("host = %q, %v; want %q", got, err, c.want)
+ }
+ })
+ }
+}
+
+// An existing file is refused without --force; the write goes through a
+// temp file and a rename, so nothing partial ever sits at the final name.
+func TestDBWriteDownloadRefusesOverwrite(t *testing.T) {
+ dir := t.TempDir()
+ dest := filepath.Join(dir, "a.db")
+ n, err := writeDownload(dest, strings.NewReader("first"), false)
+ if err != nil || n != 5 {
+ t.Fatalf("first write: %d %v", n, err)
+ }
+ if _, err := writeDownload(dest, strings.NewReader("second"), false); err == nil || !strings.Contains(err.Error(), "--force") {
+ t.Fatalf("overwrite without --force = %v, want a refusal naming --force", err)
+ }
+ if got, _ := os.ReadFile(dest); string(got) != "first" {
+ t.Fatalf("the refused write touched the file: %q", got)
+ }
+ if _, err := writeDownload(dest, strings.NewReader("second"), true); err != nil {
+ t.Fatal(err)
+ }
+ if got, _ := os.ReadFile(dest); string(got) != "second" {
+ t.Fatalf("after --force: %q", got)
+ }
+ ents, _ := os.ReadDir(dir)
+ if len(ents) != 1 {
+ t.Fatalf("temp files left behind: %v", ents)
+ }
+ // A torn download never lands.
+ if _, err := writeDownload(filepath.Join(dir, "torn.db"), &failingReader{}, false); err == nil {
+ t.Fatal("a failed copy reported success")
+ }
+ if _, err := os.Stat(filepath.Join(dir, "torn.db")); !os.IsNotExist(err) {
+ t.Fatal("a torn download sits at the final name")
+ }
+ ents, _ = os.ReadDir(dir)
+ if len(ents) != 1 {
+ t.Fatalf("temp files left behind after a torn download: %v", ents)
+ }
+}
+
+type failingReader struct{}
+
+func (failingReader) Read([]byte) (int, error) { return 0, io.ErrUnexpectedEOF }
+
+// The confirmation: --yes passes silently, a non-terminal without it is
+// refused, and at a terminal the host must be typed back exactly.
+func TestDBConfirmPush(t *testing.T) {
+ var out bytes.Buffer
+ if err := confirmDBPush("a.go.example.com", true, false, &out); err != nil || out.Len() != 0 {
+ t.Fatalf("--yes: %v (printed %q)", err, out.String())
+ }
+ if err := confirmDBPush("a.go.example.com", false, false, &out); err == nil || !strings.Contains(err.Error(), "--yes") {
+ t.Fatalf("non-interactive without --yes = %v, want a refusal naming --yes", err)
+ }
+ prev := chooserIn
+ t.Cleanup(func() { chooserIn = prev })
+ chooserIn = strings.NewReader("a.go.example.com\n")
+ out.Reset()
+ if err := confirmDBPush("a.go.example.com", false, true, &out); err != nil {
+ t.Fatalf("typed host: %v", err)
+ }
+ if !strings.Contains(out.String(), "replaces the database of a.go.example.com") {
+ t.Fatalf("the prompt did not say what it replaces: %q", out.String())
+ }
+ chooserIn = strings.NewReader("b.go.example.com\n")
+ if err := confirmDBPush("a.go.example.com", false, true, &out); err == nil || !strings.Contains(err.Error(), "nothing was pushed") {
+ t.Fatalf("wrong host typed = %v, want a refusal", err)
+ }
+ chooserIn = strings.NewReader("")
+ if err := confirmDBPush("a.go.example.com", false, true, &out); err == nil {
+ t.Fatal("EOF at the prompt confirmed a push")
+ }
+}
+
+// prepareDBUpload proves the file and packs it whole: rows written to the
+// WAL of the source are in the single-file copy; a non-database is refused.
+func TestDBPrepareUpload(t *testing.T) {
+ src := filepath.Join(t.TempDir(), "src.db")
+ db, err := sql.Open("sqlite", src)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := db.Exec(`PRAGMA journal_mode=WAL`); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := db.Exec(`CREATE TABLE t (id INTEGER PRIMARY KEY, body TEXT)`); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := db.Exec(`INSERT INTO t (body) VALUES ('in-the-wal')`); err != nil {
+ t.Fatal(err)
+ }
+ // Left open: the WAL is not checkpointed into the main file.
+ defer db.Close()
+ tmp, size, err := prepareDBUpload(src)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer os.Remove(tmp)
+ if size <= 0 {
+ t.Fatalf("size = %d", size)
+ }
+ if _, err := os.Stat(tmp + "-wal"); !os.IsNotExist(err) {
+ t.Fatal("the packed copy has a WAL beside it")
+ }
+ packed, err := sql.Open("sqlite", "file:"+tmp+"?mode=ro")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer packed.Close()
+ var body string
+ if err := packed.QueryRow(`SELECT body FROM t`).Scan(&body); err != nil || body != "in-the-wal" {
+ t.Fatalf("packed copy: %q %v — the WAL's rows were not folded in", body, err)
+ }
+
+ junk := filepath.Join(t.TempDir(), "junk.db")
+ if err := os.WriteFile(junk, []byte("this is not a database, however you look at it, no"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if _, _, err := prepareDBUpload(junk); err == nil {
+ t.Fatal("a non-database was accepted")
+ }
+ empty := filepath.Join(t.TempDir(), "empty.db")
+ edb, _ := sql.Open("sqlite", empty)
+ edb.Exec(`PRAGMA user_version=1`)
+ edb.Close()
+ if _, _, err := prepareDBUpload(empty); err == nil || !strings.Contains(err.Error(), "no tables") {
+ t.Fatalf("an empty database = %v, want the no-tables refusal", err)
+ }
+ if _, _, err := prepareDBUpload(t.TempDir()); err == nil {
+ t.Fatal("a directory was accepted")
+ }
+}
+
+// The push wire shape: PUT, the SQLite content type, the confirmation
+// header naming the host, the bytes streamed whole, a 202 view back.
+func TestDBPushSendsConfirmHeaderAndBody(t *testing.T) {
+ d := &dbTestServer{}
+ ts := httptest.NewServer(d.handler(t))
+ defer ts.Close()
+ body := bytes.Repeat([]byte("x"), 4096)
+ v, err := newTestSession(ts.URL).dbPush(context.Background(), apiAcct{Seg: "bab"}, "app", "a.go.example.com", bytes.NewReader(body), int64(len(body)))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if v.State != "requested" || v.Op != "push" || v.Bytes != 4096 {
+ t.Fatalf("view = %+v", v)
+ }
+ if !bytes.Equal(d.pushed, body) {
+ t.Fatalf("server got %d bytes", len(d.pushed))
+ }
+ if d.pushHdr.Get(dbConfirmHeader) != "a.go.example.com" || d.pushHdr.Get("Content-Type") != dbSQLiteContentType || d.pushHdr.Get("Authorization") != "Bearer cli-token" {
+ t.Fatalf("push headers: %v", d.pushHdr)
+ }
+ if d.pushHdr.Get("Content-Length") != "4096" {
+ t.Fatalf("Content-Length = %q, want the file's size (a chunked upload is what the door's cap cannot see coming)", d.pushHdr.Get("Content-Length"))
+ }
+ // A refusal comes back as the console's own sentence.
+ refuse := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(http.StatusConflict)
+ io.WriteString(w, `{"error":"a db pull is still in flight"}`)
+ }))
+ defer refuse.Close()
+ if _, err := newTestSession(refuse.URL).dbPush(context.Background(), apiAcct{Seg: "bab"}, "app", "a.go.example.com", bytes.NewReader(body), int64(len(body))); err == nil || err.Error() != "a db pull is still in flight" {
+ t.Fatalf("refusal = %v", err)
+ }
+}
+
+// pollDB waits through requested, returns on done, and turns failed and a
+// superseding transfer into errors.
+func TestDBPoll(t *testing.T) {
+ d := &dbTestServer{statuses: []apiDBView{
+ {State: "requested", ID: "0123456789abcdef", Op: "pull"},
+ {State: "requested", ID: "0123456789abcdef", Op: "pull"},
+ {State: "done", ID: "0123456789abcdef", Op: "pull", Bytes: 14},
+ }}
+ ts := httptest.NewServer(d.handler(t))
+ defer ts.Close()
+ sess := newTestSession(ts.URL)
+ tgt := dbTarget{sess: sess, acct: apiAcct{Seg: "bab"}, app: "app", host: "a.go.example.com"}
+ var progress bytes.Buffer
+ v, err := pollDB(context.Background(), sess, tgt, "0123456789abcdef", time.Millisecond, time.Second, &progress)
+ if err != nil || v.State != "done" || v.Bytes != 14 {
+ t.Fatalf("poll = %+v %v", v, err)
+ }
+ if progress.String() != ".." {
+ t.Fatalf("progress = %q, want one dot per wait", progress.String())
+ }
+ d.statuses = []apiDBView{{State: "failed", ID: "0123456789abcdef", Op: "pull", Error: "the instance has no replicated database yet"}}
+ if _, err := pollDB(context.Background(), sess, tgt, "0123456789abcdef", time.Millisecond, time.Second, nil); err == nil || !strings.Contains(err.Error(), "no replicated database") {
+ t.Fatalf("failed poll = %v", err)
+ }
+ d.statuses = []apiDBView{{State: "requested", ID: "ffffffffffffffff", Op: "push"}}
+ if _, err := pollDB(context.Background(), sess, tgt, "0123456789abcdef", time.Millisecond, time.Second, nil); err == nil || !strings.Contains(err.Error(), "superseded") {
+ t.Fatalf("superseded poll = %v", err)
+ }
+ d.statuses = []apiDBView{{State: "requested", ID: "0123456789abcdef", Op: "pull"}}
+ if _, err := pollDB(context.Background(), sess, tgt, "0123456789abcdef", time.Millisecond, 5*time.Millisecond, nil); err == nil || !strings.Contains(err.Error(), "carlos db status") {
+ t.Fatalf("timed-out poll = %v, want the status verb named", err)
+ }
+}
+
+// The download stream: 200 hands the body back for the caller to copy; a
+// non-200 is the console's sentence.
+func TestDBDownload(t *testing.T) {
+ d := &dbTestServer{}
+ ts := httptest.NewServer(d.handler(t))
+ defer ts.Close()
+ sess := newTestSession(ts.URL)
+ body, n, err := sess.dbDownload(context.Background(), dbPath(apiAcct{Seg: "bab"}, "app", "a.go.example.com", "/file?id=0123456789abcdef"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer body.Close()
+ got, _ := io.ReadAll(body)
+ if string(got) != "snapshot-bytes" || n != 14 {
+ t.Fatalf("download = %q (%d)", got, n)
+ }
+ refuse := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(http.StatusConflict)
+ io.WriteString(w, `{"error":"the pull is still in flight"}`)
+ }))
+ defer refuse.Close()
+ if _, _, err := newTestSession(refuse.URL).dbDownload(context.Background(), "/x"); err == nil || err.Error() != "the pull is still in flight" {
+ t.Fatalf("refusal = %v", err)
+ }
+}
+
+func TestDBFormatting(t *testing.T) {
+ if got := formatDBStatus("a.go.example.com", apiDBView{State: "none"}); !strings.Contains(got, "no database transfer") {
+ t.Fatalf("none = %q", got)
+ }
+ got := formatDBStatus("a.go.example.com", apiDBView{State: "failed", Op: "push", ID: "fedcba9876543210", Step: "backed-up", Backup: "20260929-120000-0badf00d", Error: "boom", RequestedAt: 1_800_000_000})
+ for _, want := range []string{"push failed (fedcba9876543210)", "step: backed-up", "backup: 20260929-120000-0badf00d", "error: boom", "requested: 2027-01-15"} {
+ if !strings.Contains(got, want) {
+ t.Fatalf("status missing %q:\n%s", want, got)
+ }
+ }
+ if got := formatDBBackups("a.go.example.com", nil); !strings.Contains(got, "no backups yet") {
+ t.Fatalf("empty backups = %q", got)
+ }
+ got = formatDBBackups("a.go.example.com", []apiDBBackup{{ID: "20260929-120000-0badf00d", Bytes: 2 << 20, CreatedAt: time.Date(2026, 9, 29, 12, 0, 0, 0, time.UTC)}})
+ if !strings.Contains(got, "20260929-120000-0badf00d") || !strings.Contains(got, "2.0MB") || !strings.Contains(got, "2026-09-29T12:00:00Z") {
+ t.Fatalf("backups = %q", got)
+ }
+}
diff --git a/docs/site/cli.md b/docs/site/cli.md
index c173c7929..c700396ff 100644
--- a/docs/site/cli.md
+++ b/docs/site/cli.md
@@ -660,6 +660,55 @@ it lists that box's own registry routes, backing and owning unit included.
Off the box, [carlos routes](/docs/cli#carlos-routes) answers the same
question through the console.
+### carlos db
+
+Copies an instance's SQLite database to your machine, or replaces it with a
+file from your machine — with a login alone, no box and no bucket access.
+
+```sh
+carlos db pull --app hello
+carlos db pull --app hello --host hello.bab.oncarlos.com --out ./hello-today.db
+carlos db push --app hello ./hello-fixed.db
+carlos db status --app hello
+carlos db backups --app hello
+```
+
+`--host` may be left out when the app has exactly one instance; with more,
+the command refuses and lists them. It works on hibernating, platform-managed
+instances only — the class whose database the platform itself keeps
+replicated — and never on an addr-backed, static, pooled or fleet-placed
+route. A transfer is capped at 256 MiB in either direction.
+
+**pull** asks the console for a snapshot; the instance's box restores the
+replicated copy, checks that it opens, and stages it for the console to
+serve, and the command downloads it to `./<host>.db` (or `--out <path>`, or a
+bare path argument). An existing file is never overwritten without
+`--force`. The snapshot is the *replica*: byte-final for a sleeping
+instance, at most the replication interval behind a running one. The
+instance is never woken or stopped for a pull.
+
+**push** is downtime, deliberately. The file is checked here first
+(`integrity_check`, then packed with `VACUUM INTO` so any write-ahead log is
+folded in), uploaded through the console, and the instance's box then holds
+the route out of service, stops the process if it is running, backs up the
+current database, replaces the replica with the upload, and drops its local
+copy — so the next request that wakes the instance restores exactly the
+pushed file. It needs confirmation: `--yes`, or the host typed back at a
+terminal. A script without `--yes` is refused rather than prompted.
+
+Every push keeps a backup of the database it replaced. The command prints
+the backup id at the end, `carlos db backups` lists them (the newest five are
+kept), and `carlos db pull --backup <id>` — or `--backup latest` — downloads
+one, which is also how a push is undone: pull the backup, push it back. A
+push of an instance that had no replicated database yet has nothing to back
+up and says so.
+
+A push is refused while a residence move is in flight, and a move is refused
+while a transfer is; a replica instance can be pulled but not pushed (push to
+the home it mirrors). `carlos db status` shows the last transfer on the
+record — verb, state, step, size, backup id, and the box's reason if it
+failed.
+
### carlos sidecars
A sidecar is a member that nothing can dial: a process the platform runs for
diff --git a/internal/console/apiinstancedb.go b/internal/console/apiinstancedb.go
new file mode 100644
index 000000000..cabb970f8
--- /dev/null
+++ b/internal/console/apiinstancedb.go
@@ -0,0 +1,362 @@
+package console
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "log"
+ "net/http"
+ "os"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/carlosframework/platform/internal/provision"
+)
+
+// The database transfer doors (`carlos db pull` / `carlos db push`): a
+// member with no box and no bucket access copies an instance's SQLite
+// database out, or a replacement in. The console writes REQUESTS onto the
+// record and stages bytes in the control bucket; the box that owns the
+// route does everything that touches data (internal/host's db worker) and
+// writes the outcome back. Seven verbs under one prefix:
+//
+// GET .../instances/{host}/db the flat state view
+// POST .../instances/{host}/db/pull request a snapshot
+// PUT .../instances/{host}/db/push upload a replacement
+// GET .../instances/{host}/db/file?id=<id> download a DONE pull
+// GET .../instances/{host}/db/backups list pre-push backups
+// GET .../instances/{host}/db/backups/{id} download one
+// DELETE .../instances/{host}/db clear a done/failed op
+//
+// Every door is registered with the ZERO role set: the payload is a whole
+// customer database, and a credential the app itself holds must not be
+// able to walk that out of (or into) the platform. A person's membership
+// is the only authority here, which is the same cut the door-key and
+// instance-token doors take.
+
+// dbView is the flat answer every db door speaks: "none" is a real state
+// so a poller needs no 404 branch; timestamps are unix seconds like the
+// move view's.
+type dbView struct {
+ State string `json:"state"`
+ ID string `json:"id,omitempty"`
+ Op string `json:"op,omitempty"`
+ Step string `json:"step,omitempty"`
+ Error string `json:"error,omitempty"`
+ Bytes int64 `json:"bytes,omitempty"`
+ Backup string `json:"backup,omitempty"`
+ RequestedAt int64 `json:"requested_at,omitempty"`
+ UpdatedAt int64 `json:"updated_at,omitempty"`
+}
+
+func dbViewOf(rec *provision.Record) dbView {
+ if rec == nil || rec.DB == nil {
+ return dbView{State: "none"}
+ }
+ d := rec.DB
+ return dbView{State: d.State, ID: d.ID, Op: d.Op, Step: d.Step, Error: d.Error, Bytes: d.Bytes,
+ Backup: d.Backup, RequestedAt: d.RequestedAt.Unix(), UpdatedAt: d.UpdatedAt.Unix()}
+}
+
+// dbGate is the family's front half: the instance gate (principal, budget,
+// bucket), the byte-identical 404, the class rail (execInstanceGate), and
+// the stage key builder — which re-checks the host shape, so no key is ever
+// built for a host the gate did not pass.
+func (s *Server) dbGate(w http.ResponseWriter, r *http.Request) (principal string, acct *Account, app, host string, rec *provision.Record, stage provision.DBStage, ok bool) {
+ principal, acct, app, host, rec, ok = s.execInstanceGate(w, r, "db push/pull")
+ if !ok {
+ return "", nil, "", "", nil, provision.DBStage{}, false
+ }
+ stage, err := provision.NewDBStage(acct.Sqid, app, host)
+ if err != nil {
+ http.NotFound(w, r)
+ return "", nil, "", "", nil, provision.DBStage{}, false
+ }
+ return principal, acct, app, host, rec, stage, true
+}
+
+// dbSQLiteContentType is the media type a push body carries and a download
+// is served as — IANA's registered type for SQLite 3 files.
+const dbSQLiteContentType = "application/vnd.sqlite3"
+
+// dbConfirmHeader is the push door's typed confirmation: the CLI sends the
+// host it is about to replace, and the door refuses a request whose header
+// does not name {host}. A replacement is a whole database gone; the
+// confirmation is the member's, restated by the client, not inferred.
+const dbConfirmHeader = "X-Carlos-Confirm-Host"
+
+func (s *Server) handleAPIInstanceDBStatus(w http.ResponseWriter, r *http.Request) {
+ _, _, _, _, rec, _, ok := s.dbGate(w, r)
+ if !ok {
+ return
+ }
+ writeJSON(w, http.StatusOK, dbViewOf(rec))
+}
+
+// dbTransition applies one guarded transition and maps its outcomes the
+// way moveTransition does: ErrDBConflict → 409 with the transition's own
+// sentence, success → the flat view + a serial bump.
+func (s *Server) dbTransition(w http.ResponseWriter, r *http.Request, acct *Account, app, host string, mut func(*provision.Record) error, okStatus int) (*provision.Record, bool) {
+ ctx := r.Context()
+ rec, err := provision.SetRecordDB(ctx, s.cfg.Releases, acct.Sqid, app, host, mut)
+ if errors.Is(err, provision.ErrDBConflict) {
+ jsonError(w, http.StatusConflict, strings.TrimPrefix(err.Error(), provision.ErrDBConflict.Error()+": "))
+ return nil, false
+ }
+ if err != nil {
+ log.Printf("console: api instance db %s/%s/%s: %v", acct.Sqid, app, host, err)
+ jsonError(w, http.StatusBadGateway, "could not write the request — try again")
+ return nil, false
+ }
+ s.bumpSerial(ctx)
+ writeJSON(w, okStatus, dbViewOf(rec))
+ return rec, true
+}
+
+// clearPriorStage deletes the staged objects of a transfer that is done or
+// failed, before a new request replaces it on the record. An in-flight one
+// is left alone: the transition will refuse the new request anyway, and
+// its objects are the box's.
+func (s *Server) clearPriorStage(r *http.Request, rec *provision.Record, stage provision.DBStage) error {
+ if rec.DB == nil || rec.DBInFlight() {
+ return nil
+ }
+ return stage.DeleteRequest(r.Context(), s.cfg.Releases, rec.DB.ID)
+}
+
+func (s *Server) handleAPIInstanceDBPull(w http.ResponseWriter, r *http.Request) {
+ principal, acct, app, host, rec, stage, ok := s.dbGate(w, r)
+ if !ok {
+ return
+ }
+ if err := s.clearPriorStage(r, rec, stage); err != nil {
+ log.Printf("console: api instance db pull %s/%s/%s: clear stage: %v", acct.Sqid, app, host, err)
+ jsonError(w, http.StatusBadGateway, "could not clear the previous transfer — try again")
+ return
+ }
+ id := provision.NewDBTransferID()
+ if _, ok := s.dbTransition(w, r, acct, app, host, provision.DBRequest(provision.DBPull, id, time.Now()), http.StatusAccepted); !ok {
+ return
+ }
+ s.audit(acct, app, principal, "db-pull", "requested a database pull of "+host+" ("+id+") (cli)")
+}
+
+// handleAPIInstanceDBPush stages the upload FIRST and writes the request
+// second: a request whose bytes are not yet there would send the box after
+// nothing. If the CAS loses, the staged object is deleted again.
+func (s *Server) handleAPIInstanceDBPush(w http.ResponseWriter, r *http.Request) {
+ principal, acct, app, host, rec, stage, ok := s.dbGate(w, r)
+ if !ok {
+ return
+ }
+ // Everything that can be refused WITHOUT reading the body is refused
+ // before it: a 256 MiB upload for a 409 is a waste on both ends.
+ if rec.Role != "" {
+ jsonError(w, http.StatusBadRequest, "a "+rec.Role+" holds a one-way mirror of the home instance — push to the home instead")
+ return
+ }
+ if rec.Move != nil {
+ jsonError(w, http.StatusConflict, "a move to "+s.homeRegionOut(rec.Move.To)+" is "+rec.Move.State+" — wait for it to finish or abort it first")
+ return
+ }
+ if rec.DBInFlight() {
+ jsonError(w, http.StatusConflict, "a db "+rec.DB.Op+" is still in flight — wait for it to finish")
+ return
+ }
+ if r.Header.Get(dbConfirmHeader) != host {
+ jsonError(w, http.StatusBadRequest, "confirm the replacement by sending "+dbConfirmHeader+": "+host)
+ return
+ }
+ ct, _, _ := strings.Cut(r.Header.Get("Content-Type"), ";")
+ if strings.TrimSpace(strings.ToLower(ct)) != dbSQLiteContentType {
+ jsonError(w, http.StatusUnsupportedMediaType, "send the database file as the body with Content-Type: "+dbSQLiteContentType)
+ return
+ }
+ // The same concurrency bound and read deadline `carlos ship` runs
+ // under (handleAPIShip): a slow upload holds a goroutine and a growing
+ // buffer, and the shared semaphore is what keeps a handful of them from
+ // being a resource-exhaustion path.
+ select {
+ case s.shipSem <- struct{}{}:
+ defer func() { <-s.shipSem }()
+ default:
+ retryAfter(w, 2*time.Second)
+ jsonError(w, http.StatusTooManyRequests, "another upload is in progress — try again shortly")
+ return
+ }
+ rc := http.NewResponseController(w)
+ if err := rc.SetReadDeadline(time.Now().Add(10 * time.Minute)); err != nil && !errors.Is(err, http.ErrNotSupported) {
+ log.Printf("console: api db push set read deadline: %v", err)
+ }
+ defer func() {
+ if err := rc.SetReadDeadline(time.Time{}); err != nil && !errors.Is(err, http.ErrNotSupported) {
+ log.Printf("console: api db push clear read deadline: %v", err)
+ }
+ }()
+ body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, provision.DBTransferMaxBytes))
+ if err != nil {
+ var tooLarge *http.MaxBytesError
+ if errors.As(err, &tooLarge) {
+ jsonError(w, http.StatusRequestEntityTooLarge, fmt.Sprintf("upload too large — transfers are capped at %d MiB", provision.DBTransferMaxBytes>>20))
+ return
+ }
+ jsonError(w, http.StatusBadRequest, "could not read the upload")
+ return
+ }
+ if !provision.LooksLikeSQLite(body) {
+ jsonError(w, http.StatusBadRequest, "the upload is not a SQLite database file (no SQLite header)")
+ return
+ }
+ if err := s.clearPriorStage(r, rec, stage); err != nil {
+ log.Printf("console: api instance db push %s/%s/%s: clear stage: %v", acct.Sqid, app, host, err)
+ jsonError(w, http.StatusBadGateway, "could not clear the previous transfer — try again")
+ return
+ }
+ id := provision.NewDBTransferID()
+ ctx := r.Context()
+ if _, err := s.cfg.Releases.Put(ctx, stage.PushKey(id), body, ""); err != nil {
+ log.Printf("console: api instance db push %s/%s/%s: stage: %v", acct.Sqid, app, host, err)
+ jsonError(w, http.StatusBadGateway, "could not stage the upload — try again")
+ return
+ }
+ size := int64(len(body))
+ mut := func(rec *provision.Record) error {
+ if err := provision.DBRequest(provision.DBPush, id, time.Now())(rec); err != nil {
+ return err
+ }
+ rec.DB.Bytes = size
+ return nil
+ }
+ if _, ok := s.dbTransition(w, r, acct, app, host, mut, http.StatusAccepted); !ok {
+ // The request did not land; the bytes must not linger for a later
+ // request to find.
+ if err := s.cfg.Releases.Delete(ctx, stage.PushKey(id)); err != nil {
+ log.Printf("console: api instance db push %s/%s/%s: unstage after a refused request: %v", acct.Sqid, app, host, err)
+ }
+ return
+ }
+ s.audit(acct, app, principal, "db-push", fmt.Sprintf("requested a database push to %s (%s, %d bytes) (cli)", host, id, size))
+}
+
+// serveDBFile writes one staged database: the type, the length, no caching
+// (a snapshot is a moment, and a backup is a customer's data), and a
+// filename a browser or curl -O would keep.
+func serveDBFile(w http.ResponseWriter, name string, body []byte) {
+ w.Header().Set("Content-Type", dbSQLiteContentType)
+ w.Header().Set("Content-Length", strconv.Itoa(len(body)))
+ w.Header().Set("Cache-Control", "no-store")
+ w.Header().Set("Content-Disposition", `attachment; filename="`+name+`"`)
+ w.WriteHeader(http.StatusOK)
+ if _, err := w.Write(body); err != nil {
+ log.Printf("console: api db file %s: write: %v", name, err)
+ }
+}
+
+// handleAPIInstanceDBFile streams a DONE pull's snapshot. The id is the
+// caller's proof they mean THIS pull: a stale link to a superseded one is a
+// 404, and a pull that has not finished is a 409 naming where it is.
+func (s *Server) handleAPIInstanceDBFile(w http.ResponseWriter, r *http.Request) {
+ _, acct, app, host, rec, stage, ok := s.dbGate(w, r)
+ if !ok {
+ return
+ }
+ id := r.URL.Query().Get("id")
+ if !provision.ValidDBTransferID(id) || rec.DB == nil || rec.DB.ID != id || rec.DB.Op != provision.DBPull {
+ jsonError(w, http.StatusNotFound, "no pull with that id on this instance")
+ return
+ }
+ switch rec.DB.State {
+ case provision.DBDone:
+ case provision.DBFailed:
+ jsonError(w, http.StatusConflict, "the pull failed: "+rec.DB.Error)
+ return
+ default:
+ jsonError(w, http.StatusConflict, "the pull is still in flight — poll the status door until it is done")
+ return
+ }
+ body, _, err := s.cfg.Releases.Get(r.Context(), stage.PullKey(id))
+ if errors.Is(err, os.ErrNotExist) {
+ jsonError(w, http.StatusNotFound, "the snapshot is no longer staged — request the pull again")
+ return
+ }
+ if err != nil {
+ log.Printf("console: api instance db file %s/%s/%s: %v", acct.Sqid, app, host, err)
+ jsonError(w, http.StatusBadGateway, "could not read the snapshot — try again")
+ return
+ }
+ serveDBFile(w, host+".db", body)
+}
+
+func (s *Server) handleAPIInstanceDBBackups(w http.ResponseWriter, r *http.Request) {
+ _, acct, app, host, _, stage, ok := s.dbGate(w, r)
+ if !ok {
+ return
+ }
+ list, err := stage.ListBackups(r.Context(), s.cfg.Releases)
+ if err != nil {
+ log.Printf("console: api instance db backups %s/%s/%s: %v", acct.Sqid, app, host, err)
+ jsonError(w, http.StatusBadGateway, "could not list the backups — try again")
+ return
+ }
+ if list == nil {
+ list = []provision.DBBackup{}
+ }
+ writeJSON(w, http.StatusOK, map[string]any{"backups": list})
+}
+
+func (s *Server) handleAPIInstanceDBBackup(w http.ResponseWriter, r *http.Request) {
+ _, acct, app, host, _, stage, ok := s.dbGate(w, r)
+ if !ok {
+ return
+ }
+ id := r.PathValue("id")
+ if !provision.ValidDBBackupID(id) {
+ jsonError(w, http.StatusNotFound, "no backup with that id on this instance")
+ return
+ }
+ body, _, err := s.cfg.Releases.Get(r.Context(), stage.BackupKey(id))
+ if errors.Is(err, os.ErrNotExist) {
+ jsonError(w, http.StatusNotFound, "no backup with that id on this instance")
+ return
+ }
+ if err != nil {
+ log.Printf("console: api instance db backup %s/%s/%s: %v", acct.Sqid, app, host, err)
+ jsonError(w, http.StatusBadGateway, "could not read the backup — try again")
+ return
+ }
+ serveDBFile(w, host+"-"+id+".db", body)
+}
+
+// handleAPIInstanceDBClear drops a done or failed transfer from the record
+// and its staged objects from the bucket. Backups are never touched: they
+// are what makes a push undoable, and a cleanup verb must not be the thing
+// that removes the undo.
+func (s *Server) handleAPIInstanceDBClear(w http.ResponseWriter, r *http.Request) {
+ principal, acct, app, host, rec, stage, ok := s.dbGate(w, r)
+ if !ok {
+ return
+ }
+ if rec.DB == nil {
+ writeJSON(w, http.StatusOK, dbViewOf(nil))
+ return
+ }
+ id := rec.DB.ID
+ ctx := r.Context()
+ if _, err := provision.SetRecordDB(ctx, s.cfg.Releases, acct.Sqid, app, host, provision.DBClear()); err != nil {
+ if errors.Is(err, provision.ErrDBConflict) {
+ jsonError(w, http.StatusConflict, strings.TrimPrefix(err.Error(), provision.ErrDBConflict.Error()+": "))
+ return
+ }
+ log.Printf("console: api instance db clear %s/%s/%s: %v", acct.Sqid, app, host, err)
+ jsonError(w, http.StatusBadGateway, "could not clear the transfer — try again")
+ return
+ }
+ if err := stage.DeleteRequest(ctx, s.cfg.Releases, id); err != nil {
+ // The record is clear; a leftover object is storage, and the next
+ // request's clearPriorStage cannot reach it any more, so say so.
+ log.Printf("console: api instance db clear %s/%s/%s: delete staged %s: %v", acct.Sqid, app, host, id, err)
+ }
+ s.audit(acct, app, principal, "db-clear", "cleared database transfer "+id+" on "+host+" (cli)")
+ writeJSON(w, http.StatusOK, dbViewOf(nil))
+}
diff --git a/internal/console/apiinstancedb_test.go b/internal/console/apiinstancedb_test.go
new file mode 100644
index 000000000..f12f19d5e
--- /dev/null
+++ b/internal/console/apiinstancedb_test.go
@@ -0,0 +1,410 @@
+package console
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "io"
+ "net/http"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/svcrole"
+)
+
+// rawBytes is rawReq for a body that is not JSON: any method, a content
+// type, extra headers, raw bytes; the response's status, headers and body.
+func rawBytes(t *testing.T, method, u, bearer, contentType string, headers map[string]string, body []byte) (int, http.Header, []byte) {
+ t.Helper()
+ var rdr io.Reader
+ if body != nil {
+ rdr = bytes.NewReader(body)
+ }
+ req, err := http.NewRequest(method, u, rdr)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if contentType != "" {
+ req.Header.Set("Content-Type", contentType)
+ }
+ for k, v := range headers {
+ req.Header.Set(k, v)
+ }
+ if bearer != "" {
+ req.Header.Set("Authorization", "Bearer "+bearer)
+ }
+ res, err := http.DefaultClient.Do(req)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer res.Body.Close()
+ out, err := io.ReadAll(res.Body)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return res.StatusCode, res.Header, out
+}
+
+// sqliteBytes is a body that passes the door's header check: the magic,
+// padded to a page.
+func sqliteBytes(n int) []byte {
+ b := make([]byte, n)
+ copy(b, provision.SQLiteMagic)
+ return b
+}
+
+func dbFixture(t *testing.T, code string) (*consoleFixture, string, string) {
+ t.Helper()
+ fx, token := instanceFixture(t, code)
+ m := titoManifest()
+ m.Regions = nil
+ m.Pool = ""
+ enableInstances(t, fx, "titogo", m)
+ u := instancesURL(fx, "bab", "titogo")
+ if code, _, out := rawReq(t, http.MethodPost, u, token, `{"host":"acme.go.tito.io","region":"ie"}`); code != http.StatusCreated {
+ t.Fatalf("create = %d %s", code, out)
+ }
+ return fx, token, u
+}
+
+// The verbs end to end against the record: request, poll, the box's half
+// written straight to the store, download, clear — and every refusal the
+// state machine answers with.
+func TestAPIInstanceDBVerbs(t *testing.T) {
+ fx, token, u := dbFixture(t, "code-inst-db")
+ ctx := context.Background()
+ dbURL := u + "/acme.go.tito.io/db"
+ stage, err := provision.NewDBStage("bab", "titogo", "acme.go.tito.io")
+ if err != nil {
+ t.Fatal(err)
+ }
+ var view struct {
+ State string `json:"state"`
+ ID string `json:"id"`
+ Op string `json:"op"`
+ Bytes int64 `json:"bytes"`
+ Backup string `json:"backup"`
+ Error string `json:"error"`
+ }
+ decode := func(out string) {
+ t.Helper()
+ view = struct {
+ State string `json:"state"`
+ ID string `json:"id"`
+ Op string `json:"op"`
+ Bytes int64 `json:"bytes"`
+ Backup string `json:"backup"`
+ Error string `json:"error"`
+ }{}
+ if err := json.Unmarshal([]byte(out), &view); err != nil {
+ t.Fatal(err)
+ }
+ }
+
+ // Quiet: 200 none, never a 404.
+ code, _, out := rawReq(t, http.MethodGet, dbURL, token, "")
+ if code != http.StatusOK || !strings.Contains(out, `"state":"none"`) {
+ t.Fatalf("db status with nothing = %d %s", code, out)
+ }
+ // A download with no pull is a 404, a clear a no-op.
+ if code, _, _ := rawReq(t, http.MethodGet, dbURL+"/file?id=0123456789abcdef", token, ""); code != http.StatusNotFound {
+ t.Fatalf("file with no pull = %d, want 404", code)
+ }
+ if code, _, out := rawReq(t, http.MethodDelete, dbURL, token, ""); code != http.StatusOK || !strings.Contains(out, `"state":"none"`) {
+ t.Fatalf("clear with nothing = %d %s", code, out)
+ }
+
+ // Pull: 202 with the flat view and an id.
+ code, _, out = rawReq(t, http.MethodPost, dbURL+"/pull", token, "")
+ if code != http.StatusAccepted {
+ t.Fatalf("pull = %d %s", code, out)
+ }
+ decode(out)
+ if view.State != provision.DBRequested || view.Op != provision.DBPull || !provision.ValidDBTransferID(view.ID) {
+ t.Fatalf("after pull request: %+v", view)
+ }
+ pullID := view.ID
+ // The record carries it verbatim; the poll door mirrors it.
+ if code, _, out := rawReq(t, http.MethodGet, u+"/acme.go.tito.io", token, ""); code != http.StatusOK || !strings.Contains(out, `"db":{"id":"`+pullID+`"`) {
+ t.Fatalf("record after request: %d %s", code, out)
+ }
+ if code, _, out := rawReq(t, http.MethodGet, dbURL, token, ""); code != http.StatusOK || !strings.Contains(out, pullID) {
+ t.Fatalf("status mid-pull = %d %s", code, out)
+ }
+ // In flight: a second pull, a push, a clear and a download all refuse.
+ if code, _, out := rawReq(t, http.MethodPost, dbURL+"/pull", token, ""); code != http.StatusConflict {
+ t.Fatalf("second pull = %d %s, want 409", code, out)
+ }
+ if code, _, out := rawBytes(t, http.MethodPut, dbURL+"/push", token, dbSQLiteContentType, map[string]string{dbConfirmHeader: "acme.go.tito.io"}, sqliteBytes(4096)); code != http.StatusConflict {
+ t.Fatalf("push during a pull = %d %s, want 409", code, out)
+ }
+ if code, _, _ := rawReq(t, http.MethodDelete, dbURL, token, ""); code != http.StatusConflict {
+ t.Fatalf("clear during a pull = %d, want 409", code)
+ }
+ if code, _, out := rawReq(t, http.MethodGet, dbURL+"/file?id="+pullID, token, ""); code != http.StatusConflict || !strings.Contains(out, "in flight") {
+ t.Fatalf("file before done = %d %s, want 409", code, out)
+ }
+ // A move during a pull: 409 from the transition itself. The armed
+ // gate needs a target that publishes a staging bucket.
+ if _, err := fx.rel.Put(ctx, "control/edges/uk-box.json", []byte(`{"v":1,"box":"uk-box","eip":"1.2.3.4","placement":"uk","data_bucket":"data-uk","data_bucket_region":"eu-west-2","hosts":{}}`), ""); err != nil {
+ t.Fatal(err)
+ }
+ if code, _, out := rawReq(t, http.MethodPost, u+"/acme.go.tito.io/move", token, `{"region":"uk"}`); code != http.StatusConflict || !strings.Contains(out, "db pull") {
+ t.Fatalf("move during a pull = %d %s, want 409 naming the pull", code, out)
+ }
+
+ // The box's half: snapshot staged, record done.
+ snapshot := sqliteBytes(8192)
+ if _, err := fx.rel.Put(ctx, stage.PullKey(pullID), snapshot, ""); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := provision.SetRecordDB(ctx, fx.rel, "bab", "titogo", "acme.go.tito.io", provision.DBComplete(pullID, int64(len(snapshot)))); err != nil {
+ t.Fatal(err)
+ }
+ code, _, out = rawReq(t, http.MethodGet, dbURL, token, "")
+ decode(out)
+ if code != http.StatusOK || view.State != provision.DBDone || view.Bytes != int64(len(snapshot)) {
+ t.Fatalf("status after done = %d %+v", code, view)
+ }
+ // Download: the bytes, the type, the length, no caching. A wrong id
+ // is a 404.
+ code, hdr, body := rawBytes(t, http.MethodGet, dbURL+"/file?id="+pullID, token, "", nil, nil)
+ if code != http.StatusOK || !bytes.Equal(body, snapshot) {
+ t.Fatalf("file = %d (%d bytes), want 200 with the snapshot", code, len(body))
+ }
+ if hdr.Get("Content-Type") != dbSQLiteContentType || hdr.Get("Content-Length") != "8192" || hdr.Get("Cache-Control") != "no-store" {
+ t.Fatalf("file headers: %v", hdr)
+ }
+ if code, _, _ := rawReq(t, http.MethodGet, dbURL+"/file?id=ffffffffffffffff", token, ""); code != http.StatusNotFound {
+ t.Fatalf("file with a stale id = %d, want 404", code)
+ }
+ if code, _, _ := rawReq(t, http.MethodGet, dbURL+"/file?id=../../x", token, ""); code != http.StatusNotFound {
+ t.Fatalf("file with a malformed id = %d, want 404", code)
+ }
+ // Clear: the field goes, the staged object goes.
+ if code, _, out := rawReq(t, http.MethodDelete, dbURL, token, ""); code != http.StatusOK || !strings.Contains(out, `"state":"none"`) {
+ t.Fatalf("clear = %d %s", code, out)
+ }
+ if keys, _ := fx.rel.List(ctx, stage.Prefix()); len(keys) != 0 {
+ t.Fatalf("staged objects survived the clear: %v", keys)
+ }
+
+ // Push. The refusals that need no body first.
+ if code, _, out := rawBytes(t, http.MethodPut, dbURL+"/push", token, dbSQLiteContentType, nil, sqliteBytes(4096)); code != http.StatusBadRequest || !strings.Contains(string(out), dbConfirmHeader) {
+ t.Fatalf("push without the confirm header = %d %s, want 400 naming it", code, out)
+ }
+ if code, _, _ := rawBytes(t, http.MethodPut, dbURL+"/push", token, dbSQLiteContentType, map[string]string{dbConfirmHeader: "other.go.tito.io"}, sqliteBytes(4096)); code != http.StatusBadRequest {
+ t.Fatalf("push confirming the wrong host = %d, want 400", code)
+ }
+ if code, _, _ := rawBytes(t, http.MethodPut, dbURL+"/push", token, "application/octet-stream", map[string]string{dbConfirmHeader: "acme.go.tito.io"}, sqliteBytes(4096)); code != http.StatusUnsupportedMediaType {
+ t.Fatalf("push with the wrong content type = %d, want 415", code)
+ }
+ confirm := map[string]string{dbConfirmHeader: "acme.go.tito.io"}
+ if code, _, out := rawBytes(t, http.MethodPut, dbURL+"/push", token, dbSQLiteContentType, confirm, []byte("not a database")); code != http.StatusBadRequest || !strings.Contains(string(out), "SQLite") {
+ t.Fatalf("push of a non-database = %d %s, want 400", code, out)
+ }
+ if code, _, _ := rawBytes(t, http.MethodPut, dbURL+"/push", token, dbSQLiteContentType, confirm, sqliteBytes(100)); code != http.StatusBadRequest {
+ t.Fatalf("push of a too-short file = %d, want 400", code)
+ }
+ // The cap: 413, and nothing staged.
+ old := provision.DBTransferMaxBytes
+ provision.DBTransferMaxBytes = 2048
+ if code, _, out := rawBytes(t, http.MethodPut, dbURL+"/push", token, dbSQLiteContentType, confirm, sqliteBytes(4096)); code != http.StatusRequestEntityTooLarge {
+ t.Fatalf("push over the cap = %d %s, want 413", code, out)
+ }
+ provision.DBTransferMaxBytes = old
+ if keys, _ := fx.rel.List(ctx, stage.Prefix()); len(keys) != 0 {
+ t.Fatalf("a refused push staged something: %v", keys)
+ }
+ // A good push: 202, the object staged FIRST, the record carrying the
+ // size.
+ upload := sqliteBytes(4096)
+ code, _, body = rawBytes(t, http.MethodPut, dbURL+"/push", token, dbSQLiteContentType, confirm, upload)
+ if code != http.StatusAccepted {
+ t.Fatalf("push = %d %s", code, body)
+ }
+ decode(string(body))
+ if view.State != provision.DBRequested || view.Op != provision.DBPush || view.Bytes != 4096 {
+ t.Fatalf("after push request: %+v", view)
+ }
+ pushID := view.ID
+ if staged, _, err := fx.rel.Get(ctx, stage.PushKey(pushID)); err != nil || !bytes.Equal(staged, upload) {
+ t.Fatalf("push.db not staged: %v", err)
+ }
+ // A move now is refused at the door, before its own transition.
+ if code, _, out := rawReq(t, http.MethodPost, u+"/acme.go.tito.io/move", token, `{"region":"uk"}`); code != http.StatusConflict {
+ t.Fatalf("move during a push = %d %s, want 409", code, out)
+ }
+ // The box fails it: the error is the view's, and a fresh push replaces
+ // it — deleting the failed push's object on the way.
+ if _, err := provision.SetRecordDB(ctx, fx.rel, "bab", "titogo", "acme.go.tito.io", provision.DBFail(pushID, "the uploaded file is not a usable SQLite database: no tables")); err != nil {
+ t.Fatal(err)
+ }
+ code, _, out = rawReq(t, http.MethodGet, dbURL, token, "")
+ decode(out)
+ if view.State != provision.DBFailed || !strings.Contains(view.Error, "no tables") {
+ t.Fatalf("status after failure = %+v", view)
+ }
+ code, _, body = rawBytes(t, http.MethodPut, dbURL+"/push", token, dbSQLiteContentType, confirm, upload)
+ if code != http.StatusAccepted {
+ t.Fatalf("push after a failure = %d %s", code, body)
+ }
+ decode(string(body))
+ if _, _, err := fx.rel.Get(ctx, stage.PushKey(pushID)); err == nil {
+ t.Fatal("the failed push's object survived the next request")
+ }
+ if _, _, err := fx.rel.Get(ctx, stage.PushKey(view.ID)); err != nil {
+ t.Fatal("the new push's object is not staged")
+ }
+ // The serial was bumped by the successful writes.
+ if _, _, err := fx.rel.Get(ctx, "control/serial"); err != nil {
+ t.Fatalf("control/serial not written: %v", err)
+ }
+
+ // Backups: listed newest first with their sidecars, streamed by id,
+ // never touched by a clear.
+ bid1 := provision.NewDBBackupID(time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC))
+ bid2 := provision.NewDBBackupID(time.Date(2026, 9, 2, 0, 0, 0, 0, time.UTC))
+ for i, bid := range []string{bid1, bid2} {
+ if _, err := fx.rel.Put(ctx, stage.BackupKey(bid), sqliteBytes(1024*(i+1)), ""); err != nil {
+ t.Fatal(err)
+ }
+ if err := stage.PutBackupMeta(ctx, fx.rel, provision.DBBackup{ID: bid, Bytes: int64(1024 * (i + 1)), CreatedAt: time.Now(), Request: pushID}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ code, _, out = rawReq(t, http.MethodGet, dbURL+"/backups", token, "")
+ if code != http.StatusOK {
+ t.Fatalf("backups = %d %s", code, out)
+ }
+ var bl struct {
+ Backups []provision.DBBackup `json:"backups"`
+ }
+ if err := json.Unmarshal([]byte(out), &bl); err != nil {
+ t.Fatal(err)
+ }
+ if len(bl.Backups) != 2 || bl.Backups[0].ID != bid2 || bl.Backups[0].Bytes != 2048 || bl.Backups[1].ID != bid1 {
+ t.Fatalf("backups = %+v, want newest first with sizes", bl.Backups)
+ }
+ code, hdr, body = rawBytes(t, http.MethodGet, dbURL+"/backups/"+bid1, token, "", nil, nil)
+ if code != http.StatusOK || len(body) != 1024 || hdr.Get("Content-Type") != dbSQLiteContentType {
+ t.Fatalf("backup download = %d (%d bytes) %v", code, len(body), hdr)
+ }
+ if code, _, _ := rawReq(t, http.MethodGet, dbURL+"/backups/20260901-000000-00000000", token, ""); code != http.StatusNotFound {
+ t.Fatalf("unknown backup = %d, want 404", code)
+ }
+ if code, _, _ := rawReq(t, http.MethodGet, dbURL+"/backups/..%2f..%2fpush.db", token, ""); code != http.StatusNotFound {
+ t.Fatalf("malformed backup id = %d, want 404", code)
+ }
+ // Even after the push finishes and is cleared, the backups stay.
+ if _, err := provision.SetRecordDB(ctx, fx.rel, "bab", "titogo", "acme.go.tito.io", provision.DBComplete(view.ID, 0)); err != nil {
+ t.Fatal(err)
+ }
+ if code, _, _ := rawReq(t, http.MethodDelete, dbURL, token, ""); code != http.StatusOK {
+ t.Fatalf("clear after push = %d", code)
+ }
+ if keys, _ := fx.rel.List(ctx, stage.BackupsPrefix()); len(keys) != 4 {
+ t.Fatalf("backups after clear: %v", keys)
+ }
+ // An empty backups list is a list, not null.
+ if code, _, out := rawReq(t, http.MethodGet, u+"/acme.go.tito.io/db/backups", token, ""); code != http.StatusOK || !strings.Contains(out, `"backups":[`) {
+ t.Fatalf("backups shape = %d %s", code, out)
+ }
+
+ // 404 discipline: an unknown host, on every verb, is the byte-identical
+ // not-found.
+ want := canonicalNotFound(t)
+ for _, c := range []struct{ method, tail string }{
+ {http.MethodGet, "/db"}, {http.MethodPost, "/db/pull"}, {http.MethodGet, "/db/file?id=" + pullID},
+ {http.MethodGet, "/db/backups"}, {http.MethodGet, "/db/backups/" + bid1}, {http.MethodDelete, "/db"},
+ } {
+ code, h, body := rawReq(t, c.method, u+"/ghost.go.tito.io"+c.tail, token, "")
+ assertNotFound(t, c.method+" "+c.tail+" on an unknown host", want, code, h, body)
+ }
+ code, h, body := rawBytes(t, http.MethodPut, u+"/ghost.go.tito.io/db/push", token, dbSQLiteContentType, map[string]string{dbConfirmHeader: "ghost.go.tito.io"}, sqliteBytes(4096))
+ assertNotFound(t, "PUT /db/push on an unknown host", want, code, h, string(body))
+}
+
+// The class rails, and the replica asymmetry: a replica may be pulled but
+// never pushed.
+func TestAPIInstanceDBClassRails(t *testing.T) {
+ fx, token, u := dbFixture(t, "code-inst-db-rails")
+ mk := func(body string) string {
+ t.Helper()
+ code, _, out := rawReq(t, http.MethodPost, u, token, body)
+ if code != http.StatusCreated {
+ t.Fatalf("create %s = %d %s", body, code, out)
+ }
+ var rec provision.Record
+ if err := json.Unmarshal([]byte(out), &rec); err != nil {
+ t.Fatal(err)
+ }
+ return rec.Host
+ }
+ for _, tc := range []struct {
+ name, body, wantIn string
+ }{
+ {"static", `{"host":"st.go.tito.io","kind":"static"}`, "kind static"},
+ {"addr-backed", `{"host":"ad.go.tito.io","addrs":["10.0.0.1:80"],"kind":"service"}`, "kind service"},
+ } {
+ host := mk(tc.body)
+ code, _, out := rawReq(t, http.MethodPost, u+"/"+host+"/db/pull", token, "")
+ if code != http.StatusBadRequest || !strings.Contains(out, tc.wantIn) || !strings.Contains(out, "database") {
+ t.Fatalf("pull %s = %d %s, want 400 in db terms mentioning %q", tc.name, code, out, tc.wantIn)
+ }
+ }
+ host := mk(`{"host":"fl.go.tito.io"}`)
+ if err := provision.SetRecordRegion(context.Background(), fx.rel, "bab", "titogo", host, "fleet/pi"); err != nil {
+ t.Fatal(err)
+ }
+ if code, _, out := rawReq(t, http.MethodGet, u+"/"+host+"/db", token, ""); code != http.StatusBadRequest || !strings.Contains(out, "fleet") {
+ t.Fatalf("db on a fleet-placed instance = %d %s", code, out)
+ }
+ pooled := provision.Record{Schema: 1, Host: "pooled.go.tito.io", Slug: "pooled", App: "titogo",
+ Account: "bab", Pool: "dedicated-x", Channel: "stable", CreatedAt: time.Now().UTC()}
+ if err := provision.CreateRecord(context.Background(), fx.rel, pooled); err != nil {
+ t.Fatal(err)
+ }
+ if code, _, out := rawReq(t, http.MethodPost, u+"/pooled.go.tito.io/db/pull", token, ""); code != http.StatusBadRequest || !strings.Contains(out, "pool") {
+ t.Fatalf("pull on a pooled instance = %d %s", code, out)
+ }
+ rep := provision.Record{Schema: 1, Host: "rep.go.tito.io", Slug: "rep", App: "titogo",
+ Account: "bab", Region: "ie", Role: "replica", Channel: "stable", CreatedAt: time.Now().UTC()}
+ if err := provision.CreateRecord(context.Background(), fx.rel, rep); err != nil {
+ t.Fatal(err)
+ }
+ if code, _, out := rawBytes(t, http.MethodPut, u+"/rep.go.tito.io/db/push", token, dbSQLiteContentType, map[string]string{dbConfirmHeader: "rep.go.tito.io"}, sqliteBytes(4096)); code != http.StatusBadRequest || !strings.Contains(string(out), "replica") {
+ t.Fatalf("push to a replica = %d %s, want 400 naming the replica", code, out)
+ }
+ if code, _, out := rawReq(t, http.MethodPost, u+"/rep.go.tito.io/db/pull", token, ""); code != http.StatusAccepted {
+ t.Fatalf("pull of a replica = %d %s, want 202", code, out)
+ }
+ // The move door's own sentences are untouched by the shared rail.
+ if code, _, out := rawReq(t, http.MethodPost, u+"/st.go.tito.io/move", token, `{"region":"uk"}`); code != http.StatusBadRequest || !strings.Contains(out, "only exec-backed instances move") {
+ t.Fatalf("move refusal changed: %d %s", code, out)
+ }
+}
+
+// No service credential reaches any db door — not the instance role that
+// creates and deletes instances, not admin.
+func TestAPIInstanceDBClosedToServiceCredentials(t *testing.T) {
+ fx, _, u := dbFixture(t, "code-inst-db-svc")
+ for _, role := range []svcrole.Role{svcrole.Instance, svcrole.Admin} {
+ tok := serviceToken(t, fx.st, OpsAccountID, "svc-"+string(role), "titogo", role)
+ for _, c := range []struct{ method, tail string }{
+ {http.MethodGet, "/db"}, {http.MethodPost, "/db/pull"}, {http.MethodGet, "/db/file?id=0123456789abcdef"},
+ {http.MethodGet, "/db/backups"}, {http.MethodDelete, "/db"},
+ } {
+ code, _, out := rawReq(t, c.method, u+"/acme.go.tito.io"+c.tail, tok, "")
+ if code != http.StatusForbidden || !strings.Contains(out, "needs a person") {
+ t.Fatalf("%s %s with role %s = %d %s, want the closed-door 403", c.method, c.tail, role, code, out)
+ }
+ }
+ code, _, out := rawBytes(t, http.MethodPut, u+"/acme.go.tito.io/db/push", tok, dbSQLiteContentType, map[string]string{dbConfirmHeader: "acme.go.tito.io"}, sqliteBytes(4096))
+ if code != http.StatusForbidden || !strings.Contains(string(out), "needs a person") {
+ t.Fatalf("PUT /db/push with role %s = %d %s, want the closed-door 403", role, code, out)
+ }
+ }
+}
diff --git a/internal/console/apiinstancemove.go b/internal/console/apiinstancemove.go
index 0204869e8..468866b35 100644
--- a/internal/console/apiinstancemove.go
+++ b/internal/console/apiinstancemove.go
@@ -28,49 +28,84 @@ import (
// byte-identical 404 for a host that names no record, and the class rail —
// only an exec-backed instance route's state is the platform's to move
// (spec §Out of scope), and fleet placements keep their deliberate
-// state-does-not-follow semantics.
+// state-does-not-follow semantics. The rail itself is execInstanceGate,
+// shared with the database transfer doors (apiinstancedb.go), which hold
+// the same class for the same reason: the activator is what manages the
+// data these verbs act on.
func (s *Server) moveGate(w http.ResponseWriter, r *http.Request) (acct *Account, app, host string, rec *provision.Record, ok bool) {
- _, acct, app, gok := s.instanceGate(w, r)
+ _, acct, app, host, rec, ok = s.execInstanceGate(w, r, "move")
+ return acct, app, host, rec, ok
+}
+
+// execInstanceGate is the rail both families share. verb names the family
+// in the refusals ("move" or "db push/pull"), so a member reading one is
+// told what was refused and why in that verb's own terms. It returns the
+// principal too, for the doors that audit.
+func (s *Server) execInstanceGate(w http.ResponseWriter, r *http.Request, verb string) (principal string, acct *Account, app, host string, rec *provision.Record, ok bool) {
+ principal, acct, app, gok := s.instanceGate(w, r)
if !gok {
- return nil, "", "", nil, false
+ return "", nil, "", "", nil, false
}
host = r.PathValue("host")
if !provision.ValidHostShape(host) {
http.NotFound(w, r)
- return nil, "", "", nil, false
+ return "", nil, "", "", nil, false
}
ctx := r.Context()
rec, err := provision.GetRecord(ctx, s.cfg.Releases, acct.Sqid, app, host)
if errors.Is(err, os.ErrNotExist) {
http.NotFound(w, r)
- return nil, "", "", nil, false
+ return "", nil, "", "", nil, false
}
if err != nil {
- log.Printf("console: api instance move %s/%s/%s: %v", acct.Sqid, app, host, err)
+ log.Printf("console: api instance %s %s/%s/%s: %v", verb, acct.Sqid, app, host, err)
jsonError(w, http.StatusBadGateway, "could not read the instance")
- return nil, "", "", nil, false
+ return "", nil, "", "", nil, false
}
+ if msg := execInstanceRefusal(*rec, verb); msg != "" {
+ jsonError(w, http.StatusBadRequest, msg)
+ return "", nil, "", "", nil, false
+ }
+ return principal, acct, app, host, rec, true
+}
+
+// execInstanceRefusal is the class rail as a sentence, or "" when the
+// record is an activator-managed exec instance. The move sentences are the
+// ones the move doors have always answered with; the db ones say what a
+// database transfer cannot reach and why.
+func execInstanceRefusal(rec provision.Record, verb string) string {
+ move := verb == "move"
if k := rec.Kind; k != "" && k != "instance" {
- jsonError(w, http.StatusBadRequest, "only exec-backed instances move — this route is kind "+k)
- return nil, "", "", nil, false
+ if move {
+ return "only exec-backed instances move — this route is kind " + k
+ }
+ return "only exec-backed instances carry a platform-managed database — this route is kind " + k
}
if rec.Addr != "" || len(rec.Addrs) > 0 {
- jsonError(w, http.StatusBadRequest, "an addr-backed route's process is not platform-managed — nothing to move")
- return nil, "", "", nil, false
+ if move {
+ return "an addr-backed route's process is not platform-managed — nothing to move"
+ }
+ return "an addr-backed route's process is not platform-managed — there is no database for db push/pull to reach"
}
if _, fleet := registry.FleetPlacement(rec.Region); fleet {
- jsonError(w, http.StatusBadRequest, "fleet-placed instances keep state-does-not-follow semantics — move is for platform regions")
- return nil, "", "", nil, false
+ if move {
+ return "fleet-placed instances keep state-does-not-follow semantics — move is for platform regions"
+ }
+ return "fleet-placed instances keep their data on the fleet's own box — db push/pull is for platform regions"
}
if rec.Pool != "" {
- jsonError(w, http.StatusBadRequest, "pooled instances do not move in v1 — their data lives on the pool's own box")
- return nil, "", "", nil, false
+ if move {
+ return "pooled instances do not move in v1 — their data lives on the pool's own box"
+ }
+ return "pooled instances keep their data on the pool's own box — db push/pull does not reach it in v1"
}
if !rec.HibernateOrDefault() {
- jsonError(w, http.StatusBadRequest, "a non-hibernating instance runs outside the activator — nothing manages its data, nothing can move it")
- return nil, "", "", nil, false
+ if move {
+ return "a non-hibernating instance runs outside the activator — nothing manages its data, nothing can move it"
+ }
+ return "a non-hibernating instance runs outside the activator — nothing manages its database, nothing can push or pull it"
}
- return acct, app, host, rec, true
+ return ""
}
// moveView is the flat answer every move door speaks — titogo's consumer
diff --git a/internal/console/authcli.go b/internal/console/authcli.go
index 9fdd2e11c..725403e51 100644
--- a/internal/console/authcli.go
+++ b/internal/console/authcli.go
@@ -626,6 +626,20 @@ func (s *Server) registerCLIRoutes() {
s.cliRoute("POST /api/cli/apps/{account}/{app}/instances/{host}/move", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveRequest)
s.cliRoute("POST /api/cli/apps/{account}/{app}/instances/{host}/move/cut", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveCut)
s.cliRoute("DELETE /api/cli/apps/{account}/{app}/instances/{host}/move", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceMoveAbort)
+ // The database transfer doors (`carlos db pull/push`, apiinstancedb.go).
+ // The zero Set, deliberately: the payload is a whole customer database
+ // — every row the instance holds, out through the pull door or replaced
+ // through the push one — and a credential the app itself holds must not
+ // be able to walk it out of (or into) the platform. A person's
+ // membership is the only authority here, the instance-token cut taken
+ // over data rather than over credentials.
+ s.cliRoute("GET /api/cli/apps/{account}/{app}/instances/{host}/db", svcrole.Set{}, s.handleAPIInstanceDBStatus)
+ s.cliRoute("POST /api/cli/apps/{account}/{app}/instances/{host}/db/pull", svcrole.Set{}, s.handleAPIInstanceDBPull)
+ s.cliRoute("PUT /api/cli/apps/{account}/{app}/instances/{host}/db/push", svcrole.Set{}, s.handleAPIInstanceDBPush)
+ s.cliRoute("GET /api/cli/apps/{account}/{app}/instances/{host}/db/file", svcrole.Set{}, s.handleAPIInstanceDBFile)
+ s.cliRoute("GET /api/cli/apps/{account}/{app}/instances/{host}/db/backups", svcrole.Set{}, s.handleAPIInstanceDBBackups)
+ s.cliRoute("GET /api/cli/apps/{account}/{app}/instances/{host}/db/backups/{id}", svcrole.Set{}, s.handleAPIInstanceDBBackup)
+ s.cliRoute("DELETE /api/cli/apps/{account}/{app}/instances/{host}/db", svcrole.Set{}, s.handleAPIInstanceDBClear)
s.cliRoute("DELETE /api/cli/apps/{account}/{app}/instances/{host}", svcrole.Allow(svcrole.Instance), s.handleAPIInstanceDelete)
// The zero Set, deliberately: a credential may never mint a credential,
// including itself. This is the escalation cut (spec §3). It takes no
diff --git a/internal/console/svcroutes_test.go b/internal/console/svcroutes_test.go
index ced377726..042e5a700 100644
--- a/internal/console/svcroutes_test.go
+++ b/internal/console/svcroutes_test.go
@@ -168,6 +168,17 @@ var closedDoors = []string{
// whose rules it never met. Shut for the same reason the manifest door
// is — an app may not rewrite the bounds it runs inside.
"POST /api/cli/apps/{account}/{app}/instances/{host}/channel",
+ // The database transfer doors carry a whole customer database in each
+ // direction. Shut to every role for the instance-token reason taken
+ // over data: a credential the app holds may run the app, not walk its
+ // tenants' rows out of the platform or replace them.
+ "GET /api/cli/apps/{account}/{app}/instances/{host}/db",
+ "POST /api/cli/apps/{account}/{app}/instances/{host}/db/pull",
+ "PUT /api/cli/apps/{account}/{app}/instances/{host}/db/push",
+ "GET /api/cli/apps/{account}/{app}/instances/{host}/db/file",
+ "GET /api/cli/apps/{account}/{app}/instances/{host}/db/backups",
+ "GET /api/cli/apps/{account}/{app}/instances/{host}/db/backups/{id}",
+ "DELETE /api/cli/apps/{account}/{app}/instances/{host}/db",
// The build probe is what `carlos deploy`'s watch TRUSTS in place of the
// X-Carlos-Version header — the header being the thing that reports a
// release before any process has restarted. A credential that could
@@ -800,8 +811,8 @@ func TestEveryClosedDoorRefusesAServiceCredential(t *testing.T) {
}
seen++
}
- if seen != len(closedDoors) || seen != 34 {
- t.Errorf("walked %d closed doors, want all %d (and 34 today: instance-token, three door-key doors, the routes door, instances-manifest, sidecars-manifest, the route-channel move, six fleets, four services, two email credentials, email pause, the build probe, the live tenancy read, five system doors, two deployment-wide antivirus, and four store-entitlement doors)",
+ if seen != len(closedDoors) || seen != 41 {
+ t.Errorf("walked %d closed doors, want all %d (and 41 today: instance-token, three door-key doors, the routes door, instances-manifest, sidecars-manifest, the route-channel move, seven database transfer doors, six fleets, four services, two email credentials, email pause, the build probe, the live tenancy read, five system doors, two deployment-wide antivirus, and four store-entitlement doors)",
seen, len(closedDoors))
}
}
diff --git a/internal/host/agent.go b/internal/host/agent.go
index 255b399d9..b8ba2233f 100644
--- a/internal/host/agent.go
+++ b/internal/host/agent.go
@@ -357,6 +357,9 @@ func RunAgent(args []string) {
eb.Mirror = m
}
armMovesPass(*cfg, dd, reg, svc)
+ // The database transfer worker (`carlos db pull/push`, dbpass.go):
+ // same coordinates, same quiesce answer, same best-effort arming.
+ armDBPass(*cfg, dd, reg, svc)
// The startup sweep (design §6.7): a leftover instance unit from a
// previous agent process, one this box's registry no longer holds
// a live wake for, is stopped before this process starts spawning
@@ -1795,6 +1798,7 @@ func fastConvergeLoop(reg *registry.Registry, svc *activator.Service, sd *schedu
}
telemetryPass(ctx, store, reg)
runMovesPass(ctx, store)
+ runDBPass(ctx, store)
// Folded once per pass and handed in, never re-read inside the
// loop: this leg runs on every serial bump (a seconds cadence),
// and the fold is a LoadBoxes plus a List plus one Get per edges
@@ -2352,6 +2356,7 @@ func instancesLoop(reg *registry.Registry) {
}
telemetryPass(ctx, store, reg)
runMovesPass(ctx, store)
+ runDBPass(ctx, store)
deploymentRoutes := edgesfile.DeploymentRoutes(ctx, store, time.Now())
if err := domains.ReconcileAliases(ctx, store, reg, time.Now, deploymentRoutes, &aliasClaimCache); err != nil {
log.Printf("agent: instances alias pass: %v", err)
diff --git a/internal/host/dbpass.go b/internal/host/dbpass.go
new file mode 100644
index 000000000..a8fb08ca4
--- /dev/null
+++ b/internal/host/dbpass.go
@@ -0,0 +1,647 @@
+package host
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "io/fs"
+ "log"
+ "os"
+ "path/filepath"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "time"
+
+ "github.com/carlosframework/platform/internal/activator"
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The database transfer worker (`carlos db pull` / `carlos db push`):
+// everything a transfer does to DATA happens here, on the box that owns the
+// route. The console only writes requests onto the record and stages bytes
+// in the control bucket, because the control bucket is the one store it can
+// reach; this box reaches both it and the data (hibernate) bucket, so it is
+// the bridge between the member's upload and the instance's replica.
+//
+// One pass = one round per requested transfer this box owns, on the same
+// trigger as the reconcile and move passes (serial bumps land in seconds).
+// Restart-safe because every step derives from the record and the buckets.
+//
+// PULL is a read: restore the host's replica (WITHOUT the lease — a reader
+// costs the holder nothing, lease.RestoreCopy) into scratch, prove it opens,
+// put it in the control bucket, mark done. It never touches the route. The
+// snapshot is the REPLICA, which for a sleeping instance is byte-final and
+// for a running one is at most the sync interval behind.
+//
+// PUSH is a swap, and its ordering is the never-two-writers argument again,
+// with the local warm cache as the thing that must not win:
+//
+// route Status "maintenance" (edge stops serving; the tick and due-wake
+// gates refuse a non-ready row — and the word
+// PERSISTS in the registry across restarts)
+// → wake fence (in-process; closes the tick race G12 closes
+// for moves; re-installed every pass)
+// → child quiesced (Quiesced(host) — retries next pass until true)
+// → local checkpoint (the replica carries the tail before it is
+// backed up)
+// → backup (replica → scratch → backups/<id>.db; Step
+// "backed-up" + the id on the record)
+// → replace (push.db → scratch, CheckDB; DELETE the
+// replica prefix; replicate scratch into it
+// under a fresh lease; purge the local DB so
+// the next wake RESTORES; Step "replaced")
+// → finish (delete push.db, prune backups, Status
+// "ready", lift fence, mark done)
+//
+// Every step is idempotent and the record's Step says where to resume. The
+// crash points, one by one:
+//
+// - Before the status write: nothing happened; the next pass starts over.
+// - After the status write, any time before "replaced": the registry
+// holds "maintenance", so no tick, sweep or request can wake the host
+// even though a restart dropped the fence. The next pass re-fences,
+// re-checks Quiesced, and continues from Step.
+// - After the backup is put but before Step records it: a second backup
+// object with a new id is written on resume; the orphan is pruned by
+// the keep-newest rule. Never a lost backup.
+// - After DeletePrefix and before the replicate completes: the replica
+// is EMPTY. This is the one window where "mark failed and release"
+// would be wrong — a wake with the local file still present would use
+// the OLD warm cache and replicate it over an empty prefix (a silent
+// un-push), and a wake with it absent would boot fresh and SEED. So an
+// error here returns without failing the record: status stays
+// maintenance, the request stays in flight, and the next pass re-runs
+// the replace (push.db is still staged; the delete is idempotent; the
+// replicate is the same fresh lineage). Proven against the vendored
+// litestream in internal/lease/replace_test.go.
+// - After the replicate, before the purge: resume re-runs the replace;
+// the local DB is still the old one but unreachable behind the status.
+// - After the purge, before Step "replaced": same, with the checkpoint
+// skipped (no local file).
+// - After "replaced": the data is done. Resume runs finish only — no
+// status write, no fence, because a wake now would restore exactly the
+// pushed database, which is the point.
+//
+// Failure classes: a dbFailure (corrupt upload, no replica, over the cap)
+// marks the transfer failed and releases the route; anything else (an S3
+// blip, a lease held for another second) is returned, logged through the
+// trouble throttle, and retried next pass with the route still held.
+
+// dbMaintenanceStatus is the route status word a push holds the route at.
+// Everything that serves or wakes a hibernating instance gates on Status ==
+// "ready" (edge.Reload, the activator's ServeScheduled and due-wake sweep,
+// the edges publisher), so any other word takes it out of service; this one
+// says why. Reconcile converges a provisioned row through narrow column
+// UPDATEs that never touch Status, so the word survives every pass until
+// this worker writes "ready" back.
+const dbMaintenanceStatus = "maintenance"
+
+// dbScratchDirName is where a transfer materializes files under DataDir:
+// restores in flight, uploads being checked. 0700, swept per transfer.
+const dbScratchDirName = ".db-transfer"
+
+// dbFailure is a transfer outcome the member needs to hear, as opposed to a
+// transient error the next pass should retry.
+type dbFailure struct{ msg string }
+
+func (f *dbFailure) Error() string { return f.msg }
+
+func failDB(format string, a ...any) error { return &dbFailure{msg: fmt.Sprintf(format, a...)} }
+
+// dbWorker is the worker's cross-pass state, RunAgent writes the pointer
+// once; both converge loops read it.
+type dbWorker struct {
+ cfg dbCfg
+ // pass runs one dbPass at a time, for the moves worker's reason: two
+ // concurrent passes could drive one push twice or lift a fence the
+ // other just installed.
+ pass sync.Mutex
+ mu sync.Mutex
+ mirror lease.MirrorStore
+ nudged sync.Map // host → true: the status→quiesce hop was nudged once
+ trouble sync.Map // host → time.Time of the last logged error
+ fenced sync.Map // host → uint64: the push's wake-fence generation
+}
+
+var armedDBWorker atomic.Pointer[dbWorker]
+
+// dbCfg carries what the worker needs, injectable for tests. The route
+// fields are the move worker's, for the same reasons (see moveCfg).
+type dbCfg struct {
+ Region string // this box's placement code, "" included (the home)
+ OwnBucket string // CARLOS_HIBERNATE_BUCKET
+ OwnBucketRegion string // CARLOS_HIBERNATE_REGION
+ OwnPrefix string // CARLOS_HIBERNATE_PREFIX
+ DataDir string // /data
+ // Mirror builds the same-region client DeletePrefix runs through. Built
+ // lazily and memoized: most passes carry no push.
+ Mirror func(ctx context.Context) (lease.MirrorStore, error)
+ Quiesced func(host string) bool
+ Fence func(ctx context.Context, host string, generation uint64) error
+ LiftFence func(host string, generation uint64) bool
+ SetStatus func(host, status string) error
+ OwnsRoute func(host string) bool
+ Route func(host string) (*registry.Route, error)
+ Checkpoint func(ctx context.Context, dbPath string) error
+ // Restore restores bucket/prefix into out WITHOUT the lease; false
+ // means the prefix holds no replica.
+ Restore func(ctx context.Context, out, bucket, prefix, region string) (bool, error)
+ // Replicate replicates dbPath into bucket/prefix under a fresh lease,
+ // waits for the remote position to catch up, and releases.
+ Replicate func(ctx context.Context, dbPath, bucket, prefix, region string) error
+ Now func() time.Time
+}
+
+// armDBPass wires the worker with what only RunAgent has in hand. Called
+// once, before svc.Start, beside armMovesPass.
+func armDBPass(hib activator.Config, dataDir string, reg *registry.Registry, svc *activator.Service) {
+ cfg := dbCfg{
+ Region: strings.TrimSpace(os.Getenv("CARLOS_REGION")),
+ OwnBucket: hib.Bucket,
+ OwnBucketRegion: hib.Region,
+ OwnPrefix: hib.Prefix,
+ DataDir: dataDir,
+ Mirror: func(ctx context.Context) (lease.MirrorStore, error) {
+ return lease.NewS3Mirror(ctx, hib.Region, hib.Region)
+ },
+ Quiesced: svc.Quiesced,
+ Fence: func(ctx context.Context, host string, generation uint64) error {
+ fctx, cancel := context.WithTimeout(ctx, 2*time.Second)
+ defer cancel()
+ if _, err := svc.FenceAndStop(fctx, host, generation); err != nil && !errors.Is(err, activator.ErrStopUnproven) {
+ return err
+ }
+ return nil
+ },
+ LiftFence: svc.LiftFence,
+ SetStatus: reg.SetStatus,
+ OwnsRoute: func(host string) bool {
+ rt, err := reg.Get(host)
+ return err == nil && rt != nil && rt.Provisioned
+ },
+ Route: reg.Get,
+ Checkpoint: func(ctx context.Context, dbPath string) error {
+ host := strings.TrimSuffix(filepath.Base(dbPath), ".db")
+ return replicateAndRelease(ctx, dbPath, hib.Bucket, activator.HibernatePrefix(hib.Prefix, host), hib.Region)
+ },
+ Restore: lease.RestoreCopy,
+ Replicate: replicateAndRelease,
+ Now: time.Now,
+ }
+ armedDBWorker.Store(&dbWorker{cfg: cfg})
+}
+
+// replicateAndRelease is one full replication of dbPath into bucket/prefix:
+// lease, sync until the remote position matches, close (final sync + lease
+// release). The move worker's checkpoint and the push's replace are the
+// same call over different files.
+func replicateAndRelease(ctx context.Context, dbPath, bucket, prefix, region string) error {
+ rep, err := lease.ReplicateWithLease(ctx, dbPath, bucket, prefix, region)
+ if err != nil {
+ return err
+ }
+ if err := rep.SyncAndWait(ctx); err != nil {
+ _ = rep.Close(ctx)
+ return err
+ }
+ return rep.Close(ctx)
+}
+
+// runDBPass is the loops' call: inert until armed.
+func runDBPass(ctx context.Context, s release.Store) {
+ w := armedDBWorker.Load()
+ if w == nil || s == nil {
+ return
+ }
+ if err := dbPass(ctx, s, w); err != nil {
+ log.Printf("agent: db pass: %v", err)
+ }
+}
+
+func (w *dbWorker) mirrorStore(ctx context.Context) (lease.MirrorStore, error) {
+ w.mu.Lock()
+ m := w.mirror
+ w.mu.Unlock()
+ if m != nil {
+ return m, nil
+ }
+ if w.cfg.Mirror == nil {
+ return nil, errors.New("db: no mirror client wired")
+ }
+ m, err := w.cfg.Mirror(ctx)
+ if err != nil {
+ return nil, err
+ }
+ w.mu.Lock()
+ w.mirror = m
+ w.mu.Unlock()
+ return m, nil
+}
+
+// dbPass runs one round over every requested transfer this box owns.
+func dbPass(ctx context.Context, s release.Store, w *dbWorker) error {
+ w.pass.Lock()
+ defer w.pass.Unlock()
+ cfg := w.cfg
+ recs, err := provision.ListAllRecords(ctx, s)
+ if err != nil {
+ return fmt.Errorf("db: list records: %w", err)
+ }
+ for _, rec := range recs {
+ if !rec.DBInFlight() || rec.Region != cfg.Region {
+ continue
+ }
+ // A move in flight owns the host (the transitions refuse the
+ // overlap; this is the belt to that brace — records have other
+ // writers).
+ if rec.Move != nil {
+ continue
+ }
+ // The class rails the console's dbGate also holds: only an
+ // activator-managed exec instance's database is this worker's to
+ // read or replace.
+ if rec.Pool != "" || !rec.HibernateOrDefault() || (rec.Kind != "" && rec.Kind != "instance") {
+ continue
+ }
+ if cfg.OwnsRoute != nil && !cfg.OwnsRoute(rec.Host) {
+ continue
+ }
+ if err := runDBRound(ctx, s, w, rec); err != nil {
+ w.logTrouble(rec.Host, err)
+ continue
+ }
+ w.trouble.Delete(rec.Host)
+ }
+ w.releaseStale(recs)
+ return nil
+}
+
+// releaseStale puts back any route this process is still holding for a
+// push that is no longer in flight — the record was cleared or rewritten
+// by another writer between two passes, so no round will ever reach
+// finishPush for it. The move worker's liftFences, for the same reason:
+// a fence with nothing behind it must not outlive the thing it guarded.
+func (w *dbWorker) releaseStale(recs []provision.Record) {
+ inFlight := map[string]bool{}
+ for _, rec := range recs {
+ if rec.DBInFlight() && rec.DB.Op == provision.DBPush {
+ inFlight[rec.Host] = true
+ }
+ }
+ w.fenced.Range(func(k, v any) bool {
+ host, gen := k.(string), v.(uint64)
+ if inFlight[host] {
+ return true
+ }
+ w.releaseRoute(host, gen)
+ log.Printf("db: %s: push no longer on the record — route released", host)
+ return true
+ })
+}
+
+// dbFenceGeneration is the push fence's generation: the request time, so
+// every pass of one push installs the same fence and a later push of the
+// same host fences above an earlier one.
+func dbFenceGeneration(d *provision.DBTransfer) uint64 {
+ if n := d.RequestedAt.UnixNano(); n > 0 {
+ return uint64(n)
+ }
+ return 1
+}
+
+// scratchPath is where one transfer's file lives while the box works on
+// it. The directory is created 0700 on first use: a restored database is
+// a whole tenant's data, and DataDir is world-readable on some boxes.
+func (w *dbWorker) scratchPath(host, id, what string) (string, error) {
+ dir := filepath.Join(w.cfg.DataDir, dbScratchDirName)
+ if err := os.MkdirAll(dir, 0o700); err != nil {
+ return "", err
+ }
+ return filepath.Join(dir, host+"-"+id+"-"+what+".db"), nil
+}
+
+// sweepScratch removes a scratch database and every sidecar litestream or
+// the driver may have left beside it. Called on every path out.
+func sweepScratch(path string) {
+ if path == "" {
+ return
+ }
+ activator.PurgeLocalDBArtifacts(path, true)
+}
+
+// routeDBPath is the row's own DBPath when the registry is wired, else the
+// legacy flat guess — the move worker's rule (isolated-exec design §9).
+func (w *dbWorker) routeDBPath(host string) string {
+ dbPath := filepath.Join(w.cfg.DataDir, host+".db")
+ if w.cfg.Route != nil {
+ if rt, err := w.cfg.Route(host); err == nil && rt != nil && rt.DBPath != "" {
+ dbPath = rt.DBPath
+ }
+ }
+ return dbPath
+}
+
+func runDBRound(ctx context.Context, s release.Store, w *dbWorker, rec provision.Record) error {
+ stage, err := provision.NewDBStage(rec.Account, rec.App, rec.Host)
+ if err != nil {
+ return err
+ }
+ switch rec.DB.Op {
+ case provision.DBPull:
+ err = w.runPull(ctx, s, rec, stage)
+ case provision.DBPush:
+ err = w.runPush(ctx, s, rec, stage)
+ default:
+ err = failDB("unknown transfer op %q", rec.DB.Op)
+ }
+ var f *dbFailure
+ if errors.As(err, &f) {
+ if _, cerr := provision.SetRecordDB(ctx, s, rec.Account, rec.App, rec.Host, provision.DBFail(rec.DB.ID, f.msg)); cerr != nil && !errors.Is(cerr, provision.ErrDBConflict) {
+ return cerr
+ }
+ nudgeSerial(ctx, s)
+ log.Printf("db: %s: %s %s failed: %s", rec.Host, rec.DB.Op, rec.DB.ID, f.msg)
+ return nil
+ }
+ return err
+}
+
+// runPull: replica → scratch → CheckDB → pull.db → done.
+func (w *dbWorker) runPull(ctx context.Context, s release.Store, rec provision.Record, stage provision.DBStage) error {
+ cfg := w.cfg
+ scratch, err := w.scratchPath(rec.Host, rec.DB.ID, "pull")
+ if err != nil {
+ return err
+ }
+ sweepScratch(scratch) // litestream refuses to restore over an existing file
+ defer sweepScratch(scratch)
+ prefix := activator.HibernatePrefix(cfg.OwnPrefix, rec.Host)
+ restored, err := cfg.Restore(ctx, scratch, cfg.OwnBucket, prefix, cfg.OwnBucketRegion)
+ if err != nil {
+ return fmt.Errorf("restore replica: %w", err)
+ }
+ if !restored {
+ return failDB("the instance has no replicated database yet")
+ }
+ if err := lease.CheckDB(scratch); err != nil {
+ return failDB("the replicated database does not open cleanly: %v", err)
+ }
+ body, err := readCapped(scratch)
+ if err != nil {
+ return err
+ }
+ // A retry after a crash between the put and the mark: the object is
+ // there under a create-only put, so clear it first. Idempotent.
+ if err := s.Delete(ctx, stage.PullKey(rec.DB.ID)); err != nil {
+ return fmt.Errorf("clear staged pull: %w", err)
+ }
+ if _, err := s.Put(ctx, stage.PullKey(rec.DB.ID), body, ""); err != nil {
+ return fmt.Errorf("stage pull: %w", err)
+ }
+ if _, err := provision.SetRecordDB(ctx, s, rec.Account, rec.App, rec.Host, provision.DBComplete(rec.DB.ID, int64(len(body)))); err != nil && !errors.Is(err, provision.ErrDBConflict) {
+ return err
+ }
+ nudgeSerial(ctx, s)
+ log.Printf("db: %s: pull %s staged (%d bytes)", rec.Host, rec.DB.ID, len(body))
+ return nil
+}
+
+// readCapped reads a scratch database whole, refusing one over the cap: the
+// control bucket store takes a byte slice, and the console serves the
+// object from memory, so the cap is a memory bound on both ends.
+func readCapped(path string) ([]byte, error) {
+ fi, err := os.Stat(path)
+ if err != nil {
+ return nil, err
+ }
+ if fi.Size() > provision.DBTransferMaxBytes {
+ return nil, failDB("the database is %d MiB; transfers are capped at %d MiB", fi.Size()>>20, provision.DBTransferMaxBytes>>20)
+ }
+ return os.ReadFile(path)
+}
+
+// runPush is the swap described in the file header. Step is the resume
+// point; each arm falls through to the next.
+func (w *dbWorker) runPush(ctx context.Context, s release.Store, rec provision.Record, stage provision.DBStage) error {
+ cfg := w.cfg
+ d := rec.DB
+ if d.Step == provision.DBStepReplaced {
+ // The data is done. Finish only — see the header's last crash
+ // point: a wake now restores the pushed database, so nothing is
+ // left to hold the route for.
+ return w.finishPush(ctx, s, rec, stage)
+ }
+ // Step 1+2: out of service, fenced, quiesced.
+ if err := cfg.SetStatus(rec.Host, dbMaintenanceStatus); err != nil {
+ return fmt.Errorf("set route status: %w", err)
+ }
+ gen := dbFenceGeneration(d)
+ if cfg.Fence != nil {
+ if err := cfg.Fence(ctx, rec.Host, gen); err != nil {
+ return fmt.Errorf("fence against wakes: %w", err)
+ }
+ w.fenced.Store(rec.Host, gen)
+ }
+ if !cfg.Quiesced(rec.Host) {
+ if _, done := w.nudged.LoadOrStore(rec.Host, true); !done {
+ nudgeSerial(ctx, s)
+ }
+ return nil // the retire is in flight; the nudged pass re-enters here
+ }
+ // The layout-migration fence, exactly as the move cut holds it: only a
+ // definite ErrNotExist counts as "no journal".
+ if _, err := os.Stat(provision.LayoutJournalPath(cfg.DataDir, rec.Host)); !errors.Is(err, fs.ErrNotExist) {
+ log.Printf("db: %s: layout migration in flight; leaving the push for the next pass", rec.Host)
+ return nil
+ }
+ dbPath := w.routeDBPath(rec.Host)
+ prefix := activator.HibernatePrefix(cfg.OwnPrefix, rec.Host)
+
+ // A failure from here on that is the member's to hear releases the
+ // route on the way out. A transient one keeps it held and retries.
+ release := func(err error) error {
+ var f *dbFailure
+ if errors.As(err, &f) {
+ w.releaseRoute(rec.Host, gen)
+ }
+ return err
+ }
+
+ if d.Step == "" {
+ // Step 3: the replica must carry the tail before it is backed up.
+ if cfg.Checkpoint != nil {
+ if _, statErr := os.Stat(dbPath); statErr == nil {
+ if err := cfg.Checkpoint(ctx, dbPath); err != nil {
+ return fmt.Errorf("final checkpoint: %w", err)
+ }
+ }
+ }
+ // Step 4: the backup.
+ backupID, err := w.backup(ctx, s, rec, stage, prefix)
+ if err != nil {
+ return release(err)
+ }
+ if _, err := provision.SetRecordDB(ctx, s, rec.Account, rec.App, rec.Host, provision.DBBackedUp(d.ID, backupID)); err != nil {
+ if errors.Is(err, provision.ErrDBConflict) {
+ return nil // someone else advanced or cleared it; re-read next pass
+ }
+ return err
+ }
+ d.Step, d.Backup = provision.DBStepBackedUp, backupID
+ if backupID == "" {
+ log.Printf("db: %s: push %s: nothing to back up (no replica yet)", rec.Host, d.ID)
+ } else {
+ log.Printf("db: %s: push %s: backup %s stored", rec.Host, d.ID, backupID)
+ }
+ }
+ if d.Step == provision.DBStepBackedUp {
+ // Step 5: the replace.
+ if err := w.replace(ctx, s, rec, stage, prefix, dbPath); err != nil {
+ return release(err)
+ }
+ if _, err := provision.SetRecordDB(ctx, s, rec.Account, rec.App, rec.Host, provision.DBReplaced(d.ID)); err != nil {
+ if errors.Is(err, provision.ErrDBConflict) {
+ return nil
+ }
+ return err
+ }
+ log.Printf("db: %s: push %s: replica replaced, warm cache purged", rec.Host, d.ID)
+ }
+ // Step 6.
+ return w.finishPush(ctx, s, rec, stage)
+}
+
+// backup restores the replica to scratch and stores it under a fresh backup
+// id. "" with a nil error means there was no replica to back up.
+func (w *dbWorker) backup(ctx context.Context, s release.Store, rec provision.Record, stage provision.DBStage, prefix string) (string, error) {
+ cfg := w.cfg
+ scratch, err := w.scratchPath(rec.Host, rec.DB.ID, "backup")
+ if err != nil {
+ return "", err
+ }
+ sweepScratch(scratch)
+ defer sweepScratch(scratch)
+ restored, err := cfg.Restore(ctx, scratch, cfg.OwnBucket, prefix, cfg.OwnBucketRegion)
+ if err != nil {
+ return "", fmt.Errorf("restore replica for backup: %w", err)
+ }
+ if !restored {
+ return "", nil
+ }
+ if err := lease.CheckDB(scratch); err != nil {
+ // A replica that does not open is still the only copy there is:
+ // refusing the push keeps it, and the member can pull it to look.
+ return "", failDB("the current replicated database does not open cleanly (%v) — refusing to replace it; pull it first", err)
+ }
+ body, err := readCapped(scratch)
+ if err != nil {
+ var f *dbFailure
+ if errors.As(err, &f) {
+ // The cap is what makes the backup restorable through the
+ // console; a push that could not be undone is refused.
+ return "", failDB("the current database cannot be backed up: %s — refusing to replace it without a backup", f.msg)
+ }
+ return "", err
+ }
+ id := provision.NewDBBackupID(cfg.Now())
+ if _, err := s.Put(ctx, stage.BackupKey(id), body, ""); err != nil {
+ return "", fmt.Errorf("store backup: %w", err)
+ }
+ if err := stage.PutBackupMeta(ctx, s, provision.DBBackup{ID: id, Bytes: int64(len(body)), CreatedAt: cfg.Now().UTC(), Request: rec.DB.ID}); err != nil {
+ return "", fmt.Errorf("store backup meta: %w", err)
+ }
+ return id, nil
+}
+
+// replace is step 5: push.db → scratch → CheckDB → DeletePrefix →
+// replicate → purge. An error after the delete is returned as-is (never a
+// dbFailure) so the caller keeps the request in flight and retries.
+func (w *dbWorker) replace(ctx context.Context, s release.Store, rec provision.Record, stage provision.DBStage, prefix, dbPath string) error {
+ cfg := w.cfg
+ body, _, err := s.Get(ctx, stage.PushKey(rec.DB.ID))
+ if errors.Is(err, os.ErrNotExist) {
+ return failDB("the uploaded database is no longer staged — push it again")
+ }
+ if err != nil {
+ return fmt.Errorf("read staged push: %w", err)
+ }
+ scratch, err := w.scratchPath(rec.Host, rec.DB.ID, "push")
+ if err != nil {
+ return err
+ }
+ sweepScratch(scratch)
+ defer sweepScratch(scratch)
+ if err := os.WriteFile(scratch, body, 0o600); err != nil {
+ return err
+ }
+ if err := lease.CheckDB(scratch); err != nil {
+ return failDB("the uploaded file is not a usable SQLite database: %v", err)
+ }
+ m, err := w.mirrorStore(ctx)
+ if err != nil {
+ return err
+ }
+ // The point of no return for this round: from here an error keeps the
+ // request in flight (see the header). The prefix must be a directory
+ // boundary for the listing, as MirrorPrefix normalizes it.
+ if err := lease.DeletePrefix(ctx, m, cfg.OwnBucket, strings.TrimSuffix(prefix, "/")+"/"); err != nil {
+ return fmt.Errorf("delete replica prefix: %w", err)
+ }
+ if err := cfg.Replicate(ctx, scratch, cfg.OwnBucket, prefix, cfg.OwnBucketRegion); err != nil {
+ return fmt.Errorf("replicate pushed database (replica prefix is empty until this succeeds — retrying): %w", err)
+ }
+ // The warm-cache purge: the next wake finds no local file and RESTORES
+ // from the replica, through the ordinary CheckDB + ownership path.
+ activator.PurgeLocalDBArtifacts(dbPath, true)
+ return nil
+}
+
+// finishPush is step 6: staging gone, backups pruned, route back, fence
+// lifted, record done.
+func (w *dbWorker) finishPush(ctx context.Context, s release.Store, rec provision.Record, stage provision.DBStage) error {
+ if err := stage.DeleteRequest(ctx, s, rec.DB.ID); err != nil {
+ return fmt.Errorf("delete staged push: %w", err)
+ }
+ if pruned, err := stage.PruneBackups(ctx, s, provision.DBBackupsKept); err != nil {
+ return fmt.Errorf("prune backups: %w", err)
+ } else if len(pruned) > 0 {
+ log.Printf("db: %s: pruned %d old backup(s): %v", rec.Host, len(pruned), pruned)
+ }
+ w.releaseRoute(rec.Host, dbFenceGeneration(rec.DB))
+ if _, err := provision.SetRecordDB(ctx, s, rec.Account, rec.App, rec.Host, provision.DBComplete(rec.DB.ID, rec.DB.Bytes)); err != nil && !errors.Is(err, provision.ErrDBConflict) {
+ return err
+ }
+ w.nudged.Delete(rec.Host)
+ nudgeSerial(ctx, s)
+ log.Printf("db: %s: push %s complete — the next wake restores the pushed database", rec.Host, rec.DB.ID)
+ return nil
+}
+
+// releaseRoute puts the route back in service and lifts this push's fence.
+// Status first: a fence lifted before the status write would open a window
+// where a tick could wake the host and find it "maintenance" — harmless,
+// but the other order is the one every reader expects.
+func (w *dbWorker) releaseRoute(host string, gen uint64) {
+ if err := w.cfg.SetStatus(host, "ready"); err != nil {
+ log.Printf("db: %s: restore route status: %v", host, err)
+ }
+ if w.cfg.LiftFence != nil {
+ if _, held := w.fenced.LoadAndDelete(host); held {
+ w.cfg.LiftFence(host, gen)
+ }
+ }
+}
+
+// logTrouble rate-limits per-host errors to one line a minute.
+func (w *dbWorker) logTrouble(host string, err error) {
+ now := time.Now()
+ if last, ok := w.trouble.Load(host); ok && now.Sub(last.(time.Time)) < time.Minute {
+ return
+ }
+ w.trouble.Store(host, now)
+ log.Printf("db: %s: %v", host, err)
+}
diff --git a/internal/host/dbpass_test.go b/internal/host/dbpass_test.go
new file mode 100644
index 000000000..708926174
--- /dev/null
+++ b/internal/host/dbpass_test.go
@@ -0,0 +1,767 @@
+package host
+
+import (
+ "context"
+ "database/sql"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/lease"
+ "github.com/carlosframework/platform/internal/provision"
+ "github.com/carlosframework/platform/internal/registry"
+ "github.com/carlosframework/platform/internal/release"
+
+ _ "modernc.org/sqlite"
+)
+
+// dbHarness fakes the two sides the worker bridges: the control bucket is a
+// real DirStore; the data bucket is a map of replica prefix → the SQLite
+// file that a restore of it yields (plus the fake mirror's key listing, so
+// DeletePrefix has something to delete and the test can see it happen).
+type dbHarness struct {
+ s release.Store
+ f *fakeMoveMirror
+ cfg dbCfg
+ replicas map[string]string // prefix → path of the DB a restore yields
+ statuses []string
+ ops []string
+ quiet bool
+ owns bool
+ fences map[string]uint64
+ // replicateErr, when set, is what Replicate answers — the
+ // DeletePrefix-then-crash window.
+ replicateErr error
+ restoreErr error
+}
+
+func newDBHarness(t *testing.T) *dbHarness {
+ t.Helper()
+ h := &dbHarness{s: release.NewDirStore(t.TempDir()), f: newFakeMoveMirror(), quiet: true, owns: true,
+ replicas: map[string]string{}, fences: map[string]uint64{}}
+ dataDir := t.TempDir()
+ h.cfg = dbCfg{
+ Region: "ie",
+ OwnBucket: "data-ie",
+ OwnBucketRegion: "eu-west-1",
+ OwnPrefix: "instances",
+ DataDir: dataDir,
+ Mirror: func(context.Context) (lease.MirrorStore, error) { return &loggingMirror{h: h}, nil },
+ Quiesced: func(string) bool { return h.quiet },
+ Fence: func(_ context.Context, host string, gen uint64) error {
+ h.fences[host] = gen
+ h.ops = append(h.ops, "fence "+host)
+ return nil
+ },
+ LiftFence: func(host string, gen uint64) bool {
+ if h.fences[host] != gen {
+ return false
+ }
+ delete(h.fences, host)
+ h.ops = append(h.ops, "lift "+host)
+ return true
+ },
+ SetStatus: func(host, status string) error {
+ h.statuses = append(h.statuses, host+"="+status)
+ h.ops = append(h.ops, "status "+status)
+ return nil
+ },
+ OwnsRoute: func(string) bool { return h.owns },
+ Checkpoint: func(_ context.Context, dbPath string) error {
+ h.ops = append(h.ops, "checkpoint "+filepath.Base(dbPath))
+ return nil
+ },
+ Restore: func(_ context.Context, out, bucket, prefix, region string) (bool, error) {
+ if h.restoreErr != nil {
+ return false, h.restoreErr
+ }
+ src, ok := h.replicas[prefix]
+ if !ok {
+ return false, nil
+ }
+ body, err := os.ReadFile(src)
+ if err != nil {
+ return false, err
+ }
+ h.ops = append(h.ops, "restore "+prefix)
+ return true, os.WriteFile(out, body, 0o600)
+ },
+ Replicate: func(_ context.Context, dbPath, bucket, prefix, region string) error {
+ if h.replicateErr != nil {
+ return h.replicateErr
+ }
+ // The lineage rule the worker relies on: the prefix must be
+ // EMPTY when the new database is replicated into it.
+ if left, _ := h.f.List(context.Background(), bucket, prefix+"/"); len(left) != 0 {
+ return fmt.Errorf("replicate into a non-empty prefix: %v", left)
+ }
+ body, err := os.ReadFile(dbPath)
+ if err != nil {
+ return err
+ }
+ keep := filepath.Join(h.cfg.DataDir, "replica-"+strings.ReplaceAll(prefix, "/", "_")+".db")
+ if err := os.WriteFile(keep, body, 0o600); err != nil {
+ return err
+ }
+ h.replicas[prefix] = keep
+ h.f.put(bucket, prefix+"/generations/new/snapshot", "pushed")
+ h.ops = append(h.ops, "replicate "+prefix)
+ return nil
+ },
+ Now: time.Now,
+ }
+ return h
+}
+
+func (h *dbHarness) worker() *dbWorker { return &dbWorker{cfg: h.cfg} }
+
+// loggingMirror is the harness's fake data bucket with its deletes folded
+// into the harness op log, so the test can assert the delete's place in the
+// push sequence beside the status, fence and replicate ops.
+type loggingMirror struct{ h *dbHarness }
+
+func (m *loggingMirror) List(ctx context.Context, bucket, prefix string) (map[string]string, error) {
+ return m.h.f.List(ctx, bucket, prefix)
+}
+func (m *loggingMirror) Copy(ctx context.Context, sb, sk, db, dk string) error {
+ return m.h.f.Copy(ctx, sb, sk, db, dk)
+}
+func (m *loggingMirror) Delete(ctx context.Context, bucket, key string) error {
+ m.h.ops = append(m.h.ops, "delete "+bucket+"/"+key)
+ return m.h.f.Delete(ctx, bucket, key)
+}
+
+// seedReplica gives host a replica: a real SQLite file with one table
+// holding `rows` rows, listed in the fake data bucket under the prefix.
+func (h *dbHarness) seedReplica(t *testing.T, host string, rows []string) {
+ t.Helper()
+ prefix := "instances/" + host
+ path := filepath.Join(h.cfg.DataDir, "seed-"+host+".db")
+ makeSQLite(t, path, "t", rows)
+ h.replicas[prefix] = path
+ h.f.put(h.cfg.OwnBucket, prefix+"/generations/old/snapshot", "e1")
+ h.f.put(h.cfg.OwnBucket, prefix+"/lock.json", "e2")
+}
+
+func (h *dbHarness) record(t *testing.T, host string, mut ...func(*provision.Record) error) provision.Record {
+ t.Helper()
+ rec := provision.Record{Schema: 1, Host: host, Slug: "s", App: "app", Account: "bab",
+ Region: "ie", Channel: "stable", CreatedAt: time.Now().UTC()}
+ if err := provision.CreateRecord(context.Background(), h.s, rec); err != nil {
+ t.Fatal(err)
+ }
+ for _, m := range mut {
+ if _, err := provision.SetRecordDB(context.Background(), h.s, "bab", "app", host, m); err != nil {
+ t.Fatal(err)
+ }
+ }
+ return h.read(t, host)
+}
+
+func (h *dbHarness) read(t *testing.T, host string) provision.Record {
+ t.Helper()
+ got, err := provision.GetRecord(context.Background(), h.s, "bab", "app", host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return *got
+}
+
+func (h *dbHarness) stage(t *testing.T, host string) provision.DBStage {
+ t.Helper()
+ st, err := provision.NewDBStage("bab", "app", host)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return st
+}
+
+func (h *dbHarness) pass(t *testing.T, w *dbWorker) {
+ t.Helper()
+ if err := dbPass(context.Background(), h.s, w); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func makeSQLite(t *testing.T, path, table string, rows []string) {
+ t.Helper()
+ db, err := sql.Open("sqlite", path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer db.Close()
+ if _, err := db.Exec(`CREATE TABLE ` + table + ` (id INTEGER PRIMARY KEY, body TEXT)`); err != nil {
+ t.Fatal(err)
+ }
+ for _, r := range rows {
+ if _, err := db.Exec(`INSERT INTO `+table+` (body) VALUES (?)`, r); err != nil {
+ t.Fatal(err)
+ }
+ }
+}
+
+func readRows(t *testing.T, path, table string) []string {
+ t.Helper()
+ db, err := sql.Open("sqlite", "file:"+path+"?mode=ro")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer db.Close()
+ rows, err := db.Query(`SELECT body FROM ` + table + ` ORDER BY id`)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer rows.Close()
+ var out []string
+ for rows.Next() {
+ var s string
+ if err := rows.Scan(&s); err != nil {
+ t.Fatal(err)
+ }
+ out = append(out, s)
+ }
+ return out
+}
+
+func scratchFiles(t *testing.T, dataDir string) []string {
+ t.Helper()
+ ents, err := os.ReadDir(filepath.Join(dataDir, dbScratchDirName))
+ if err != nil && !errors.Is(err, os.ErrNotExist) {
+ t.Fatal(err)
+ }
+ var out []string
+ for _, e := range ents {
+ out = append(out, e.Name())
+ }
+ return out
+}
+
+func TestDBPassPull(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"one", "two"})
+ id := provision.NewDBTransferID()
+ h.record(t, host, provision.DBRequest(provision.DBPull, id, time.Now()))
+
+ h.pass(t, h.worker())
+ got := h.read(t, host)
+ if got.DB.State != provision.DBDone || got.DB.Bytes == 0 {
+ t.Fatalf("after pull: %+v", got.DB)
+ }
+ raw, _, err := h.s.Get(context.Background(), h.stage(t, host).PullKey(id))
+ if err != nil {
+ t.Fatalf("pull.db not staged: %v", err)
+ }
+ if int64(len(raw)) != got.DB.Bytes {
+ t.Fatalf("bytes %d on the record, %d staged", got.DB.Bytes, len(raw))
+ }
+ out := filepath.Join(t.TempDir(), "pulled.db")
+ if err := os.WriteFile(out, raw, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if rows := readRows(t, out, "t"); fmt.Sprint(rows) != "[one two]" {
+ t.Fatalf("staged pull = %v", rows)
+ }
+ if files := scratchFiles(t, h.cfg.DataDir); len(files) != 0 {
+ t.Fatalf("scratch left behind: %v", files)
+ }
+ // A pull never touches the route.
+ if len(h.statuses) != 0 || len(h.fences) != 0 {
+ t.Fatalf("a pull touched the route: statuses=%v fences=%v", h.statuses, h.fences)
+ }
+ // And the pass is inert for a done record.
+ before := len(h.ops)
+ h.pass(t, h.worker())
+ if len(h.ops) != before {
+ t.Fatal("a done pull ran again")
+ }
+ if _, _, err := h.s.Get(context.Background(), "control/serial"); err != nil {
+ t.Fatal("the serial was not nudged")
+ }
+}
+
+func TestDBPassPullFailures(t *testing.T) {
+ // No replica: failed with the sentence, no object staged.
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ id := provision.NewDBTransferID()
+ h.record(t, host, provision.DBRequest(provision.DBPull, id, time.Now()))
+ h.pass(t, h.worker())
+ if got := h.read(t, host); got.DB.State != provision.DBFailed || !strings.Contains(got.DB.Error, "no replicated database") {
+ t.Fatalf("pull with no replica: %+v", got.DB)
+ }
+ // A corrupt replica: failed, and the scratch swept.
+ h = newDBHarness(t)
+ junk := filepath.Join(h.cfg.DataDir, "junk.db")
+ if err := os.WriteFile(junk, []byte("not a database at all, not even close, really"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ h.replicas["instances/"+host] = junk
+ h.record(t, host, provision.DBRequest(provision.DBPull, id, time.Now()))
+ h.pass(t, h.worker())
+ if got := h.read(t, host); got.DB.State != provision.DBFailed || !strings.Contains(got.DB.Error, "does not open cleanly") {
+ t.Fatalf("pull of a corrupt replica: %+v", got.DB)
+ }
+ if files := scratchFiles(t, h.cfg.DataDir); len(files) != 0 {
+ t.Fatalf("scratch left behind: %v", files)
+ }
+ // Over the cap: failed naming the cap.
+ h = newDBHarness(t)
+ h.seedReplica(t, host, []string{"one"})
+ old := provision.DBTransferMaxBytes
+ provision.DBTransferMaxBytes = 1024
+ t.Cleanup(func() { provision.DBTransferMaxBytes = old })
+ h.record(t, host, provision.DBRequest(provision.DBPull, id, time.Now()))
+ h.pass(t, h.worker())
+ if got := h.read(t, host); got.DB.State != provision.DBFailed || !strings.Contains(got.DB.Error, "capped") {
+ t.Fatalf("pull over the cap: %+v", got.DB)
+ }
+ // A transient restore error: still requested, retried next pass.
+ provision.DBTransferMaxBytes = old
+ h = newDBHarness(t)
+ h.seedReplica(t, host, []string{"one"})
+ h.restoreErr = errors.New("s3 blip")
+ h.record(t, host, provision.DBRequest(provision.DBPull, id, time.Now()))
+ w := h.worker()
+ h.pass(t, w)
+ if got := h.read(t, host); got.DB.State != provision.DBRequested {
+ t.Fatalf("a transient error settled the pull: %+v", got.DB)
+ }
+ h.restoreErr = nil
+ h.pass(t, w)
+ if got := h.read(t, host); got.DB.State != provision.DBDone {
+ t.Fatalf("the retry did not complete: %+v", got.DB)
+ }
+}
+
+// stagePush uploads a real SQLite file as the push and writes the request,
+// the way the console does.
+func (h *dbHarness) stagePush(t *testing.T, host string, rows []string) string {
+ t.Helper()
+ id := provision.NewDBTransferID()
+ path := filepath.Join(t.TempDir(), "upload.db")
+ makeSQLite(t, path, "pushed", rows)
+ body, err := os.ReadFile(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := h.s.Put(context.Background(), h.stage(t, host).PushKey(id), body, ""); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := provision.SetRecordDB(context.Background(), h.s, "bab", "app", host, provision.DBRequest(provision.DBPush, id, time.Now())); err != nil {
+ t.Fatal(err)
+ }
+ return id
+}
+
+func TestDBPassPushHappyPath(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"old-1", "old-2"})
+ h.record(t, host)
+ local := filepath.Join(h.cfg.DataDir, host+".db")
+ if err := writeFileForTest(local, "warm cache"); err != nil {
+ t.Fatal(err)
+ }
+ st := h.stage(t, host)
+ // Six older backups, so the prune has something to do.
+ base := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+ for i := 0; i < 6; i++ {
+ bid := provision.NewDBBackupID(base.Add(time.Duration(i) * time.Hour))
+ if _, err := h.s.Put(context.Background(), st.BackupKey(bid), []byte("x"), ""); err != nil {
+ t.Fatal(err)
+ }
+ }
+ id := h.stagePush(t, host, []string{"new-1", "new-2", "new-3"})
+
+ h.pass(t, h.worker())
+ got := h.read(t, host)
+ if got.DB.State != provision.DBDone || got.DB.Step != provision.DBStepReplaced || got.DB.Backup == "" {
+ t.Fatalf("after push: %+v", got.DB)
+ }
+ // The order: status, fence, checkpoint, restore (backup), delete the
+ // old prefix, replicate, status ready, lift.
+ joined := strings.Join(h.ops, "\n")
+ want := []string{"status maintenance", "fence " + host, "checkpoint " + host + ".db", "restore instances/" + host,
+ "delete data-ie/instances/" + host + "/generations/old/snapshot", "replicate instances/" + host, "status ready", "lift " + host}
+ last := -1
+ for _, op := range want {
+ i := strings.Index(joined, op)
+ if i < 0 || i < last {
+ t.Fatalf("op order: %q missing or out of place in\n%s", op, joined)
+ }
+ last = i
+ }
+ // The lease lock was deleted with the prefix (a deleted prefix holds
+ // nothing back).
+ if left, _ := h.f.List(context.Background(), "data-ie", "instances/"+host+"/"); len(left) != 1 {
+ t.Fatalf("old prefix after replace: %v (want only the new snapshot)", left)
+ }
+ // The replica now restores the pushed content.
+ if rows := readRows(t, h.replicas["instances/"+host], "pushed"); fmt.Sprint(rows) != "[new-1 new-2 new-3]" {
+ t.Fatalf("replica after push = %v", rows)
+ }
+ // The backup is the OLD content, with its sidecar.
+ raw, _, err := h.s.Get(context.Background(), st.BackupKey(got.DB.Backup))
+ if err != nil {
+ t.Fatalf("backup not stored: %v", err)
+ }
+ bk := filepath.Join(t.TempDir(), "backup.db")
+ if err := os.WriteFile(bk, raw, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if rows := readRows(t, bk, "t"); fmt.Sprint(rows) != "[old-1 old-2]" {
+ t.Fatalf("backup = %v", rows)
+ }
+ backups, err := st.ListBackups(context.Background(), h.s)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(backups) != provision.DBBackupsKept || backups[0].ID != got.DB.Backup || backups[0].Request != id || backups[0].Bytes != int64(len(raw)) {
+ t.Fatalf("backups after push: %+v", backups)
+ }
+ // The warm cache is gone, the staged push is gone, scratch is swept.
+ if fileExistsForTest(local) {
+ t.Fatal("the local DB survived the push — the next wake would use the old warm cache")
+ }
+ if _, _, err := h.s.Get(context.Background(), st.PushKey(id)); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("push.db after finish: %v", err)
+ }
+ if files := scratchFiles(t, h.cfg.DataDir); len(files) != 0 {
+ t.Fatalf("scratch left behind: %v", files)
+ }
+ if len(h.fences) != 0 {
+ t.Fatalf("fence still held: %v", h.fences)
+ }
+ if h.statuses[len(h.statuses)-1] != host+"=ready" {
+ t.Fatalf("statuses = %v, want ready last", h.statuses)
+ }
+ if _, _, err := h.s.Get(context.Background(), "control/serial"); err != nil {
+ t.Fatal("the serial was not nudged")
+ }
+}
+
+func TestDBPassPushWaitsForQuiesce(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"old"})
+ h.record(t, host)
+ h.stagePush(t, host, []string{"new"})
+ h.quiet = false
+ w := h.worker()
+ h.pass(t, w)
+ if fmt.Sprint(h.ops) != "[status maintenance fence "+host+"]" {
+ t.Fatalf("un-quiesced push did more than hold the route: %v", h.ops)
+ }
+ if got := h.read(t, host); got.DB.State != provision.DBRequested || got.DB.Step != "" {
+ t.Fatalf("un-quiesced push advanced: %+v", got.DB)
+ }
+ if _, _, err := h.s.Get(context.Background(), "control/serial"); err != nil {
+ t.Fatal("the status→quiesce hop was not nudged")
+ }
+ // A second un-quiesced pass does not nudge again (one per host).
+ _ = h.s.Delete(context.Background(), "control/serial")
+ h.pass(t, w)
+ if _, _, err := h.s.Get(context.Background(), "control/serial"); err == nil {
+ t.Fatal("a wedged stop turned the loop into a self-triggering cycle")
+ }
+ h.quiet = true
+ h.pass(t, w)
+ if got := h.read(t, host); got.DB.State != provision.DBDone {
+ t.Fatalf("quiesced pass did not finish: %+v", got.DB)
+ }
+}
+
+// Resume from each step: a push that died after backed-up never takes a
+// second backup; one that died after replaced never re-fences.
+func TestDBPassPushResumes(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"old"})
+ h.record(t, host)
+ id := h.stagePush(t, host, []string{"new"})
+ if _, err := provision.SetRecordDB(context.Background(), h.s, "bab", "app", host, provision.DBBackedUp(id, "20260101-000000-deadbeef")); err != nil {
+ t.Fatal(err)
+ }
+ h.pass(t, h.worker())
+ got := h.read(t, host)
+ if got.DB.State != provision.DBDone || got.DB.Backup != "20260101-000000-deadbeef" {
+ t.Fatalf("resume from backed-up: %+v", got.DB)
+ }
+ for _, op := range h.ops {
+ if strings.HasPrefix(op, "restore ") {
+ t.Fatalf("a resumed push took a second backup: %v", h.ops)
+ }
+ }
+ if rows := readRows(t, h.replicas["instances/"+host], "pushed"); fmt.Sprint(rows) != "[new]" {
+ t.Fatalf("replica after resumed push = %v", rows)
+ }
+
+ // From replaced: finish only.
+ h = newDBHarness(t)
+ h.seedReplica(t, host, []string{"old"})
+ h.record(t, host)
+ id = h.stagePush(t, host, []string{"new"})
+ for _, m := range []func(*provision.Record) error{provision.DBBackedUp(id, ""), provision.DBReplaced(id)} {
+ if _, err := provision.SetRecordDB(context.Background(), h.s, "bab", "app", host, m); err != nil {
+ t.Fatal(err)
+ }
+ }
+ h.pass(t, h.worker())
+ if got := h.read(t, host); got.DB.State != provision.DBDone {
+ t.Fatalf("resume from replaced: %+v", got.DB)
+ }
+ if fmt.Sprint(h.ops) != "[status ready]" {
+ t.Fatalf("resume from replaced did more than finish: %v", h.ops)
+ }
+ if _, _, err := h.s.Get(context.Background(), h.stage(t, host).PushKey(id)); !errors.Is(err, os.ErrNotExist) {
+ t.Fatal("push.db survived the finish")
+ }
+}
+
+// A corrupt upload fails cleanly: the replica is untouched, the route is
+// back in service, the fence lifted, the backup recorded.
+func TestDBPassPushCorruptUpload(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"old"})
+ h.record(t, host)
+ id := provision.NewDBTransferID()
+ junk := make([]byte, 4096)
+ copy(junk, provision.SQLiteMagic) // passes the console's magic check, fails CheckDB
+ if _, err := h.s.Put(context.Background(), h.stage(t, host).PushKey(id), junk, ""); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := provision.SetRecordDB(context.Background(), h.s, "bab", "app", host, provision.DBRequest(provision.DBPush, id, time.Now())); err != nil {
+ t.Fatal(err)
+ }
+ h.pass(t, h.worker())
+ got := h.read(t, host)
+ if got.DB.State != provision.DBFailed || !strings.Contains(got.DB.Error, "not a usable SQLite database") || got.DB.Backup == "" {
+ t.Fatalf("corrupt upload: %+v", got.DB)
+ }
+ for _, op := range h.ops {
+ if strings.HasPrefix(op, "delete ") || strings.HasPrefix(op, "replicate ") {
+ t.Fatalf("the replica was touched by a corrupt upload: %v", h.ops)
+ }
+ }
+ if h.statuses[len(h.statuses)-1] != host+"=ready" || len(h.fences) != 0 {
+ t.Fatalf("route not released: statuses=%v fences=%v", h.statuses, h.fences)
+ }
+ if files := scratchFiles(t, h.cfg.DataDir); len(files) != 0 {
+ t.Fatalf("scratch left behind: %v", files)
+ }
+}
+
+// The DeletePrefix-then-crash window: a replicate that fails leaves the
+// request IN FLIGHT with the route held, and the next pass re-runs the
+// replace to completion. Never failed-and-released over an empty replica.
+func TestDBPassPushHoldsTheRouteAcrossAnEmptyReplica(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"old"})
+ h.record(t, host)
+ local := filepath.Join(h.cfg.DataDir, host+".db")
+ if err := writeFileForTest(local, "warm cache"); err != nil {
+ t.Fatal(err)
+ }
+ h.stagePush(t, host, []string{"new"})
+ h.replicateErr = errors.New("s3: connection reset")
+ w := h.worker()
+ h.pass(t, w)
+ got := h.read(t, host)
+ if got.DB.State != provision.DBRequested || got.DB.Step != provision.DBStepBackedUp {
+ t.Fatalf("after a failed replicate: %+v, want still requested at backed-up", got.DB)
+ }
+ if left, _ := h.f.List(context.Background(), "data-ie", "instances/"+host+"/"); len(left) != 0 {
+ t.Fatalf("the prefix was not deleted before the replicate: %v", left)
+ }
+ if h.statuses[len(h.statuses)-1] != host+"=maintenance" || h.fences[host] == 0 {
+ t.Fatalf("route released over an empty replica: statuses=%v fences=%v", h.statuses, h.fences)
+ }
+ if !fileExistsForTest(local) {
+ t.Fatal("the local DB was purged before the replica carried the new lineage")
+ }
+ // The next pass: same step, replicate works, done.
+ h.replicateErr = nil
+ h.pass(t, w)
+ got = h.read(t, host)
+ if got.DB.State != provision.DBDone || got.DB.Step != provision.DBStepReplaced {
+ t.Fatalf("retry after the empty-replica window: %+v", got.DB)
+ }
+ if rows := readRows(t, h.replicas["instances/"+host], "pushed"); fmt.Sprint(rows) != "[new]" {
+ t.Fatalf("replica after retry = %v", rows)
+ }
+ if fileExistsForTest(local) || len(h.fences) != 0 || h.statuses[len(h.statuses)-1] != host+"=ready" {
+ t.Fatalf("after the retry: local=%v fences=%v statuses=%v", fileExistsForTest(local), h.fences, h.statuses)
+ }
+ // Exactly one backup was taken across both passes.
+ backups, _ := h.stage(t, host).ListBackups(context.Background(), h.s)
+ if len(backups) != 1 || backups[0].ID != got.DB.Backup {
+ t.Fatalf("backups after retry: %+v", backups)
+ }
+}
+
+// A push onto an instance with no replica yet: no backup, the replace
+// still runs, Backup stays empty on the record.
+func TestDBPassPushWithoutAReplica(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.record(t, host)
+ h.stagePush(t, host, []string{"new"})
+ h.pass(t, h.worker())
+ got := h.read(t, host)
+ if got.DB.State != provision.DBDone || got.DB.Backup != "" {
+ t.Fatalf("push with no replica: %+v", got.DB)
+ }
+ if rows := readRows(t, h.replicas["instances/"+host], "pushed"); fmt.Sprint(rows) != "[new]" {
+ t.Fatalf("replica after push = %v", rows)
+ }
+ if backups, _ := h.stage(t, host).ListBackups(context.Background(), h.s); len(backups) != 0 {
+ t.Fatalf("a backup of nothing: %+v", backups)
+ }
+}
+
+// A backup that cannot be made (the replica is over the cap) refuses the
+// push rather than replacing without one.
+func TestDBPassPushRefusesWithoutABackup(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"old"})
+ h.record(t, host)
+ h.stagePush(t, host, []string{"new"})
+ old := provision.DBTransferMaxBytes
+ provision.DBTransferMaxBytes = 1024
+ t.Cleanup(func() { provision.DBTransferMaxBytes = old })
+ h.pass(t, h.worker())
+ got := h.read(t, host)
+ if got.DB.State != provision.DBFailed || !strings.Contains(got.DB.Error, "without a backup") {
+ t.Fatalf("push over the backup cap: %+v", got.DB)
+ }
+ if left, _ := h.f.List(context.Background(), "data-ie", "instances/"+host+"/"); len(left) != 2 {
+ t.Fatalf("the replica was touched: %v", left)
+ }
+ if h.statuses[len(h.statuses)-1] != host+"=ready" || len(h.fences) != 0 {
+ t.Fatalf("route not released: statuses=%v fences=%v", h.statuses, h.fences)
+ }
+}
+
+// The rails: a move in flight, a pooled or non-hibernating record, a route
+// another box owns — skipped untouched.
+func TestDBPassRailsAndOwnership(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"old"})
+ h.record(t, host, provision.DBRequest(provision.DBPull, provision.NewDBTransferID(), time.Now()))
+
+ h.owns = false
+ h.pass(t, h.worker())
+ if len(h.ops) != 0 || h.read(t, host).DB.State != provision.DBRequested {
+ t.Fatalf("a non-owning box acted: %v", h.ops)
+ }
+ h.owns = true
+
+ // A move stamped on the record by another writer: skipped.
+ rec := h.read(t, host)
+ rec.Move = &provision.MoveState{To: "uk", State: provision.MovePreparing}
+ _, etag, _ := h.s.Get(context.Background(), provision.RecordKey("bab", "app", host))
+ if err := provision.UpdateRecord(context.Background(), h.s, rec, etag); err != nil {
+ t.Fatal(err)
+ }
+ h.pass(t, h.worker())
+ if len(h.ops) != 0 {
+ t.Fatalf("a record mid-move was acted on: %v", h.ops)
+ }
+
+ pooled := "p.go.example.com"
+ prec := provision.Record{Schema: 1, Host: pooled, Slug: "p", App: "app", Account: "bab",
+ Region: "ie", Pool: "dedicated-x", Channel: "stable", CreatedAt: time.Now().UTC()}
+ if err := provision.CreateRecord(context.Background(), h.s, prec); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := provision.SetRecordDB(context.Background(), h.s, "bab", "app", pooled, provision.DBRequest(provision.DBPull, provision.NewDBTransferID(), time.Now())); err != nil {
+ t.Fatal(err)
+ }
+ h.pass(t, h.worker())
+ if h.read(t, pooled).DB.State != provision.DBRequested {
+ t.Fatal("a pooled record's pull ran on the platform box")
+ }
+}
+
+// A push checkpoints and purges the row's OWN DBPath when the registry is
+// wired, the move worker's rule.
+func TestDBPassPushUsesRouteDBPath(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"old"})
+ h.record(t, host)
+ nested := provision.NestedDBPath(h.cfg.DataDir, host)
+ if err := os.MkdirAll(filepath.Dir(nested), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := writeFileForTest(nested, "stale"); err != nil {
+ t.Fatal(err)
+ }
+ h.cfg.Route = func(string) (*registry.Route, error) { return &registry.Route{Host: host, DBPath: nested}, nil }
+ h.stagePush(t, host, []string{"new"})
+ h.pass(t, h.worker())
+ found := false
+ for _, op := range h.ops {
+ if op == "checkpoint "+filepath.Base(nested) {
+ found = true
+ }
+ }
+ if !found {
+ t.Fatalf("checkpoint did not use the route's DBPath: %v", h.ops)
+ }
+ if fileExistsForTest(nested) {
+ t.Fatal("the route's own DB survived the push")
+ }
+}
+
+func TestDBPassRunsOneAtATime(t *testing.T) {
+ h := newDBHarness(t)
+ w := h.worker()
+ w.pass.Lock()
+ done := make(chan error, 1)
+ go func() { done <- dbPass(context.Background(), h.s, w) }()
+ select {
+ case <-done:
+ t.Fatal("a second pass ran while the first held the worker")
+ case <-time.After(50 * time.Millisecond):
+ }
+ w.pass.Unlock()
+ if err := <-done; err != nil {
+ t.Fatal(err)
+ }
+}
+
+// A route held for a push whose record was rewritten out from under it is
+// released on the next pass rather than stranded at "maintenance".
+func TestDBPassReleasesARouteWhosePushVanished(t *testing.T) {
+ h := newDBHarness(t)
+ host := "a.go.example.com"
+ h.seedReplica(t, host, []string{"old"})
+ h.record(t, host)
+ h.stagePush(t, host, []string{"new"})
+ h.quiet = false
+ w := h.worker()
+ h.pass(t, w) // held: status maintenance, fenced, waiting for quiesce
+ if h.fences[host] == 0 {
+ t.Fatal("the push did not fence")
+ }
+ rec := h.read(t, host)
+ rec.DB = nil
+ _, etag, _ := h.s.Get(context.Background(), provision.RecordKey("bab", "app", host))
+ if err := provision.UpdateRecord(context.Background(), h.s, rec, etag); err != nil {
+ t.Fatal(err)
+ }
+ h.pass(t, w)
+ if len(h.fences) != 0 || h.statuses[len(h.statuses)-1] != host+"=ready" {
+ t.Fatalf("route not released after the record lost its push: fences=%v statuses=%v", h.fences, h.statuses)
+ }
+}
diff --git a/internal/lease/check.go b/internal/lease/check.go
index 8fdac6469..9700c368d 100644
--- a/internal/lease/check.go
+++ b/internal/lease/check.go
@@ -107,6 +107,19 @@ func VerifyReplica(ctx context.Context, bucket, prefix, region, dir string) (uin
return ChangeCounter(out)
}
+// RestoreCopy restores s3://bucket/prefix into out and KEEPS the file — a
+// pull's snapshot, a push's pre-swap backup — without taking the lease.
+// Restore's contract demands the lease because it materializes the LIVE
+// database; VerifyReplica's argument (above) is why a reader needs none:
+// what the lease forbids is a second WRITER, and a copy that is never
+// replicated from cannot shadow anything. The caller owns the copy and its
+// sidecars (out, out-wal, out-shm) and must never replicate from it under
+// the same prefix without the lease. Same false-means-no-replica answer as
+// Restore.
+func RestoreCopy(ctx context.Context, out, bucket, prefix, region string) (bool, error) {
+ return Restore(ctx, out, bucket, prefix, region)
+}
+
// ChangeCounter reads the SQLite file-change counter: bytes 24-27 of the
// database header, big-endian. SQLite increments it on every write transaction
// that modifies the main database file, which makes it the cheapest honest
diff --git a/internal/lease/replace_test.go b/internal/lease/replace_test.go
new file mode 100644
index 000000000..b70b05382
--- /dev/null
+++ b/internal/lease/replace_test.go
@@ -0,0 +1,169 @@
+package lease
+
+import (
+ "context"
+ "database/sql"
+ "fmt"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/benbjohnson/litestream"
+ lsfile "github.com/benbjohnson/litestream/file"
+)
+
+// The push worker's replace step (internal/host/dbpass.go, step 5) is:
+// delete the replica prefix, then replicate the pushed file into that same
+// prefix under a fresh lease, so the NEXT wake — which restores only when
+// the local file is absent — materializes exactly the pushed database. This
+// test runs that sequence against the vendored litestream with a
+// file-backed replica client (the same seam the checkpoint tests use), so
+// what it proves is litestream's own behaviour, not a fake's.
+//
+// Two properties, both required by the worker's crash-point argument:
+//
+// - after delete + replicate, a restore yields the pushed content and
+// nothing of the old lineage;
+// - the replace is idempotent — running delete + replicate a second time
+// (what a resumed push does after a crash between the two) still
+// restores the pushed content.
+func TestReplaceReplicaRestoresThePushedDatabase(t *testing.T) {
+ replica := t.TempDir()
+
+ // The old lineage: 50 rows, replicated and released the way an instance
+ // going to sleep leaves its prefix.
+ old := filepath.Join(t.TempDir(), "old.db")
+ seedDB(t, old)
+ replicateOnce(t, old, replica)
+ if rows := restoreRows(t, replica, filepath.Join(t.TempDir(), "check-old.db")); len(rows) != 50 || rows[0] != "row-0" {
+ t.Fatalf("control: the old lineage did not restore: %d rows %v", len(rows), rows)
+ }
+
+ // The pushed database: a different table shape and content, so a
+ // mixed lineage would be unmistakable.
+ pushed := filepath.Join(t.TempDir(), "pushed.db")
+ seedPushed(t, pushed, []string{"alpha", "beta", "gamma"})
+
+ // Step 5 as the worker runs it: delete the prefix, replicate the
+ // scratch file into it under a fresh lease, sync, close.
+ deletePrefixDir(t, replica)
+ replicateOnce(t, pushed, replica)
+ rows := restoreRows(t, replica, filepath.Join(t.TempDir(), "check-new.db"))
+ if fmt.Sprint(rows) != "[alpha beta gamma]" {
+ t.Fatalf("after replace, restore = %v, want exactly the pushed rows", rows)
+ }
+
+ // The resumed-push case: the same sequence again over the already
+ // replaced prefix. Same answer.
+ deletePrefixDir(t, replica)
+ replicateOnce(t, pushed, replica)
+ rows = restoreRows(t, replica, filepath.Join(t.TempDir(), "check-again.db"))
+ if fmt.Sprint(rows) != "[alpha beta gamma]" {
+ t.Fatalf("after a repeated replace, restore = %v, want exactly the pushed rows", rows)
+ }
+
+ // And the pushed file itself is untouched as a database — litestream
+ // switches it to WAL mode, which is what the worker's CheckDB-before
+ // and the later restore both tolerate.
+ if err := CheckDB(pushed); err != nil {
+ t.Fatalf("the pushed scratch file after replication: %v", err)
+ }
+}
+
+// replicateOnce is what the worker's Replicate closure does in production:
+// ReplicateWithLease + SyncAndWait + Close, here through the Held seam so
+// the replica is a directory rather than S3.
+func replicateOnce(t *testing.T, dbPath, replica string) {
+ t.Helper()
+ h := localHeld(t, &lockStore{}, 10e9)
+ h.replicaClient = lsfile.NewReplicaClient(replica)
+ rep, err := h.Replicate(context.Background(), dbPath, 0)
+ if err != nil {
+ t.Fatalf("replicate %s: %v", dbPath, err)
+ }
+ if err := rep.SyncAndWait(context.Background()); err != nil {
+ t.Fatalf("sync %s: %v", dbPath, err)
+ }
+ if err := rep.Close(context.Background()); err != nil {
+ t.Fatalf("close %s: %v", dbPath, err)
+ }
+}
+
+// deletePrefixDir is DeletePrefix for the file-backed replica: every object
+// under the prefix gone, the prefix itself still addressable.
+func deletePrefixDir(t *testing.T, replica string) {
+ t.Helper()
+ if err := os.RemoveAll(replica); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.MkdirAll(replica, 0o700); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func seedPushed(t *testing.T, path string, names []string) {
+ t.Helper()
+ db, err := sql.Open("sqlite", path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer db.Close()
+ if _, err := db.Exec(`CREATE TABLE pushed (id INTEGER PRIMARY KEY, name TEXT)`); err != nil {
+ t.Fatal(err)
+ }
+ for _, n := range names {
+ if _, err := db.Exec(`INSERT INTO pushed (name) VALUES (?)`, n); err != nil {
+ t.Fatal(err)
+ }
+ }
+}
+
+// restoreRows restores the replica and reads back whichever of the two
+// tables it carries, in id order — so a lineage mix-up shows up as the
+// wrong table, not merely the wrong count.
+func restoreRows(t *testing.T, replica, out string) []string {
+ t.Helper()
+ r := litestream.NewReplicaWithClient(nil, lsfile.NewReplicaClient(replica))
+ opt := litestream.NewRestoreOptions()
+ opt.OutputPath = out
+ if err := r.Restore(context.Background(), opt); err != nil {
+ t.Fatalf("restore: %v", err)
+ }
+ if err := CheckDB(out); err != nil {
+ t.Fatalf("restored file: %v", err)
+ }
+ db, err := sql.Open("sqlite", "file:"+out+"?mode=ro")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer db.Close()
+ var table string
+ // litestream keeps two bookkeeping tables of its own (_litestream_seq,
+ // _litestream_lock) beside the data; they are not the lineage.
+ if err := db.QueryRow(`SELECT group_concat(name, ',') FROM sqlite_master WHERE type='table' AND name NOT LIKE '\_litestream\_%' ESCAPE '\'`).Scan(&table); err != nil {
+ t.Fatal(err)
+ }
+ var q string
+ switch table {
+ case "t":
+ q = `SELECT body FROM t ORDER BY id`
+ case "pushed":
+ q = `SELECT name FROM pushed ORDER BY id`
+ default:
+ t.Fatalf("restored tables = %q, want exactly one of t / pushed", table)
+ }
+ rows, err := db.Query(q)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer rows.Close()
+ var out2 []string
+ for rows.Next() {
+ var s string
+ if err := rows.Scan(&s); err != nil {
+ t.Fatal(err)
+ }
+ out2 = append(out2, s)
+ }
+ return out2
+}
diff --git a/internal/provision/dbstage.go b/internal/provision/dbstage.go
new file mode 100644
index 000000000..18773af1a
--- /dev/null
+++ b/internal/provision/dbstage.go
@@ -0,0 +1,219 @@
+package provision
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "regexp"
+ "sort"
+ "strings"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The staging keys a database transfer moves through, all in the CONTROL
+// bucket (release.Store) — the one store the console can reach. Boxes reach
+// both it and their data (hibernate) bucket, so the box is the bridge: a pull
+// restores the replica and puts the snapshot here; a push takes the upload
+// from here and replicates it into the data bucket.
+//
+// apps/<sqid>/<app>/db/<host>/<id>/pull.db the box's snapshot for a pull
+// apps/<sqid>/<app>/db/<host>/<id>/push.db the member's upload for a push
+// apps/<sqid>/<app>/db/<host>/backups/<bid>.db pre-push backups (newest 5 kept)
+// apps/<sqid>/<app>/db/<host>/backups/<bid>.json each backup's size + time
+//
+// `db/` is a literal sibling of `instances/` under the app prefix, never
+// under it: ParseRecordKey walks instances/ and must never mistake a staged
+// database for a record. The host is shape-checked before any key is built
+// — it arrives from a URL segment, and RecordKey's own rule (path-traversal
+// defence, not tidiness) applies to every key that joins it.
+
+// DBStage names one instance's staging area. Built only through NewDBStage,
+// which is where the host shape is checked.
+type DBStage struct{ acct, app, host string }
+
+// NewDBStage validates the coordinates and returns the key builder.
+func NewDBStage(acct, app, host string) (DBStage, error) {
+ if acct == "" || app == "" {
+ return DBStage{}, errors.New("provision: db stage needs an account and an app")
+ }
+ if !ValidHostShape(host) {
+ return DBStage{}, fmt.Errorf("provision: db stage: %q is not a valid host", host)
+ }
+ return DBStage{acct: acct, app: app, host: host}, nil
+}
+
+// Host is the instance the stage belongs to.
+func (d DBStage) Host() string { return d.host }
+
+// Prefix is everything the instance stages, requests and backups alike.
+func (d DBStage) Prefix() string {
+ return release.AppPrefix(d.acct, d.app) + "db/" + d.host + "/"
+}
+
+// RequestPrefix is one request's own directory. The id is checked here
+// too: an id reaches this from the record (which only DBRequest writes) or
+// from a query string (which anyone writes).
+func (d DBStage) RequestPrefix(id string) string {
+ if !ValidDBTransferID(id) {
+ return d.Prefix() + "invalid/"
+ }
+ return d.Prefix() + id + "/"
+}
+
+// PullKey is where the box puts a pull's snapshot.
+func (d DBStage) PullKey(id string) string { return d.RequestPrefix(id) + "pull.db" }
+
+// PushKey is where the console puts a push's upload.
+func (d DBStage) PushKey(id string) string { return d.RequestPrefix(id) + "push.db" }
+
+// BackupsPrefix holds every pre-push backup of the instance.
+func (d DBStage) BackupsPrefix() string { return d.Prefix() + "backups/" }
+
+// BackupKey is one backup's database; BackupMetaKey its sidecar.
+func (d DBStage) BackupKey(id string) string { return d.BackupsPrefix() + id + ".db" }
+func (d DBStage) BackupMetaKey(id string) string { return d.BackupsPrefix() + id + ".json" }
+
+// DeleteRequest removes one request's staged objects (pull.db, push.db —
+// never a backup). Idempotent, like Store.Delete.
+func (d DBStage) DeleteRequest(ctx context.Context, s release.Store, id string) error {
+ if !ValidDBTransferID(id) {
+ return nil
+ }
+ keys, err := s.List(ctx, d.RequestPrefix(id))
+ if err != nil {
+ return err
+ }
+ for _, k := range keys {
+ if err := s.Delete(ctx, k); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// Backups are named by when they were taken, so a plain string sort is a
+// time sort: "20260929-120000-1a2b3c4d". The random tail keeps two backups
+// taken in the same second apart.
+var dbBackupIDRe = regexp.MustCompile(`^[0-9]{8}-[0-9]{6}-[0-9a-f]{8}$`)
+
+// ValidDBBackupID reports whether id has the shape NewDBBackupID mints.
+func ValidDBBackupID(id string) bool { return dbBackupIDRe.MatchString(id) }
+
+// NewDBBackupID mints a backup id for a backup taken at now.
+func NewDBBackupID(now time.Time) string {
+ return now.UTC().Format("20060102-150405") + "-" + NewDBTransferID()[:8]
+}
+
+// DBBackup is one listed backup: what `carlos db backups` prints.
+type DBBackup struct {
+ ID string `json:"id"`
+ Bytes int64 `json:"bytes"`
+ CreatedAt time.Time `json:"created_at"`
+ // Request is the push that took it, so a member can tie a backup to
+ // the push they remember making.
+ Request string `json:"request,omitempty"`
+}
+
+// DBBackupsKept is how many backups the box keeps per instance: after a
+// successful push it prunes everything older than the newest five.
+const DBBackupsKept = 5
+
+// PutBackupMeta writes a backup's sidecar. Create-only, like every staged
+// object: ids are unique, so a second write of one would be a bug.
+func (d DBStage) PutBackupMeta(ctx context.Context, s release.Store, b DBBackup) error {
+ if !ValidDBBackupID(b.ID) {
+ return fmt.Errorf("provision: backup id %q malformed", b.ID)
+ }
+ body, err := json.Marshal(b)
+ if err != nil {
+ return err
+ }
+ _, err = s.Put(ctx, d.BackupMetaKey(b.ID), body, "")
+ return err
+}
+
+// ListBackups lists the instance's backups newest first. A backup whose
+// sidecar is missing or unreadable is listed with what its id says (the
+// time) and zero bytes rather than hidden: the .db is what a restore needs,
+// and the sidecar is bookkeeping.
+func (d DBStage) ListBackups(ctx context.Context, s release.Store) ([]DBBackup, error) {
+ keys, err := s.List(ctx, d.BackupsPrefix())
+ if err != nil {
+ return nil, err
+ }
+ var ids []string
+ for _, k := range keys {
+ rel := strings.TrimPrefix(k, d.BackupsPrefix())
+ id, ok := strings.CutSuffix(rel, ".db")
+ if !ok || !ValidDBBackupID(id) {
+ continue
+ }
+ ids = append(ids, id)
+ }
+ sort.Sort(sort.Reverse(sort.StringSlice(ids)))
+ out := make([]DBBackup, 0, len(ids))
+ for _, id := range ids {
+ b := DBBackup{ID: id}
+ if t, err := time.Parse("20060102-150405", id[:15]); err == nil {
+ b.CreatedAt = t
+ }
+ if raw, _, err := s.Get(ctx, d.BackupMetaKey(id)); err == nil {
+ var meta DBBackup
+ if json.Unmarshal(raw, &meta) == nil {
+ b.Bytes, b.Request = meta.Bytes, meta.Request
+ if !meta.CreatedAt.IsZero() {
+ b.CreatedAt = meta.CreatedAt
+ }
+ }
+ } else if !errors.Is(err, os.ErrNotExist) {
+ return nil, err
+ }
+ out = append(out, b)
+ }
+ return out, nil
+}
+
+// PruneBackups deletes every backup beyond the newest keep. Returns the ids
+// it removed.
+func (d DBStage) PruneBackups(ctx context.Context, s release.Store, keep int) ([]string, error) {
+ all, err := d.ListBackups(ctx, s)
+ if err != nil {
+ return nil, err
+ }
+ var pruned []string
+ for i := keep; i < len(all); i++ {
+ id := all[i].ID
+ if err := s.Delete(ctx, d.BackupKey(id)); err != nil {
+ return pruned, err
+ }
+ if err := s.Delete(ctx, d.BackupMetaKey(id)); err != nil {
+ return pruned, err
+ }
+ pruned = append(pruned, id)
+ }
+ return pruned, nil
+}
+
+// SQLiteMagic is the 16-byte header every SQLite 3 database file starts
+// with; SQLiteMinBytes is the smallest file that can carry one page. The
+// console checks both before it stages an upload, so a stray text file is
+// refused at the door rather than by the box after a downtime window has
+// already opened.
+const (
+ SQLiteMagic = "SQLite format 3\x00"
+ SQLiteMinBytes = 512
+)
+
+// LooksLikeSQLite is that check.
+func LooksLikeSQLite(b []byte) bool {
+ return len(b) >= SQLiteMinBytes && string(b[:len(SQLiteMagic)]) == SQLiteMagic
+}
+
+// DBTransferMaxBytes caps a transfer in either direction: the file is
+// staged through the console and held in memory on both sides of the wire.
+// A var so a test can shrink it; production never writes it.
+var DBTransferMaxBytes int64 = 256 << 20
diff --git a/internal/provision/dbtransfer.go b/internal/provision/dbtransfer.go
new file mode 100644
index 000000000..11c552850
--- /dev/null
+++ b/internal/provision/dbtransfer.go
@@ -0,0 +1,242 @@
+package provision
+
+import (
+ "context"
+ "crypto/rand"
+ "encoding/hex"
+ "errors"
+ "fmt"
+ "regexp"
+ "strings"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The database transfer field (`carlos db pull` / `carlos db push`): a
+// member with no box and no bucket access asks the console for a copy of an
+// instance's SQLite database, or hands it a replacement. The console writes
+// a REQUEST onto the record and bumps the serial; the box that owns the
+// route does everything that touches data (internal/host's db worker) and
+// writes the outcome back through the same guarded transitions.
+//
+// The field is additive + omitempty: a record at rest before this existed
+// parses unchanged, and a record whose last transfer is done or failed keeps
+// the field as the answer to "what happened" until the next request or a
+// DELETE replaces it. Written ONLY through SetRecordDB's transitions.
+
+// DBTransfer is the record's database transfer: the last one requested,
+// whatever state it reached. State is one of the DB* constants; Op is
+// DBPull or DBPush; Step is the push worker's resume marker (a push that
+// died mid-way picks up at its last durable milestone rather than at the
+// top).
+type DBTransfer struct {
+ ID string `json:"id"` // random, URL-safe: 16 hex chars
+ Op string `json:"op"` // "pull" | "push"
+ State string `json:"state"` // "requested" | "done" | "failed"
+ // Step is the push's progress marker: "" until the backup is stored,
+ // DBStepBackedUp once it is, DBStepReplaced once the replica carries the
+ // pushed database. A pull never sets it.
+ Step string `json:"step,omitempty"`
+ Error string `json:"error,omitempty"`
+ // Bytes is the size of the staged file (pull: the snapshot the box
+ // uploaded; push: the file the member uploaded).
+ Bytes int64 `json:"bytes,omitempty"`
+ // Backup is the id of the backup the push took before the swap. Empty on
+ // a push of an instance that had no replicated database yet, and on
+ // every pull.
+ Backup string `json:"backup,omitempty"`
+ RequestedAt time.Time `json:"requested_at"`
+ // UpdatedAt is when the state or step last changed — stamped by
+ // SetRecordDB itself, the way SetRecordMove stamps a move.
+ UpdatedAt time.Time `json:"updated_at"`
+}
+
+// The ops, states and push steps.
+const (
+ DBPull = "pull"
+ DBPush = "push"
+
+ DBRequested = "requested"
+ DBDone = "done"
+ DBFailed = "failed"
+
+ DBStepBackedUp = "backed-up"
+ DBStepReplaced = "replaced"
+)
+
+// ErrDBConflict is a transition refused because the record's current state
+// does not admit it: a second request while one is in flight, a request
+// while a move is in flight, a completion for an id that is not the one on
+// the record. The console maps it to 409; the box treats it as "someone
+// else already advanced this" and re-reads.
+var ErrDBConflict = errors.New("provision: db transfer state conflict")
+
+// DBInFlight reports whether the record carries a transfer the box has not
+// finished with — the one state that excludes a new request AND a move.
+func (r Record) DBInFlight() bool {
+ return r.DB != nil && r.DB.State == DBRequested
+}
+
+// SetRecordDB applies one guarded transfer transition: the same narrow
+// read-CAS-write with one retry SetRecordMove uses, for the same reason —
+// re-reading and re-applying a transition on a lost race changes nothing
+// about what it meant, and an ILLEGAL transition after the re-read is
+// ErrDBConflict, which is the answer rather than a race.
+func SetRecordDB(ctx context.Context, s release.Store, acct, app, host string, mut func(*Record) error) (*Record, error) {
+ return updateRecord(ctx, s, acct, app, host, func(rec *Record) error {
+ if err := mut(rec); err != nil {
+ return err
+ }
+ if rec.DB != nil {
+ rec.DB.UpdatedAt = time.Now().UTC()
+ }
+ return nil
+ })
+}
+
+// dbTransferIDRe is the id shape: 16 lowercase hex characters, minted by
+// NewDBTransferID. Checked wherever an id arrives from outside (a query
+// string, a path segment) before it is joined into a bucket key.
+var dbTransferIDRe = regexp.MustCompile(`^[0-9a-f]{16}$`)
+
+// ValidDBTransferID reports whether id has the shape NewDBTransferID mints.
+func ValidDBTransferID(id string) bool { return dbTransferIDRe.MatchString(id) }
+
+// NewDBTransferID mints a transfer id: 8 random bytes as hex.
+func NewDBTransferID() string {
+ var b [8]byte
+ if _, err := rand.Read(b[:]); err != nil {
+ panic("provision: crypto/rand failed: " + err.Error())
+ }
+ return hex.EncodeToString(b[:])
+}
+
+// The transitions, each a mut for SetRecordDB. They are the ONLY writers of
+// Record.DB.
+
+// DBRequest starts a transfer. Refused while another transfer is in flight
+// and while a residence move is (the move's cut purges the very database a
+// push would be replacing, and a pull mid-cut could read either lineage).
+// A done or failed transfer is simply replaced: the console deletes its
+// staged objects before writing the new request.
+func DBRequest(op, id string, now time.Time) func(*Record) error {
+ return func(rec *Record) error {
+ if op != DBPull && op != DBPush {
+ return fmt.Errorf("%w: unknown op %q", ErrDBConflict, op)
+ }
+ if !ValidDBTransferID(id) {
+ return fmt.Errorf("%w: malformed transfer id", ErrDBConflict)
+ }
+ if rec.Move != nil {
+ return fmt.Errorf("%w: a move to %s is %s — wait for it to finish or abort it first", ErrDBConflict, rec.Move.To, rec.Move.State)
+ }
+ if rec.DBInFlight() {
+ return fmt.Errorf("%w: a db %s requested %s is still in flight", ErrDBConflict, rec.DB.Op, rec.DB.RequestedAt.UTC().Format(time.RFC3339))
+ }
+ // A replica never gets a route on a platform box, so no box would
+ // ever own a push of one; pulling its replica is fine (the box
+ // side reads the data bucket, which the home's replicator keeps
+ // current). Checked here as well as at the console's door because
+ // records have other writers.
+ if op == DBPush && rec.Role != "" {
+ return fmt.Errorf("%w: a %s holds a one-way mirror — push to the home instance instead", ErrDBConflict, rec.Role)
+ }
+ rec.DB = &DBTransfer{ID: id, Op: op, State: DBRequested, RequestedAt: now.UTC()}
+ return nil
+ }
+}
+
+// dbInFlightFor is the guard every box-side transition shares: the record
+// must carry THIS request, still in flight.
+func dbInFlightFor(rec *Record, id string) error {
+ if rec.DB == nil || rec.DB.ID != id {
+ return fmt.Errorf("%w: the record no longer carries transfer %s", ErrDBConflict, id)
+ }
+ if rec.DB.State != DBRequested {
+ return fmt.Errorf("%w: transfer %s is already %s", ErrDBConflict, id, rec.DB.State)
+ }
+ return nil
+}
+
+// DBBackedUp is the push worker's first milestone: the pre-swap backup is
+// stored under backupID ("" when the instance had nothing to back up).
+// Legal only from the empty step — a resumed push that already recorded it
+// never re-records a second backup over the first.
+func DBBackedUp(id, backupID string) func(*Record) error {
+ return func(rec *Record) error {
+ if err := dbInFlightFor(rec, id); err != nil {
+ return err
+ }
+ if rec.DB.Op != DBPush || rec.DB.Step != "" {
+ return fmt.Errorf("%w: backed-up is only reached from a push's first step", ErrDBConflict)
+ }
+ rec.DB.Step = DBStepBackedUp
+ rec.DB.Backup = backupID
+ return nil
+ }
+}
+
+// DBReplaced is the push worker's second milestone: the replica's lineage
+// IS the pushed database and the warm cache is purged. Legal only from
+// backed-up.
+func DBReplaced(id string) func(*Record) error {
+ return func(rec *Record) error {
+ if err := dbInFlightFor(rec, id); err != nil {
+ return err
+ }
+ if rec.DB.Op != DBPush || rec.DB.Step != DBStepBackedUp {
+ return fmt.Errorf("%w: replaced is only reached from backed-up", ErrDBConflict)
+ }
+ rec.DB.Step = DBStepReplaced
+ return nil
+ }
+}
+
+// DBComplete marks a transfer done. bytes is the staged file's size (a pull
+// reports the snapshot it uploaded; a push carries the size the console
+// recorded at upload, so the box passes it back through).
+func DBComplete(id string, bytes int64) func(*Record) error {
+ return func(rec *Record) error {
+ if err := dbInFlightFor(rec, id); err != nil {
+ return err
+ }
+ rec.DB.State = DBDone
+ rec.DB.Error = ""
+ if bytes > 0 {
+ rec.DB.Bytes = bytes
+ }
+ return nil
+ }
+}
+
+// DBFail marks a transfer failed with the sentence the member will read.
+func DBFail(id, msg string) func(*Record) error {
+ return func(rec *Record) error {
+ if err := dbInFlightFor(rec, id); err != nil {
+ return err
+ }
+ rec.DB.State = DBFailed
+ rec.DB.Error = strings.TrimSpace(msg)
+ if rec.DB.Error == "" {
+ rec.DB.Error = "failed"
+ }
+ return nil
+ }
+}
+
+// DBClear removes a done or failed transfer from the record — the console's
+// DELETE, after it has deleted that request's staged objects. Refused while
+// the transfer is in flight: the box is still writing to it.
+func DBClear() func(*Record) error {
+ return func(rec *Record) error {
+ if rec.DB == nil {
+ return nil
+ }
+ if rec.DB.State == DBRequested {
+ return fmt.Errorf("%w: a db %s is still in flight — wait for it to finish", ErrDBConflict, rec.DB.Op)
+ }
+ rec.DB = nil
+ return nil
+ }
+}
diff --git a/internal/provision/dbtransfer_test.go b/internal/provision/dbtransfer_test.go
new file mode 100644
index 000000000..f41c8dff5
--- /dev/null
+++ b/internal/provision/dbtransfer_test.go
@@ -0,0 +1,299 @@
+package provision
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/carlosframework/platform/internal/release"
+)
+
+// The transfer transitions, as a table over the record's state: which mut
+// is legal where, and what it leaves behind. Every 409 the console answers
+// and every "someone else advanced this" the box sees is a row here.
+func TestSetRecordDBTransitions(t *testing.T) {
+ ctx := context.Background()
+ s := release.NewDirStore(t.TempDir())
+ rec := Record{Schema: 1, Host: "a.go.example.com", Slug: "a", App: "app", Account: "bab",
+ Region: "ie", Channel: "stable", CreatedAt: time.Now().UTC()}
+ if err := CreateRecord(ctx, s, rec); err != nil {
+ t.Fatal(err)
+ }
+ apply := func(mut func(*Record) error) error {
+ _, err := SetRecordDB(ctx, s, "bab", "app", "a.go.example.com", mut)
+ return err
+ }
+ read := func() Record {
+ t.Helper()
+ got, err := GetRecord(ctx, s, "bab", "app", "a.go.example.com")
+ if err != nil {
+ t.Fatal(err)
+ }
+ return *got
+ }
+ now := time.Now()
+ pull, push := NewDBTransferID(), NewDBTransferID()
+
+ // Nothing in flight: the box-side transitions all refuse.
+ for name, mut := range map[string]func(*Record) error{
+ "backed-up": DBBackedUp(push, "x"),
+ "replaced": DBReplaced(push),
+ "complete": DBComplete(pull, 1),
+ "fail": DBFail(pull, "x"),
+ } {
+ if err := apply(mut); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("%s with nothing in flight = %v, want ErrDBConflict", name, err)
+ }
+ }
+ // Clear with nothing there is a no-op, not a conflict.
+ if err := apply(DBClear()); err != nil {
+ t.Fatal(err)
+ }
+ // A malformed op or id never lands.
+ if err := apply(DBRequest("sync", pull, now)); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("unknown op = %v", err)
+ }
+ if err := apply(DBRequest(DBPull, "../x", now)); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("malformed id = %v", err)
+ }
+
+ // Pull: request → done.
+ if err := apply(DBRequest(DBPull, pull, now)); err != nil {
+ t.Fatal(err)
+ }
+ got := read()
+ if got.DB == nil || got.DB.ID != pull || got.DB.Op != DBPull || got.DB.State != DBRequested || got.DB.UpdatedAt.IsZero() || !got.DBInFlight() {
+ t.Fatalf("after pull request: %+v", got.DB)
+ }
+ if err := apply(DBRequest(DBPush, push, now)); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("second request while in flight = %v, want ErrDBConflict", err)
+ }
+ if err := apply(DBClear()); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("clear while in flight = %v, want ErrDBConflict", err)
+ }
+ if err := apply(MoveRequest("uk", now)); !errors.Is(err, ErrMoveConflict) {
+ t.Fatalf("move while a db op is in flight = %v, want ErrMoveConflict", err)
+ }
+ // The push milestones do not apply to a pull, and a wrong id is refused.
+ if err := apply(DBBackedUp(pull, "b")); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("backed-up on a pull = %v", err)
+ }
+ if err := apply(DBComplete(push, 1)); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("complete with the wrong id = %v", err)
+ }
+ if err := apply(DBComplete(pull, 4096)); err != nil {
+ t.Fatal(err)
+ }
+ if got = read(); got.DB.State != DBDone || got.DB.Bytes != 4096 || got.DBInFlight() {
+ t.Fatalf("after complete: %+v", got.DB)
+ }
+ if err := apply(DBComplete(pull, 1)); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("double complete = %v, want ErrDBConflict", err)
+ }
+
+ // A done transfer is replaced by a new request. Push: the milestones
+ // in order, each refusing to be skipped or repeated.
+ if err := apply(DBRequest(DBPush, push, now)); err != nil {
+ t.Fatal(err)
+ }
+ if err := apply(DBReplaced(push)); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("replaced before backed-up = %v", err)
+ }
+ if err := apply(DBBackedUp(push, "20260929-120000-0badf00d")); err != nil {
+ t.Fatal(err)
+ }
+ if got = read(); got.DB.Step != DBStepBackedUp || got.DB.Backup != "20260929-120000-0badf00d" {
+ t.Fatalf("after backed-up: %+v", got.DB)
+ }
+ if err := apply(DBBackedUp(push, "other")); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("second backed-up = %v, want ErrDBConflict (never a second backup over the first)", err)
+ }
+ if err := apply(DBReplaced(push)); err != nil {
+ t.Fatal(err)
+ }
+ if err := apply(DBReplaced(push)); !errors.Is(err, ErrDBConflict) {
+ t.Fatalf("second replaced = %v", err)
+ }
+ if err := apply(DBFail(push, " the upload is not a SQLite database ")); err != nil {
+ t.Fatal(err)
+ }
+ if got = read(); got.DB.State != DBFailed || got.DB.Error != "the upload is not a SQLite database" || got.DB.Backup == "" {
+ t.Fatalf("after fail: %+v (the backup id must survive a failure)", got.DB)
+ }
+ // A failed transfer clears, and a move is legal again.
+ if err := apply(DBClear()); err != nil {
+ t.Fatal(err)
+ }
+ if got = read(); got.DB != nil {
+ t.Fatalf("after clear: %+v", got.DB)
+ }
+ if err := apply(MoveRequest("uk", now)); err != nil {
+ t.Fatal(err)
+ }
+ // And with a move in flight, no transfer starts.
+ if err := apply(DBRequest(DBPull, NewDBTransferID(), now)); !errors.Is(err, ErrDBConflict) || !strings.Contains(err.Error(), "move") {
+ t.Fatalf("request during a move = %v, want ErrDBConflict naming the move", err)
+ }
+}
+
+// A replica may be pulled (its mirror is real data) but never pushed (the
+// home would overwrite it on the next batch, and a box never owns it).
+func TestDBRequestReplicaRail(t *testing.T) {
+ rec := &Record{Region: "uk", Role: "replica"}
+ if err := DBRequest(DBPush, NewDBTransferID(), time.Now())(rec); !errors.Is(err, ErrDBConflict) || rec.DB != nil {
+ t.Fatalf("push to a replica = %v, db %+v; want ErrDBConflict and no request", err, rec.DB)
+ }
+ if err := DBRequest(DBPull, NewDBTransferID(), time.Now())(rec); err != nil || rec.DB == nil {
+ t.Fatalf("pull of a replica = %v, db %+v; want accepted", err, rec.DB)
+ }
+}
+
+// A record written before the field existed round-trips without growing a
+// db key — the wire-shape rule.
+func TestDBFieldAbsentOnPreTransferRecords(t *testing.T) {
+ raw := []byte(`{"schema":1,"host":"a.go.example.com","slug":"a","app":"app","account":"bab","region":"ie","channel":"stable","created_at":"2026-08-01T00:00:00Z"}`)
+ var rec Record
+ if err := json.Unmarshal(raw, &rec); err != nil {
+ t.Fatal(err)
+ }
+ if rec.DB != nil {
+ t.Fatalf("pre-transfer record decoded a DB: %+v", rec.DB)
+ }
+ out, err := json.Marshal(rec)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if strings.Contains(string(out), `"db"`) {
+ t.Fatalf("re-marshal grew a db key: %s", out)
+ }
+ // And a record carrying one keeps it through an unrelated writer.
+ ctx := context.Background()
+ s := release.NewDirStore(t.TempDir())
+ rec.CreatedAt = time.Now().UTC()
+ if err := CreateRecord(ctx, s, rec); err != nil {
+ t.Fatal(err)
+ }
+ id := NewDBTransferID()
+ if _, err := SetRecordDB(ctx, s, "bab", "app", "a.go.example.com", DBRequest(DBPull, id, time.Now())); err != nil {
+ t.Fatal(err)
+ }
+ if err := SetRecordRegion(ctx, s, "bab", "app", "a.go.example.com", "se"); err != nil {
+ t.Fatal(err)
+ }
+ got, err := GetRecord(ctx, s, "bab", "app", "a.go.example.com")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got.DB == nil || got.DB.ID != id || got.Region != "se" {
+ t.Fatalf("an unrelated writer dropped the transfer: %+v region=%q", got.DB, got.Region)
+ }
+}
+
+// The staging keys: shape-checked coordinates, request objects deleted as a
+// unit, backups listed newest first and pruned to the kept count.
+func TestDBStageKeysAndBackups(t *testing.T) {
+ ctx := context.Background()
+ s := release.NewDirStore(t.TempDir())
+ if _, err := NewDBStage("bab", "app", "../x"); err == nil {
+ t.Fatal("a traversal host built a stage")
+ }
+ if _, err := NewDBStage("", "app", "a.go.example.com"); err == nil {
+ t.Fatal("an empty account built a stage")
+ }
+ st, err := NewDBStage("bab", "app", "a.go.example.com")
+ if err != nil {
+ t.Fatal(err)
+ }
+ id := NewDBTransferID()
+ if got, want := st.PullKey(id), "apps/bab/app/db/a.go.example.com/"+id+"/pull.db"; got != want {
+ t.Fatalf("pull key = %s, want %s", got, want)
+ }
+ if got, want := st.PushKey(id), "apps/bab/app/db/a.go.example.com/"+id+"/push.db"; got != want {
+ t.Fatalf("push key = %s, want %s", got, want)
+ }
+ if !strings.HasPrefix(st.RequestPrefix("../../x"), st.Prefix()+"invalid/") {
+ t.Fatalf("a malformed id escaped the stage: %s", st.RequestPrefix("../../x"))
+ }
+ // A record key never parses as one of ours: db/ is a sibling of
+ // instances/, so the record walker cannot mistake a staged file.
+ if _, _, _, ok := ParseRecordKey(st.PullKey(id), map[string]bool{"bab": true}); ok {
+ t.Fatal("a staged database parsed as an instance record")
+ }
+ if _, err := s.Put(ctx, st.PullKey(id), []byte("x"), ""); err != nil {
+ t.Fatal(err)
+ }
+ if err := st.DeleteRequest(ctx, s, id); err != nil {
+ t.Fatal(err)
+ }
+ if keys, _ := s.List(ctx, st.Prefix()); len(keys) != 0 {
+ t.Fatalf("request objects survived the delete: %v", keys)
+ }
+ if err := st.DeleteRequest(ctx, s, "nope"); err != nil {
+ t.Fatalf("delete of a malformed id = %v, want a no-op", err)
+ }
+
+ // Backups.
+ base := time.Date(2026, 9, 29, 12, 0, 0, 0, time.UTC)
+ var ids []string
+ for i := 0; i < 7; i++ {
+ bid := NewDBBackupID(base.Add(time.Duration(i) * time.Minute))
+ if !ValidDBBackupID(bid) {
+ t.Fatalf("minted an invalid backup id %q", bid)
+ }
+ ids = append(ids, bid)
+ if _, err := s.Put(ctx, st.BackupKey(bid), []byte("db"), ""); err != nil {
+ t.Fatal(err)
+ }
+ if i != 3 { // one backup with no sidecar still lists
+ if err := st.PutBackupMeta(ctx, s, DBBackup{ID: bid, Bytes: int64(100 + i), CreatedAt: base.Add(time.Duration(i) * time.Minute), Request: id}); err != nil {
+ t.Fatal(err)
+ }
+ }
+ }
+ list, err := st.ListBackups(ctx, s)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(list) != 7 || list[0].ID != ids[6] || list[6].ID != ids[0] {
+ t.Fatalf("backups not newest first: %+v", list)
+ }
+ if list[0].Bytes != 106 || list[0].Request != id || !list[0].CreatedAt.Equal(base.Add(6*time.Minute)) {
+ t.Fatalf("newest backup meta: %+v", list[0])
+ }
+ if list[3].ID != ids[3] || list[3].Bytes != 0 || list[3].CreatedAt.IsZero() {
+ t.Fatalf("sidecar-less backup: %+v (want listed, zero bytes, time from the id)", list[3])
+ }
+ pruned, err := st.PruneBackups(ctx, s, DBBackupsKept)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(pruned) != 2 || pruned[0] != ids[1] || pruned[1] != ids[0] {
+ t.Fatalf("pruned %v, want the two oldest", pruned)
+ }
+ list, _ = st.ListBackups(ctx, s)
+ if len(list) != 5 || list[4].ID != ids[2] {
+ t.Fatalf("after prune: %+v", list)
+ }
+ if keys, _ := s.List(ctx, st.BackupsPrefix()); len(keys) != 9 { // 5 db + 4 sidecars (ids[3] has none)
+ t.Fatalf("objects after prune: %v", keys)
+ }
+}
+
+func TestLooksLikeSQLite(t *testing.T) {
+ good := make([]byte, SQLiteMinBytes)
+ copy(good, SQLiteMagic)
+ if !LooksLikeSQLite(good) {
+ t.Fatal("a minimal header was refused")
+ }
+ if LooksLikeSQLite(good[:100]) {
+ t.Fatal("a short file passed")
+ }
+ bad := make([]byte, 4096)
+ copy(bad, "SQLite format 4\x00")
+ if LooksLikeSQLite(bad) {
+ t.Fatal("a wrong magic passed")
+ }
+}
diff --git a/internal/provision/provision.go b/internal/provision/provision.go
index 2b399c424..cbfe8ed04 100644
--- a/internal/provision/provision.go
+++ b/internal/provision/provision.go
@@ -57,6 +57,14 @@ type Record struct {
// unchanged. Written ONLY through SetRecordMove's guarded transitions.
Move *MoveState `json:"move,omitempty"`
+ // DB is the instance's last database transfer (`carlos db pull` /
+ // `carlos db push`, dbtransfer.go): requested by the console, carried
+ // out by the box that owns the route, and left on the record as done
+ // or failed until the next request or a DELETE clears it. Additive +
+ // omitempty — pre-transfer records parse unchanged. Written ONLY
+ // through SetRecordDB's guarded transitions.
+ DB *DBTransfer `json:"db,omitempty"`
+
// --- shape fields (2026-08-10 route-records design §3) --------------
// Every one is omitempty with a zero value that reproduces the mint
// reconcile hardcoded before they existed: a record written by an older
@@ -971,6 +979,13 @@ func MoveRequest(to string, now time.Time) func(*Record) error {
if rec.Region == to {
return fmt.Errorf("%w: the instance is already in %s", ErrMoveConflict, to)
}
+ // A database transfer in flight holds the route out of service
+ // (push) or is reading the replica the cut would purge (pull);
+ // the two never overlap. Refused in the transition, not only at
+ // the console's door, because records have other writers.
+ if rec.DBInFlight() {
+ return fmt.Errorf("%w: a db %s is in flight — wait for it to finish", ErrMoveConflict, rec.DB.Op)
+ }
// A replica never gets a route on a platform box (reconcile's mint
// gate), so no box owns its move and it would wedge in preparing
// (G14). Checked here as well as at the console's door because
diff --git a/main.go b/main.go
index b3b52dff7..169f5f082 100644
--- a/main.go
+++ b/main.go
@@ -73,6 +73,8 @@ func main() {
runDeploy(args)
case "restart":
runRestart(args)
+ case "db":
+ runDB(args)
case "steering":
runSteering(args)
case "schedule":
@@ -197,6 +199,9 @@ Your apps — ship and run software on a CARLOS deployment
nothing the app already serves changes: deploy | ls | rm
restart cycle an app's instances — no new version, no config change;
--host or --environment narrows it, --dry-run names the set
+ db copy an instance's SQLite database down (pull) or a
+ replacement up (push, with a backup and a confirmation);
+ status | backups read what those left behind
schedule recurring or one-shot work per app: ls | set | rm | run
store object storage: create | status | apply | approvals | approve | decline | grant | rotate
relay the platform STUN/TURN relay: serve (what the unit runs)