amadan
lease: honour CARLOS_S3_ENDPOINT, so hibernation (and instance provisioning) works off AWS (#88)
* lease: honour CARLOS_S3_ENDPOINT, so hibernation works off AWS
release.NewS3Store has honoured CARLOS_S3_ENDPOINT since the first
S3-compatible deployment; internal/lease never did. Every S3 client it
built — the leaser's lock.json client and litestream's replica client on
both the Replicate and Restore paths — dialled real AWS regardless.
That gap is not confined to hibernation. The provisioning pass mints
auto-created instance routes with Hibernate:true; the activator only arms
when CARLOS_HIBERNATE_BUCKET is set, and arming means acquiring the lease
here. On a non-AWS store the acquire could never succeed, so e.act stayed
nil and an exec-backed route with nothing to spawn it 502'd forever —
taking `carlos instances enable` / `carlos instances create`, the whole
console-driven provisioning path, down with it. The same clients back the
agent's status-publisher election (internal/host/agent.go), so the fix
has to reach all three construction sites, not one.
New internal/lease/endpoint.go owns the override:
- newLeaserClient applies BaseEndpoint + path-style + when-required
checksums, matching release.NewS3Store's reasoning (aws-chunked framing
some compatibles reject) and litestream's own gating. Applied ONLY under
an override, so the flagship's pure-AWS path is byte-identical.
- newReplicaClient sets Endpoint/ForcePathStyle. litestream reads
LITESTREAM_S3_ENDPOINT only on its URL-parsing path, which this package
never takes.
- effectiveRegion closes the asymmetry an override exposes: litestream's
replica client defaults its own region for a non-AWS store, but the
leaser is handed a ready-made client and has no chain to fall back on,
so an empty region left it signing for nothing at all. Resolved once in
acquireTTL and carried on the Held, so both halves of one lease —
lock.json and the LTX files beside it — sign identically.
Tests need no AWS and no CARLOS_TEST_S3_BUCKET: the point of the override
is that traffic can be pointed anywhere, so TestAcquire_ReachesEndpointOverride
points it at an httptest server and asserts a full acquire-then-release
arrives there with path-style keys. Verified it catches the bug — with the
override neutered the request goes to real AWS and returns InvalidAccessKeyId.
lease_test.go's live-gated helper now goes through the package's own
constructor; a verification client that dialled stock AWS while the code
under test honoured the override would fail every assertion against a
compatible store and strand the objects it was meant to delete.
go build/vet/test ./... clean, 20 packages; gofmt clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* lease: let AWS_REGION outrank the non-AWS default region
Caught while preparing to deploy this branch to the private CARLOS box:
its host.env sets AWS_REGION=nbg1, and the first cut of effectiveRegion
would have overridden that with litestream's us-east-1 whenever
CARLOS_HIBERNATE_REGION was unset — which it is there.
The premise was wrong. "Region empty under an endpoint override" does not
mean "nothing to fall back on": the AWS default chain still resolves
AWS_REGION, and that is the region release.NewS3Store already signs for
against the very same bucket. Substituting a constant would have made the
lease sign differently from every other S3 client in the process, against
a store that may well check — a regression shipped straight into the one
deployment this branch exists to unblock.
Replaced with an explicit order of authority (regionFor):
1. the caller's explicit region (CARLOS_HIBERNATE_REGION)
2. the AWS default chain (AWS_REGION, profile)
3. only if both are empty AND an endpoint override is in play,
litestream's non-AWS default
Step 3 is the only part an override adds. newLeaserClient now returns the
region it resolved so acquireTTL can carry that exact value onto the Held,
and Restore resolves the same way — otherwise litestream's replica client
substitutes a default of its own and a restore signs differently from the
lease guarding it.
Tests: regionFor's table pins all three tiers; TestResolveRegion_PrefersAWSRegionEnv
drives the real AWS chain with AWS_REGION set, which is the exact shape of
the box's host.env. fakeCreds now clears AWS_REGION so region resolution
can't depend on whatever machine runs the suite.
go build/vet/test ./... clean; gofmt clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
4 files changed,
+427
−22
internal/lease/endpoint.go+138 −0internal/lease/endpoint_test.go+263 −0internal/lease/lease.go+19 −19internal/lease/lease_test.go+7 −3
diff --git a/internal/lease/endpoint.go b/internal/lease/endpoint.go| new file mode 100644 |
| index 000000000..b78a8addc |
| --- /dev/null |
| +++ b/internal/lease/endpoint.go |
| @@ -0,0 +1,138 @@ |
| +package lease |
| + |
| +// CARLOS_S3_ENDPOINT support for the hibernation lease. |
| +// |
| +// release.NewS3Store has honoured CARLOS_S3_ENDPOINT since the first |
| +// S3-compatible deployment; this package did not, and every S3 client it |
| +// builds dialled real AWS regardless. That gap is not cosmetic: an activator |
| +// route is minted with Hibernate:true, the activator only arms when |
| +// CARLOS_HIBERNATE_BUCKET is set, and arming means acquiring the lease here. |
| +// On a non-AWS store (Hetzner Object Storage, R2, MinIO) the acquire could |
| +// never succeed, so the activator stayed nil and an exec-backed route with |
| +// nothing to spawn it 502'd forever — which took `carlos instances create`, |
| +// the whole console-driven provisioning path, down with it. The same clients |
| +// back the agent's status-publisher election (internal/host/agent.go), so the |
| +// override has to reach all three construction sites, not just the one. |
| + |
| +import ( |
| + "context" |
| + "fmt" |
| + "os" |
| + "strings" |
| + |
| + "github.com/aws/aws-sdk-go-v2/aws" |
| + awsconfig "github.com/aws/aws-sdk-go-v2/config" |
| + awss3 "github.com/aws/aws-sdk-go-v2/service/s3" |
| + lss3 "github.com/benbjohnson/litestream/s3" |
| +) |
| + |
| +// s3Endpoint reports the CARLOS_S3_ENDPOINT override, or "" for stock AWS. |
| +// Read per call rather than cached at init so a test (and a process that |
| +// re-execs its config) sees the value it just set. |
| +func s3Endpoint() string { |
| + return strings.TrimSpace(os.Getenv("CARLOS_S3_ENDPOINT")) |
| +} |
| + |
| +// resolveRegion decides the region every client in this package signs for, in |
| +// strict order of authority: |
| +// |
| +// 1. the caller's explicit region (CARLOS_HIBERNATE_REGION), if any; |
| +// 2. whatever the AWS default chain yields — AWS_REGION, or a profile. This |
| +// step is why the resolution cannot be a pure function of the argument: a |
| +// compatible-store deployment sets AWS_REGION to its own location (the |
| +// private CARLOS uses nbg1), and that is the region release.NewS3Store |
| +// already signs for against the same bucket. Substituting a constant here |
| +// would make the lease sign differently from every other S3 client in the |
| +// process, against a store that may well check; |
| +// 3. only if BOTH are empty and an endpoint override is in play, litestream's |
| +// non-AWS default — better a working guess than a client signing for no |
| +// region at all. |
| +// |
| +// Step 3 is the one an override adds. Without it the AWS chain is authoritative |
| +// and an empty result is fine: stock AWS resolves the region per-request from |
| +// the bucket's own endpoint, which a compatible store cannot do. |
| +// |
| +// The resolved value is returned, not just applied, because it has to travel: |
| +// litestream's replica client does its OWN region defaulting (Init substitutes |
| +// the same non-AWS constant for an empty Region), so handing it the resolved |
| +// region is what stops the two halves of one lease — lock.json and the LTX |
| +// files beside it under the same prefix — from signing differently. |
| +func resolveRegion(ctx context.Context, region string) (string, error) { |
| + if region != "" { |
| + return region, nil // no config load needed: step 1 already decided |
| + } |
| + cfg, err := awsconfig.LoadDefaultConfig(ctx) |
| + if err != nil { |
| + return "", fmt.Errorf("load aws config: %w", err) |
| + } |
| + return regionFor(region, cfg.Region), nil |
| +} |
| + |
| +// regionFor is resolveRegion's decision with the chain lookup already done — |
| +// the shared rule, so the leaser's path (which loads a config anyway, and must |
| +// not load a second one just to re-ask the same question) and Restore's cannot |
| +// drift apart. |
| +func regionFor(explicit, chainResolved string) string { |
| + if explicit != "" { |
| + return explicit |
| + } |
| + if chainResolved == "" && s3Endpoint() != "" { |
| + return lss3.DefaultRegion |
| + } |
| + return chainResolved |
| +} |
| + |
| +// newLeaserClient builds the S3 client the litestream leaser writes lock.json |
| +// with. Unlike the replica client, which builds its own from its Endpoint |
| +// field, the leaser takes a ready-made client (lss3.Leaser.SetClient) — so the |
| +// override has to be applied right here or the lock lands on AWS while the |
| +// data it guards lands on the compatible store. |
| +// |
| +// An endpoint implies path-style addressing and when-required checksums, for |
| +// the reason release.NewS3Store gives: default-on flexible checksums wrap |
| +// bodies in aws-chunked framing that some compatibles reject. Both are applied |
| +// ONLY under an override, matching what litestream's own replica client does |
| +// (replica_client.go Init: the checksum knobs are gated on Endpoint != ""), so |
| +// the pure-AWS path stays byte-identical to before this file existed. |
| +// It returns the region it resolved alongside the client, so acquireTTL can |
| +// carry that exact value onto the Held for Replicate's replica client rather |
| +// than letting litestream substitute a different default of its own. |
| +func newLeaserClient(ctx context.Context, region string) (*awss3.Client, string, error) { |
| + var opts []func(*awsconfig.LoadOptions) error |
| + if region != "" { |
| + opts = append(opts, awsconfig.WithRegion(region)) |
| + } |
| + cfg, err := awsconfig.LoadDefaultConfig(ctx, opts...) |
| + if err != nil { |
| + return nil, "", fmt.Errorf("load aws config: %w", err) |
| + } |
| + cfg.Region = regionFor(region, cfg.Region) |
| + return awss3.NewFromConfig(cfg, func(o *awss3.Options) { |
| + if ep := s3Endpoint(); ep != "" { |
| + o.BaseEndpoint = &ep |
| + o.UsePathStyle = true |
| + o.RequestChecksumCalculation = aws.RequestChecksumCalculationWhenRequired |
| + o.ResponseChecksumValidation = aws.ResponseChecksumValidationWhenRequired |
| + } |
| + }), cfg.Region, nil |
| +} |
| + |
| +// newReplicaClient builds litestream's replica client for the LTX files at |
| +// s3://bucket/prefix. litestream reads its OWN env var (LITESTREAM_S3_ENDPOINT) |
| +// only on the URL-parsing path, which this package never takes — it sets the |
| +// fields directly — so the override is applied here too. Endpoint/ForcePathStyle |
| +// is all it takes: litestream's Init derives the rest (the checksum knobs and |
| +// the non-AWS default region) from a non-empty Endpoint on its own, and the |
| +// signed-payload and content-MD5 defaults NewReplicaClient already ships are |
| +// the ones every compatible in litestream's provider table wants. |
| +func newReplicaClient(bucket, prefix, region string) *lss3.ReplicaClient { |
| + c := lss3.NewReplicaClient() |
| + c.Bucket = bucket |
| + c.Path = prefix |
| + c.Region = region |
| + if ep := s3Endpoint(); ep != "" { |
| + c.Endpoint = ep |
| + c.ForcePathStyle = true |
| + } |
| + return c |
| +} |
diff --git a/internal/lease/endpoint_test.go b/internal/lease/endpoint_test.go| new file mode 100644 |
| index 000000000..935e9cb00 |
| --- /dev/null |
| +++ b/internal/lease/endpoint_test.go |
| @@ -0,0 +1,263 @@ |
| +package lease |
| + |
| +// CARLOS_S3_ENDPOINT tests. Unlike lease_test.go these need no AWS and no |
| +// CARLOS_TEST_S3_BUCKET: the point of the override is that S3 traffic can be |
| +// pointed anywhere, so the test points it at an httptest server and asserts |
| +// the request actually arrives. A regression here — the leaser silently going |
| +// back to real AWS — is exactly what cost this deployment hibernation and the |
| +// whole `carlos instances create` provisioning path. |
| + |
| +import ( |
| + "context" |
| + "encoding/json" |
| + "io" |
| + "net/http" |
| + "net/http/httptest" |
| + "sync" |
| + "testing" |
| + |
| + "github.com/benbjohnson/litestream" |
| + lss3 "github.com/benbjohnson/litestream/s3" |
| +) |
| + |
| +// fakeS3 is just enough of S3 for the leaser's three verbs (GET/PUT/DELETE on |
| +// one key) to complete a full acquire-then-release, recording every request so |
| +// a test can assert on where the traffic went. |
| +type fakeS3 struct { |
| + mu sync.Mutex |
| + reqs []string // "METHOD /path" |
| + body []byte // last PUT body |
| +} |
| + |
| +func (f *fakeS3) record(method, path string) { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + f.reqs = append(f.reqs, method+" "+path) |
| +} |
| + |
| +func (f *fakeS3) requests() []string { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + return append([]string(nil), f.reqs...) |
| +} |
| + |
| +func (f *fakeS3) putBody() []byte { |
| + f.mu.Lock() |
| + defer f.mu.Unlock() |
| + return append([]byte(nil), f.body...) |
| +} |
| + |
| +func newFakeS3(t *testing.T) (*fakeS3, string) { |
| + t.Helper() |
| + f := &fakeS3{} |
| + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
| + f.record(r.Method, r.URL.Path) |
| + switch r.Method { |
| + case http.MethodGet: |
| + // No lease on the key yet. The XML body matters: litestream |
| + // classifies "absent" by the S3 error CODE, and a bare 404 with no |
| + // body would read as a hard failure instead. |
| + w.Header().Set("Content-Type", "application/xml") |
| + w.WriteHeader(http.StatusNotFound) |
| + _, _ = w.Write([]byte(`<?xml version="1.0" encoding="UTF-8"?>` + |
| + `<Error><Code>NoSuchKey</Code><Message>no such key</Message></Error>`)) |
| + case http.MethodPut: |
| + b, _ := io.ReadAll(r.Body) |
| + f.mu.Lock() |
| + f.body = b |
| + f.mu.Unlock() |
| + w.Header().Set("ETag", `"lease-etag"`) |
| + w.WriteHeader(http.StatusOK) |
| + case http.MethodDelete: |
| + w.WriteHeader(http.StatusNoContent) |
| + default: |
| + w.WriteHeader(http.StatusMethodNotAllowed) |
| + } |
| + })) |
| + t.Cleanup(srv.Close) |
| + return f, srv.URL |
| +} |
| + |
| +// fakeCreds plants credentials the AWS default chain will resolve without |
| +// touching the operator's real profile or IMDS — the same isolation instinct |
| +// as carlosTestEnv, applied to the SDK instead of the carlos binary. |
| +func fakeCreds(t *testing.T) { |
| + t.Helper() |
| + t.Setenv("AWS_ACCESS_KEY_ID", "test-key") |
| + t.Setenv("AWS_SECRET_ACCESS_KEY", "test-secret") |
| + t.Setenv("AWS_SESSION_TOKEN", "") |
| + // Cleared, not left ambient: region resolution is under test, so a region |
| + // inherited from whatever machine runs the suite would make these results |
| + // depend on the box. Tests that care set it back explicitly. |
| + t.Setenv("AWS_REGION", "") |
| + t.Setenv("AWS_DEFAULT_REGION", "") |
| + t.Setenv("AWS_EC2_METADATA_DISABLED", "true") |
| +} |
| + |
| +// TestAcquire_ReachesEndpointOverride is the regression test for the bug this |
| +// file exists for: with CARLOS_S3_ENDPOINT set, the lease's lock.json must be |
| +// written through the override — not to real AWS. It asserts on the recorded |
| +// requests, so it fails loudly (rather than hanging on a live AWS dial) if the |
| +// leaser ever stops honouring the endpoint. |
| +func TestAcquire_ReachesEndpointOverride(t *testing.T) { |
| + f, url := newFakeS3(t) |
| + fakeCreds(t) |
| + t.Setenv("CARLOS_S3_ENDPOINT", url) |
| + |
| + // Region deliberately empty, and fakeCreds leaves the chain empty too, so |
| + // this also exercises the last resort: a client signing for no region at |
| + // all would fail every request against the fake. |
| + held, err := Acquire(context.Background(), "test-bucket", "inst/example.com", "") |
| + if err != nil { |
| + t.Fatalf("Acquire against %s: %v", url, err) |
| + } |
| + |
| + got := f.requests() |
| + // Path-style addressing: bucket in the path, not the hostname — a |
| + // compatible store has no per-bucket virtual host. |
| + wantKey := "/test-bucket/inst/example.com/lock.json" |
| + var sawGet, sawPut bool |
| + for _, r := range got { |
| + switch r { |
| + case "GET " + wantKey: |
| + sawGet = true |
| + case "PUT " + wantKey: |
| + sawPut = true |
| + } |
| + } |
| + if !sawGet || !sawPut { |
| + t.Fatalf("lease traffic did not reach the override: got %v, want a GET and a PUT of %s", got, wantKey) |
| + } |
| + |
| + // The PUT body is the lock itself — proof the acquire really happened |
| + // here rather than the server merely being probed. |
| + var lease litestream.Lease |
| + if err := json.Unmarshal(f.putBody(), &lease); err != nil { |
| + t.Fatalf("decode written lock.json (%q): %v", f.putBody(), err) |
| + } |
| + if lease.Owner == "" { |
| + t.Errorf("written lease carries no owner: %+v", lease) |
| + } |
| + |
| + if err := held.Release(context.Background()); err != nil { |
| + t.Fatalf("Release: %v", err) |
| + } |
| + var sawDelete bool |
| + for _, r := range f.requests() { |
| + if r == "DELETE "+wantKey { |
| + sawDelete = true |
| + } |
| + } |
| + if !sawDelete { |
| + t.Errorf("release did not reach the override: got %v", f.requests()) |
| + } |
| +} |
| + |
| +// TestNewLeaserClient_EndpointConfig pins the client options either side of the |
| +// override, because the no-override case is the flagship's production path and |
| +// must stay byte-identical to what it was before endpoint.go existed. |
| +func TestNewLeaserClient_EndpointConfig(t *testing.T) { |
| + fakeCreds(t) |
| + |
| + t.Run("override sets a path-style base endpoint", func(t *testing.T) { |
| + t.Setenv("CARLOS_S3_ENDPOINT", "https://fsn1.your-objectstorage.com") |
| + c, _, err := newLeaserClient(context.Background(), "eu-central-1") |
| + if err != nil { |
| + t.Fatalf("newLeaserClient: %v", err) |
| + } |
| + o := c.Options() |
| + if o.BaseEndpoint == nil || *o.BaseEndpoint != "https://fsn1.your-objectstorage.com" { |
| + t.Errorf("BaseEndpoint = %v, want the override", o.BaseEndpoint) |
| + } |
| + if !o.UsePathStyle { |
| + t.Error("UsePathStyle = false, want true under an endpoint override") |
| + } |
| + }) |
| + |
| + t.Run("no override leaves stock AWS addressing", func(t *testing.T) { |
| + t.Setenv("CARLOS_S3_ENDPOINT", "") |
| + c, _, err := newLeaserClient(context.Background(), "eu-west-1") |
| + if err != nil { |
| + t.Fatalf("newLeaserClient: %v", err) |
| + } |
| + o := c.Options() |
| + if o.BaseEndpoint != nil { |
| + t.Errorf("BaseEndpoint = %q, want nil (stock AWS)", *o.BaseEndpoint) |
| + } |
| + if o.UsePathStyle { |
| + t.Error("UsePathStyle = true, want false without an override") |
| + } |
| + }) |
| +} |
| + |
| +// TestNewReplicaClient_EndpointConfig covers the other half of a lease: the |
| +// LTX files beside lock.json. A replica client that ignored the override while |
| +// the leaser honoured it would split one lease across two stores. |
| +func TestNewReplicaClient_EndpointConfig(t *testing.T) { |
| + t.Run("override", func(t *testing.T) { |
| + t.Setenv("CARLOS_S3_ENDPOINT", "https://fsn1.your-objectstorage.com") |
| + c := newReplicaClient("b", "p", "eu-central-1") |
| + if c.Endpoint != "https://fsn1.your-objectstorage.com" { |
| + t.Errorf("Endpoint = %q, want the override", c.Endpoint) |
| + } |
| + if !c.ForcePathStyle { |
| + t.Error("ForcePathStyle = false, want true under an endpoint override") |
| + } |
| + if c.Bucket != "b" || c.Path != "p" || c.Region != "eu-central-1" { |
| + t.Errorf("bucket/path/region = %q/%q/%q, want b/p/eu-central-1", c.Bucket, c.Path, c.Region) |
| + } |
| + }) |
| + |
| + t.Run("no override", func(t *testing.T) { |
| + t.Setenv("CARLOS_S3_ENDPOINT", "") |
| + c := newReplicaClient("b", "p", "eu-west-1") |
| + if c.Endpoint != "" || c.ForcePathStyle { |
| + t.Errorf("Endpoint/ForcePathStyle = %q/%v, want empty/false without an override", c.Endpoint, c.ForcePathStyle) |
| + } |
| + }) |
| +} |
| + |
| +// TestRegionFor pins the order of authority. The case that matters most is |
| +// "chain wins over the non-AWS default": a compatible-store deployment sets |
| +// AWS_REGION to its own location (carlos.exciting.io uses nbg1) and that is |
| +// what release.NewS3Store already signs for against the same bucket — |
| +// substituting a constant would make the lease sign differently from every |
| +// other S3 client in the process. |
| +func TestRegionFor(t *testing.T) { |
| + tests := []struct { |
| + name, endpoint, explicit, chain, want string |
| + }{ |
| + {"explicit wins over everything", "https://s3.example.com", "fsn1", "nbg1", "fsn1"}, |
| + {"chain wins over the non-AWS default", "https://s3.example.com", "", "nbg1", "nbg1"}, |
| + {"non-AWS default only fills a total blank", "https://s3.example.com", "", "", lss3.DefaultRegion}, |
| + {"no override leaves a blank blank", "", "", "", ""}, |
| + {"no override keeps the chain", "", "", "eu-west-1", "eu-west-1"}, |
| + } |
| + for _, tt := range tests { |
| + t.Run(tt.name, func(t *testing.T) { |
| + t.Setenv("CARLOS_S3_ENDPOINT", tt.endpoint) |
| + if got := regionFor(tt.explicit, tt.chain); got != tt.want { |
| + t.Errorf("regionFor(%q, %q) with endpoint %q = %q, want %q", |
| + tt.explicit, tt.chain, tt.endpoint, got, tt.want) |
| + } |
| + }) |
| + } |
| +} |
| + |
| +// TestResolveRegion_PrefersAWSRegionEnv is the end-to-end version of the case |
| +// above, through the real AWS chain: with AWS_REGION set and no explicit |
| +// region, the lease must sign for AWS_REGION — not for litestream's non-AWS |
| +// constant. This is the exact shape of the private CARLOS box's host.env. |
| +func TestResolveRegion_PrefersAWSRegionEnv(t *testing.T) { |
| + fakeCreds(t) |
| + t.Setenv("CARLOS_S3_ENDPOINT", "https://nbg1.your-objectstorage.com") |
| + t.Setenv("AWS_REGION", "nbg1") |
| + |
| + got, err := resolveRegion(context.Background(), "") |
| + if err != nil { |
| + t.Fatalf("resolveRegion: %v", err) |
| + } |
| + if got != "nbg1" { |
| + t.Errorf("resolveRegion = %q, want %q (AWS_REGION must outrank the non-AWS default)", got, "nbg1") |
| + } |
| +} |
diff --git a/internal/lease/lease.go b/internal/lease/lease.go| index 4827481ce..3a108ed81 100644 |
| --- a/internal/lease/lease.go |
| +++ b/internal/lease/lease.go |
| @@ -19,8 +19,6 @@ import ( |
| "sync" |
| "time" |
| - awsconfig "github.com/aws/aws-sdk-go-v2/config" |
| - awss3 "github.com/aws/aws-sdk-go-v2/service/s3" |
| "github.com/benbjohnson/litestream" |
| lss3 "github.com/benbjohnson/litestream/s3" |
| ) |
| @@ -184,7 +182,8 @@ type Held struct { |
| // and starts renewing it. If another process holds the lease it fails LOUDLY |
| // with a *HeldError (check with errors.As) — this is the |
| // signal the activator reacts to. Region may be empty; the AWS default chain |
| -// (env/profile) resolves it. |
| +// (env/profile) resolves it, or — under a CARLOS_S3_ENDPOINT override, where |
| +// there is no chain to resolve from — endpoint.go's non-AWS default. |
| func Acquire(ctx context.Context, bucket, prefix, region string) (*Held, error) { |
| return acquireTTL(ctx, bucket, prefix, region, LeaseTTL) |
| } |
| @@ -197,18 +196,18 @@ func Acquire(ctx context.Context, bucket, prefix, region string) (*Held, error) |
| // fast, which is what makes "the holder went dark" observable. |
| func acquireTTL(ctx context.Context, bucket, prefix, region string, ttl time.Duration) (*Held, error) { |
| // The leaser wants a ready-made S3 client (unlike the replica client, |
| - // which builds its own). Default AWS credential chain. |
| - var opts []func(*awsconfig.LoadOptions) error |
| - if region != "" { |
| - opts = append(opts, awsconfig.WithRegion(region)) |
| - } |
| - cfg, err := awsconfig.LoadDefaultConfig(ctx, opts...) |
| + // which builds its own). Default AWS credential chain, and the |
| + // CARLOS_S3_ENDPOINT override applied — see endpoint.go for why the lock |
| + // and the data it guards must resolve to the same store. The region is |
| + // resolved ONCE, here, and carried on the Held: Replicate's replica client |
| + // has to sign for whatever this client signed for. |
| + client, region, err := newLeaserClient(ctx, region) |
| if err != nil { |
| - return nil, fmt.Errorf("load aws config: %w", err) |
| + return nil, err |
| } |
| leaser := lss3.NewLeaser() |
| - leaser.SetClient(awss3.NewFromConfig(cfg)) |
| + leaser.SetClient(client) |
| leaser.Bucket = bucket |
| leaser.Path = prefix // lock lives at <prefix>/lock.json, colocated with the replica |
| leaser.TTL = ttl |
| @@ -267,10 +266,7 @@ func (h *Held) Replicate(ctx context.Context, dbPath string, syncInterval time.D |
| // namespace ends the class. |
| db.SetMetaPath(filepath.Join(filepath.Dir(dbPath), ".activator-"+filepath.Base(dbPath)+"-litestream")) |
| db.MonitorInterval = dbMonitorInterval |
| - client := lss3.NewReplicaClient() |
| - client.Bucket = h.leaser.Bucket |
| - client.Path = h.leaser.Path |
| - client.Region = h.region |
| + client := newReplicaClient(h.leaser.Bucket, h.leaser.Path, h.region) |
| db.Replica = litestream.NewReplicaWithClient(db, client) |
| if syncInterval > 0 { |
| db.Replica.SyncInterval = syncInterval |
| @@ -380,10 +376,14 @@ func (r *Replicator) Close(ctx context.Context) error { |
| // it could resurrect a stale copy that later shadows the true holder's newer |
| // state. |
| func Restore(ctx context.Context, dbPath, bucket, prefix, region string) (bool, error) { |
| - client := lss3.NewReplicaClient() |
| - client.Bucket = bucket |
| - client.Path = prefix |
| - client.Region = region |
| + // Resolved the same way the leaser's client resolves it — a restore that |
| + // signed for a different region than the lease guarding it would be a |
| + // silent split against a store that checks. |
| + region, err := resolveRegion(ctx, region) |
| + if err != nil { |
| + return false, err |
| + } |
| + client := newReplicaClient(bucket, prefix, region) |
| // A nil *DB is litestream's own pattern for URL restores (the CLI does the |
| // same): Restore only needs the replica client and the output path. |
diff --git a/internal/lease/lease_test.go b/internal/lease/lease_test.go| index 8129595d3..ca31c0733 100644 |
| --- a/internal/lease/lease_test.go |
| +++ b/internal/lease/lease_test.go |
| @@ -19,7 +19,6 @@ import ( |
| "time" |
| "github.com/aws/aws-sdk-go-v2/aws" |
| - awsconfig "github.com/aws/aws-sdk-go-v2/config" |
| awss3 "github.com/aws/aws-sdk-go-v2/service/s3" |
| s3types "github.com/aws/aws-sdk-go-v2/service/s3/types" |
| @@ -65,13 +64,18 @@ func seedDB(t *testing.T, path string) { |
| } |
| } |
| +// s3Client is the verification client the live tests list and clean up with. |
| +// It deliberately goes through the package's OWN constructor rather than |
| +// building its own: a helper that dialled stock AWS while the code under test |
| +// honoured CARLOS_S3_ENDPOINT would fail every assertion against a compatible |
| +// store and strand the objects it was meant to delete. |
| func s3Client(t *testing.T, region string) *awss3.Client { |
| t.Helper() |
| - cfg, err := awsconfig.LoadDefaultConfig(context.Background(), awsconfig.WithRegion(region)) |
| + client, _, err := newLeaserClient(context.Background(), region) |
| if err != nil { |
| t.Fatalf("aws config: %v", err) |
| } |
| - return awss3.NewFromConfig(cfg) |
| + return client |
| } |
| func listKeys(t *testing.T, bucket, prefix, region string) []string { |