nwsync: upload sink, key-addressed CLI, fail-closed publication marker (#73)
build-binaries / build-binaries (push) Successful in 2m18s
build-binaries / build-binaries (push) Successful in 2m18s
Builds the code half of #60, and closes the CLI gap the #53 sweep found: PR #71 shipped #53's original surface rather than the one #56, #62 and #65 settled. ## What lands **`depot.KeyStore`** — `ProbeKey` / `PutReader` / `GetKey`, addressing the zone by object key rather than by depot sha. NWSync cannot use the sha-addressed path: a blob is named after the sha1 of its *uncompressed* bytes while the body uploaded is the compressed form, and Bunny's `Checksum` header is sha256 of the body. Per #55 this reuses `internal/depot`'s `httpBackend` — same IPv4-pinned transport, same retry, same tri-state probe — and the sha-addressed `Backend` is now rewritten on top of it. No second HTTP client, no per-instance hash-function fields: the caller passes the key and the checksum, which turned out simpler than #55 expected. **A sink in `internal/nwsync`** — the zone by default, a local tree under `--out DIR` as the conformance path. Blobs upload as they are produced and the index lands last, so the presence of an index is the publication marker. A blob already in the zone is skipped via #55's probe *without* paying for compression (the body is a thunk) — which matters for the backfill, where compression is the expensive part. **The settled CLI** ``` nwsync emit [--as NAME] [--out DIR] <artifact-key> <file> nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>... ``` Artifact keys are depot keys; an index lives beside its artifact with the extension replaced (#62), derived in exactly one place so `emit` and `assemble` cannot disagree. Flags may now follow positionals — Go's `flag` stops at the first non-flag argument, which cost a run during #59. **Fail-closed in two places** — an artifact key whose embedded digest does not match the file is refused (publishing an index under the wrong key silently pairs a manifest with the wrong artifact), and `assemble` refuses an artifact with no index rather than publishing a manifest missing a hak. ## Checks `make check` green. Six new tests run against a Bunny-shaped `httptest` zone that verifies the `Checksum` header the way Bunny does: blobs-then-index ordering, skip-if-present, no index after a failed upload, key/file mismatch, and assemble reading indexes back out of the zone. Conformance re-run through the new CLI against upstream `nwn_nwsync_write` 2.1.2 over `sow_vfxs_01.hak` (#59's oracle): the manifest is still **byte-identical**. ## Not in this PR - **Live upload against the real zone.** The nwsync zone and its credential are #61, still open. Everything here is proven against a fake zone only. - **Consumer wiring** — #65, in the three producer repos. - **The mid-hak failure *policy*.** The mechanism is here (fail closed, orphan blobs left, re-run resumes); whether a module release may proceed when an emit failed is a human call, still open on #60. 🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #73 Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
This commit was merged in pull request #73.
This commit is contained in:
@@ -10,7 +10,6 @@ import (
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -71,16 +70,7 @@ type httpBackend struct {
|
||||
|
||||
func (b *httpBackend) Name() string { return b.name }
|
||||
|
||||
func (b *httpBackend) storageURL(sha string) string {
|
||||
host := b.cfg.StorageHost
|
||||
// StorageHost is normally a bare host ("storage.bunnycdn.com"); allow a
|
||||
// full scheme (used by tests against httptest.NewServer) to pass through
|
||||
// unchanged.
|
||||
if strings.Contains(host, "://") {
|
||||
return fmt.Sprintf("%s/%s/%s", strings.TrimSuffix(host, "/"), b.cfg.StorageZone, BlobKey(sha))
|
||||
}
|
||||
return fmt.Sprintf("https://%s/%s/%s", host, b.cfg.StorageZone, BlobKey(sha))
|
||||
}
|
||||
func (b *httpBackend) storageURL(sha string) string { return b.keyURL(BlobKey(sha)) }
|
||||
|
||||
func (b *httpBackend) cdnURL(sha string) string {
|
||||
return fmt.Sprintf("%s/%s", b.cfg.CDNBase, BlobKey(sha))
|
||||
@@ -140,7 +130,8 @@ func (b *httpBackend) Probe(ctx context.Context, sha string) (ProbeState, bool,
|
||||
return b.rangeProbe(ctx, b.storageURL(sha), map[string]string{"AccessKey": b.cfg.ReadKey})
|
||||
}
|
||||
|
||||
// Put uploads src for sha. cdn is read-only.
|
||||
// Put uploads src for sha. cdn is read-only. A depot object is named after the
|
||||
// sha256 of its own bytes, so the key's sha doubles as the Checksum header.
|
||||
func (b *httpBackend) Put(ctx context.Context, sha, src string) error {
|
||||
if b.name == "cdn" {
|
||||
return errors.New("cdn backend is read-only")
|
||||
@@ -157,30 +148,7 @@ func (b *httpBackend) Put(ctx context.Context, sha, src string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
f, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPut, b.storageURL(sha), f)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req.Header.Set("AccessKey", b.cfg.WriteKey)
|
||||
req.Header.Set("Checksum", strings.ToUpper(sha))
|
||||
|
||||
resp, err := b.client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
return fmt.Errorf("bunny put %s: unexpected status %d", sha, resp.StatusCode)
|
||||
}
|
||||
return nil
|
||||
return b.putFile(ctx, BlobKey(sha), src, sha)
|
||||
}
|
||||
|
||||
// Get fetches sha into dest via temp file + rename, re-hashing and deleting
|
||||
|
||||
Reference in New Issue
Block a user