nwsync: upload sink, key-addressed CLI, fail-closed publication marker (#73)
build-binaries / build-binaries (push) Successful in 2m18s

Builds the code half of #60, and closes the CLI gap the #53 sweep found: PR #71 shipped #53's original surface rather than the one #56, #62 and #65 settled.

## What lands

**`depot.KeyStore`** — `ProbeKey` / `PutReader` / `GetKey`, addressing the zone by object key rather than by depot sha. NWSync cannot use the sha-addressed path: a blob is named after the sha1 of its *uncompressed* bytes while the body uploaded is the compressed form, and Bunny's `Checksum` header is sha256 of the body. Per #55 this reuses `internal/depot`'s `httpBackend` — same IPv4-pinned transport, same retry, same tri-state probe — and the sha-addressed `Backend` is now rewritten on top of it. No second HTTP client, no per-instance hash-function fields: the caller passes the key and the checksum, which turned out simpler than #55 expected.

**A sink in `internal/nwsync`** — the zone by default, a local tree under `--out DIR` as the conformance path. Blobs upload as they are produced and the index lands last, so the presence of an index is the publication marker. A blob already in the zone is skipped via #55's probe *without* paying for compression (the body is a thunk) — which matters for the backfill, where compression is the expensive part.

**The settled CLI**

```
nwsync emit     [--as NAME] [--out DIR] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
```

Artifact keys are depot keys; an index lives beside its artifact with the extension replaced (#62), derived in exactly one place so `emit` and `assemble` cannot disagree. Flags may now follow positionals — Go's `flag` stops at the first non-flag argument, which cost a run during #59.

**Fail-closed in two places** — an artifact key whose embedded digest does not match the file is refused (publishing an index under the wrong key silently pairs a manifest with the wrong artifact), and `assemble` refuses an artifact with no index rather than publishing a manifest missing a hak.

## Checks

`make check` green. Six new tests run against a Bunny-shaped `httptest` zone that verifies the `Checksum` header the way Bunny does: blobs-then-index ordering, skip-if-present, no index after a failed upload, key/file mismatch, and assemble reading indexes back out of the zone.

Conformance re-run through the new CLI against upstream `nwn_nwsync_write` 2.1.2 over `sow_vfxs_01.hak` (#59's oracle): the manifest is still **byte-identical**.

## Not in this PR

- **Live upload against the real zone.** The nwsync zone and its credential are #61, still open. Everything here is proven against a fake zone only.
- **Consumer wiring** — #65, in the three producer repos.
- **The mid-hak failure *policy*.** The mechanism is here (fail closed, orphan blobs left, re-run resumes); whether a module release may proceed when an emit failed is a human call, still open on #60.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #73

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
This commit was merged in pull request #73.
This commit is contained in:
2026-07-30 21:56:06 +00:00
committed by archvillainette
parent a131b25e5b
commit f9051a2c01
9 changed files with 841 additions and 203 deletions
+43 -26
View File
@@ -4,18 +4,21 @@ import (
"crypto/sha1"
"encoding/json"
"fmt"
"os"
"path/filepath"
"path"
)
// AssembleOptions describes one merged manifest.
type AssembleOptions struct {
Order []string // artifact names, highest priority first
EntriesDir string // directory holding <name>.nsym and <name>.nsym.json
OutDir string // repository root; the manifest lands in <out>/manifests
GroupID int // 1 = current, 2 = testing; 0 means absent
ModuleName string
Description string
// ArtifactKeys are the depot keys of the artifacts to merge, in
// Mod_HakList order — highest priority first. Each one's index is read
// from the key beside it.
ArtifactKeys []string
TLKKey string // the TLK's key, if the manifest carries one
OutDir string // write locally instead of uploading — the conformance path
GroupID int // 1 = current, 2 = testing; 0 means absent
ModuleName string
Description string
Sink sink // test seam; nil means OutDir or the zone
}
// AssembleResult reports what one assemble run produced.
@@ -33,25 +36,43 @@ type AssembleResult struct {
// the game resolves it (upstream's resman adds haks in reverse and lets the
// last one win). Get this backwards and the wrong texture ships silently.
func Assemble(options AssembleOptions) (AssembleResult, error) {
if len(options.Order) == 0 {
return AssembleResult{}, fmt.Errorf("assemble: --order names no artifacts")
if len(options.ArtifactKeys) == 0 {
return AssembleResult{}, fmt.Errorf("assemble: no artifact keys given")
}
target, err := openSink(options.OutDir, options.Sink)
if err != nil {
return AssembleResult{}, err
}
// The TLK carries no precedence — it is not a hak and shadows nothing —
// so it merges last, after every hak has had its say.
keys := append([]string{}, options.ArtifactKeys...)
if options.TLKKey != "" {
keys = append(keys, options.TLKKey)
}
merged := make([]Entry, 0, 1024)
winner := make(map[Identity]bool, 1024)
var onDiskBytes int64
for _, name := range options.Order {
manifestPath := filepath.Join(options.EntriesDir, name+".nsym")
data, err := os.ReadFile(manifestPath)
for _, artifactKey := range keys {
key, err := resolveIndexKey(artifactKey, options.OutDir)
if err != nil {
return AssembleResult{}, fmt.Errorf("assemble: no index for %q: %w", name, err)
return AssembleResult{}, err
}
data, sidecarBody, err := target.getIndex(key)
if err != nil {
// An artifact with no index is an artifact whose emit never
// finished. Publishing a manifest without it would ship a release
// missing a hak, so this fails closed.
return AssembleResult{}, fmt.Errorf("assemble: no index for %s: %w", artifactKey, err)
}
entries, err := readManifest(data)
if err != nil {
return AssembleResult{}, fmt.Errorf("%s: %w", manifestPath, err)
return AssembleResult{}, fmt.Errorf("%s: %w", target.describe(key), err)
}
sidecar, err := readSidecar(manifestPath + ".json")
sidecar, err := parseSidecar(target.describe(key), sidecarBody)
if err != nil {
return AssembleResult{}, err
}
@@ -61,7 +82,7 @@ func Assemble(options AssembleOptions) (AssembleResult, error) {
if sidecar.EmitterVersion != emitterVersion {
return AssembleResult{}, fmt.Errorf(
"assemble: emitter version mismatch: %s was emitted by emitter %q, this is emitter %q",
name, sidecar.EmitterVersion, emitterVersion)
artifactKey, sidecar.EmitterVersion, emitterVersion)
}
// on_disk_bytes overcounts by the handful of cross-artifact
// duplicates. It is a display statistic; no dedupe pass for it.
@@ -86,26 +107,22 @@ func Assemble(options AssembleOptions) (AssembleResult, error) {
return AssembleResult{}, err
}
sha1Hex := fmt.Sprintf("%x", sha1.Sum(data))
manifestPath := filepath.Join(options.OutDir, "manifests", sha1Hex)
manifestKey := path.Join("manifests", sha1Hex)
sidecar := Sidecar{
ModuleName: options.ModuleName,
Description: options.Description,
GroupID: options.GroupID,
}
if err := writeManifestPair(manifestPath, data, merged, onDiskBytes, sidecar); err != nil {
if err := putManifestPair(target, manifestKey, data, merged, onDiskBytes, sidecar); err != nil {
return AssembleResult{}, err
}
return AssembleResult{SHA1: sha1Hex, ManifestPath: manifestPath, Entries: len(merged)}, nil
return AssembleResult{SHA1: sha1Hex, ManifestPath: target.describe(manifestKey), Entries: len(merged)}, nil
}
func readSidecar(path string) (Sidecar, error) {
data, err := os.ReadFile(path)
if err != nil {
return Sidecar{}, fmt.Errorf("assemble: missing sidecar: %w", err)
}
func parseSidecar(where string, data []byte) (Sidecar, error) {
var sidecar Sidecar
if err := json.Unmarshal(data, &sidecar); err != nil {
return Sidecar{}, fmt.Errorf("%s: %w", path, err)
return Sidecar{}, fmt.Errorf("%s: %w", where, err)
}
return sidecar, nil
}