nwsync: upload sink, key-addressed CLI, fail-closed publication marker #73

Merged
archvillainette merged 1 commits from feat/nwsync-upload-sink into main 2026-07-30 21:56:07 +00:00
Owner

Builds the code half of #60, and closes the CLI gap the #53 sweep found: PR #71 shipped #53's original surface rather than the one #56, #62 and #65 settled.

What lands

depot.KeyStoreProbeKey / PutReader / GetKey, addressing the zone by object key rather than by depot sha. NWSync cannot use the sha-addressed path: a blob is named after the sha1 of its uncompressed bytes while the body uploaded is the compressed form, and Bunny's Checksum header is sha256 of the body. Per #55 this reuses internal/depot's httpBackend — same IPv4-pinned transport, same retry, same tri-state probe — and the sha-addressed Backend is now rewritten on top of it. No second HTTP client, no per-instance hash-function fields: the caller passes the key and the checksum, which turned out simpler than #55 expected.

A sink in internal/nwsync — the zone by default, a local tree under --out DIR as the conformance path. Blobs upload as they are produced and the index lands last, so the presence of an index is the publication marker. A blob already in the zone is skipped via #55's probe without paying for compression (the body is a thunk) — which matters for the backfill, where compression is the expensive part.

The settled CLI

nwsync emit     [--as NAME] [--out DIR] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...

Artifact keys are depot keys; an index lives beside its artifact with the extension replaced (#62), derived in exactly one place so emit and assemble cannot disagree. Flags may now follow positionals — Go's flag stops at the first non-flag argument, which cost a run during #59.

Fail-closed in two places — an artifact key whose embedded digest does not match the file is refused (publishing an index under the wrong key silently pairs a manifest with the wrong artifact), and assemble refuses an artifact with no index rather than publishing a manifest missing a hak.

Checks

make check green. Six new tests run against a Bunny-shaped httptest zone that verifies the Checksum header the way Bunny does: blobs-then-index ordering, skip-if-present, no index after a failed upload, key/file mismatch, and assemble reading indexes back out of the zone.

Conformance re-run through the new CLI against upstream nwn_nwsync_write 2.1.2 over sow_vfxs_01.hak (#59's oracle): the manifest is still byte-identical.

Not in this PR

  • Live upload against the real zone. The nwsync zone and its credential are #61, still open. Everything here is proven against a fake zone only.
  • Consumer wiring#65, in the three producer repos.
  • The mid-hak failure policy. The mechanism is here (fail closed, orphan blobs left, re-run resumes); whether a module release may proceed when an emit failed is a human call, still open on #60.

🤖 Generated with Claude Code

Builds the code half of #60, and closes the CLI gap the #53 sweep found: PR #71 shipped #53's original surface rather than the one #56, #62 and #65 settled. ## What lands **`depot.KeyStore`** — `ProbeKey` / `PutReader` / `GetKey`, addressing the zone by object key rather than by depot sha. NWSync cannot use the sha-addressed path: a blob is named after the sha1 of its *uncompressed* bytes while the body uploaded is the compressed form, and Bunny's `Checksum` header is sha256 of the body. Per #55 this reuses `internal/depot`'s `httpBackend` — same IPv4-pinned transport, same retry, same tri-state probe — and the sha-addressed `Backend` is now rewritten on top of it. No second HTTP client, no per-instance hash-function fields: the caller passes the key and the checksum, which turned out simpler than #55 expected. **A sink in `internal/nwsync`** — the zone by default, a local tree under `--out DIR` as the conformance path. Blobs upload as they are produced and the index lands last, so the presence of an index is the publication marker. A blob already in the zone is skipped via #55's probe *without* paying for compression (the body is a thunk) — which matters for the backfill, where compression is the expensive part. **The settled CLI** ``` nwsync emit [--as NAME] [--out DIR] <artifact-key> <file> nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>... ``` Artifact keys are depot keys; an index lives beside its artifact with the extension replaced (#62), derived in exactly one place so `emit` and `assemble` cannot disagree. Flags may now follow positionals — Go's `flag` stops at the first non-flag argument, which cost a run during #59. **Fail-closed in two places** — an artifact key whose embedded digest does not match the file is refused (publishing an index under the wrong key silently pairs a manifest with the wrong artifact), and `assemble` refuses an artifact with no index rather than publishing a manifest missing a hak. ## Checks `make check` green. Six new tests run against a Bunny-shaped `httptest` zone that verifies the `Checksum` header the way Bunny does: blobs-then-index ordering, skip-if-present, no index after a failed upload, key/file mismatch, and assemble reading indexes back out of the zone. Conformance re-run through the new CLI against upstream `nwn_nwsync_write` 2.1.2 over `sow_vfxs_01.hak` (#59's oracle): the manifest is still **byte-identical**. ## Not in this PR - **Live upload against the real zone.** The nwsync zone and its credential are #61, still open. Everything here is proven against a fake zone only. - **Consumer wiring** — #65, in the three producer repos. - **The mid-hak failure *policy*.** The mechanism is here (fail closed, orphan blobs left, re-run resumes); whether a module release may proceed when an emit failed is a human call, still open on #60. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
archvillainette added 1 commit 2026-07-30 18:17:10 +00:00
Resolves the build half of sow-tools#60 and closes the CLI gap sow-tools#53's
sweep found: PR #71 shipped #53's original surface, not the one #56, #62 and
#65 settled.

- `depot.KeyStore` (`ProbeKey`/`PutReader`/`GetKey`) addresses the zone by
  object key instead of by depot sha, reusing the existing IPv4-pinned
  transport, retry and tri-state probe. The sha-addressed `Backend` now rides
  on it; no new HTTP client.
- `nwsync` gains a sink: the zone by default, a local tree with `--out DIR` as
  the conformance path. Blobs upload as they are produced, the index lands
  last, and a blob already in the zone is skipped without paying for
  compression.
- CLI is now `emit [--as NAME] [--out DIR] <artifact-key> <file>` and
  `assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...`.
  Indexes live beside their artifact with the extension replaced, derived in
  one place. Flags may follow positionals, which cost a run during sow-tools#59.
- Fail-closed: an artifact key whose digest does not match the file is refused,
  and `assemble` refuses an artifact with no index rather than publishing a
  manifest missing a hak.

Conformance re-checked through the new CLI against upstream nwn_nwsync_write
2.1.2 on sow_vfxs_01.hak: the manifest is still byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
archvillainette scheduled this pull request to auto merge when all checks succeed 2026-07-30 21:55:31 +00:00
xtul approved these changes 2026-07-30 21:56:04 +00:00
archvillainette merged commit f9051a2c01 into main 2026-07-30 21:56:07 +00:00
archvillainette deleted branch feat/nwsync-upload-sink 2026-07-30 21:56:07 +00:00
Sign in to join this conversation.