docs(nwsync): verify is what tells you which keys to purge

Running the repair for sow-tools#88 disproved the advice #90 had just
landed. "Purge the zone, then believe verify" assumed the stale set was
unknowable. It is not: verify reads the edge, so a run straight after a
repair names every key the edge is still serving stale — a survey, not a
verdict. Purge those, re-run, and the second run is the verdict.

The measured numbers are the argument. The repair rewrote 2,603 blobs at
the origin; 8 were stale at the edge, all of them ones a failed player
sync had pulled ninety minutes earlier. The edge only caches what someone
fetched, so purging the whole zone would have cooled 69,169 objects to
fix 8.

Refs #88, #89, #75.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.gitea/workflows/build-binaries.yml

@@ -65,3 +65,17 @@ jobs:
                 "${api}/releases/${id}/assets?name=$(basename "$f")"
             done
           '
+
+      # Gitea has no asset retention (#52): without this every tag keeps its
+      # ~57 MB binary set forever. Best-effort — a stale asset is cheaper than
+      # a blocked publish, so a failure here never fails the release.
+      - name: Prune assets of older releases
+        continue-on-error: true
+        env:
+          TOKEN: ${{ secrets.GITEA_TOKEN }}
+          REPO: ${{ github.repository }}
+          SERVER: ${{ github.server_url }}
+        run: |
+          nix develop --command bash -c '
+            API="${SERVER}/api/v1/repos/${REPO}" scripts/prune-release-assets.sh
+          '

AGENTS.md

@@ -87,3 +87,21 @@ make smoke                  # assert fail-closed contract
 ## Tests
 
 Tests must survive harmless changes to constants, defaults, wording, ordering, fixture data, and internal implementation details. A test that fails merely because a basic value changed is usually a bad test. Only assert exact values when the value is part of a documented public contract, external protocol, compatibility requirement, security rule, migration, or business rule.
+
+## Agent skills
+
+### Issue tracker
+
+Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`.
+
+### Triage labels
+
+Default label vocabulary (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
+
+### Domain docs
+
+Single-context: `CONTEXT.md` at the repo root plus `docs/adr/`. See `docs/agents/domain.md`.
+
+### Where work lives
+
+Markdown here is **reference, law, or an ADR — nothing else** (`sow-codebase` ADR-0001). Live work -> wayfinder maps + Gitea issues (closeable, assignable, queryable). Settled decisions -> ADR files in `docs/adr/` (immutable, never closed, only superseded). Standing law -> `DOCTRINE.md` / `AGENTS.md` / `CONTEXT.md`. Current-state reference -> docs describing what the code does now. Everything else — plans, specs, concepts, handoffs, trackers — is process: it belongs in a Gitea issue, not a file. Harvest unfinished intent to an issue before deleting a process doc. `sow-docs` is deprecated and read-only.

Makefile

@@ -7,6 +7,7 @@ check: vet test
 	shellcheck scripts/*.sh
 	yamllint .gitea
 	bash tests/workflow-contract.sh
+	bash tests/prune-release-assets.sh
 
 vet:
 	go vet ./...

README.md

@@ -19,6 +19,7 @@ Crucible is how the artifact repos turn source into artifacts.
 | `crucible-depot`   | `crucible depot`   | content-addressed depot blob verify/move        |
 | `crucible-hak`     | `crucible hak`     | ERF/HAK pack/unpack + hak manifests             |
 | `crucible-module`  | `crucible module`  | build/extract/validate/compare the `.mod`       |
+| `crucible-nwsync`  | `crucible nwsync`  | NWSync blob emit + manifest assemble + verify   |
 | `crucible-topdata` | `crucible topdata` | compile 2da/tlk topdata + packages              |
 | `crucible-wiki`    | `crucible wiki`    | render + deploy mechanical wiki pages           |
 
@@ -85,7 +86,9 @@ https://git.westgate.pw/ShadowsOverWestgate/sow-docs).
 - `ci.yml` — vet, test, shellcheck, yamllint, binary smoke, and cross-build all
   targets once per pull request.
 - `build-binaries.yml` — on a `v*` tag, cross-build and upload the binaries,
-  `SHA256SUMS`, and the wrappers to the Gitea release.
+  `SHA256SUMS`, and the wrappers to the Gitea release, then delete the assets
+  of every release except the newest two — Gitea keeps them forever otherwise,
+  and every binary is reproducible from its tag.
 - `sync-wrappers.yml` — on a `main` push that touches `wrappers/`, auto-PR the
   canonical wrappers to the consumer repos in `wrappers/consumers.txt`.
   Consumer drift checks run after those PRs merge to `main`, not on the PRs

cmd/crucible-nwsync/main.go

@@ -0,0 +1,11 @@
+// Command crucible-nwsync is the standalone nwsync builder (equivalent to
+// `crucible nwsync`). A single-token binary keeps consumer wrapper scripts simple.
+package main
+
+import (
+	"os"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/dispatch"
+)
+
+func main() { os.Exit(dispatch.RunBuilder("nwsync", os.Args[1:])) }

docs/adr/0001-markdown-is-reference-or-law-only.md

@@ -0,0 +1,55 @@
+# ADR-0001: Markdown in this repo is reference, law, or an ADR — nothing else
+
+- **Status:** Accepted
+- **Date:** 2026-07-24
+- **Origin:** Adopted workspace-wide from `sow-codebase` ADR-0001, which records
+  the full context (a `docs/` tree that had grown to 434 files, ~27MB, most of
+  it dead process artifacts). This repo adopts the same law so the rule is
+  local, not a cross-repo reference.
+
+## Context
+
+Markdown that *claims to describe current state or a plan* rots, because
+reality diverges and nobody edits the file. Markdown that claims neither — a
+dated, immutable decision record, or a standing rule — does not rot.
+
+Two different things get conflated:
+
+- **Rationale** — why a system is shaped the way it is. Worth keeping.
+- **Process artifacts** — plans, specs, concepts, handoffs, trackers. A record
+  of how work happened, not what is true now.
+
+Left unchecked, every completed effort leaves its planning behind and the repo
+accumulates a "historical" pile that agents and humans must route around to
+find the few live docs.
+
+## Decision
+
+Markdown may exist in this repo only as one of three genres:
+
+1. **Current-state reference** — describes what the code does now, kept honest
+   by code review touching it.
+2. **Standing law** — `DOCTRINE.md`, root and folder-scoped `AGENTS.md`,
+   `CONTEXT.md`. States rules and vocabulary, not plans.
+3. **Immutable decision record** — ADRs under `docs/adr/`. Append-only; never
+   edited, only superseded by a later ADR that points back.
+
+Anything else — implementation plans, design specs, concepts, handoffs,
+progress trackers, scratch — is **process** and does not live in the repo. It
+lives in Gitea issues and wayfinder maps, where it can be assigned, closed, and
+superseded. Small tasks need no written plan at all.
+
+Deleting a process document is not destroying history: `git log -- <path>`
+recovers it. The exception is *unfinished intent* (a design never built, an
+open question still wanted) — that is live, not history, and must be harvested
+to a Gitea ticket before its file is deleted. A citation from a living doc or
+from code must be resolved before the cited file is deleted.
+
+## Consequences
+
+- The documentation map lists only current truth; there is no "historical" pile
+  to route around.
+- No agent, under any plugin or skill, writes process-markdown into the repo.
+  The rule is plugin-agnostic on purpose — it binds the agent, not a named tool.
+- History of deleted process docs is in git; unfinished intent is in the issue
+  tracker; rationale is in ADRs and law. Each thing has exactly one home.

docs/adr/0002-sow-tools-becomes-the-crucible-suite.md

@@ -0,0 +1,12 @@
+# ADR-0002: `sow-tools` becomes the Crucible suite
+
+- **Status:** Accepted
+- **Date:** 2026-06-11
+- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D11**, on the
+  retirement of `sow-docs`. Content is the original decision, unchanged.
+
+- **Decision:** Rename the future toolchain surface to Crucible: one Go module,
+  multiple `cmd/` binaries, plus a dispatcher so wrapper commands can stay
+  stable.
+- **Rationale:** Depot, HAK, module, topdata, and wiki tooling share internals
+  but have different command surfaces. One module avoids premature repo splits.

docs/adr/0003-crucible-binary-contract-standalone-shims-fail-closed-scaffold.md

@@ -0,0 +1,29 @@
+# ADR-0003: Crucible binary contract: standalone shims, fail-closed scaffold, no committed binaries
+
+- **Status:** Accepted
+- **Date:** 2026-06-11
+- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D19**, on the
+  retirement of `sow-docs`. Content is the original decision, unchanged.
+
+- **Decision:** Phase 5 builds `migration/sow-tools` (Crucible, D11) as a
+  multi-binary Go scaffold:
+  - One `crucible` dispatcher plus standalone `crucible-{depot,hak,module,topdata,wiki}`
+    binaries sharing one registry (`internal/dispatch`). The dispatcher is for
+    humans/CI; the **standalone shims are canonical for consumers** so wrapper
+    scripts resolve a single-token command and `"$builder" args` quoting stays
+    correct.
+  - Consumer resolution order: explicit env override
+    (`$SOW_MODULE_BUILD`/`$SOW_TOPDATA_BUILD`/`$CRUCIBLE`) → `crucible-<name>` →
+    legacy `sow-<name>-build`. CI runs inside the pinned `crucible:<sha>` image.
+  - Every builder is **unwired** in the scaffold and fails closed (exit 70);
+    it never fakes an artifact. The `internal/` logic is migrated from
+    `gitea/sow-tools` by the operator at cutover (hard rule: no source
+    transplant by tooling).
+  - **Binaries are never committed** — they are CI artifacts / image layers.
+    Retires the committed `nwn-tool` / `tools/sow-toolkit`.
+  - Builders take `NWN_ROOT` only via explicit env/flag; no `$HOME` defaulting.
+- **Rationale:** Locks the command surface, container, and CI shape so migrated
+  logic drops into a stable frame; aligns the three artifact repos onto one
+  Crucible contract while keeping back-compat with the pre-D11 skeletons.
+- **Image name:** `registry.westgate.pw/deployment/crucible:<git-sha>` (the bare
+  `crucible:<sha>` is the local/dev tag).

docs/adr/0004-nix-built-oci-images-start-server-side-with.md

@@ -0,0 +1,24 @@
+# ADR-0004: Nix-built OCI images start server-side with Crucible; local Anvil images deferred
+
+- **Status:** Accepted
+- **Date:** 2026-06-12
+- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D29**, on the
+  retirement of `sow-docs`. Content is the original decision, unchanged.
+
+- **Decision:** Introduce Nix-built OCI images as an artifact-production option,
+  starting with `sow-tools`/Crucible in server-side CI. `sow-tools` will grow a
+  Nix-built `crucible` package plus `crucible-image`; PR CI builds/smokes it and
+  `main` publishes the normal pinned OCI image
+  `registry.westgate.pw/deployment/crucible:<sha>`. `docker/Dockerfile` stays during
+  parity and as the client-compatible fallback.
+- **Local rule:** Nix users may get optional wrappers that provide tools and
+  call the existing Dockerfiles for local Anvil/NWServer testing. A true
+  `nix build .#nwserver-test-image` is explicitly deferred until real
+  `sow-codebase/src` exists and the Dockerfile image is proven.
+- **Rationale:** Server-side image build shape is harder to change once deploy
+  promotion and registry gates are active, so prove Nix image builds before
+  cutover. Crucible is the lowest-risk pilot because it is a Go toolchain image;
+  NodeBB and Anvil/NWServer depend more heavily on upstream container filesystem
+  semantics.
+- **Non-goals:** no source builds in `sow-platform`; no `nix-sidecar` or
+  Kubernetes-style runtime cache layer; no removal of client Dockerfiles.

docs/agents/domain.md

@@ -0,0 +1,51 @@
+# Domain Docs
+
+How the engineering skills should consume this repo's domain documentation when exploring the codebase.
+
+## Before exploring, read these
+
+- **`CONTEXT.md`** at the repo root, or
+- **`CONTEXT-MAP.md`** at the repo root if it exists — it points at one `CONTEXT.md` per context. Read each one relevant to the topic.
+- **`docs/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src/<context>/docs/adr/` for context-scoped decisions.
+
+If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
+
+## File structure
+
+Single-context repo (most repos):
+
+```
+/
+├── CONTEXT.md
+├── docs/adr/
+│   ├── 0001-event-sourced-orders.md
+│   └── 0002-postgres-for-write-model.md
+└── src/
+```
+
+Multi-context repo (presence of `CONTEXT-MAP.md` at the root):
+
+```
+/
+├── CONTEXT-MAP.md
+├── docs/adr/                          ← system-wide decisions
+└── src/
+    ├── ordering/
+    │   ├── CONTEXT.md
+    │   └── docs/adr/                  ← context-specific decisions
+    └── billing/
+        ├── CONTEXT.md
+        └── docs/adr/
+```
+
+## Use the glossary's vocabulary
+
+When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary explicitly avoids.
+
+If the concept you need isn't in the glossary yet, that's a signal — either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`).
+
+## Flag ADR conflicts
+
+If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
+
+> _Contradicts ADR-0007 (event-sourced orders) — but worth reopening because…_

docs/agents/issue-tracker.md

@@ -0,0 +1,88 @@
+# Issue tracker: Gitea (via tea)
+
+Issues for this repo live in Gitea at `git.westgate.pw`, repo
+`ShadowsOverWestgate/sow-tools`. Use the `tea` CLI for all operations —
+`gh` does not work here. For anything `tea` lacks a subcommand for, use
+`tea api <endpoint>` (Gitea's API mirrors GitHub's closely).
+
+Authenticate with your own `tea` login (`tea login add`); never commit tokens
+or tea config into this repo. Note Gitea blocks self-review, so approving a PR
+needs a different account than the one that opened it.
+
+## Where work lives
+
+Markdown in this repo is **reference, law, or an ADR — nothing else**
+(`sow-codebase` ADR-0001). Four homes, no overlap:
+
+- **Live work** → wayfinder maps + Gitea issues. Closeable, assignable,
+  queryable. Never a markdown file.
+- **Settled decisions** → ADR files in `docs/adr/`. Immutable, findable, never
+  closed, never edited — only superseded by a later ADR pointing back. When an
+  issue ends in a durable decision, write the ADR, then close the issue
+  pointing at it. Decisions spanning repos go to `sow-platform/docs/adr/`.
+- **Standing law** → `DOCTRINE.md`, `AGENTS.md`, `CONTEXT.md`. Rules that are
+  always true and vocabulary everyone shares — not the record of one decision.
+- **Current-state reference** → docs describing what the code does now, kept
+  honest by review touching them.
+
+Anything else — implementation plans, design specs, concepts, handoffs,
+progress trackers, scratch — is **process**. It does not live in this repo. It
+lives in a Gitea issue or a wayfinder map, where it can be assigned, closed,
+and superseded. Small tasks need no written plan at all.
+
+Deleting a process doc is not destroying history — `git log -- <path>` recovers
+it. But *unfinished intent* (a design never built, an open question still
+wanted) is live, not history: harvest it to a Gitea issue before deleting.
+
+`sow-docs` is deprecated and read-only. Never add to it, never send work there.
+
+## Which repo gets the issue
+
+Issues follow ownership. File the issue in the repo that **owns the work** —
+see the Repo/Owns/Produces table in the workspace root `AGENTS.md`. Standing
+in one repo is not a reason to file there.
+
+If work spans repos, file it in the repo that owns the *outcome* and reference
+the others from it. A wrong-repo issue is a routing bug, not a filing
+preference — move it.
+
+## Conventions
+
+- **Create an issue**: `tea issues create --title "..." --description "..."`
+- **Read an issue**: `tea issues <number>` and
+  `tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/comments` for comments.
+- **List issues**: `tea issues list --state open` (add `--labels ...` to filter).
+- **Comment**: `tea comment <number> "..." </dev/null`
+  Always redirect stdin. `tea` reads stdin to EOF and appends it to the body,
+  so any non-interactive shell (every agent) hangs forever without
+  `</dev/null`. Same trap on `tea issues create --description` and
+  `tea pr create`.
+- **Apply / remove labels**: `tea api --method PATCH` on the issue, or
+  `tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/labels` endpoints.
+- **Close**: `tea issues close <number>`
+
+`tea` infers the repo from the git remote when run inside the clone.
+Gitea shares one number space across issues and PRs.
+
+## Pull requests as a triage surface
+
+**PRs as a request surface: no.**
+
+## When a skill says "publish to the issue tracker"
+
+Create a Gitea issue with `tea issues create`.
+
+## When a skill says "fetch the relevant ticket"
+
+Run `tea issues <number>` plus the comments API call above.
+
+## Wayfinding operations
+
+Used by `/wayfinder`. The **map** is a single issue with **child** issues as tickets.
+
+- **Map**: a single issue labelled `wayfinder:map`, holding the Notes / Decisions-so-far / Fog body. `tea issues create --title "..." --description "..." --labels wayfinder:map`.
+- **Child ticket**: this Gitea instance (v1.27) has no native sub-issue hierarchy, so a child issue carries `Part of #<map>` at the top of its description, and is also added to a task list in the map body. Labels: `wayfinder:<type>` (`research`/`prototype`/`grilling`/`task`). Once claimed, the ticket is assigned to the driving dev.
+- **Blocking**: Gitea's **native dependencies API** — the canonical, UI-visible representation (shows as "Depends on" / "Blocks" on the issue page). Add an edge with `tea api -X POST repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies -f owner=ShadowsOverWestgate -f repo=sow-tools -F index=<blocker>`, where `<blocker>` is the blocker's issue **index** (its `#number` — Gitea's dependency API takes the index directly, unlike GitHub's numeric database id). Check status with `tea api repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies` (GET) — a ticket is unblocked when every returned issue's `state` is `closed`.
+- **Frontier query**: list the map's open children (`tea issues list --state open`, keep the ones whose description contains `Part of #<map>`), drop any with an open dependency (per the GET above) or an assignee; first in map order wins.
+- **Claim**: `tea issues edit <n> --add-assignees <username>` — the session's first write.
+- **Resolve**: `tea comment <n> "<answer>" </dev/null`, then `tea issues close <n>`, then append a context pointer (gist + link) to the map's Decisions-so-far.

docs/agents/triage-labels.md

@@ -0,0 +1,15 @@
+# Triage Labels
+
+The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker (Gitea — see `issue-tracker.md` for how to apply labels with `tea`).
+
+| Label in mattpocock/skills | Label in our tracker | Meaning                                  |
+| -------------------------- | -------------------- | ---------------------------------------- |
+| `needs-triage`             | `needs-triage`       | Maintainer needs to evaluate this issue  |
+| `needs-info`               | `needs-info`         | Waiting on reporter for more information |
+| `ready-for-agent`          | `ready-for-agent`    | Fully specified, ready for an AFK agent  |
+| `ready-for-human`          | `ready-for-human`    | Requires human implementation            |
+| `wontfix`                  | `wontfix`            | Will not be actioned                     |
+
+When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
+
+Edit the right-hand column to match whatever vocabulary you actually use.

docs/command-surface.md

@@ -29,9 +29,136 @@ aliases.
 | `assets` | `fix-mdl` | Lowercase `.mdl` names and rewrite model identity to match. |
 | `assets` | `check-dupes` | Report runtime-name (basename) collisions across dirs. |
 | `assets` | `clean-dupes` | Delete clean-tree files whose basename collides with primary. |
+| `depot` | `status` | Report referenced-vs-present drift against a backend. |
+| `depot` | `push` | Upload referenced-but-absent blobs from a local depot. |
+| `depot` | `verify` | Existence sweep plus sampled download re-hash. |
+| `depot` | `get` | Fetch one blob with sha re-verify. |
+| `depot` | `pull` | Incremental verified pull of every referenced blob. |
+| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
+| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
+| `nwsync` | `verify` | Decompress and hash a published manifest's blobs through the pull zone. |
 
-`crucible-depot` remains registered but unwired. It fails closed with exit `70`
+`depot status` and `depot get` pick their backend either with `--out DIR`, a
-and never emits placeholder artifacts.
+depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
+flags are mutually exclusive.
+
+`nwsync emit` runs where an artifact is born (a `.hak`/`.erf`, or a loose file
+such as the TLK); `nwsync assemble` runs at module release and reads only the
+small per-artifact indexes. Both take **depot keys**: an artifact's index lives
+beside the artifact itself with the extension replaced, so `emit` and
+`assemble` agree on where it is without being told.
+
+```
+nwsync emit     [--as NAME] [--out DIR] [--jobs N] [--verify] <artifact-key> <file>
+nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
+nwsync verify   [--sample N] [--base URL] [--jobs N] <manifest-sha1>
+```
+
+`emit` is latency-bound, not CPU-bound: every blob costs an existence probe
+plus an upload, and a measured backfill spent 26 seconds of CPU across 9.5
+minutes of wall clock. `--jobs N` (default 16) sets how many resources are in
+flight at once. The manifest is byte-identical at any value — the number of
+workers is never observable in the output. Peak memory is `N` times the
+per-resource limit of 15 MB plus its compressed copy, so raising `N` far past
+the default costs real memory for little gain: the transport keeps 16 idle
+connections per host, and past that a worker pays a fresh TLS handshake.
+
+Both verbs upload by default; nothing bulky is ever written to the runner's
+disk. `--out DIR` writes a local repository tree instead, which is the
+conformance path against upstream `nwn_nwsync_write`. The zone comes from
+`NWSYNC_STORAGE_ZONE` and `NWSYNC_STORAGE_PASSWORD`, with the host from
+`BUNNY_STORAGE_HOST` — NWSync data is a separate zone from the asset depot.
+
+`assemble`'s artifact keys are in `Mod_HakList` order, highest priority first: a
+resref in more than one artifact resolves to the earliest one, the way the game
+resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
+`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
+out of the sidecar.
+
+`nwsync verify` is the only check on a published blob upstream of a player's
+client. It reads the **pull zone**, not the storage API, and needs no
+credential: what matters is the bytes a client is served, edge behaviour
+included. Every blob is decompressed and hashed, and the zstd frame is asserted
+to declare its content size. Neither half is optional — a `Content-Length` check
+passes a byte-correct-looking object whose contents are short, and a round-trip
+check alone passes a frame the game client cannot decode but Go's decoder can.
+Failures are reported per blob as missing, malformed framing, size mismatch or
+hash mismatch, and the exit code is 1.
+
+A full sweep of the live manifest is roughly 69,000 blobs and 15 GB, so
+`--sample N` exists to make verifying routine; the default is a full sweep.
+`--base URL` (or `NWSYNC_PULL_BASE`) overrides the public host.
+
+`emit --verify` applies the same check where `emit` would otherwise skip. `emit`
+normally reads a blob's presence as proof of its contents, decided by a 1-byte
+range GET, so an object written truncated — or written by an emitter since found
+broken — is skipped by every later run forever and no backfill repairs it. With
+`--verify` the stored copy is read back, unwrapped, hashed against its own name,
+and replaced when it does not match. It costs a full GET per existing blob, so
+it is a repair pass, not the default.
+
+**After a repair, `verify` is what tells you which keys to purge.** A repair is
+the one thing that makes a key serve different bytes than it did before, and the
+edge caches these objects for 30 days precisely because that normally cannot
+happen. The two commands look at different copies on purpose: `emit --verify`
+repairs the **origin**, `verify` reads the **edge**. So a `verify` run straight
+after a repair is not a verdict — it is a survey, and every blob it still calls
+bad is one the edge is serving stale. Purge exactly those, then re-run it; only
+that second run is the verdict.
+
+Purging the keys `verify` names beats purging the zone, because the edge only
+ever cached what somebody actually fetched: the 2026-08-01 repair rewrote 2,603
+blobs at the origin and left 8 stale at the edge. The purge belongs in the
+repair procedure rather than in `emit`, which reports how many blobs it wrote
+and never which ones — so it could not target one even with a CDN credential,
+which it deliberately does not hold (#89; the procedure itself is in
+sow-platform's NWSync runbook).
+
+`emit` uploads blobs first and the index last, so the presence of an index is
+the publication marker: an artifact whose emit died halfway leaves real blobs in
+the zone and no index. Blob names are content hashes, so re-running skips
+whatever already landed, and `assemble` fails closed on an artifact with no
+index rather than publishing a manifest that is missing a hak.
+
+### What an emitted tree looks like
+
+`--out DIR` produces the same tree `emit` would upload, which makes it the way
+to check a zone by hand without touching one:
+
+```
+<artifact-sha>.nsym          binary index
+<artifact-sha>.nsym.json     the same index, readable
+data/sha1/a7/4a/a74aa84a...  one blob per resource, two-level fanout
+```
+
+A blob's name is the SHA-1 of the resource's **original** bytes, but the file on
+disk is not those bytes: each blob is wrapped in NWCompressedBuffer framing, a
+24-byte `NSYC` header followed by a zstd frame. Hashing the file directly will
+not match its name, which is the obvious
+first thing to try and the obvious first thing to be confused by. Strip the
+header first:
+
+```
+tail -c +25 <blob> | zstd -dc | sha1sum   # == the blob's filename
+```
+
+The header carries the uncompressed length as a little-endian `uint32` at offset
+12, so the decompressed size is checkable without decompressing. Compression is
+worth roughly a 4:1
+saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
+against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
+`total_bytes`.
+
+The zstd frame always declares its `Frame_Content_Size`. The game client sizes
+its output buffer from that field and cannot decode a frame without one, but the
+Go encoder omits it below 256 bytes, so `emit` re-headers those frames into the
+shape reference libzstd emits: `Single_Segment_flag` set, `Window_Descriptor`
+dropped, and a one-byte content size in its place. `zstd -l <frame>` must print a
+decompressed size; a blank column there is the fault, and it is invisible to any
+check that only decompresses, because both `zstd -dc` and Go's decoder stream
+such a frame happily. This is what the sidecar's `emitter_version` counts:
+version 1 omitted the field and no client could sync past such a blob, version 2
+declares it. `assemble` refuses to merge indexes that disagree.
 
 ## Hidden compatibility aliases
 
@@ -49,6 +176,7 @@ but omitted from routine help and the interactive menu:
 | `topdata` | `build-top-package` | `build-top-package` |
 | `topdata` | `compare-topdata` | `compare-topdata` |
 | `topdata` | `convert-topdata` | `convert-topdata` |
+| `depot` | `--target local` | `--out $DEPOT_DIR` on `status`/`get` |
 | `wiki` | `build-wiki` | `build-wiki` |
 | `wiki` | `deploy-wiki` | `deploy-wiki` |
 

flake.nix

@@ -22,12 +22,13 @@
             pname = "crucible";
             inherit version;
             src = ./.;
-            vendorHash = "sha256-hm6mrNAtXv0LidzHUfz4eukTFZouizGtxkZ8gKJFUVI=";
+            vendorHash = "sha256-0I8j7On9YGD2GK9xbj/KkgBrlkMJ6Y6XQv+KCLTgBBU=";
             subPackages = [
               "cmd/crucible"
               "cmd/crucible-depot"
               "cmd/crucible-hak"
               "cmd/crucible-module"
+              "cmd/crucible-nwsync"
               "cmd/crucible-topdata"
               "cmd/crucible-wiki"
             ];

go.mod

@@ -6,3 +6,5 @@ require (
 	golang.org/x/text v0.35.0
 	gopkg.in/yaml.v3 v3.0.1
 )
+
+require github.com/klauspost/compress v1.19.1

go.sum

@@ -1,3 +1,5 @@
+github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
+github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
 golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=

internal/app/app.go

@@ -385,7 +385,7 @@ func refreshBuildModuleManifest(ctx context, p *project.Project, progress func(s
 	}
 
 	progress("Refreshing hak list from the latest published sow-assets manifest...")
-	if err := runProjectScript(ctx, p, []string{"scripts", "fetch-hak-manifest"}, manifestPath); err != nil {
+	if err := runProjectScript(ctx, p, []string{"scripts", "fetch-upstream-manifests"}, manifestPath); err != nil {
 		return "", "", err
 	}
 	if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil {

internal/depot/keystore.go

@@ -0,0 +1,121 @@
+package depot
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+)
+
+// KeyStore is the zone addressed by object key rather than by depot sha. The
+// depot names every object after the sha256 of its contents; NWSync does not —
+// a blob is named after the sha1 of its *uncompressed* bytes while the body
+// uploaded is the compressed form, and a per-artifact index is named after its
+// artifact. Both addressing modes want the same transport, retry and probe
+// discipline, so the sha-addressed Backend rides on this rather than the other
+// way round.
+type KeyStore interface {
+	// ProbeKey returns the existence state of one key. transient=true means a
+	// retry might change the answer — never read it as "missing, re-upload".
+	ProbeKey(ctx context.Context, key string) (state ProbeState, transient bool, err error)
+	// PutReader uploads size bytes read from r to key. checksum is the
+	// uppercase hex sha256 of those bytes, which Bunny verifies server-side.
+	PutReader(ctx context.Context, key string, r io.Reader, size int64, checksum string) error
+	// GetKey fetches the whole object at key. Small objects only — it holds
+	// the body in memory and does no hash check, because a key is not always
+	// a content hash.
+	GetKey(ctx context.Context, key string) ([]byte, error)
+}
+
+// NewKeyStore returns a KeyStore for cfg's storage zone. Fails closed on a
+// missing host or read key, matching NewBackend.
+func NewKeyStore(cfg Config) (KeyStore, error) {
+	if cfg.StorageHost == "" {
+		return nil, errors.New("storage backend requires a storage host")
+	}
+	if cfg.StorageZone == "" {
+		return nil, errors.New("storage backend requires a storage zone")
+	}
+	if cfg.ReadKey == "" {
+		return nil, errors.New("storage backend requires a read key")
+	}
+	return &httpBackend{name: "bunny", client: newHTTPClient(cfg), cfg: cfg}, nil
+}
+
+// keyURL is the storage URL of one object key.
+func (b *httpBackend) keyURL(key string) string {
+	host := b.cfg.StorageHost
+	// StorageHost is normally a bare host ("storage.bunnycdn.com"); allow a
+	// full scheme (used by tests against httptest.NewServer) to pass through
+	// unchanged.
+	if !strings.Contains(host, "://") {
+		host = "https://" + host
+	}
+	return fmt.Sprintf("%s/%s/%s", strings.TrimSuffix(host, "/"), b.cfg.StorageZone, key)
+}
+
+func (b *httpBackend) ProbeKey(ctx context.Context, key string) (ProbeState, bool, error) {
+	return b.rangeProbe(ctx, b.keyURL(key), map[string]string{"AccessKey": b.cfg.ReadKey})
+}
+
+func (b *httpBackend) PutReader(ctx context.Context, key string, r io.Reader, size int64, checksum string) error {
+	if b.name == "cdn" {
+		return errors.New("cdn backend is read-only")
+	}
+	if b.cfg.WriteKey == "" {
+		return errors.New("storage backend requires a write key to write")
+	}
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodPut, b.keyURL(key), r)
+	if err != nil {
+		return err
+	}
+	req.ContentLength = size
+	req.Header.Set("AccessKey", b.cfg.WriteKey)
+	// Bunny defines Checksum as sha256 of the body and rejects a mismatch, so
+	// this is server-side integrity checking, not decoration.
+	req.Header.Set("Checksum", strings.ToUpper(checksum))
+
+	resp, err := b.client.Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	_, _ = io.Copy(io.Discard, resp.Body)
+
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return fmt.Errorf("put %s: unexpected status %d", key, resp.StatusCode)
+	}
+	return nil
+}
+
+func (b *httpBackend) GetKey(ctx context.Context, key string) ([]byte, error) {
+	resp, err := b.get(ctx, b.keyURL(key), map[string]string{"AccessKey": b.cfg.ReadKey})
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		_, _ = io.Copy(io.Discard, resp.Body)
+		return nil, fmt.Errorf("get %s: unexpected status %d", key, resp.StatusCode)
+	}
+	return io.ReadAll(resp.Body)
+}
+
+// putFile uploads the file at src to key, streaming it. checksum is the
+// uppercase hex sha256 of the file's bytes.
+func (b *httpBackend) putFile(ctx context.Context, key, src, checksum string) error {
+	f, err := os.Open(src)
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	info, err := f.Stat()
+	if err != nil {
+		return err
+	}
+	return b.PutReader(ctx, key, f, info.Size(), checksum)
+}

internal/depot/remote.go

@@ -10,7 +10,6 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
-	"strings"
 	"time"
 )
 
@@ -71,16 +70,7 @@ type httpBackend struct {
 
 func (b *httpBackend) Name() string { return b.name }
 
-func (b *httpBackend) storageURL(sha string) string {
+func (b *httpBackend) storageURL(sha string) string { return b.keyURL(BlobKey(sha)) }
-	host := b.cfg.StorageHost
-	// StorageHost is normally a bare host ("storage.bunnycdn.com"); allow a
-	// full scheme (used by tests against httptest.NewServer) to pass through
-	// unchanged.
-	if strings.Contains(host, "://") {
-		return fmt.Sprintf("%s/%s/%s", strings.TrimSuffix(host, "/"), b.cfg.StorageZone, BlobKey(sha))
-	}
-	return fmt.Sprintf("https://%s/%s/%s", host, b.cfg.StorageZone, BlobKey(sha))
-}
 
 func (b *httpBackend) cdnURL(sha string) string {
 	return fmt.Sprintf("%s/%s", b.cfg.CDNBase, BlobKey(sha))
@@ -140,7 +130,8 @@ func (b *httpBackend) Probe(ctx context.Context, sha string) (ProbeState, bool,
 	return b.rangeProbe(ctx, b.storageURL(sha), map[string]string{"AccessKey": b.cfg.ReadKey})
 }
 
-// Put uploads src for sha. cdn is read-only.
+// Put uploads src for sha. cdn is read-only. A depot object is named after the
+// sha256 of its own bytes, so the key's sha doubles as the Checksum header.
 func (b *httpBackend) Put(ctx context.Context, sha, src string) error {
 	if b.name == "cdn" {
 		return errors.New("cdn backend is read-only")
@@ -157,30 +148,7 @@ func (b *httpBackend) Put(ctx context.Context, sha, src string) error {
 		return nil
 	}
 
-	f, err := os.Open(src)
+	return b.putFile(ctx, BlobKey(sha), src, sha)
-	if err != nil {
-		return err
-	}
-	defer f.Close()
-
-	req, err := http.NewRequestWithContext(ctx, http.MethodPut, b.storageURL(sha), f)
-	if err != nil {
-		return err
-	}
-	req.Header.Set("AccessKey", b.cfg.WriteKey)
-	req.Header.Set("Checksum", strings.ToUpper(sha))
-
-	resp, err := b.client.Do(req)
-	if err != nil {
-		return err
-	}
-	defer resp.Body.Close()
-	_, _ = io.Copy(io.Discard, resp.Body)
-
-	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
-		return fmt.Errorf("bunny put %s: unexpected status %d", sha, resp.StatusCode)
-	}
-	return nil
 }
 
 // Get fetches sha into dest via temp file + rename, re-hashing and deleting

internal/depot/run.go

@@ -11,6 +11,7 @@ import (
 	"os"
 	"path/filepath"
 	"sort"
+	"strings"
 	"sync"
 	"time"
 )
@@ -51,13 +52,15 @@ func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) in
 
 func printRunUsage(w io.Writer) {
 	fmt.Fprint(w, `usage:
-  depot status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
+  depot status [--manifests DIR] --target bunny|cdn | --out DIR
-  depot push   [--manifests DIR] --source LOCAL_DEPOT   --target bunny
+  depot push   [--manifests DIR] --source LOCAL_DEPOT --target bunny
-  depot verify [--manifests DIR] --target bunny|cdn     [--sample N]
+  depot verify [--manifests DIR] --target bunny|cdn   [--sample N]
-  depot get    <sha> <dest>       --target cdn|bunny|local
+  depot get    <sha> <dest>      --target cdn|bunny   | --out DIR
-  depot pull   [--manifests DIR] --dest DIR             --target cdn|bunny
+  depot pull   [--manifests DIR] --dest DIR           --target cdn|bunny
 
---target local uses the DEPOT_DIR environment variable as the local root.
+--out DIR reads and writes a depot tree on disk at DIR instead of a remote
+backend. --target names a remote backend only; the two are mutually exclusive.
+--target local is a deprecated alias for --out $DEPOT_DIR.
 `)
 }
 
@@ -65,16 +68,27 @@ func printRunUsage(w io.Writer) {
 // than internal/backend failures (exit 70).
 var errUsage = errors.New("usage error")
 
-// resolveBackend builds the named backend, resolving "local" against DEPOT_DIR.
+// resolveBackend picks the backend for a command that can work either against a
-func resolveBackend(target string, getenv func(string) string, cfg Config) (Backend, error) {
+// depot tree on disk (--out DIR) or a remote backend (--target bunny|cdn).
-	root := ""
+// "--target local" stays as a deprecated alias for --out $DEPOT_DIR so older
-	if target == "local" {
+// callers keep working.
-		root = getenv("DEPOT_DIR")
+func resolveBackend(out, target string, getenv func(string) string, cfg Config) (Backend, error) {
+	switch {
+	case out != "" && target != "":
+		return nil, fmt.Errorf("--out and --target are mutually exclusive: %w", errUsage)
+	case out != "":
+		return NewBackend("local", out, cfg)
+	case target == "local":
+		root := getenv("DEPOT_DIR")
 		if root == "" {
-			return nil, fmt.Errorf("--target local requires DEPOT_DIR to be set: %w", errUsage)
+			return nil, fmt.Errorf("--target local requires DEPOT_DIR to be set (prefer --out DIR): %w", errUsage)
 		}
+		return NewBackend("local", root, cfg)
+	case target == "":
+		return nil, fmt.Errorf("--out DIR or --target bunny|cdn is required: %w", errUsage)
+	default:
+		return NewBackend(target, "", cfg)
 	}
-	return NewBackend(target, root, cfg)
 }
 
 // backendErrExit maps a resolveBackend error onto the exit contract.
@@ -119,18 +133,18 @@ func runStatus(args []string, stdout, stderr io.Writer, getenv func(string) stri
 	fs := flag.NewFlagSet("status", flag.ContinueOnError)
 	fs.SetOutput(stderr)
 	manifests := fs.String("manifests", "assets", "directory of *.yml manifests")
-	_ = fs.String("source", "", "local depot root (unused for status target=local; see DEPOT_DIR)")
+	source := fs.String("source", "", "deprecated and ignored (use --out DIR for a depot tree on disk)")
-	target := fs.String("target", "", "bunny|cdn|local")
+	out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
+	target := fs.String("target", "", "bunny|cdn")
 	if err := fs.Parse(args); err != nil {
 		return exitUsage
 	}
-	if *target == "" {
+	if *source != "" {
-		fmt.Fprintln(stderr, "depot status: --target is required")
+		fmt.Fprintln(stderr, "depot status: --source is ignored; use --out DIR")
-		return exitUsage
 	}
 
 	cfg := LoadConfig(getenv)
-	backend, err := resolveBackend(*target, getenv, cfg)
+	backend, err := resolveBackend(*out, *target, getenv, cfg)
 	if err != nil {
 		fmt.Fprintln(stderr, "depot status:", err)
 		return backendErrExit(err)
@@ -309,13 +323,21 @@ func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) stri
 func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
 	fs := flag.NewFlagSet("get", flag.ContinueOnError)
 	fs.SetOutput(stderr)
-	target := fs.String("target", "", "cdn|bunny|local")
+	out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
-	if err := fs.Parse(args); err != nil {
+	target := fs.String("target", "", "cdn|bunny")
+	// Positionals come first in the documented usage, and Go's flag package
+	// stops parsing at the first one, so split them off by hand.
+	split := 0
+	for split < len(args) && !strings.HasPrefix(args[split], "-") {
+		split++
+	}
+	positional := args[:split]
+	if err := fs.Parse(args[split:]); err != nil {
 		return exitUsage
 	}
-	positional := fs.Args()
+	positional = append(positional, fs.Args()...)
 	if len(positional) != 2 {
-		fmt.Fprintln(stderr, "depot get: usage: depot get <sha> <dest> --target cdn|bunny|local")
+		fmt.Fprintln(stderr, "depot get: usage: depot get <sha> <dest> --target cdn|bunny | --out DIR")
 		return exitUsage
 	}
 	sha, dest := positional[0], positional[1]
@@ -323,13 +345,9 @@ func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string)
 		fmt.Fprintf(stderr, "depot get: invalid sha %q\n", sha)
 		return exitUsage
 	}
-	if *target == "" {
-		fmt.Fprintln(stderr, "depot get: --target is required")
-		return exitUsage
-	}
 
 	cfg := LoadConfig(getenv)
-	backend, err := resolveBackend(*target, getenv, cfg)
+	backend, err := resolveBackend(*out, *target, getenv, cfg)
 	if err != nil {
 		fmt.Fprintln(stderr, "depot get:", err)
 		return backendErrExit(err)

internal/depot/run_test.go

@@ -54,6 +54,91 @@ func TestRunUsage(t *testing.T) {
 	})
 }
 
+func TestOutFlag(t *testing.T) {
+	t.Run("status --out reads the given tree", func(t *testing.T) {
+		sha := shaOf("out-blob")
+		manifests := t.TempDir()
+		writeManifest(t, manifests, sha, 8)
+		depotDir := t.TempDir()
+		writeBlob(t, depotDir, sha, "out-blob")
+
+		var out, errb bytes.Buffer
+		code := Run([]string{"status", "--manifests", manifests, "--out", depotDir}, &out, &errb, testGetenv(nil))
+		if code != 0 {
+			t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
+		}
+		if !bytesContains(out.String(), "present=1") {
+			t.Fatalf("expected present=1, got %s", out.String())
+		}
+	})
+
+	t.Run("get --out fetches from the given tree", func(t *testing.T) {
+		sha := shaOf("get-blob")
+		depotDir := t.TempDir()
+		writeBlob(t, depotDir, sha, "get-blob")
+		dest := filepath.Join(t.TempDir(), "fetched")
+
+		var out, errb bytes.Buffer
+		code := Run([]string{"get", sha, dest, "--out", depotDir}, &out, &errb, testGetenv(nil))
+		if code != 0 {
+			t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
+		}
+		got, err := os.ReadFile(dest)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if string(got) != "get-blob" {
+			t.Fatalf("got %q", got)
+		}
+	})
+
+	t.Run("--out with any --target is rejected", func(t *testing.T) {
+		for _, target := range []string{"bunny", "cdn", "local"} {
+			var out, errb bytes.Buffer
+			code := Run([]string{"status", "--manifests", t.TempDir(), "--out", t.TempDir(), "--target", target}, &out, &errb, testGetenv(nil))
+			if code != 64 {
+				t.Fatalf("--target %s: expected 64, got %d (stderr=%s)", target, code, errb.String())
+			}
+		}
+	})
+
+	t.Run("get accepts flags before the positionals", func(t *testing.T) {
+		sha := shaOf("flags-first-blob")
+		depotDir := t.TempDir()
+		writeBlob(t, depotDir, sha, "flags-first-blob")
+		dest := filepath.Join(t.TempDir(), "fetched")
+
+		var out, errb bytes.Buffer
+		code := Run([]string{"get", "--out", depotDir, sha, dest}, &out, &errb, testGetenv(nil))
+		if code != 0 {
+			t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
+		}
+	})
+
+	t.Run("neither --out nor --target is rejected", func(t *testing.T) {
+		var out, errb bytes.Buffer
+		code := Run([]string{"status", "--manifests", t.TempDir()}, &out, &errb, testGetenv(nil))
+		if code != 64 {
+			t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
+		}
+	})
+
+	t.Run("--target local still works as a hidden alias", func(t *testing.T) {
+		sha := shaOf("alias-blob")
+		manifests := t.TempDir()
+		writeManifest(t, manifests, sha, 11)
+		depotDir := t.TempDir()
+		writeBlob(t, depotDir, sha, "alias-blob")
+
+		var out, errb bytes.Buffer
+		code := Run([]string{"status", "--manifests", manifests, "--target", "local"}, &out, &errb,
+			testGetenv(map[string]string{"DEPOT_DIR": depotDir}))
+		if code != 0 {
+			t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
+		}
+	})
+}
+
 func TestGetInvalidSHA(t *testing.T) {
 	var out, errb bytes.Buffer
 	dir := t.TempDir()
@@ -172,6 +257,17 @@ func bytesContains(s, substr string) bool {
 	return bytes.Contains([]byte(s), []byte(substr))
 }
 
+func writeBlob(t *testing.T, root, sha, content string) {
+	t.Helper()
+	path := filepath.Join(root, BlobKey(sha))
+	if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path, []byte(content), 0644); err != nil {
+		t.Fatal(err)
+	}
+}
+
 func writeManifest(t *testing.T, dir, sha string, size int64) {
 	t.Helper()
 	content := fmt.Sprintf("assets:\n  - path: foo\n    sha256: %s\n    size: %d\n", sha, size)

internal/dispatch/dispatch.go

@@ -21,6 +21,7 @@ import (
 	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
 	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/depot"
 	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/menu"
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/nwsync"
 )
 
 // Exit codes follow the sysexits(3) convention so CI can distinguish
@@ -94,16 +95,27 @@ var Registry = []Builder{
 	{
 		Name:    "depot",
 		Bin:     "crucible-depot",
-		Summary: "content-addressed asset depot (local/cdn/bunny)",
+		Summary: "content-addressed asset depot (disk/cdn/bunny)",
 		Commands: []Command{
-			{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] [--source DIR] --target bunny|cdn|local"},
+			{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] --target bunny|cdn | --out DIR"},
 			{Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"},
 			{Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"},
-			{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny|local"},
+			{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny | --out DIR"},
 			{Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"},
 		},
 		Wired: true,
 	},
+	{
+		Name:    "nwsync",
+		Bin:     "crucible-nwsync",
+		Summary: "publish NWSync blobs and manifests (emit/assemble/verify)",
+		Commands: []Command{
+			{Name: "emit", Summary: "explode one artifact into blobs plus its own NSYM manifest", Usage: "crucible nwsync emit <artifact> --out DIR"},
+			{Name: "assemble", Summary: "merge per-artifact NSYM manifests into one", Usage: "crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]"},
+			{Name: "verify", Summary: "read a published manifest's blobs back through the pull zone and hash them", Usage: "crucible nwsync verify <manifest-sha1> [--sample N]"},
+		},
+		Wired: true,
+	},
 	{
 		Name:    "assets",
 		Bin:     "crucible-assets",
@@ -241,7 +253,7 @@ var Registry = []Builder{
 					"2da-to-module [flags] <input.2da> [output.json]",
 					"json-to-2da <input.json> <output.2da>",
 				},
-				Aliases:    []CommandAlias{{Name: "convert-topdata"}},
+				Aliases: []CommandAlias{{Name: "convert-topdata"}},
 			},
 		},
 		Wired: true,
@@ -392,7 +404,7 @@ func runBuilder(name string, args []string, out, errw io.Writer) int {
 		}
 	}
 	if b.Wired {
-		// depot and assets are self-contained builders: they parse their own
+		// depot, assets and nwsync are self-contained builders: they parse their own
 		// subcommands and own their exit contract, bypassing the
 		// b.Commands/delegateLegacy legacy routing entirely.
 		switch b.Name {
@@ -400,6 +412,8 @@ func runBuilder(name string, args []string, out, errw io.Writer) int {
 			return depot.Run(args, out, errw, os.Getenv)
 		case "assets":
 			return assets.Run(args, out, errw, os.Getenv)
+		case "nwsync":
+			return nwsync.Run(args, out, errw)
 		}
 	}
 	if !b.Wired {

internal/dispatch/dispatch_test.go

@@ -148,6 +148,10 @@ func TestBuilderHelpIsOK(t *testing.T) {
 	}
 }
 
+// selfContained builders parse their own subcommands instead of delegating to
+// the legacy internal/app surface.
+var selfContained = map[string]bool{"depot": true, "assets": true, "nwsync": true}
+
 func TestCanonicalCommandSurface(t *testing.T) {
 	want := map[string][]string{
 		"depot":   {"status", "push", "verify", "get", "pull"},
@@ -156,6 +160,7 @@ func TestCanonicalCommandSurface(t *testing.T) {
 		"module":  {"build", "extract", "validate", "compare", "manifest"},
 		"topdata": {"validate", "build", "package", "compare", "convert"},
 		"wiki":    {"build", "deploy"},
+		"nwsync":  {"emit", "assemble", "verify"},
 	}
 	for _, builder := range Registry {
 		got := builder.subcommands()
@@ -173,10 +178,10 @@ func TestRegistryCommandNamesAndAliasesAreUnambiguous(t *testing.T) {
 	for _, builder := range Registry {
 		seen := map[string]bool{}
 		for _, command := range builder.Commands {
-			// depot and assets parse their own subcommands and bypass AppCommand
+			// depot, assets and nwsync parse their own subcommands and bypass
-			// routing entirely (see the self-contained-builder branch in
+			// AppCommand routing entirely (see the self-contained-builder branch
-			// runBuilder), so their Commands carry no AppCommand.
+			// in runBuilder), so their Commands carry no AppCommand.
-			requireAppCommand := builder.Name != "depot" && builder.Name != "assets"
+			requireAppCommand := !selfContained[builder.Name]
 			if command.Name == "" || command.Summary == "" || command.Usage == "" || (requireAppCommand && command.AppCommand == "") {
 				t.Errorf("%s has incomplete command metadata: %#v", builder.Name, command)
 			}

internal/erf/erf.go

@@ -67,6 +67,19 @@ type resourceEntry struct {
 	Size   uint32
 }
 
+// extensionTypes and typeExtensions must stay exact inverses of each other;
+// init() panics if they drift apart.
+//
+// Numbers below 0x0BB8 follow upstream neverwinter.nim
+// (neverwinter/restype.nim). The 0x0BB8 and up entries are lyt/vis/mdx: real
+// Aurora archive types that NWN1 ships in its own data/*.bif but upstream
+// happens not to register. xoreos corroborates those three numbers
+// (src/aurora/types.h).
+//
+// This table is NWN:EE only. Do not add NWN2 formats (mdb, gr2, wlk, xml):
+// NWN:EE either gives that number to something else (2070 is xbc here and mdb
+// in NWN2) or has no number for it at all, so packing one into a HAK writes a
+// resource the game misreads. See the reserved list in erf_test.go.
 var extensionTypes = map[string]uint16{
 	"res": 0x0000,
 	"bmp": 0x0001,
@@ -96,12 +109,13 @@ var extensionTypes = map[string]uint16{
 	"utt": 0x07F0,
 	"dds": 0x07F1,
 	"uts": 0x07F3,
+	"ltr": 0x07F4,
+	"gff": 0x07F5,
 	"fac": 0x07F6,
-	"gff": 0x07F7,
 	"ute": 0x07F8,
 	"utd": 0x07FA,
 	"utp": 0x07FC,
-	"dfa": 0x07FD,
+	"dft": 0x07FD,
 	"gic": 0x07FE,
 	"gui": 0x07FF,
 	"utm": 0x0803,
@@ -117,20 +131,15 @@ var extensionTypes = map[string]uint16{
 	"ndb": 0x0810,
 	"ptm": 0x0811,
 	"ptt": 0x0812,
-	"ltr": 0x0813,
 	"shd": 0x0815,
-	"mdb": 0x0816,
 	"mtr": 0x0818,
-	"jpg": 0x081C,
 	"lod": 0x081E,
 	"gif": 0x081F,
 	"png": 0x0820,
+	"jpg": 0x0821,
 	"lyt": 0x0BB8,
 	"vis": 0x0BB9,
 	"mdx": 0x0BC0,
-	"wlk": 0x0BCC,
-	"xml": 0x0BCD,
-	"gr2": 0x0FA3,
 }
 
 var typeExtensions = map[uint16]string{
@@ -162,12 +171,13 @@ var typeExtensions = map[uint16]string{
 	0x07F0: "utt",
 	0x07F1: "dds",
 	0x07F3: "uts",
+	0x07F4: "ltr",
+	0x07F5: "gff",
 	0x07F6: "fac",
-	0x07F7: "gff",
 	0x07F8: "ute",
 	0x07FA: "utd",
 	0x07FC: "utp",
-	0x07FD: "dfa",
+	0x07FD: "dft",
 	0x07FE: "gic",
 	0x07FF: "gui",
 	0x0803: "utm",
@@ -183,20 +193,15 @@ var typeExtensions = map[uint16]string{
 	0x0810: "ndb",
 	0x0811: "ptm",
 	0x0812: "ptt",
-	0x0813: "ltr",
 	0x0815: "shd",
-	0x0816: "mdb",
 	0x0818: "mtr",
-	0x081C: "jpg",
 	0x081E: "lod",
 	0x081F: "gif",
 	0x0820: "png",
+	0x0821: "jpg",
 	0x0BB8: "lyt",
 	0x0BB9: "vis",
 	0x0BC0: "mdx",
-	0x0BCC: "wlk",
-	0x0BCD: "xml",
-	0x0FA3: "gr2",
 }
 
 func init() {
@@ -205,8 +210,13 @@ func init() {
 		if !ok {
 			panic(fmt.Sprintf("missing canonical extension for resource type 0x%04X", resourceType))
 		}
-		if canonicalExt == ext {
+		if canonicalExt != ext {
-			continue
+			panic(fmt.Sprintf("resource type 0x%04X is %q in extensionTypes but %q in typeExtensions", resourceType, ext, canonicalExt))
+		}
+	}
+	for resourceType, ext := range typeExtensions {
+		if registered, ok := extensionTypes[ext]; !ok || registered != resourceType {
+			panic(fmt.Sprintf("extension %q is 0x%04X in typeExtensions but 0x%04X in extensionTypes (registered=%v)", ext, resourceType, registered, ok))
 		}
 	}
 }
@@ -301,63 +311,110 @@ func Write(w io.Writer, archive Archive) error {
 	return nil
 }
 
+// IndexEntry locates one resource inside an archive without holding its
+// payload. Streaming callers read one payload at a time from these, so peak
+// memory tracks the largest resource instead of the whole archive.
+type IndexEntry struct {
+	Name   string
+	Type   uint16
+	Offset int64
+	Size   int64
+}
+
+// Index is the header plus the resource table of an ERF: everything except the
+// payloads.
+type Index struct {
+	FileType string
+	Version  string
+	Entries  []IndexEntry
+}
+
+// ReadIndex parses the tables of an ERF of the given size, reading only the
+// header, the key list and the resource list.
+func ReadIndex(r io.ReaderAt, size int64) (Index, error) {
+	if size < headerSize {
+		return Index{}, fmt.Errorf("erf file too small: %d bytes", size)
+	}
+
+	var hdr header
+	if err := binary.Read(io.NewSectionReader(r, 0, headerSize), binary.LittleEndian, &hdr); err != nil {
+		return Index{}, fmt.Errorf("decode erf header: %w", err)
+	}
+
+	if int64(hdr.KeyListOffset)+int64(hdr.EntryCount)*24 > size {
+		return Index{}, fmt.Errorf("erf key list exceeds file bounds")
+	}
+	keys := make([]keyEntry, hdr.EntryCount)
+	keyReader := io.NewSectionReader(r, int64(hdr.KeyListOffset), int64(hdr.EntryCount)*24)
+	if err := binary.Read(keyReader, binary.LittleEndian, &keys); err != nil {
+		return Index{}, fmt.Errorf("decode key list: %w", err)
+	}
+
+	if int64(hdr.ResourceListOffset)+int64(hdr.EntryCount)*8 > size {
+		return Index{}, fmt.Errorf("erf resource list exceeds file bounds")
+	}
+	entries := make([]resourceEntry, hdr.EntryCount)
+	entryReader := io.NewSectionReader(r, int64(hdr.ResourceListOffset), int64(hdr.EntryCount)*8)
+	if err := binary.Read(entryReader, binary.LittleEndian, &entries); err != nil {
+		return Index{}, fmt.Errorf("decode resource list: %w", err)
+	}
+
+	index := Index{
+		FileType: string(hdr.FileType[:]),
+		Version:  string(hdr.Version[:]),
+		Entries:  make([]IndexEntry, 0, hdr.EntryCount),
+	}
+	for position, key := range keys {
+		entry := entries[position]
+		if int64(entry.Offset)+int64(entry.Size) > size {
+			return Index{}, fmt.Errorf("resource %d exceeds file bounds", position)
+		}
+		index.Entries = append(index.Entries, IndexEntry{
+			Name:   string(bytes.TrimRight(key.ResRef[:], "\x00")),
+			Type:   key.ResourceType,
+			Offset: int64(entry.Offset),
+			Size:   int64(entry.Size),
+		})
+	}
+	return index, nil
+}
+
+// ReadPayload returns one resource's bytes.
+func ReadPayload(r io.ReaderAt, entry IndexEntry) ([]byte, error) {
+	payload := make([]byte, entry.Size)
+	if _, err := r.ReadAt(payload, entry.Offset); err != nil {
+		return nil, fmt.Errorf("read resource %q: %w", entry.Name, err)
+	}
+	return payload, nil
+}
+
+// Read materialises a whole archive. Payloads are subslices of the buffer the
+// archive was read into, so nothing is copied twice: a caller must not mutate
+// Data. Callers that only need one resource at a time should use ReadIndex
+// instead, which never holds the archive at all.
 func Read(r io.Reader) (Archive, error) {
 	data, err := io.ReadAll(r)
 	if err != nil {
 		return Archive{}, fmt.Errorf("read erf: %w", err)
 	}
-	if len(data) < headerSize {
+	index, err := ReadIndex(bytes.NewReader(data), int64(len(data)))
-		return Archive{}, fmt.Errorf("erf file too small: %d bytes", len(data))
+	if err != nil {
+		return Archive{}, err
 	}
 
-	var hdr header
+	resources := make([]Resource, 0, len(index.Entries))
-	if err := binary.Read(bytes.NewReader(data[:headerSize]), binary.LittleEndian, &hdr); err != nil {
+	for _, entry := range index.Entries {
-		return Archive{}, fmt.Errorf("decode erf header: %w", err)
-	}
-
-	keyStart := int(hdr.KeyListOffset)
-	keyEnd := keyStart + int(hdr.EntryCount)*24
-	if keyEnd > len(data) {
-		return Archive{}, fmt.Errorf("erf key list exceeds file bounds")
-	}
-	keys := make([]keyEntry, hdr.EntryCount)
-	if err := binary.Read(bytes.NewReader(data[keyStart:keyEnd]), binary.LittleEndian, &keys); err != nil {
-		return Archive{}, fmt.Errorf("decode key list: %w", err)
-	}
-
-	resourceStart := int(hdr.ResourceListOffset)
-	resourceEnd := resourceStart + int(hdr.EntryCount)*8
-	if resourceEnd > len(data) {
-		return Archive{}, fmt.Errorf("erf resource list exceeds file bounds")
-	}
-	entries := make([]resourceEntry, hdr.EntryCount)
-	if err := binary.Read(bytes.NewReader(data[resourceStart:resourceEnd]), binary.LittleEndian, &entries); err != nil {
-		return Archive{}, fmt.Errorf("decode resource list: %w", err)
-	}
-
-	resources := make([]Resource, 0, hdr.EntryCount)
-	for index, key := range keys {
-		entry := entries[index]
-		start := int(entry.Offset)
-		end := start + int(entry.Size)
-		if end > len(data) {
-			return Archive{}, fmt.Errorf("resource %d exceeds file bounds", index)
-		}
-
-		resref := string(bytes.TrimRight(key.ResRef[:], "\x00"))
-		payload := make([]byte, entry.Size)
-		copy(payload, data[start:end])
 		resources = append(resources, Resource{
-			Name: resref,
+			Name: entry.Name,
-			Type: key.ResourceType,
+			Type: entry.Type,
-			Data: payload,
+			Data: data[entry.Offset : entry.Offset+entry.Size],
-			Size: int64(entry.Size),
+			Size: entry.Size,
 		})
 	}
 
 	return Archive{
-		FileType:  string(hdr.FileType[:]),
+		FileType:  index.FileType,
-		Version:   string(hdr.Version[:]),
+		Version:   index.Version,
 		Resources: resources,
 	}, nil
 }

internal/erf/erf_test.go

@@ -134,14 +134,10 @@ func TestExtensionMappingsSupportModernAssetTypes(t *testing.T) {
 		"mtr": 0x0818,
 		"shd": 0x0815,
 		"txi": 0x07E6,
-		"jpg": 0x081C,
+		"jpg": 0x0821,
-		"mdb": 0x0816,
 		"lyt": 0x0BB8,
 		"vis": 0x0BB9,
 		"mdx": 0x0BC0,
-		"xml": 0x0BCD,
-		"wlk": 0x0BCC,
-		"gr2": 0x0FA3,
 	}
 	for ext, wantType := range cases {
 		gotType, ok := ResourceTypeForExtension(ext)
@@ -191,3 +187,71 @@ func TestExtensionMappingsSupportAllUTBlueprintTypes(t *testing.T) {
 		}
 	}
 }
+
+// TestRestypeTableMatchesUpstream pins the restype numbers Crucible shares with
+// neverwinter.nim's restype.nim. The values below are upstream's; a mismatch
+// means a HAK we write is mislabelled for the game.
+func TestRestypeTableMatchesUpstream(t *testing.T) {
+	upstream := map[string]uint16{
+		"res": 0, "bmp": 1, "mve": 2, "tga": 3, "wav": 4, "plt": 6, "ini": 7,
+		"bmu": 8, "txt": 10, "mdl": 2002, "nss": 2009, "ncs": 2010, "are": 2012,
+		"set": 2013, "ifo": 2014, "bic": 2015, "wok": 2016, "2da": 2017,
+		"tlk": 2018, "txi": 2022, "git": 2023, "uti": 2025, "utc": 2027,
+		"dlg": 2029, "itp": 2030, "utt": 2032, "dds": 2033, "uts": 2035,
+		"ltr": 2036, "gff": 2037, "fac": 2038, "ute": 2040, "utd": 2042,
+		"utp": 2044, "dft": 2045, "gic": 2046, "gui": 2047, "utm": 2051,
+		"dwk": 2052, "pwk": 2053, "utg": 2055, "jrl": 2056, "utw": 2058,
+		"ssf": 2060, "hak": 2061, "nwm": 2062, "bik": 2063, "ndb": 2064,
+		"ptm": 2065, "ptt": 2066, "shd": 2069, "mtr": 2072, "lod": 2078,
+		"gif": 2079, "png": 2080, "jpg": 2081,
+	}
+	for ext, want := range upstream {
+		got, ok := ResourceTypeForExtension(ext)
+		if !ok {
+			t.Errorf("%s: not registered, upstream has %d", ext, want)
+			continue
+		}
+		if got != want {
+			t.Errorf("%s: registered as %d, upstream has %d", ext, got, want)
+		}
+	}
+	// Extensions upstream registers that Crucible must not reuse for anything else.
+	reserved := map[uint16]string{2039: "bte", 2067: "bak", 2070: "xbc", 2076: "tml"}
+	for number, upstreamExt := range reserved {
+		if ext, ok := ExtensionForResourceType(number); ok {
+			t.Errorf("%d: registered as %s, upstream reserves it for %s", number, ext, upstreamExt)
+		}
+	}
+}
+
+// TestNWN2FormatsAreNotRegistered keeps NWN2 file formats out of an NWN:EE
+// table. mdb and gr2 have Aurora numbers that mean something else (or nothing)
+// in NWN:EE; wlk and xml have no archive number in any Aurora game, so the
+// values Crucible used for them were invented. Packing any of these into a HAK
+// writes a resource the game misreads.
+func TestNWN2FormatsAreNotRegistered(t *testing.T) {
+	for _, ext := range []string{"mdb", "gr2", "wlk", "xml"} {
+		if number, ok := ResourceTypeForExtension(ext); ok {
+			t.Errorf("%s is an NWN2 format but is registered as 0x%04X", ext, number)
+		}
+	}
+}
+
+// TestRestypeTablesAreMutualInverses is the check init() is meant to enforce.
+func TestRestypeTablesAreMutualInverses(t *testing.T) {
+	for ext, number := range extensionTypes {
+		canonical, ok := typeExtensions[number]
+		if !ok {
+			t.Errorf("%s: number 0x%04X missing from typeExtensions", ext, number)
+			continue
+		}
+		if canonical != ext {
+			t.Errorf("%s: maps to 0x%04X, which maps back to %s", ext, number, canonical)
+		}
+	}
+	for number, ext := range typeExtensions {
+		if got, ok := extensionTypes[ext]; !ok || got != number {
+			t.Errorf("0x%04X: maps to %s, which maps back to 0x%04X (ok=%v)", number, ext, got, ok)
+		}
+	}
+}

internal/nwsync/assemble.go

@@ -0,0 +1,128 @@
+package nwsync
+
+import (
+	"crypto/sha1"
+	"encoding/json"
+	"fmt"
+	"path"
+)
+
+// AssembleOptions describes one merged manifest.
+type AssembleOptions struct {
+	// ArtifactKeys are the depot keys of the artifacts to merge, in
+	// Mod_HakList order — highest priority first. Each one's index is read
+	// from the key beside it.
+	ArtifactKeys []string
+	TLKKey       string // the TLK's key, if the manifest carries one
+	OutDir       string // write locally instead of uploading — the conformance path
+	GroupID      int    // 1 = current, 2 = testing; 0 means absent
+	ModuleName   string
+	Description  string
+	Sink         sink // test seam; nil means OutDir or the zone
+}
+
+// AssembleResult reports what one assemble run produced.
+type AssembleResult struct {
+	SHA1         string
+	ManifestPath string
+	Entries      int
+}
+
+// Assemble merges the per-artifact NSYM manifests named by Order into one
+// manifest. It reads no bulk data at all — only the small index files.
+//
+// Merge rule is resref shadowing, not concatenation: a resref present in more
+// than one artifact resolves to the earliest artifact in Order, which is how
+// the game resolves it (upstream's resman adds haks in reverse and lets the
+// last one win). Get this backwards and the wrong texture ships silently.
+func Assemble(options AssembleOptions) (AssembleResult, error) {
+	if len(options.ArtifactKeys) == 0 {
+		return AssembleResult{}, fmt.Errorf("assemble: no artifact keys given")
+	}
+
+	target, err := openSink(options.OutDir, options.Sink)
+	if err != nil {
+		return AssembleResult{}, err
+	}
+
+	// The TLK carries no precedence — it is not a hak and shadows nothing —
+	// so it merges last, after every hak has had its say.
+	keys := append([]string{}, options.ArtifactKeys...)
+	if options.TLKKey != "" {
+		keys = append(keys, options.TLKKey)
+	}
+
+	merged := make([]Entry, 0, 1024)
+	winner := make(map[Identity]bool, 1024)
+	var onDiskBytes int64
+
+	for _, artifactKey := range keys {
+		key, err := resolveIndexKey(artifactKey, options.OutDir)
+		if err != nil {
+			return AssembleResult{}, err
+		}
+		data, sidecarBody, err := target.getIndex(key)
+		if err != nil {
+			// An artifact with no index is an artifact whose emit never
+			// finished. Publishing a manifest without it would ship a release
+			// missing a hak, so this fails closed.
+			return AssembleResult{}, fmt.Errorf("assemble: no index for %s: %w", artifactKey, err)
+		}
+		entries, err := readManifest(data)
+		if err != nil {
+			return AssembleResult{}, fmt.Errorf("%s: %w", target.describe(key), err)
+		}
+		sidecar, err := parseSidecar(target.describe(key), sidecarBody)
+		if err != nil {
+			return AssembleResult{}, err
+		}
+		// Two producers of blobs means a skewed emitter can write blobs the
+		// merged manifest quietly disagrees with. Refuse to merge across
+		// mismatched emitter versions.
+		if sidecar.EmitterVersion != emitterVersion {
+			return AssembleResult{}, fmt.Errorf(
+				"assemble: emitter version mismatch: %s was emitted by emitter %q, this is emitter %q",
+				artifactKey, sidecar.EmitterVersion, emitterVersion)
+		}
+		// on_disk_bytes overcounts by the handful of cross-artifact
+		// duplicates. It is a display statistic; no dedupe pass for it.
+		onDiskBytes += sidecar.OnDiskBytes
+
+		for _, entry := range entries {
+			identity := entry.identity()
+			if winner[identity] {
+				continue
+			}
+			winner[identity] = true
+			merged = append(merged, entry)
+		}
+	}
+
+	if len(merged) == 0 {
+		return AssembleResult{}, fmt.Errorf("assemble: merged manifest is empty")
+	}
+
+	data, err := writeManifest(merged)
+	if err != nil {
+		return AssembleResult{}, err
+	}
+	sha1Hex := fmt.Sprintf("%x", sha1.Sum(data))
+	manifestKey := path.Join("manifests", sha1Hex)
+	sidecar := Sidecar{
+		ModuleName:  options.ModuleName,
+		Description: options.Description,
+		GroupID:     options.GroupID,
+	}
+	if err := putManifestPair(target, manifestKey, data, merged, onDiskBytes, sidecar); err != nil {
+		return AssembleResult{}, err
+	}
+	return AssembleResult{SHA1: sha1Hex, ManifestPath: target.describe(manifestKey), Entries: len(merged)}, nil
+}
+
+func parseSidecar(where string, data []byte) (Sidecar, error) {
+	var sidecar Sidecar
+	if err := json.Unmarshal(data, &sidecar); err != nil {
+		return Sidecar{}, fmt.Errorf("%s: %w", where, err)
+	}
+	return sidecar, nil
+}

internal/nwsync/compressedbuf.go

@@ -0,0 +1,179 @@
+package nwsync
+
+import (
+	"bytes"
+	"encoding/binary"
+	"encoding/hex"
+	"fmt"
+
+	"github.com/klauspost/compress/zstd"
+)
+
+// NWCompressedBuffer framing, as upstream's neverwinter/compressedbuf.nim
+// writes it for NWSync blobs. All fields are little-endian uint32:
+//
+//	magic "NSYC", version 3, algorithm 2 (zstd), uncompressed size,
+//	zstd header version 1, dictionary 0, then the raw zstd frame.
+const (
+	blobMagic       = 0x4359534E // "NSYC" little-endian
+	blobVersion     = 3
+	algorithmZstd   = 2
+	zstdHeaderVer   = 1
+	zstdDictionary  = 0
+	blobHeaderBytes = 24
+)
+
+// EncodeAll/DecodeAll are single-threaded per call, so the default pool of one
+// encoder per CPU only buys idle memory: each holds a window-sized history, so
+// on a 24-core runner that is ~200 MB of live heap doing nothing. Concurrency 1
+// produces byte-identical output.
+var (
+	blobEncoder, _ = zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
+	blobDecoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(1))
+)
+
+// zstd frame header bits we care about. A frame starts with the magic, then a
+// one-byte Frame_Header_Descriptor: bits 7-6 size the Frame_Content_Size field,
+// bit 5 is Single_Segment_flag, bits 1-0 size the Dictionary_ID field.
+const (
+	zstdFrameMagic      = "\x28\xb5\x2f\xfd"
+	frameSingleSegment  = 1 << 5
+	frameDictionaryMask = 0x03
+	// oneByteContentSizeCeiling is the size above which a Frame_Content_Size no
+	// longer fits in one byte. Below it the field's size flag is 0, which is
+	// what lets klauspost/compress leave the field out entirely.
+	oneByteContentSizeCeiling = 256
+)
+
+// compressBlob wraps data in NWCompressedBuffer framing.
+func compressBlob(data []byte) []byte {
+	var out bytes.Buffer
+	header := []uint32{blobMagic, blobVersion, algorithmZstd, uint32(len(data)), zstdHeaderVer, zstdDictionary}
+	for _, field := range header {
+		_ = binary.Write(&out, binary.LittleEndian, field)
+	}
+	frame := declareFrameContentSize(blobEncoder.EncodeAll(data, nil), len(data))
+	// Fail closed rather than publish a blob no client can decode. An encoder
+	// upgrade that finds a new way to omit the field would otherwise reproduce
+	// #86 in silence, and a blob is skipped by every later emit once written.
+	if !frameDeclaresContentSize(frame) {
+		panic(fmt.Sprintf("nwsync: refusing to emit a %d-byte blob whose zstd frame declares no content size (descriptor %#x)",
+			len(data), frame[4]))
+	}
+	out.Write(frame)
+	return out.Bytes()
+}
+
+// inspectBlob unwraps a stored blob the way the game client reads it, and is the
+// only reader that should be trusted to judge a published blob.
+//
+// It asserts the frame property on top of the round trip. Go's decoder — like
+// the zstd CLI — streams a frame that declares no content size, so a check that
+// only decompresses and hashes is a *more* capable decoder than the client's: it
+// certifies exactly the blobs the client rejects, which is how #86 reached
+// production and survived an audit.
+func inspectBlob(blob []byte) ([]byte, error) {
+	data, err := decompressBlob(blob)
+	if err != nil {
+		return nil, fmt.Errorf("malformed framing: %w", err)
+	}
+	if len(blob) > blobHeaderBytes && !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
+		return nil, fmt.Errorf("malformed framing: the zstd frame declares no content size, which the game client cannot decode")
+	}
+	return data, nil
+}
+
+// blobMatchesName holds a stored blob to its own file name: a blob is named
+// after the sha1 of its uncompressed bytes, so the name is a complete statement
+// about the contents and nothing else is needed to check it.
+func blobMatchesName(blob []byte, sha1Hex string) error {
+	data, err := inspectBlob(blob)
+	if err != nil {
+		return err
+	}
+	if got := hex.EncodeToString(sha1Sum(data)); got != sha1Hex {
+		return fmt.Errorf("blob %s holds the contents of %s", sha1Hex, got)
+	}
+	return nil
+}
+
+// frameDeclaresContentSize reports whether a zstd frame states how many bytes it
+// decompresses to. A frame with a zero-sized Frame_Content_Size field declares
+// one only when Single_Segment_flag is set; otherwise the size is unknown.
+func frameDeclaresContentSize(frame []byte) bool {
+	if len(frame) < 5 || string(frame[:4]) != zstdFrameMagic {
+		return false
+	}
+	descriptor := frame[4]
+	return descriptor>>6 != 0 || descriptor&frameSingleSegment != 0
+}
+
+// declareFrameContentSize rewrites a frame that does not declare its
+// Frame_Content_Size so that it does, and returns any other frame unchanged.
+//
+// klauspost/compress omits the field for inputs under 256 bytes, which the spec
+// permits. Reference libzstd never does, so the NWN client — which sizes its
+// output buffer from ZSTD_getFrameContentSize and has therefore never met a
+// frame without one — rejects the blob outright with an empty "potential
+// compression error" (#86). No encoder option changes this, so the frame is
+// re-headered here.
+//
+// The result is the shape libzstd itself emits for the same input: setting
+// Single_Segment_flag drops the Window_Descriptor byte, and the freed byte pays
+// for a one-byte Frame_Content_Size. Window_Size then equals the content size,
+// which is sound because the content is under 256 bytes and every match in it
+// therefore falls inside that window. Same length in, same length out.
+func declareFrameContentSize(frame []byte, size int) []byte {
+	if size <= 0 || size >= oneByteContentSizeCeiling || len(frame) < 6 || string(frame[:4]) != zstdFrameMagic {
+		return frame
+	}
+	descriptor := frame[4]
+	// Rewrite only the exact shape a small input produces: no declared size, no
+	// single segment, no dictionary. Anything else either declares a size
+	// already or is not a frame this reinterpretation is safe on.
+	if descriptor>>6 != 0 || descriptor&frameSingleSegment != 0 || descriptor&frameDictionaryMask != 0 {
+		return frame
+	}
+	reframed := make([]byte, len(frame))
+	copy(reframed, frame)
+	reframed[4] = descriptor | frameSingleSegment
+	reframed[5] = byte(size) // replaces Window_Descriptor
+	return reframed
+}
+
+// decompressBlob unwraps NWCompressedBuffer framing. It exists so a blob this
+// package wrote — or one upstream wrote — can be compared by its uncompressed
+// bytes, which is the only comparison that is meaningful across zstd
+// implementations.
+func decompressBlob(blob []byte) ([]byte, error) {
+	if len(blob) < blobHeaderBytes {
+		return nil, fmt.Errorf("blob too small: %d bytes", len(blob))
+	}
+	header := make([]uint32, 6)
+	if err := binary.Read(bytes.NewReader(blob[:blobHeaderBytes]), binary.LittleEndian, header); err != nil {
+		return nil, fmt.Errorf("decode blob header: %w", err)
+	}
+	switch {
+	case header[0] != blobMagic:
+		return nil, fmt.Errorf("invalid blob magic: %#x", header[0])
+	case header[1] != blobVersion:
+		return nil, fmt.Errorf("unsupported blob version: %d", header[1])
+	case header[2] != algorithmZstd:
+		return nil, fmt.Errorf("unsupported compression algorithm: %d", header[2])
+	case header[4] != zstdHeaderVer:
+		return nil, fmt.Errorf("unsupported zstd header version: %d", header[4])
+	case header[5] != zstdDictionary:
+		return nil, fmt.Errorf("zstd dictionaries are not supported")
+	}
+	if header[3] == 0 {
+		return nil, nil
+	}
+	data, err := blobDecoder.DecodeAll(blob[blobHeaderBytes:], nil)
+	if err != nil {
+		return nil, fmt.Errorf("decompress blob: %w", err)
+	}
+	if uint32(len(data)) != header[3] {
+		return nil, fmt.Errorf("blob size mismatch: header says %d, got %d", header[3], len(data))
+	}
+	return data, nil
+}

internal/nwsync/emit.go

@@ -0,0 +1,365 @@
+package nwsync
+
+import (
+	"context"
+	"crypto/sha1"
+	"fmt"
+	"io"
+	"os"
+	"path"
+	"path/filepath"
+	"slices"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+)
+
+// fileSizeLimit matches upstream's --limit-file-size default of 15 MB. A
+// resource over it is a hard failure, not a skip: upstream quit(1)s and so do
+// we. Our largest resource today is 13.66 MiB, so the headroom is thin.
+const fileSizeLimit = 15 * 1024 * 1024
+
+// skippedTypes are never published, matching upstream's GobalResTypeSkipList.
+var skippedTypes = resTypes("nss", "ndb", "gic")
+
+// emitterVersion identifies the blob/manifest byte format this package
+// produces. assemble refuses to merge indexes that disagree on it, because two
+// producers of blobs mean a skewed emitter can otherwise write blobs the merged
+// manifest quietly disagrees with. Bump it only when emitted bytes change — it
+// is deliberately not the build revision, which would invalidate every
+// published index on every unrelated commit.
+// Version 2 declares Frame_Content_Size on every blob (#86); version 1 omitted
+// it below 256 bytes and no client could sync past such a blob.
+const emitterVersion = "2"
+
+// serverTypes are loaded only server-side; a manifest holding nothing else
+// has no client contents. Mirrors upstream's GlobalResTypeServerList, whose
+// trailing 0 is RESTYPE_INVALID.
+var serverTypes = append(resTypes(
+	"are", "dlg", "fac", "gic", "git", "ifo", "itp", "jrl", "ncs", "ndb",
+	"nss", "ptm", "utc", "utd", "ute", "uti", "utm", "utp", "uts", "utt", "utw",
+), 0)
+
+func resTypes(extensions ...string) []uint16 {
+	types := make([]uint16, 0, len(extensions))
+	for _, extension := range extensions {
+		restype, ok := erf.ResourceTypeForExtension(extension)
+		if !ok {
+			panic("nwsync: unknown restype " + extension)
+		}
+		types = append(types, restype)
+	}
+	return types
+}
+
+// EmitResult reports what one emit run produced.
+type EmitResult struct {
+	Name         string // artifact name, without extension
+	ManifestPath string
+	Entries      int
+	BlobsWritten int
+}
+
+// defaultEmitJobs is how many resources are hashed, compressed and stored at
+// once. Emit is latency-bound, not CPU-bound: a blob costs a probe round-trip
+// plus an upload round-trip, and a measured backfill spent 26 s of CPU across
+// 9.5 minutes of wall clock. The figure matches depot's DEPOT_JOBS default and
+// the transport's MaxIdleConnsPerHost, so a worker per connection needs no new
+// TLS handshake.
+const defaultEmitJobs = 16
+
+// EmitOptions describes one emit run.
+type EmitOptions struct {
+	ArtifactKey  string // depot key of the artifact; the NSYM key is derived from it
+	ArtifactPath string // the file on disk
+	As           string // name override, for a TLK whose filename is not its published name
+	OutDir       string // write locally instead of uploading — the conformance path
+	Jobs         int    // resources in flight at once; 0 means defaultEmitJobs
+	Verify       bool   // hash what would be skipped instead of trusting presence
+	Sink         sink   // test seam; nil means OutDir or the zone
+}
+
+// Emit explodes one artifact — a .hak/.erf or a loose file such as the TLK —
+// into NWSync blobs plus a NSYM manifest describing only that artifact.
+//
+// Blobs go up as they are produced and the index lands last, so the presence of
+// an index is the publication marker: an artifact whose emit died halfway has
+// real blobs in the zone and no index, which is unambiguous. Blob names are
+// content hashes, so re-running skips whatever already landed.
+func Emit(options EmitOptions) (EmitResult, error) {
+	artifact, err := os.Open(options.ArtifactPath)
+	if err != nil {
+		return EmitResult{}, fmt.Errorf("read artifact: %w", err)
+	}
+	defer artifact.Close()
+	info, err := artifact.Stat()
+	if err != nil {
+		return EmitResult{}, fmt.Errorf("read artifact: %w", err)
+	}
+	// A section reader, not the file itself: hashing must not move the file
+	// offset out from under everything that reads the artifact afterwards.
+	if err := checkArtifactKey(options.ArtifactKey, io.NewSectionReader(artifact, 0, info.Size())); err != nil {
+		return EmitResult{}, err
+	}
+	name := options.As
+	if name == "" {
+		name = path.Base(options.ArtifactKey)
+	}
+	extension := path.Ext(name)
+	name = strings.TrimSuffix(name, extension)
+
+	key, err := resolveIndexKey(options.ArtifactKey, options.OutDir)
+	if err != nil {
+		return EmitResult{}, err
+	}
+
+	index, err := readArtifactIndex(options.ArtifactPath, artifact, info.Size(), name)
+	if err != nil {
+		return EmitResult{}, err
+	}
+
+	target, err := openSink(options.OutDir, options.Sink)
+	if err != nil {
+		return EmitResult{}, err
+	}
+
+	jobs := options.Jobs
+	if jobs < 1 {
+		jobs = defaultEmitJobs
+	}
+	entries, blobs, onDiskBytes, err := emitResources(artifact, index, target, jobs, options.Verify)
+	if err != nil {
+		return EmitResult{}, err
+	}
+	if len(entries) == 0 {
+		return EmitResult{}, fmt.Errorf("%s: nothing to index (no publishable resources)", options.ArtifactPath)
+	}
+
+	data, err := writeManifest(entries)
+	if err != nil {
+		return EmitResult{}, err
+	}
+	if err := putManifestPair(target, key, data, entries, onDiskBytes, Sidecar{ModuleName: name}); err != nil {
+		return EmitResult{}, err
+	}
+	return EmitResult{Name: name, ManifestPath: target.describe(key), Entries: len(entries), BlobsWritten: blobs}, nil
+}
+
+// openSink returns the zone sink, or a local directory when outDir is set.
+func openSink(outDir string, injected sink) (sink, error) {
+	if injected != nil {
+		return injected, nil
+	}
+	if outDir != "" {
+		return dirSink{root: outDir}, nil
+	}
+	return newZoneSink(context.Background(), os.Getenv)
+}
+
+// readArtifactIndex locates the resources of an ERF/HAK/MOD, or the single
+// resource a loose file represents, without reading any payload. Upstream's
+// resman does the same dispatch on the file's first three bytes. name is the
+// artifact's published name, which for a loose file is also its resref.
+func readArtifactIndex(path string, artifact io.ReaderAt, size int64, name string) ([]erf.IndexEntry, error) {
+	magic := make([]byte, 3)
+	if size >= 3 {
+		if _, err := artifact.ReadAt(magic, 0); err != nil {
+			return nil, fmt.Errorf("%s: %w", path, err)
+		}
+	}
+	switch string(magic) {
+	case "ERF", "HAK":
+		index, err := erf.ReadIndex(artifact, size)
+		if err != nil {
+			return nil, fmt.Errorf("%s: %w", path, err)
+		}
+		return index.Entries, nil
+	case "MOD":
+		// A persistent world never publishes module contents, so the .mod
+		// contributes no bytes to a manifest — it only says which haks and
+		// which TLK the manifest covers.
+		return nil, fmt.Errorf("%s: a module is never emitted; a manifest is haks plus the TLK", path)
+	}
+	extension := filepath.Ext(filepath.Base(path))
+	restype, ok := erf.ResourceTypeForExtension(extension)
+	if !ok {
+		return nil, fmt.Errorf("%s: unknown resource type %q", path, extension)
+	}
+	return []erf.IndexEntry{{Name: name, Type: restype, Offset: 0, Size: size}}, nil
+}
+
+// emitResources hashes, compresses and stores resources, reading each payload
+// from the artifact only when its turn comes. Peak memory tracks the resources
+// in flight, not the archive: a 2 GB hak must emit inside a runner's few spare
+// GB. jobs of them are in flight at once, so the ceiling is jobs multiplied by
+// fileSizeLimit and its compressed copy — bounded, and bounded by a constant
+// this package enforces itself.
+//
+// The returned entries are in artifact order whatever order the workers finish
+// in, because a manifest's bytes are promised deterministic by emitterVersion.
+func emitResources(artifact io.ReaderAt, index []erf.IndexEntry, target sink, jobs int, verify bool) ([]Entry, int, int64, error) {
+	// A resref appearing twice inside one artifact resolves to the last one,
+	// the way resman lets the last container added win.
+	order := make([]Identity, 0, len(index))
+	latest := make(map[Identity]erf.IndexEntry, len(index))
+	var tooBig []string
+	for _, entry := range index {
+		if _, ok := erf.ExtensionForResourceType(entry.Type); !ok {
+			return nil, 0, 0, fmt.Errorf("resref %s is not resolvable (unknown restype %d)", entry.Name, entry.Type)
+		}
+		if slices.Contains(skippedTypes, entry.Type) {
+			continue
+		}
+		if entry.Size > fileSizeLimit {
+			tooBig = append(tooBig, fmt.Sprintf("%s: %d bytes > %d", entry.Name, entry.Size, fileSizeLimit))
+			continue
+		}
+		identity := Identity{ResRef: strings.ToLower(entry.Name), ResType: entry.Type}
+		if _, seen := latest[identity]; !seen {
+			order = append(order, identity)
+		}
+		latest[identity] = entry
+	}
+	if len(tooBig) > 0 {
+		sort.Strings(tooBig)
+		return nil, 0, 0, fmt.Errorf("resources exceed the file size limit:\n  %s", strings.Join(tooBig, "\n  "))
+	}
+
+	// Index-addressed, never appended to: a worker owns entries[i] alone, so
+	// the slice comes back in artifact order and needs no lock.
+	entries := make([]Entry, len(order))
+	var blobs int
+	var onDiskBytes int64
+	var mu sync.Mutex
+	var firstErr error
+	// Two resrefs in one artifact can hold identical bytes, and therefore one
+	// blob. Serially the sink's existence check absorbed that; in parallel both
+	// workers would probe, both miss, and both upload. Claiming the sha1 here
+	// restores the dedupe and skips the probe round-trip as well.
+	claimed := make(map[[20]byte]bool, len(order))
+
+	failed := func() bool {
+		mu.Lock()
+		defer mu.Unlock()
+		return firstErr != nil
+	}
+
+	store := func(i int) {
+		identity := order[i]
+		payload, err := erf.ReadPayload(artifact, latest[identity])
+		if err != nil {
+			mu.Lock()
+			if firstErr == nil {
+				firstErr = err
+			}
+			mu.Unlock()
+			return
+		}
+		sum := sha1.Sum(payload)
+		entries[i] = Entry{
+			SHA1:    sum,
+			Size:    uint32(len(payload)),
+			ResRef:  identity.ResRef,
+			ResType: identity.ResType,
+		}
+		mu.Lock()
+		duplicate := claimed[sum]
+		claimed[sum] = true
+		mu.Unlock()
+		if duplicate {
+			return
+		}
+		written, err := target.putBlob(fmt.Sprintf("%x", sum), verify, func() []byte { return compressBlob(payload) })
+		mu.Lock()
+		defer mu.Unlock()
+		if err != nil {
+			if firstErr == nil {
+				firstErr = err
+			}
+			return
+		}
+		if written > 0 {
+			blobs++
+			onDiskBytes += written
+		}
+	}
+
+	if jobs < 1 {
+		jobs = 1
+	}
+	work := make(chan int)
+	var wg sync.WaitGroup
+	for range jobs {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			for i := range work {
+				// After a failure the run is over — the caller discards
+				// everything and no index is written. Draining the rest of the
+				// channel cheaply, rather than returning, keeps the feeder from
+				// blocking on workers that have gone away.
+				if failed() {
+					continue
+				}
+				store(i)
+			}
+		}()
+	}
+	for i := range order {
+		work <- i
+	}
+	close(work)
+	wg.Wait()
+
+	if firstErr != nil {
+		return nil, 0, 0, firstErr
+	}
+	return entries, blobs, onDiskBytes, nil
+}
+
+// created is the sidecar timestamp. SOURCE_DATE_EPOCH pins it so a build can
+// be reproduced byte for byte; the manifest itself is deterministic already.
+func created() int64 {
+	if raw := os.Getenv("SOURCE_DATE_EPOCH"); raw != "" {
+		if seconds, err := strconv.ParseInt(raw, 10, 64); err == nil {
+			return seconds
+		}
+	}
+	return time.Now().Unix()
+}
+
+// putManifestPair stores a NSYM manifest and its .json sidecar at key. The
+// caller supplies the sidecar fields it knows; the rest are derived from the
+// entries. data must be the serialised form of entries.
+func putManifestPair(target sink, key string, data []byte, entries []Entry, onDiskBytes int64, sidecar Sidecar) error {
+	var totalBytes int64
+	clientContents := false
+	for _, entry := range entries {
+		totalBytes += int64(entry.Size)
+		if !slices.Contains(serverTypes, entry.ResType) {
+			clientContents = true
+		}
+	}
+	sidecar.Version = manifestVersion
+	sidecar.SHA1 = fmt.Sprintf("%x", sha1.Sum(data))
+	sidecar.HashTreeDepth = hashTreeDepth
+	sidecar.IncludesModuleContents = false
+	sidecar.IncludesClientContents = clientContents
+	sidecar.TotalFiles = len(entries)
+	sidecar.TotalBytes = totalBytes
+	sidecar.OnDiskBytes = onDiskBytes
+	sidecar.Created = created()
+	sidecar.CreatedWith = buildinfo.String()
+	sidecar.EmitterVersion = emitterVersion
+
+	body, err := marshalSidecar(sidecar)
+	if err != nil {
+		return err
+	}
+	return target.putIndex(key, data, body)
+}

internal/nwsync/jobs_test.go

@@ -0,0 +1,134 @@
+package nwsync
+
+import (
+	"bytes"
+	"fmt"
+	"path/filepath"
+	"runtime/debug"
+	"strings"
+	"testing"
+)
+
+// manyResources is a hak body with enough distinct resources that a worker pool
+// actually interleaves. Payloads differ so nothing is deduplicated away.
+func manyResources(count int) map[string][]byte {
+	contents := make(map[string][]byte, count)
+	for i := range count {
+		contents[fmt.Sprintf("res%05d.tga", i)] = []byte(fmt.Sprintf("payload %d", i))
+	}
+	return contents
+}
+
+// TestEmitProducesTheSameIndexAtEveryJobCount is the contract that lets emit be
+// parallel at all: emitterVersion promises a manifest's bytes are a function of
+// its artifact, so the number of workers must not be observable in the output.
+func TestEmitProducesTheSameIndexAtEveryJobCount(t *testing.T) {
+	// The sidecar stamps a wall-clock time unless this is set, which would make
+	// two runs differ for a reason that has nothing to do with job count.
+	t.Setenv("SOURCE_DATE_EPOCH", "1700000000")
+
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, manyResources(64))
+	key := artifactKey(t, hak)
+
+	emit := func(jobs int) (manifest, sidecar []byte, result EmitResult) {
+		out := filepath.Join(t.TempDir(), "out")
+		result, err := Emit(EmitOptions{
+			ArtifactKey:  key,
+			ArtifactPath: hak,
+			OutDir:       out,
+			Jobs:         jobs,
+		})
+		if err != nil {
+			t.Fatalf("emit at -jobs %d: %v", jobs, err)
+		}
+		manifest, sidecar, err = dirSink{root: out}.getIndex(filepath.Base(result.ManifestPath))
+		if err != nil {
+			t.Fatalf("read index at -jobs %d: %v", jobs, err)
+		}
+		return manifest, sidecar, result
+	}
+
+	serialManifest, serialSidecar, serial := emit(1)
+	parallelManifest, parallelSidecar, parallel := emit(16)
+
+	if !bytes.Equal(serialManifest, parallelManifest) {
+		t.Errorf("manifest bytes differ between -jobs 1 and -jobs 16")
+	}
+	if !bytes.Equal(serialSidecar, parallelSidecar) {
+		t.Errorf("sidecar bytes differ between -jobs 1 and -jobs 16:\n %s\n %s", serialSidecar, parallelSidecar)
+	}
+	if serial.Entries != parallel.Entries || serial.BlobsWritten != parallel.BlobsWritten {
+		t.Errorf("-jobs 1 wrote %d entries/%d blobs, -jobs 16 wrote %d/%d",
+			serial.Entries, serial.BlobsWritten, parallel.Entries, parallel.BlobsWritten)
+	}
+}
+
+// TestEmitLeavesNoIndexWhenAParallelUploadFails is the fail-closed check with
+// workers in flight: several uploads are in the air when the first one fails,
+// and the index must still never appear. Run under -race this also covers the
+// shared counters.
+func TestEmitLeavesNoIndexWhenAParallelUploadFails(t *testing.T) {
+	fixture := newZoneFixture(t)
+	fixture.zone.failOn = func(key string) bool { return strings.HasPrefix(key, "data/sha1/") }
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, manyResources(64))
+
+	if _, err := Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, hak),
+		ArtifactPath: hak,
+		Sink:         fixture.sink,
+		Jobs:         16,
+	}); err == nil {
+		t.Fatal("emit reported success after an upload failed")
+	}
+	fixture.zone.mu.Lock()
+	defer fixture.zone.mu.Unlock()
+	for key := range fixture.zone.objects {
+		if strings.HasSuffix(key, ".nsym") {
+			t.Errorf("a half-emitted artifact published an index: %s", key)
+		}
+	}
+}
+
+// TestEmitPeakMemoryIsBoundedByJobCount pins the ceiling the parallel emit
+// rests on. Peak still must not track the archive — it tracks the resources in
+// flight, so a bigger hak at the same job count costs the same.
+func TestEmitPeakMemoryIsBoundedByJobCount(t *testing.T) {
+	if testing.Short() {
+		t.Skip("writes a 64 MB fixture")
+	}
+	defer debug.SetGCPercent(debug.SetGCPercent(10))
+
+	measure := func(count, jobs int) uint64 {
+		dir := t.TempDir()
+		hak := filepath.Join(dir, "big.hak")
+		writeStreamedHak(t, hak, count)
+		options := EmitOptions{
+			ArtifactKey:  artifactKey(t, hak),
+			ArtifactPath: hak,
+			As:           filepath.Base(hak),
+			OutDir:       filepath.Join(dir, "out"),
+			Jobs:         jobs,
+		}
+		return peakHeapDuring(func() {
+			if _, err := Emit(options); err != nil {
+				t.Fatalf("emit %d resources at -jobs %d: %v", count, jobs, err)
+			}
+		})
+	}
+
+	const jobs = 8
+	small := measure(8, jobs)  // 8 MB
+	large := measure(64, jobs) // 64 MB
+	// Each worker may hold one resourceSize payload plus its compressed copy,
+	// so the pool itself is the slack — not the archive.
+	const slack = 24 << 20
+
+	t.Logf("peak heap at -jobs %d: 8 MB hak %d bytes, 64 MB hak %d bytes", jobs, small, large)
+	if large > small+slack {
+		t.Fatalf("peak heap scaled with artifact size at -jobs %d: 8 MB hak peaked at %d bytes, 64 MB hak at %d", jobs, small, large)
+	}
+}

internal/nwsync/manifest.go

@@ -0,0 +1,208 @@
+package nwsync
+
+import (
+	"bytes"
+	"encoding/binary"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"io"
+	"path"
+	"path/filepath"
+	"sort"
+	"strings"
+)
+
+// NSYM manifest, version 3, exactly as upstream neverwinter/nwsync.nim writes
+// it. Everything is little-endian:
+//
+//	"NSYM", uint32 version, uint32 entry count, uint32 mapping count,
+//	entries:  byte[20] raw sha1, uint32 size, char[16] resref, uint16 restype
+//	mappings: uint32 entry index,             char[16] resref, uint16 restype
+//
+// Entries are sorted by lowercase sha1 hex then resref, and a resource whose
+// sha1 was already written becomes a mapping instead of a second entry.
+const (
+	manifestVersion = 3
+	hashTreeDepth   = 2
+	resRefBytes     = 16
+)
+
+// Entry is one resource in a manifest.
+type Entry struct {
+	SHA1    [20]byte
+	Size    uint32
+	ResRef  string // lowercase, no extension, at most 16 characters
+	ResType uint16
+}
+
+func (e Entry) sha1Hex() string { return hex.EncodeToString(e.SHA1[:]) }
+
+// Identity is what a resref resolves by: name plus type. It is the merge key
+// inside one artifact and across artifacts alike.
+type Identity struct {
+	ResRef  string
+	ResType uint16
+}
+
+func (e Entry) identity() Identity { return Identity{ResRef: e.ResRef, ResType: e.ResType} }
+
+// writeManifest serialises entries into NSYM v3 bytes.
+func writeManifest(entries []Entry) ([]byte, error) {
+	sorted := make([]Entry, len(entries))
+	copy(sorted, entries)
+	sort.SliceStable(sorted, func(i, j int) bool {
+		left, right := sorted[i].sha1Hex(), sorted[j].sha1Hex()
+		if left != right {
+			return left < right
+		}
+		return sorted[i].ResRef < sorted[j].ResRef
+	})
+
+	var body, mappings bytes.Buffer
+	seen := make(map[string]uint32, len(sorted))
+	var entryCount, mappingCount uint32
+	for _, entry := range sorted {
+		padded, err := padResRef(entry.ResRef)
+		if err != nil {
+			return nil, err
+		}
+		if index, ok := seen[entry.sha1Hex()]; ok {
+			_ = binary.Write(&mappings, binary.LittleEndian, index)
+			mappings.Write(padded)
+			_ = binary.Write(&mappings, binary.LittleEndian, entry.ResType)
+			mappingCount++
+			continue
+		}
+		seen[entry.sha1Hex()] = entryCount
+		entryCount++
+		body.Write(entry.SHA1[:])
+		_ = binary.Write(&body, binary.LittleEndian, entry.Size)
+		body.Write(padded)
+		_ = binary.Write(&body, binary.LittleEndian, entry.ResType)
+	}
+
+	var out bytes.Buffer
+	out.WriteString("NSYM")
+	for _, field := range []uint32{manifestVersion, entryCount, mappingCount} {
+		_ = binary.Write(&out, binary.LittleEndian, field)
+	}
+	out.Write(body.Bytes())
+	out.Write(mappings.Bytes())
+	return out.Bytes(), nil
+}
+
+// readManifest parses NSYM v3 bytes. Mappings are expanded back into entries,
+// the way upstream's reader does, so a caller sees one entry per resref.
+func readManifest(data []byte) ([]Entry, error) {
+	reader := bytes.NewReader(data)
+	magic := make([]byte, 4)
+	if _, err := io.ReadFull(reader, magic); err != nil || string(magic) != "NSYM" {
+		return nil, fmt.Errorf("not a manifest (invalid magic bytes)")
+	}
+	var version, entryCount, mappingCount uint32
+	for _, field := range []*uint32{&version, &entryCount, &mappingCount} {
+		if err := binary.Read(reader, binary.LittleEndian, field); err != nil {
+			return nil, fmt.Errorf("truncated manifest header: %w", err)
+		}
+	}
+	if version != manifestVersion {
+		return nil, fmt.Errorf("unsupported manifest version %d", version)
+	}
+
+	entries := make([]Entry, 0, entryCount+mappingCount)
+	for i := uint32(0); i < entryCount; i++ {
+		var entry Entry
+		if _, err := io.ReadFull(reader, entry.SHA1[:]); err != nil {
+			return nil, fmt.Errorf("truncated entry %d: %w", i, err)
+		}
+		if err := binary.Read(reader, binary.LittleEndian, &entry.Size); err != nil {
+			return nil, fmt.Errorf("truncated entry %d: %w", i, err)
+		}
+		resref, restype, err := readResRef(reader)
+		if err != nil {
+			return nil, fmt.Errorf("truncated entry %d: %w", i, err)
+		}
+		entry.ResRef, entry.ResType = resref, restype
+		entries = append(entries, entry)
+	}
+	for i := uint32(0); i < mappingCount; i++ {
+		var index uint32
+		if err := binary.Read(reader, binary.LittleEndian, &index); err != nil {
+			return nil, fmt.Errorf("truncated mapping %d: %w", i, err)
+		}
+		if index >= entryCount {
+			return nil, fmt.Errorf("mapping %d references non-existent entry %d", i, index)
+		}
+		resref, restype, err := readResRef(reader)
+		if err != nil {
+			return nil, fmt.Errorf("truncated mapping %d: %w", i, err)
+		}
+		target := entries[index]
+		entries = append(entries, Entry{SHA1: target.SHA1, Size: target.Size, ResRef: resref, ResType: restype})
+	}
+	return entries, nil
+}
+
+func readResRef(reader *bytes.Reader) (string, uint16, error) {
+	raw := make([]byte, resRefBytes)
+	if _, err := io.ReadFull(reader, raw); err != nil {
+		return "", 0, err
+	}
+	var restype uint16
+	if err := binary.Read(reader, binary.LittleEndian, &restype); err != nil {
+		return "", 0, err
+	}
+	return strings.ToLower(string(bytes.TrimRight(raw, "\x00"))), restype, nil
+}
+
+func padResRef(resref string) ([]byte, error) {
+	if len(resref) > resRefBytes {
+		return nil, fmt.Errorf("resref %q exceeds %d characters", resref, resRefBytes)
+	}
+	padded := make([]byte, resRefBytes)
+	copy(padded, strings.ToLower(resref))
+	return padded, nil
+}
+
+// Sidecar is the .json file written next to every manifest. Clients fetch it
+// (nwn_nwsync_fetch.nim), so the field names and order match upstream.
+// Upstream omits an integer meta field whose value is 0, hence group_id's
+// omitempty: 0 means absent, not "group zero".
+type Sidecar struct {
+	Version                int    `json:"version"`
+	SHA1                   string `json:"sha1"`
+	HashTreeDepth          int    `json:"hash_tree_depth"`
+	ModuleName             string `json:"module_name"`
+	Description            string `json:"description"`
+	IncludesModuleContents bool   `json:"includes_module_contents"`
+	IncludesClientContents bool   `json:"includes_client_contents"`
+	TotalFiles             int    `json:"total_files"`
+	TotalBytes             int64  `json:"total_bytes"`
+	OnDiskBytes            int64  `json:"on_disk_bytes"`
+	Created                int64  `json:"created"`
+	CreatedWith            string `json:"created_with"`
+	EmitterVersion         string `json:"emitter_version,omitempty"`
+	GroupID                int    `json:"group_id,omitempty"`
+}
+
+func marshalSidecar(sidecar Sidecar) ([]byte, error) {
+	body, err := json.MarshalIndent(sidecar, "", "  ")
+	if err != nil {
+		return nil, err
+	}
+	// Upstream terminates the file with CRLF; match it.
+	return append(body, '\r', '\n'), nil
+}
+
+// blobKey is where a blob lives in a zone, hash tree depth 2. emit writes it,
+// verify reads it and the game client requests it, so the rule lives here and
+// nowhere else.
+func blobKey(sha1Hex string) string {
+	return path.Join("data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
+}
+
+// blobPath is the same location inside a local repository tree.
+func blobPath(root, sha1Hex string) string {
+	return filepath.Join(root, filepath.FromSlash(blobKey(sha1Hex)))
+}

internal/nwsync/memory_test.go

@@ -0,0 +1,150 @@
+package nwsync
+
+import (
+	"bytes"
+	"fmt"
+	"math/rand"
+	"os"
+	"path/filepath"
+	"runtime"
+	"runtime/debug"
+	"testing"
+	"time"
+
+	"github.com/klauspost/compress/zstd"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+)
+
+// TestSingleThreadedEncoderMatchesDefault pins the claim the blob encoder's
+// concurrency setting rests on: it saves memory only, and a published blob is
+// the same bytes either way.
+func TestSingleThreadedEncoderMatchesDefault(t *testing.T) {
+	standard, err := zstd.NewWriter(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer standard.Close()
+
+	body := make([]byte, 4<<20)
+	random := rand.New(rand.NewSource(1))
+	random.Read(body[:len(body)/2])
+	for _, size := range []int{0, 1, 4 << 10, len(body)} {
+		if !bytes.Equal(blobEncoder.EncodeAll(body[:size], nil), standard.EncodeAll(body[:size], nil)) {
+			t.Fatalf("%d bytes compress differently at concurrency 1", size)
+		}
+	}
+}
+
+// resourceSize is one payload in the memory fixtures. Real haks hold a few MB
+// per resource, and peak memory is meant to track that, not the archive.
+const resourceSize = 1 << 20
+
+// writeStreamedHak builds a hak of count resources without ever holding the
+// archive in memory, so the fixture itself does not decide the measurement.
+// Payloads are distinct, so no blob is deduplicated away.
+func writeStreamedHak(t *testing.T, path string, count int) {
+	t.Helper()
+	payload := filepath.Join(t.TempDir(), "payload.bin")
+	body := make([]byte, resourceSize)
+	for index := range body {
+		body[index] = byte(index)
+	}
+
+	resources := make([]erf.Resource, 0, count)
+	for index := range count {
+		// A distinct first byte per resource is enough to give every payload
+		// its own sha1 while still streaming from one file per resource.
+		unique := filepath.Join(filepath.Dir(payload), fmt.Sprintf("p%d.bin", index))
+		body[0] = byte(index)
+		body[1] = byte(index >> 8)
+		if err := os.WriteFile(unique, body, 0o644); err != nil {
+			t.Fatal(err)
+		}
+		resources = append(resources, erf.Resource{
+			Name:       fmt.Sprintf("res%05d", index),
+			Type:       restype(t, "tga"),
+			SourcePath: unique,
+			Size:       resourceSize,
+		})
+	}
+
+	file, err := os.Create(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer file.Close()
+	if err := erf.Write(file, erf.New("HAK", resources)); err != nil {
+		t.Fatalf("write hak: %v", err)
+	}
+}
+
+// peakHeapDuring runs work while sampling the heap, and returns the largest
+// live heap it saw.
+func peakHeapDuring(work func()) uint64 {
+	runtime.GC()
+	done := make(chan struct{})
+	peak := make(chan uint64, 1)
+	go func() {
+		var highest uint64
+		var stats runtime.MemStats
+		for {
+			select {
+			case <-done:
+				peak <- highest
+				return
+			default:
+			}
+			runtime.ReadMemStats(&stats)
+			if stats.HeapAlloc > highest {
+				highest = stats.HeapAlloc
+			}
+			time.Sleep(time.Millisecond)
+		}
+	}()
+	work()
+	close(done)
+	return <-peak
+}
+
+// TestEmitPeakMemoryDoesNotScaleWithArtifactSize is the regression check for
+// the OOM kills on large haks: emit used to hold the whole archive (twice), so
+// a 2 GB hak needed about 10 GB. Emitting an archive 8× bigger must not cost
+// meaningfully more memory.
+func TestEmitPeakMemoryDoesNotScaleWithArtifactSize(t *testing.T) {
+	if testing.Short() {
+		t.Skip("writes a 64 MB fixture")
+	}
+	// A lazy GC lets garbage pile up in proportion to the live heap, which
+	// hides the thing under test. Collecting eagerly makes the sampled heap
+	// track what emit actually holds.
+	defer debug.SetGCPercent(debug.SetGCPercent(10))
+
+	measure := func(count int) uint64 {
+		dir := t.TempDir()
+		hak := filepath.Join(dir, "big.hak")
+		writeStreamedHak(t, hak, count)
+		// The key is computed outside the measurement: the test helper reads
+		// the whole file to hash it, which emit itself no longer does.
+		options := EmitOptions{
+			ArtifactKey:  artifactKey(t, hak),
+			ArtifactPath: hak,
+			As:           filepath.Base(hak),
+			OutDir:       filepath.Join(dir, "out"),
+		}
+		return peakHeapDuring(func() {
+			if _, err := Emit(options); err != nil {
+				t.Fatalf("emit %d resources: %v", count, err)
+			}
+		})
+	}
+
+	small := measure(8)  // 8 MB
+	large := measure(64) // 64 MB
+	const slack = 24 << 20
+
+	t.Logf("peak heap: 8 MB hak %d bytes, 64 MB hak %d bytes", small, large)
+	if large > small+slack {
+		t.Fatalf("peak heap scaled with artifact size: 8 MB hak peaked at %d bytes, 64 MB hak at %d", small, large)
+	}
+}

internal/nwsync/nwsync_test.go

@@ -0,0 +1,531 @@
+package nwsync
+
+import (
+	"bytes"
+	"crypto/sha1"
+	"crypto/sha256"
+	"encoding/binary"
+	"encoding/hex"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+)
+
+func restype(t *testing.T, extension string) uint16 {
+	t.Helper()
+	value, ok := erf.ResourceTypeForExtension(extension)
+	if !ok {
+		t.Fatalf("unknown restype %q", extension)
+	}
+	return value
+}
+
+// writeHak builds a HAK fixture from resref.ext => body pairs.
+func writeHak(t *testing.T, path string, contents map[string][]byte) {
+	t.Helper()
+	resources := make([]erf.Resource, 0, len(contents))
+	for name, body := range contents {
+		stem, extension, _ := strings.Cut(name, ".")
+		resources = append(resources, erf.Resource{
+			Name: stem,
+			Type: restype(t, extension),
+			Data: body,
+			Size: int64(len(body)),
+		})
+	}
+	var out bytes.Buffer
+	if err := erf.Write(&out, erf.New("HAK", resources)); err != nil {
+		t.Fatalf("write hak: %v", err)
+	}
+	if err := os.WriteFile(path, out.Bytes(), 0o644); err != nil {
+		t.Fatalf("write hak file: %v", err)
+	}
+}
+
+// artifactKey is the depot key a file would be published under: the sha256 of
+// its bytes, hash-tree depth 2, keeping the extension.
+func artifactKey(t *testing.T, path string) string {
+	t.Helper()
+	body, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	sum := sha256.Sum256(body)
+	digest := hex.EncodeToString(sum[:])
+	return "artifacts/haks/sha256/" + digest[0:2] + "/" + digest[2:4] + "/" + digest + filepath.Ext(path)
+}
+
+// emitLocal emits one artifact into a local tree, the conformance path.
+func emitLocal(t *testing.T, path, out string) (EmitResult, error) {
+	t.Helper()
+	return Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, path),
+		ArtifactPath: path,
+		As:           filepath.Base(path),
+		OutDir:       out,
+	})
+}
+
+func TestBlobFramingRoundTrips(t *testing.T) {
+	data := []byte("the quick brown fox jumps over the lazy dog, repeatedly and at length")
+	blob := compressBlob(data)
+
+	header := make([]uint32, 6)
+	if err := binary.Read(bytes.NewReader(blob[:blobHeaderBytes]), binary.LittleEndian, header); err != nil {
+		t.Fatalf("read header: %v", err)
+	}
+	want := []uint32{blobMagic, 3, 2, uint32(len(data)), 1, 0}
+	for i := range want {
+		if header[i] != want[i] {
+			t.Errorf("header field %d = %d, want %d", i, header[i], want[i])
+		}
+	}
+	if string(blob[:4]) != "NSYC" {
+		t.Errorf("magic bytes = %q, want NSYC", blob[:4])
+	}
+
+	got, err := decompressBlob(blob)
+	if err != nil {
+		t.Fatalf("decompress: %v", err)
+	}
+	if !bytes.Equal(got, data) {
+		t.Errorf("round trip mismatch: %q", got)
+	}
+}
+
+func TestManifestBytesMatchUpstreamLayout(t *testing.T) {
+	shared := sha1.Sum([]byte("shared"))
+	other := sha1.Sum([]byte("other"))
+	// Deliberately out of order, and with two resrefs sharing one sha1: the
+	// second one must become a mapping, not a second entry.
+	entries := []Entry{
+		{SHA1: other, Size: 5, ResRef: "zzz", ResType: 1},
+		{SHA1: shared, Size: 6, ResRef: "bbb", ResType: 2},
+		{SHA1: shared, Size: 6, ResRef: "aaa", ResType: 3},
+	}
+	data, err := writeManifest(entries)
+	if err != nil {
+		t.Fatalf("write manifest: %v", err)
+	}
+
+	if string(data[:4]) != "NSYM" {
+		t.Fatalf("magic = %q", data[:4])
+	}
+	var version, entryCount, mappingCount uint32
+	reader := bytes.NewReader(data[4:16])
+	for _, field := range []*uint32{&version, &entryCount, &mappingCount} {
+		_ = binary.Read(reader, binary.LittleEndian, field)
+	}
+	if version != 3 || entryCount != 2 || mappingCount != 1 {
+		t.Fatalf("header = version %d, %d entries, %d mappings; want 3/2/1", version, entryCount, mappingCount)
+	}
+	wantSize := 16 + int(entryCount)*(20+4+16+2) + int(mappingCount)*(4+16+2)
+	if len(data) != wantSize {
+		t.Fatalf("manifest is %d bytes, want %d", len(data), wantSize)
+	}
+
+	// Sorted by sha1 hex then resref, so the shared hash's "aaa" is the entry
+	// and "bbb" is demoted to a mapping.
+	round, err := readManifest(data)
+	if err != nil {
+		t.Fatalf("read manifest: %v", err)
+	}
+	if len(round) != 3 {
+		t.Fatalf("round trip returned %d entries, want 3", len(round))
+	}
+	byResRef := map[string]Entry{}
+	for _, entry := range round {
+		byResRef[entry.ResRef] = entry
+	}
+	for _, entry := range entries {
+		got, ok := byResRef[entry.ResRef]
+		if !ok {
+			t.Fatalf("resref %q lost in round trip", entry.ResRef)
+		}
+		if got != entry {
+			t.Errorf("resref %q = %+v, want %+v", entry.ResRef, got, entry)
+		}
+	}
+	if round[0].ResRef != "aaa" && round[1].ResRef != "aaa" {
+		t.Errorf("entries are not sorted by sha1 then resref: %+v", round)
+	}
+}
+
+func TestEmitWritesBlobsAndManifest(t *testing.T) {
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	body := []byte("texture bytes")
+	writeHak(t, hak, map[string][]byte{
+		"bloodstain1.tga": body,
+		"copy1.txi":       body, // same content, different resref: one blob
+		"script1.nss":     []byte("void main() {}"),
+		"debug1.ndb":      []byte("debug"),
+		"comment1.gic":    []byte("comment"),
+	})
+
+	out := filepath.Join(dir, "out")
+	result, err := emitLocal(t, hak, out)
+	if err != nil {
+		t.Fatalf("emit: %v", err)
+	}
+	if result.Entries != 2 {
+		t.Errorf("emitted %d entries, want 2 (nss/ndb/gic are always skipped)", result.Entries)
+	}
+	if result.BlobsWritten != 1 {
+		t.Errorf("wrote %d blobs, want 1 (identical content shares a blob)", result.BlobsWritten)
+	}
+
+	// The blob is named by the sha1 of the uncompressed bytes, under a depth-2
+	// hash tree, and decompresses back to exactly those bytes.
+	sum := sha1.Sum(body)
+	name := hex.EncodeToString(sum[:])
+	path := filepath.Join(out, "data", "sha1", name[0:2], name[2:4], name)
+	blob, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatalf("blob missing at %s: %v", path, err)
+	}
+	got, err := decompressBlob(blob)
+	if err != nil {
+		t.Fatalf("decompress blob: %v", err)
+	}
+	if !bytes.Equal(got, body) {
+		t.Errorf("blob decompressed to %q, want %q", got, body)
+	}
+
+	entries := readEmitted(t, result.ManifestPath)
+	for _, entry := range entries {
+		if entry.ResType == restype(t, "nss") || entry.ResType == restype(t, "ndb") || entry.ResType == restype(t, "gic") {
+			t.Errorf("skipped restype leaked into the manifest: %+v", entry)
+		}
+	}
+
+	var sidecar Sidecar
+	body2, err := os.ReadFile(result.ManifestPath + ".json")
+	if err != nil {
+		t.Fatalf("sidecar missing: %v", err)
+	}
+	if err := json.Unmarshal(body2, &sidecar); err != nil {
+		t.Fatalf("sidecar json: %v", err)
+	}
+	if sidecar.TotalFiles != 2 || sidecar.HashTreeDepth != 2 || sidecar.Version != 3 {
+		t.Errorf("sidecar = %+v", sidecar)
+	}
+	if sidecar.IncludesModuleContents {
+		t.Error("sidecar claims module contents; a published manifest never has them")
+	}
+	if strings.Contains(string(body2), "group_id") {
+		t.Error("per-artifact sidecar should omit group_id (0 means absent)")
+	}
+	if _, err := os.Stat(filepath.Join(out, "latest")); err == nil {
+		t.Error("a latest file was written; there must never be one")
+	}
+}
+
+func readEmitted(t *testing.T, indexPath string) []Entry {
+	t.Helper()
+	data, err := os.ReadFile(indexPath)
+	if err != nil {
+		t.Fatalf("read emitted manifest: %v", err)
+	}
+	entries, err := readManifest(data)
+	if err != nil {
+		t.Fatalf("parse emitted manifest: %v", err)
+	}
+	return entries
+}
+
+func TestEmitFailsClosedOnOversizeResource(t *testing.T) {
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "big.hak")
+	writeHak(t, hak, map[string][]byte{"huge1.tga": make([]byte, fileSizeLimit+1)})
+
+	if _, err := emitLocal(t, hak, filepath.Join(dir, "out")); err == nil {
+		t.Fatal("emit accepted a resource over the 15 MB limit")
+	}
+}
+
+func TestEmitLooseFile(t *testing.T) {
+	dir := t.TempDir()
+	tlk := filepath.Join(dir, "sow_tlk.tlk")
+	if err := os.WriteFile(tlk, []byte("TLK V3.0 payload"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	out := filepath.Join(dir, "out")
+	result, err := emitLocal(t, tlk, out)
+	if err != nil {
+		t.Fatalf("emit tlk: %v", err)
+	}
+	entries := readEmitted(t, result.ManifestPath)
+	if len(entries) != 1 || entries[0].ResRef != "sow_tlk" || entries[0].ResType != restype(t, "tlk") {
+		t.Fatalf("tlk emitted as %+v", entries)
+	}
+	if result.BlobsWritten != 1 {
+		t.Errorf("wrote %d blobs, want 1", result.BlobsWritten)
+	}
+}
+
+// emitFixture emits two haks that share a resref, so the merge rule is
+// observable: "top" holds the winning body, "assets" the shadowed one.
+func emitFixture(t *testing.T) (out string, keys map[string]string, topBody, assetBody []byte) {
+	t.Helper()
+	dir := t.TempDir()
+	topBody = []byte("2da from sow_top")
+	assetBody = []byte("2da from the asset hak")
+	writeHak(t, filepath.Join(dir, "sow_top.hak"), map[string][]byte{"appearance.2da": topBody})
+	writeHak(t, filepath.Join(dir, "sow_core_01.hak"), map[string][]byte{
+		"appearance.2da":  assetBody,
+		"bloodstain1.tga": []byte("blood"),
+	})
+	out = filepath.Join(dir, "out")
+	keys = map[string]string{}
+	for _, name := range []string{"sow_top", "sow_core_01"} {
+		path := filepath.Join(dir, name+".hak")
+		keys[name] = artifactKey(t, path)
+		if _, err := emitLocal(t, path, out); err != nil {
+			t.Fatalf("emit %s: %v", name, err)
+		}
+	}
+	return out, keys, topBody, assetBody
+}
+
+func TestAssembleShadowsByOrder(t *testing.T) {
+	entriesDir, keys, topBody, assetBody := emitFixture(t)
+
+	result, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{keys["sow_top"], keys["sow_core_01"]},
+		OutDir:       entriesDir,
+		GroupID:      2,
+	})
+	if err != nil {
+		t.Fatalf("assemble: %v", err)
+	}
+	if result.Entries != 2 {
+		t.Fatalf("merged %d entries, want 2 (appearance.2da is shadowed, not duplicated)", result.Entries)
+	}
+
+	data, err := os.ReadFile(result.ManifestPath)
+	if err != nil {
+		t.Fatalf("read merged manifest: %v", err)
+	}
+	merged := sha1.Sum(data)
+	if hex.EncodeToString(merged[:]) != result.SHA1 || filepath.Base(result.ManifestPath) != result.SHA1 {
+		t.Errorf("manifest is not named by its own sha1: %s", result.ManifestPath)
+	}
+	entries, err := readManifest(data)
+	if err != nil {
+		t.Fatalf("parse merged manifest: %v", err)
+	}
+	for _, entry := range entries {
+		if entry.ResRef != "appearance" {
+			continue
+		}
+		if entry.SHA1 != sha1.Sum(topBody) {
+			t.Errorf("appearance.2da resolved to the wrong hak; want the earliest in --order")
+		}
+		if entry.SHA1 == sha1.Sum(assetBody) {
+			t.Error("appearance.2da resolved to the shadowed hak")
+		}
+	}
+
+	sidecar := Sidecar{}
+	body, err := os.ReadFile(result.ManifestPath + ".json")
+	if err != nil {
+		t.Fatalf("merged sidecar missing: %v", err)
+	}
+	if err := json.Unmarshal(body, &sidecar); err != nil {
+		t.Fatalf("merged sidecar json: %v", err)
+	}
+	if sidecar.GroupID != 2 {
+		t.Errorf("group_id = %d, want 2 (testing)", sidecar.GroupID)
+	}
+	if sidecar.SHA1 != result.SHA1 {
+		t.Errorf("sidecar sha1 = %s, want %s", sidecar.SHA1, result.SHA1)
+	}
+	if sidecar.TotalFiles != 2 {
+		t.Errorf("total_files = %d, want 2", sidecar.TotalFiles)
+	}
+}
+
+func TestAssembleReversedOrderPicksTheOtherHak(t *testing.T) {
+	entriesDir, keys, topBody, assetBody := emitFixture(t)
+
+	result, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{keys["sow_core_01"], keys["sow_top"]},
+		OutDir:       entriesDir,
+	})
+	if err != nil {
+		t.Fatalf("assemble: %v", err)
+	}
+	data, _ := os.ReadFile(result.ManifestPath)
+	entries, err := readManifest(data)
+	if err != nil {
+		t.Fatalf("parse merged manifest: %v", err)
+	}
+	for _, entry := range entries {
+		if entry.ResRef == "appearance" && entry.SHA1 != sha1.Sum(assetBody) {
+			t.Errorf("appearance.2da did not follow --order; still resolves to %x", entry.SHA1)
+		}
+	}
+	_ = topBody
+}
+
+func TestAssembleRefusesMismatchedEmitterVersions(t *testing.T) {
+	entriesDir, keys, _, _ := emitFixture(t)
+
+	index, err := resolveIndexKey(keys["sow_core_01"], entriesDir)
+	if err != nil {
+		t.Fatal(err)
+	}
+	path := filepath.Join(entriesDir, index+".json")
+	var sidecar Sidecar
+	body, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := json.Unmarshal(body, &sidecar); err != nil {
+		t.Fatal(err)
+	}
+	sidecar.EmitterVersion = "0"
+	patched, err := marshalSidecar(sidecar)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path, patched, 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	_, err = Assemble(AssembleOptions{
+		ArtifactKeys: []string{keys["sow_top"], keys["sow_core_01"]},
+		OutDir:       entriesDir,
+	})
+	if err == nil || !strings.Contains(err.Error(), "emitter version mismatch") {
+		t.Fatalf("assemble merged across emitter versions: %v", err)
+	}
+}
+
+func TestEmitHonoursSourceDateEpoch(t *testing.T) {
+	t.Setenv("SOURCE_DATE_EPOCH", "1700000000")
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "pinned.hak")
+	writeHak(t, hak, map[string][]byte{"one1.tga": []byte("body")})
+	result, err := emitLocal(t, hak, filepath.Join(dir, "out"))
+	if err != nil {
+		t.Fatalf("emit: %v", err)
+	}
+	body, err := os.ReadFile(result.ManifestPath + ".json")
+	if err != nil {
+		t.Fatal(err)
+	}
+	var sidecar Sidecar
+	if err := json.Unmarshal(body, &sidecar); err != nil {
+		t.Fatal(err)
+	}
+	if sidecar.Created != 1700000000 {
+		t.Errorf("created = %d, want the pinned SOURCE_DATE_EPOCH", sidecar.Created)
+	}
+}
+
+func TestEmitRejectsAModule(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "sow.mod")
+	var out bytes.Buffer
+	if err := erf.Write(&out, erf.New("MOD", []erf.Resource{
+		{Name: "module", Type: restype(t, "ifo"), Data: []byte("ifo"), Size: 3},
+	})); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path, out.Bytes(), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := emitLocal(t, path, filepath.Join(dir, "out")); err == nil {
+		t.Fatal("emit accepted a .mod; a manifest never carries module contents")
+	}
+}
+
+func TestAssembleFailsClosedOnMissingIndex(t *testing.T) {
+	entriesDir, keys, _, _ := emitFixture(t)
+	missing := "artifacts/haks/sha256/00/11/" + strings.Repeat("0", 64) + ".hak"
+	_, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{keys["sow_top"], missing},
+		OutDir:       entriesDir,
+	})
+	if err == nil || !strings.Contains(err.Error(), missing) {
+		t.Fatalf("assemble did not fail closed and name the missing artifact: %v", err)
+	}
+}
+
+// Callers capture a script's stdout as a value: `dir="$(pack-haks.sh)"`. A
+// summary line on stdout gets glued onto that value, so both summaries belong
+// on stderr.
+func TestRunKeepsSummariesOffStdout(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "sow_top.hak")
+	writeHak(t, path, map[string][]byte{"appearance.2da": []byte("2da from sow_top")})
+	key := artifactKey(t, path)
+	out := filepath.Join(dir, "out")
+
+	for _, args := range [][]string{
+		{"emit", "--out", out, "--as", "sow_top.hak", key, path},
+		{"assemble", "--out", out, key},
+	} {
+		var stdout, stderr bytes.Buffer
+		if code := Run(args, &stdout, &stderr); code != exitOK {
+			t.Fatalf("Run(%v) exit=%d: %s", args, code, stderr.String())
+		}
+		if stdout.Len() != 0 {
+			t.Errorf("Run(%v) wrote to stdout: %q", args, stdout.String())
+		}
+		if stderr.Len() == 0 {
+			t.Errorf("Run(%v) reported no summary on stderr", args)
+		}
+	}
+}
+
+func TestRunUsageErrors(t *testing.T) {
+	cases := [][]string{
+		nil,
+		{"nope"},
+		{"emit"},
+		{"emit", "artifact-key.hak"},
+		{"emit", "a", "b", "c"},
+		{"assemble", "--out", "y"},
+	}
+	for _, args := range cases {
+		var out, errw bytes.Buffer
+		if code := Run(args, &out, &errw); code != exitUsage {
+			t.Errorf("Run(%v) exit=%d, want %d", args, code, exitUsage)
+		}
+	}
+}
+
+// TestEveryBlobDeclaresItsFrameContentSize guards the fault that stopped every
+// client sync (#86): klauspost/compress omits Frame_Content_Size for inputs
+// under 256 bytes, and the game client cannot decode a frame without it. This
+// asserts a frame property, not a round trip — the zstd CLI and Go's decoder
+// both stream such a frame happily, so round-tripping cannot see the defect.
+func TestEveryBlobDeclaresItsFrameContentSize(t *testing.T) {
+	// 230 and 175 are real sizes from the manifest that failed to sync; 255/256
+	// straddle the encoder's threshold.
+	for _, size := range []int{1, 32, 175, 230, 255, 256, 257, 1024, 5000} {
+		payload := make([]byte, size)
+		for i := range payload {
+			payload[i] = byte('a' + i%26)
+		}
+		blob := compressBlob(payload)
+		if !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
+			t.Errorf("blob of %d bytes declares no frame content size (descriptor %#x)",
+				size, blob[blobHeaderBytes+4])
+		}
+		got, err := decompressBlob(blob)
+		if err != nil {
+			t.Fatalf("decompress %d-byte blob: %v", size, err)
+		}
+		if !bytes.Equal(got, payload) {
+			t.Errorf("%d-byte blob did not round trip", size)
+		}
+	}
+}

internal/nwsync/run.go

@@ -0,0 +1,202 @@
+// Package nwsync publishes NWSync repository data: blobs and a per-artifact
+// NSYM manifest at each artifact's birth (emit), and one merged manifest at
+// module release (assemble).
+//
+// The split exists because upstream nwn_nwsync_write wants every hak, the TLK
+// and the module present in one run on one disk, which our build hosts cannot
+// hold. Upstream stays the conformance oracle: manifests compare byte for
+// byte, blobs compare after decompression.
+package nwsync
+
+import (
+	"flag"
+	"fmt"
+	"io"
+	"os"
+)
+
+const (
+	exitOK       = 0
+	exitUsage    = 64
+	exitInternal = 70
+	// exitDrift says the command worked and the zone is wrong, which is a
+	// different thing for CI to act on than the command failing. It matches
+	// depot's code for the same meaning.
+	exitDrift = 1
+)
+
+// Run executes an nwsync subcommand. args[0] is the subcommand (emit|assemble);
+// returns the process exit code.
+func Run(args []string, stdout, stderr io.Writer) int {
+	if len(args) == 0 {
+		printRunUsage(stderr)
+		return exitUsage
+	}
+	switch args[0] {
+	case "emit":
+		return runEmit(args[1:], stderr)
+	case "assemble":
+		return runAssemble(args[1:], stderr)
+	case "verify":
+		return runVerify(args[1:], stdout, stderr, os.Getenv)
+	case "-h", "--help", "help":
+		printRunUsage(stdout)
+		return exitOK
+	default:
+		fmt.Fprintf(stderr, "nwsync: unknown subcommand %q\n\n", args[0])
+		printRunUsage(stderr)
+		return exitUsage
+	}
+}
+
+func printRunUsage(w io.Writer) {
+	fmt.Fprint(w, `usage:
+  nwsync emit     [--as NAME] [--out DIR] [--verify] <artifact-key> <file>
+  nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
+  nwsync verify   [--sample N] [--base URL] <manifest-sha1>
+
+emit explodes one .hak/.erf or one loose file (the TLK) into NWSync blobs plus
+a NSYM index covering only that artifact, and uploads both. assemble merges
+those indexes into one manifest, reading no bulk data. Artifact keys are depot
+keys; an index lives beside its artifact, with the extension replaced.
+
+verify reads a published manifest and its blobs back through the public pull
+zone, with no credential, and decompresses and hashes every one. It is the only
+check on a published blob upstream of a player's client.
+
+--verify makes emit hash what it would otherwise skip. emit normally treats a
+blob's presence as proof of its contents, so without this an object written
+truncated, or written by an emitter since found broken, is skipped forever.
+--verify repairs the storage zone, while verify reads the edge in front of it.
+So a verify run right after a repair is a survey, not a verdict: it names the
+keys the edge still serves stale. Purge those, then run it again.
+
+--out DIR writes to a local repository tree instead of uploading, which is the
+conformance path against upstream nwn_nwsync_write. Without it, the zone comes
+from NWSYNC_STORAGE_ZONE, NWSYNC_STORAGE_PASSWORD and BUNNY_STORAGE_HOST.
+verify needs none of those; its base comes from --base or NWSYNC_PULL_BASE.
+`)
+}
+
+// parseArgs parses flags that may appear before, after or between positionals.
+// Go's flag package stops at the first non-flag argument, which turns
+// `emit <key> <file> --out DIR` into a confusing arity error.
+func parseArgs(fs *flag.FlagSet, args []string) ([]string, error) {
+	var positional []string
+	for {
+		if err := fs.Parse(args); err != nil {
+			return nil, err
+		}
+		rest := fs.Args()
+		if len(rest) == 0 {
+			return positional, nil
+		}
+		positional = append(positional, rest[0])
+		args = rest[1:]
+	}
+}
+
+func runEmit(args []string, stderr io.Writer) int {
+	fs := flag.NewFlagSet("emit", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	as := fs.String("as", "", "published name of the artifact, when it differs from the key")
+	out := fs.String("out", "", "write to a local repository tree instead of uploading")
+	jobs := fs.Int("jobs", defaultEmitJobs, "resources to hash, compress and store at once")
+	verify := fs.Bool("verify", false, "read back and hash blobs that already exist instead of trusting their presence")
+	positional, err := parseArgs(fs, args)
+	if err != nil {
+		return exitUsage
+	}
+	if len(positional) != 2 {
+		fmt.Fprintf(stderr, "nwsync emit: <artifact-key> and <file> are both required\n")
+		return exitUsage
+	}
+	if *jobs < 1 {
+		fmt.Fprintf(stderr, "nwsync emit: -jobs must be at least 1, got %d\n", *jobs)
+		return exitUsage
+	}
+
+	result, err := Emit(EmitOptions{
+		ArtifactKey:  positional[0],
+		ArtifactPath: positional[1],
+		As:           *as,
+		OutDir:       *out,
+		Jobs:         *jobs,
+		Verify:       *verify,
+	})
+	if err != nil {
+		fmt.Fprintf(stderr, "nwsync emit: %v\n", err)
+		return exitInternal
+	}
+	fmt.Fprintf(stderr, "emitted %s: %d resources, %d new blobs, index %s\n",
+		result.Name, result.Entries, result.BlobsWritten, result.ManifestPath)
+	return exitOK
+}
+
+func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
+	fs := flag.NewFlagSet("verify", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	base := fs.String("base", getenv("NWSYNC_PULL_BASE"), "pull zone base URL to read through")
+	sample := fs.Int("sample", 0, "check this many random blobs instead of all of them")
+	jobs := fs.Int("jobs", defaultEmitJobs, "blobs to fetch and hash at once")
+	positional, err := parseArgs(fs, args)
+	if err != nil {
+		return exitUsage
+	}
+	if len(positional) != 1 {
+		fmt.Fprintf(stderr, "nwsync verify: exactly one <manifest-sha1> is required\n")
+		return exitUsage
+	}
+
+	result, err := Verify(VerifyOptions{
+		ManifestSHA1: positional[0],
+		Base:         *base,
+		Sample:       *sample,
+		Jobs:         *jobs,
+		Log:          stderr,
+	})
+	if err != nil {
+		fmt.Fprintf(stderr, "nwsync verify: %v\n", err)
+		return exitInternal
+	}
+	fmt.Fprintf(stdout, "verified %d of %d blobs behind %d resources: %d failures, %d bytes checked\n",
+		result.Checked, result.Blobs, result.Entries, result.Failures, result.Bytes)
+	if result.Failures > 0 {
+		return exitDrift
+	}
+	return exitOK
+}
+
+func runAssemble(args []string, stderr io.Writer) int {
+	fs := flag.NewFlagSet("assemble", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	tlkKey := fs.String("tlk-key", "", "depot key of the TLK, which shadows nothing and merges last")
+	out := fs.String("out", "", "write to a local repository tree instead of uploading")
+	groupID := fs.Int("group-id", 0, "NWSync group id (1 = current, 2 = testing; 0 omits it)")
+	moduleName := fs.String("module-name", "", "module name recorded in the sidecar")
+	description := fs.String("description", "", "description recorded in the sidecar")
+	positional, err := parseArgs(fs, args)
+	if err != nil {
+		return exitUsage
+	}
+	if len(positional) == 0 {
+		fmt.Fprintf(stderr, "nwsync assemble: at least one artifact key is required\n")
+		return exitUsage
+	}
+
+	result, err := Assemble(AssembleOptions{
+		ArtifactKeys: positional,
+		TLKKey:       *tlkKey,
+		OutDir:       *out,
+		GroupID:      *groupID,
+		ModuleName:   *moduleName,
+		Description:  *description,
+	})
+	if err != nil {
+		fmt.Fprintf(stderr, "nwsync assemble: %v\n", err)
+		return exitInternal
+	}
+	fmt.Fprintf(stderr, "assembled manifest %s: %d resources, %s\n",
+		result.SHA1, result.Entries, result.ManifestPath)
+	return exitOK
+}

internal/nwsync/sink.go

@@ -0,0 +1,241 @@
+package nwsync
+
+import (
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"os"
+	"path"
+	"path/filepath"
+	"strings"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/depot"
+)
+
+// sink is where an emit or assemble run puts what it produces. The zone is the
+// production sink; a local directory exists only as the conformance path, so
+// upstream's output and ours can be diffed on a developer machine.
+type sink interface {
+	// putBlob stores one NWCompressedBuffer blob under its sha1 name and
+	// returns the bytes stored, or 0 if a good copy was already there. Blob
+	// names are content hashes, so an existing name is normally taken as
+	// existing content — which is why body is a thunk: compression is the
+	// expensive part of emit and a blob that is already stored must not pay
+	// for it.
+	//
+	// verify stops trusting presence: the stored copy is read back, unwrapped
+	// and hashed, and replaced when it is not what its name claims. Without it
+	// an object written truncated, or written by an emitter since found broken,
+	// is skipped by every later emit forever and no backfill can repair it.
+	putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error)
+	// putIndex stores a NSYM manifest and its sidecar under key, which is
+	// either an artifact-derived object key or a local path.
+	putIndex(key string, manifest, sidecar []byte) error
+	// getIndex reads back a NSYM manifest and its sidecar.
+	getIndex(key string) (manifest, sidecar []byte, err error)
+	// describe names the sink for messages.
+	describe(key string) string
+}
+
+// dirSink writes a local NWSync repository tree.
+type dirSink struct{ root string }
+
+func (s dirSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) {
+	blob := blobPath(s.root, sha1Hex)
+	if !verify {
+		// Stat, not read: the common path must not pay to open every blob that
+		// is already there.
+		if _, err := os.Stat(blob); err == nil {
+			return 0, nil
+		}
+	} else if stored, err := os.ReadFile(blob); err == nil && blobMatchesName(stored, sha1Hex) == nil {
+		return 0, nil
+	}
+	if err := os.MkdirAll(filepath.Dir(blob), 0o755); err != nil {
+		return 0, fmt.Errorf("create blob directory: %w", err)
+	}
+	data := body()
+	if err := os.WriteFile(blob, data, 0o644); err != nil {
+		return 0, fmt.Errorf("write blob: %w", err)
+	}
+	return int64(len(data)), nil
+}
+
+func (s dirSink) putIndex(key string, manifest, sidecar []byte) error {
+	target := filepath.Join(s.root, filepath.FromSlash(key))
+	if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+		return fmt.Errorf("create manifest directory: %w", err)
+	}
+	if err := os.WriteFile(target, manifest, 0o644); err != nil {
+		return fmt.Errorf("write manifest: %w", err)
+	}
+	if err := os.WriteFile(target+".json", sidecar, 0o644); err != nil {
+		return fmt.Errorf("write sidecar: %w", err)
+	}
+	return nil
+}
+
+func (s dirSink) getIndex(key string) ([]byte, []byte, error) {
+	target := filepath.Join(s.root, filepath.FromSlash(key))
+	manifest, err := os.ReadFile(target)
+	if err != nil {
+		return nil, nil, fmt.Errorf("read index: %w", err)
+	}
+	sidecar, err := os.ReadFile(target + ".json")
+	if err != nil {
+		return nil, nil, fmt.Errorf("read sidecar: %w", err)
+	}
+	return manifest, sidecar, nil
+}
+
+func (s dirSink) describe(key string) string {
+	return filepath.Join(s.root, filepath.FromSlash(key))
+}
+
+// zoneSink uploads straight to the NWSync storage zone. Nothing bulky is ever
+// written to the runner's disk: the working set is one resource at a time.
+type zoneSink struct {
+	store depot.KeyStore
+	ctx   context.Context
+	zone  string
+}
+
+func (s zoneSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) {
+	key := blobKey(sha1Hex)
+	// A throttled probe must never be read as "missing, re-upload" or as
+	// "present, skip", so only a confirmed Present skips the upload.
+	state, _, err := s.store.ProbeKey(s.ctx, key)
+	if err != nil {
+		return 0, fmt.Errorf("probe blob %s: %w", sha1Hex, err)
+	}
+	if state == depot.Present {
+		if !verify {
+			return 0, nil
+		}
+		// The probe only proved the object exists. Read it back and hold it to
+		// its own name.
+		//
+		// This reads the storage API rather than the pull zone: emit holds the
+		// write credential, and a repair decision has to be made against the
+		// copy it is about to overwrite, not against an edge cache of it. A
+		// read that fails outright is a fault, not a verdict — treating it as
+		// "bad, re-upload" would turn a throttled zone into a full backfill.
+		stored, err := s.store.GetKey(s.ctx, key)
+		if err != nil {
+			return 0, fmt.Errorf("read back blob %s: %w", sha1Hex, err)
+		}
+		if blobMatchesName(stored, sha1Hex) == nil {
+			return 0, nil
+		}
+	}
+	data := body()
+	if err := s.put(key, data); err != nil {
+		return 0, fmt.Errorf("upload blob %s: %w", sha1Hex, err)
+	}
+	return int64(len(data)), nil
+}
+
+func (s zoneSink) putIndex(key string, manifest, sidecar []byte) error {
+	// The manifest lands last: its presence is the publication marker, so it
+	// must never appear before the blobs it names.
+	if err := s.put(key+".json", sidecar); err != nil {
+		return fmt.Errorf("upload sidecar %s: %w", key, err)
+	}
+	if err := s.put(key, manifest); err != nil {
+		return fmt.Errorf("upload index %s: %w", key, err)
+	}
+	return nil
+}
+
+func (s zoneSink) getIndex(key string) ([]byte, []byte, error) {
+	manifest, err := s.store.GetKey(s.ctx, key)
+	if err != nil {
+		return nil, nil, fmt.Errorf("read index: %w", err)
+	}
+	sidecar, err := s.store.GetKey(s.ctx, key+".json")
+	if err != nil {
+		return nil, nil, fmt.Errorf("read sidecar: %w", err)
+	}
+	return manifest, sidecar, nil
+}
+
+func (s zoneSink) describe(key string) string { return s.zone + "/" + key }
+
+func (s zoneSink) put(key string, body []byte) error {
+	sum := sha256.Sum256(body)
+	return s.store.PutReader(s.ctx, key, bytes.NewReader(body), int64(len(body)), hex.EncodeToString(sum[:]))
+}
+
+// newZoneSink builds the upload sink from the environment. NWSync data lives
+// in its own zone, separate from the asset depot, so it has its own zone and
+// credential; only the host is shared, and Crucible has no default host.
+func newZoneSink(ctx context.Context, getenv func(string) string) (sink, error) {
+	cfg := depot.LoadConfig(getenv)
+	cfg.StorageZone = getenv("NWSYNC_STORAGE_ZONE")
+	cfg.WriteKey = getenv("NWSYNC_STORAGE_PASSWORD")
+	cfg.ReadKey = cfg.WriteKey
+	if cfg.StorageZone == "" {
+		return nil, fmt.Errorf("NWSYNC_STORAGE_ZONE is unset (or pass --out DIR to write locally)")
+	}
+	if cfg.WriteKey == "" {
+		return nil, fmt.Errorf("NWSYNC_STORAGE_PASSWORD is unset (or pass --out DIR to write locally)")
+	}
+	if cfg.StorageHost == "" {
+		return nil, fmt.Errorf("BUNNY_STORAGE_HOST is unset")
+	}
+	store, err := depot.NewKeyStore(cfg)
+	if err != nil {
+		return nil, err
+	}
+	return zoneSink{store: store, ctx: ctx, zone: cfg.StorageZone}, nil
+}
+
+// indexKey is where an artifact's NSYM lives: beside the artifact itself, with
+// the final extension replaced. emit and assemble must agree on this one rule,
+// so it lives here and nowhere else.
+//
+// artifacts/haks/sha256/30/46/3046….hak -> artifacts/haks/sha256/30/46/3046….nsym
+func indexKey(artifactKey string) (string, error) {
+	extension := path.Ext(artifactKey)
+	if extension == "" {
+		return "", fmt.Errorf("artifact key %q has no extension", artifactKey)
+	}
+	return strings.TrimSuffix(artifactKey, extension) + ".nsym", nil
+}
+
+// resolveIndexKey is where emit writes an artifact's index and where assemble
+// reads it from. On the zone that is beside the artifact; locally the indexes
+// sit flat beside the data tree, so upstream's output and ours diff directly.
+func resolveIndexKey(artifactKey, outDir string) (string, error) {
+	key, err := indexKey(artifactKey)
+	if err != nil {
+		return "", err
+	}
+	if outDir != "" {
+		return path.Base(key), nil
+	}
+	return key, nil
+}
+
+// checkArtifactKey fails closed when the key's embedded digest is not the
+// digest of the bytes being emitted. Publishing an index under the wrong key
+// silently pairs a manifest with the wrong artifact.
+// artifact is hashed by streaming, so a multi-gigabyte hak is never resident.
+func checkArtifactKey(artifactKey string, artifact io.Reader) error {
+	base := path.Base(artifactKey)
+	digest := strings.TrimSuffix(base, path.Ext(base))
+	if len(digest) != 64 {
+		return fmt.Errorf("artifact key %q does not name a sha256", artifactKey)
+	}
+	hash := sha256.New()
+	if _, err := io.Copy(hash, artifact); err != nil {
+		return fmt.Errorf("hash artifact: %w", err)
+	}
+	if got := hex.EncodeToString(hash.Sum(nil)); got != digest {
+		return fmt.Errorf("artifact key %q names digest %s but the file hashes to %s", artifactKey, digest, got)
+	}
+	return nil
+}

internal/nwsync/verify.go

@@ -0,0 +1,233 @@
+package nwsync
+
+import (
+	"crypto/sha1"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"io"
+	"math/rand/v2"
+	"net/http"
+	"path"
+	"sort"
+	"strconv"
+	"sync"
+	"time"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+)
+
+// defaultPullBase is the public NWSync host, which is a Bunny pull zone fronting
+// the storage zone. Verify reads through it rather than through the storage API
+// on purpose: what matters is the bytes a client is served, edge behaviour
+// included, not what the origin believes it holds.
+const defaultPullBase = "https://nwsync.westgate.pw"
+
+// errBlobMissing marks an object the zone does not serve at all, as distinct
+// from one it serves badly.
+var errBlobMissing = errors.New("missing")
+
+// blobSource reads one object out of the zone by key. Verify never writes and
+// never authenticates, so this is deliberately narrower than sink.
+type blobSource interface {
+	get(key string) ([]byte, error)
+	describe(key string) string
+}
+
+// pullZone reads the zone over plain HTTP, with no credential.
+type pullZone struct {
+	base   string
+	client *http.Client
+}
+
+func newPullZone(base string) blobSource {
+	if base == "" {
+		base = defaultPullBase
+	}
+	return pullZone{
+		base: base,
+		// A full sweep is tens of thousands of small requests, so connections
+		// have to be reused; the default transport does that already.
+		client: &http.Client{Timeout: 60 * time.Second},
+	}
+}
+
+func (z pullZone) describe(key string) string { return z.base + "/" + key }
+
+func (z pullZone) get(key string) ([]byte, error) {
+	resp, err := z.client.Get(z.describe(key))
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode == http.StatusNotFound || resp.StatusCode == http.StatusGone {
+		_, _ = io.Copy(io.Discard, resp.Body)
+		return nil, errBlobMissing
+	}
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		_, _ = io.Copy(io.Discard, resp.Body)
+		return nil, fmt.Errorf("unexpected status %d", resp.StatusCode)
+	}
+	return io.ReadAll(resp.Body)
+}
+
+// VerifyOptions describes one verify run.
+type VerifyOptions struct {
+	ManifestSHA1 string     // the merged manifest to verify
+	Base         string     // pull zone base URL; empty means defaultPullBase
+	Sample       int        // check this many random blobs; 0 means all of them
+	Jobs         int        // blobs in flight at once; 0 means defaultEmitJobs
+	Source       blobSource // test seam; nil means the pull zone at Base
+	Log          io.Writer  // per-blob failures land here; nil discards them
+}
+
+// VerifyResult reports what one verify run found.
+type VerifyResult struct {
+	Entries  int   // resources the manifest names
+	Blobs    int   // distinct blobs behind those resources
+	Checked  int   // blobs actually fetched
+	Failures int   // blobs that failed a check
+	Bytes    int64 // uncompressed bytes verified
+}
+
+// Verify reads a published manifest and its blobs the way a client reads them,
+// and reports every blob that is not what the manifest says it is.
+//
+// Presence is not correctness. emit skips an object that already exists on the
+// strength of a 1-byte range GET, so a truncated or wrongly framed object is
+// skipped by every later emit forever and the backfill cannot repair it. This is
+// the only thing upstream of a player's client that can tell that has happened.
+//
+// Every blob is decompressed and hashed. A Content-Length check would pass the
+// exact failure mode being hunted — a byte-correct-looking object whose contents
+// are wrong — and a round-trip check alone would pass a frame that omits its
+// content size, because Go's decoder is more capable than the client's (#86).
+func Verify(options VerifyOptions) (VerifyResult, error) {
+	// The argument is interpolated straight into a URL path, so it is checked
+	// rather than trusted: exactly 20 bytes of hex, nothing else.
+	if sum, err := hex.DecodeString(options.ManifestSHA1); err != nil || len(sum) != sha1.Size {
+		return VerifyResult{}, fmt.Errorf("%q is not a manifest sha1", options.ManifestSHA1)
+	}
+	source := options.Source
+	if source == nil {
+		source = newPullZone(options.Base)
+	}
+	log := options.Log
+	if log == nil {
+		log = io.Discard
+	}
+
+	manifestKey := path.Join("manifests", options.ManifestSHA1)
+	data, err := source.get(manifestKey)
+	if err != nil {
+		return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
+	}
+	// A manifest is named after its own sha1, so this catches the zone serving
+	// a different manifest — or a truncated one — before any blob is fetched.
+	if got := hex.EncodeToString(sha1Sum(data)); got != options.ManifestSHA1 {
+		return VerifyResult{}, fmt.Errorf("%s hashes to %s, not the manifest asked for",
+			source.describe(manifestKey), got)
+	}
+	entries, err := readManifest(data)
+	if err != nil {
+		return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
+	}
+
+	// A manifest names one blob many times over: mappings share a sha1, and so
+	// do resrefs with identical contents. Fetch each blob once.
+	blobs := make([]Entry, 0, len(entries))
+	seen := make(map[[20]byte]bool, len(entries))
+	for _, entry := range entries {
+		if seen[entry.SHA1] {
+			continue
+		}
+		seen[entry.SHA1] = true
+		blobs = append(blobs, entry)
+	}
+
+	result := VerifyResult{Entries: len(entries), Blobs: len(blobs)}
+	checking := blobs
+	if options.Sample > 0 && options.Sample < len(blobs) {
+		// A full sweep of the live manifest is ~69,000 objects and ~15 GB, so
+		// sampling is what makes verifying a routine act rather than an event.
+		picks := rand.Perm(len(blobs))[:options.Sample]
+		checking = make([]Entry, 0, options.Sample)
+		for _, i := range picks {
+			checking = append(checking, blobs[i])
+		}
+	}
+	result.Checked = len(checking)
+
+	jobs := options.Jobs
+	if jobs < 1 {
+		jobs = defaultEmitJobs
+	}
+	var (
+		mu       sync.Mutex
+		failures []string
+	)
+	work := make(chan Entry)
+	var wg sync.WaitGroup
+	for range jobs {
+		wg.Go(func() {
+			for entry := range work {
+				fault := checkEntry(source, entry)
+				mu.Lock()
+				if fault != "" {
+					failures = append(failures, fault)
+				} else {
+					result.Bytes += int64(entry.Size)
+				}
+				mu.Unlock()
+			}
+		})
+	}
+	for _, entry := range checking {
+		work <- entry
+	}
+	close(work)
+	wg.Wait()
+
+	// Workers finish in any order; a report an operator can diff must not.
+	sort.Strings(failures)
+	for _, fault := range failures {
+		fmt.Fprintln(log, fault)
+	}
+	result.Failures = len(failures)
+	return result, nil
+}
+
+// checkEntry fetches one blob and returns a one-line fault, or "" if it is
+// exactly what the manifest entry says it is.
+func checkEntry(source blobSource, entry Entry) string {
+	// Name the resource, not just the hash: an operator has to find the thing
+	// in a hak, and a bare sha1 says nothing about where to look.
+	extension, ok := erf.ExtensionForResourceType(entry.ResType)
+	if !ok {
+		extension = strconv.Itoa(int(entry.ResType))
+	}
+	where := fmt.Sprintf("%s (%s.%s)", entry.sha1Hex(), entry.ResRef, extension)
+	blob, err := source.get(blobKey(entry.sha1Hex()))
+	if err != nil {
+		if errors.Is(err, errBlobMissing) {
+			return where + ": missing"
+		}
+		return where + ": unreadable: " + err.Error()
+	}
+	data, err := inspectBlob(blob)
+	if err != nil {
+		return where + ": " + err.Error()
+	}
+	if uint32(len(data)) != entry.Size {
+		return fmt.Sprintf("%s: size mismatch: %d bytes, manifest says %d", where, len(data), entry.Size)
+	}
+	if sha1.Sum(data) != entry.SHA1 {
+		return fmt.Sprintf("%s: hash mismatch: contents hash to %x", where, sha1.Sum(data))
+	}
+	return ""
+}
+
+func sha1Sum(data []byte) []byte {
+	sum := sha1.Sum(data)
+	return sum[:]
+}

internal/nwsync/verify_test.go

@@ -0,0 +1,285 @@
+package nwsync
+
+import (
+	"bytes"
+	"crypto/sha1"
+	"encoding/hex"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/klauspost/compress/zstd"
+)
+
+// verifyFixture emits two haks and a TLK into a fake zone, assembles them, and
+// hands back a verifier reading that zone the way a client would.
+type verifyFixture struct {
+	*zoneSinkFixture
+	manifestSHA1 string
+}
+
+func newVerifyFixture(t *testing.T) *verifyFixture {
+	t.Helper()
+	zone := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	// A payload under 256 bytes is the one the frame-header check exists for.
+	writeHak(t, hak, map[string][]byte{
+		"bloodstain1.tga": []byte("small"),
+		"appearance.2da":  bytes.Repeat([]byte("2DA V2.0\n"), 200),
+	})
+	tlk := filepath.Join(dir, "sow_tlk.tlk")
+	if err := os.WriteFile(tlk, []byte("TLK V3.0 payload"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	zone.emit(t, hak)
+	if _, err := Emit(EmitOptions{
+		ArtifactKey: artifactKey(t, tlk), ArtifactPath: tlk, As: "sow_tlk.tlk", Sink: zone.sink,
+	}); err != nil {
+		t.Fatalf("emit tlk: %v", err)
+	}
+	assembled, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{artifactKey(t, hak)}, TLKKey: artifactKey(t, tlk), Sink: zone.sink,
+	})
+	if err != nil {
+		t.Fatalf("assemble: %v", err)
+	}
+	return &verifyFixture{zoneSinkFixture: zone, manifestSHA1: assembled.SHA1}
+}
+
+func (f *verifyFixture) verify(t *testing.T, sample int) (VerifyResult, string, error) {
+	t.Helper()
+	var log bytes.Buffer
+	result, err := Verify(VerifyOptions{
+		ManifestSHA1: f.manifestSHA1,
+		Sample:       sample,
+		Source:       f.zone.pullZone(),
+		Log:          &log,
+	})
+	return result, log.String(), err
+}
+
+// keyOf is where a resource's blob lives, addressed by the sha1 of its
+// uncompressed bytes — the same path the client requests.
+func keyOf(body []byte) string {
+	sum := sha1.Sum(body)
+	return blobKey(hex.EncodeToString(sum[:]))
+}
+
+func TestVerifyPassesACleanZone(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 0 {
+		t.Errorf("verify reported %d failures on a clean zone: %s", result.Failures, log)
+	}
+	// A default run is a full sweep, so it must reach every blob the manifest
+	// names — not some of them.
+	if result.Checked != result.Blobs || result.Blobs == 0 {
+		t.Errorf("checked %d of %d blobs; a full sweep must check all of them", result.Checked, result.Blobs)
+	}
+}
+
+func TestVerifyReportsAMissingBlob(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	key := keyOf([]byte("small"))
+	fixture.zone.mu.Lock()
+	delete(fixture.zone.objects, key)
+	fixture.zone.mu.Unlock()
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "missing") {
+		t.Errorf("a deleted blob was not reported as missing: %s", log)
+	}
+}
+
+func TestVerifyReportsATruncatedBlob(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	key := keyOf([]byte("small"))
+	fixture.zone.mu.Lock()
+	fixture.zone.objects[key] = fixture.zone.objects[key][:blobHeaderBytes+4]
+	fixture.zone.mu.Unlock()
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "framing") {
+		t.Errorf("a truncated blob was not reported as malformed framing: %s", log)
+	}
+}
+
+// TestVerifyRejectsABlobWithNoDeclaredFrameContentSize is the check that #86
+// slipped past: the blob decompresses to exactly the right bytes, so a verifier
+// that only round-trips certifies it, yet the client cannot decode it.
+func TestVerifyRejectsABlobWithNoDeclaredFrameContentSize(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	body := []byte("small")
+	key := keyOf(body)
+
+	fixture.zone.mu.Lock()
+	good := fixture.zone.objects[key]
+	encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
+	if err != nil {
+		t.Fatal(err)
+	}
+	bad := append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
+	fixture.zone.objects[key] = bad
+	fixture.zone.mu.Unlock()
+
+	if frameDeclaresContentSize(bad[blobHeaderBytes:]) {
+		t.Fatal("the fixture blob declares a content size; it cannot exercise the check")
+	}
+	if got, err := decompressBlob(bad); err != nil || !bytes.Equal(got, body) {
+		t.Fatalf("the fixture blob must round trip, or it proves nothing: %v", err)
+	}
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "content size") {
+		t.Errorf("undeclared frame content size was not the reported reason: %s", log)
+	}
+}
+
+func TestVerifyReportsWrongContents(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	key := keyOf([]byte("small"))
+	fixture.zone.mu.Lock()
+	// Valid framing, valid zstd, wrong bytes: only decompressing and hashing
+	// can see this, which is why Content-Length is not enough.
+	fixture.zone.objects[key] = compressBlob([]byte("wrong"))
+	fixture.zone.mu.Unlock()
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "hash mismatch") {
+		t.Errorf("wrong contents were not reported as a hash mismatch: %s", log)
+	}
+}
+
+func TestVerifyReportsAShortBlob(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	key := keyOf([]byte("small"))
+	fixture.zone.mu.Lock()
+	// Well-formed all the way down and simply too short — the shape a killed
+	// upload leaves behind, and the one a Content-Length check would pass.
+	fixture.zone.objects[key] = compressBlob([]byte("sma"))
+	fixture.zone.mu.Unlock()
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "size mismatch") {
+		t.Errorf("a short blob was not reported as a size mismatch: %s", log)
+	}
+}
+
+func TestVerifyFailsWhenTheManifestIsNotTheOneAsked(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	fixture.zone.mu.Lock()
+	fixture.zone.objects["manifests/"+fixture.manifestSHA1] = []byte("NSYM garbage")
+	fixture.zone.mu.Unlock()
+
+	if _, _, err := fixture.verify(t, 0); err == nil {
+		t.Fatal("verify accepted a manifest that is not the one requested")
+	}
+}
+
+func TestVerifySampleChecksFewerBlobs(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	result, log, err := fixture.verify(t, 1)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Checked != 1 {
+		t.Errorf("--sample 1 checked %d blobs, want 1: %s", result.Checked, log)
+	}
+	if result.Blobs <= result.Checked {
+		t.Errorf("sampling %d of %d blobs is not a sample", result.Checked, result.Blobs)
+	}
+}
+
+// TestEmitVerifyReplacesABlobThatIsNotItsName covers the reason #86 could not be
+// fixed by the encoder alone: emit skips whatever is already present, so every
+// blob published by the broken encoder stays broken until emit stops trusting
+// presence.
+func TestEmitVerifyReplacesABlobThatIsNotItsName(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	body := []byte("blood")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": body})
+
+	emit := func(verify bool) EmitResult {
+		t.Helper()
+		result, err := Emit(EmitOptions{
+			ArtifactKey:  artifactKey(t, hak),
+			ArtifactPath: hak,
+			Sink:         fixture.sink,
+			Verify:       verify,
+		})
+		if err != nil {
+			t.Fatalf("emit (verify=%v): %v", verify, err)
+		}
+		return result
+	}
+
+	emit(false)
+	key := keyOf(body)
+	encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
+	if err != nil {
+		t.Fatal(err)
+	}
+	fixture.zone.mu.Lock()
+	good := fixture.zone.objects[key]
+	fixture.zone.objects[key] = append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
+	fixture.zone.mu.Unlock()
+
+	if plain := emit(false); plain.BlobsWritten != 0 {
+		t.Fatalf("a plain re-emit wrote %d blobs; it is supposed to trust presence", plain.BlobsWritten)
+	}
+	if verified := emit(true); verified.BlobsWritten != 1 {
+		t.Fatalf("--verify wrote %d blobs, want 1 (the bad copy must be replaced)", verified.BlobsWritten)
+	}
+
+	fixture.zone.mu.Lock()
+	repaired := fixture.zone.objects[key]
+	fixture.zone.mu.Unlock()
+	if !bytes.Equal(repaired, good) {
+		t.Error("the replaced blob is not what the current encoder produces")
+	}
+	if _, err := inspectBlob(repaired); err != nil {
+		t.Errorf("the replaced blob still fails inspection: %v", err)
+	}
+
+	// A second verifying run has nothing left to repair.
+	if again := emit(true); again.BlobsWritten != 0 {
+		t.Errorf("--verify rewrote %d good blobs", again.BlobsWritten)
+	}
+}

internal/nwsync/zone_test.go

@@ -0,0 +1,257 @@
+package nwsync
+
+import (
+	"crypto/sha1"
+	"crypto/sha256"
+	"encoding/hex"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+)
+
+// fakeZone is a Bunny-shaped object store: PUT stores, GET reads, and the
+// Checksum header is verified the way Bunny verifies it.
+type fakeZone struct {
+	mu      sync.Mutex
+	objects map[string][]byte
+	puts    []string
+	failOn  func(key string) bool // when true, the PUT fails
+	url     string                // base the same objects are readable at
+}
+
+// pullZone reads the fake zone the way the public pull zone is read: plain
+// unauthenticated GETs, no storage API.
+func (z *fakeZone) pullZone() blobSource {
+	return newPullZone(z.url)
+}
+
+func newFakeZone(t *testing.T) (*fakeZone, func(string) string) {
+	t.Helper()
+	zone := &fakeZone{objects: map[string][]byte{}}
+	server := httptest.NewServer(zone)
+	t.Cleanup(server.Close)
+	zone.url = server.URL + "/sow-nwsync"
+	getenv := func(name string) string {
+		switch name {
+		case "NWSYNC_STORAGE_ZONE":
+			return "sow-nwsync"
+		case "NWSYNC_STORAGE_PASSWORD":
+			return "write-key"
+		case "BUNNY_STORAGE_HOST":
+			return server.URL
+		}
+		return ""
+	}
+	return zone, getenv
+}
+
+func (z *fakeZone) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	key := strings.TrimPrefix(r.URL.Path, "/sow-nwsync/")
+	switch r.Method {
+	case http.MethodPut:
+		if z.failOn != nil && z.failOn(key) {
+			http.Error(w, "boom", http.StatusInternalServerError)
+			return
+		}
+		body, err := io.ReadAll(r.Body)
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+		sum := sha256.Sum256(body)
+		if want := strings.ToUpper(hex.EncodeToString(sum[:])); r.Header.Get("Checksum") != want {
+			http.Error(w, "checksum mismatch", http.StatusBadRequest)
+			return
+		}
+		z.mu.Lock()
+		z.objects[key] = body
+		z.puts = append(z.puts, key)
+		z.mu.Unlock()
+		w.WriteHeader(http.StatusCreated)
+	case http.MethodGet:
+		z.mu.Lock()
+		body, ok := z.objects[key]
+		z.mu.Unlock()
+		if !ok {
+			http.Error(w, "not found", http.StatusNotFound)
+			return
+		}
+		_, _ = w.Write(body)
+	default:
+		http.Error(w, "unsupported", http.StatusMethodNotAllowed)
+	}
+}
+
+func (z *zoneSinkFixture) emit(t *testing.T, path string) EmitResult {
+	t.Helper()
+	result, err := Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, path),
+		ArtifactPath: path,
+		Sink:         z.sink,
+	})
+	if err != nil {
+		t.Fatalf("emit %s: %v", path, err)
+	}
+	return result
+}
+
+type zoneSinkFixture struct {
+	zone *fakeZone
+	sink sink
+}
+
+func newZoneFixture(t *testing.T) *zoneSinkFixture {
+	t.Helper()
+	zone, getenv := newFakeZone(t)
+	target, err := newZoneSink(t.Context(), getenv)
+	if err != nil {
+		t.Fatalf("zone sink: %v", err)
+	}
+	return &zoneSinkFixture{zone: zone, sink: target}
+}
+
+func sha1Of(body []byte) [20]byte { return sha1.Sum(body) }
+
+func TestEmitUploadsBlobsThenIndex(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	body := []byte("texture bytes")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": body, "copy1.txi": body})
+
+	key := artifactKey(t, hak)
+	result := fixture.emit(t, hak)
+
+	index, err := indexKey(key)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, ok := fixture.zone.objects[index]; !ok {
+		t.Fatalf("no index at %s; zone holds %v", index, fixture.zone.puts)
+	}
+	if result.BlobsWritten != 1 {
+		t.Errorf("uploaded %d blobs, want 1 (identical content shares a blob)", result.BlobsWritten)
+	}
+
+	// The index is the publication marker, so it must land after every blob it
+	// names — including its own sidecar.
+	last := fixture.zone.puts[len(fixture.zone.puts)-1]
+	if last != index {
+		t.Errorf("index landed at position %d of %d; it must be last", len(fixture.zone.puts), len(fixture.zone.puts))
+	}
+	for _, key := range fixture.zone.puts[:len(fixture.zone.puts)-1] {
+		if strings.HasPrefix(key, "data/sha1/") || key == index+".json" {
+			continue
+		}
+		t.Errorf("unexpected object uploaded before the index: %s", key)
+	}
+}
+
+func TestEmitSkipsBlobsAlreadyInTheZone(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": []byte("blood")})
+
+	first := fixture.emit(t, hak)
+	if first.BlobsWritten != 1 {
+		t.Fatalf("first emit uploaded %d blobs, want 1", first.BlobsWritten)
+	}
+	second := fixture.emit(t, hak)
+	if second.BlobsWritten != 0 {
+		t.Errorf("re-emit uploaded %d blobs, want 0 (a blob name is its content)", second.BlobsWritten)
+	}
+}
+
+func TestEmitLeavesNoIndexWhenAnUploadFails(t *testing.T) {
+	fixture := newZoneFixture(t)
+	fixture.zone.failOn = func(key string) bool { return strings.HasPrefix(key, "data/sha1/") }
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": []byte("blood")})
+
+	_, err := Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, hak),
+		ArtifactPath: hak,
+		Sink:         fixture.sink,
+	})
+	if err == nil {
+		t.Fatal("emit reported success after an upload failed")
+	}
+	for key := range fixture.zone.objects {
+		if strings.HasSuffix(key, ".nsym") {
+			t.Errorf("a half-emitted artifact published an index: %s", key)
+		}
+	}
+}
+
+func TestEmitRejectsAKeyThatDoesNotMatchTheFile(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": []byte("blood")})
+
+	wrong := "artifacts/haks/sha256/00/11/" + strings.Repeat("0", 64) + ".hak"
+	_, err := Emit(EmitOptions{ArtifactKey: wrong, ArtifactPath: hak, Sink: fixture.sink})
+	if err == nil || !strings.Contains(err.Error(), "hashes to") {
+		t.Fatalf("emit published under a key that names another artifact: %v", err)
+	}
+}
+
+func TestAssembleReadsIndexesFromTheZone(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	topBody := []byte("2da from sow_top")
+	assetBody := []byte("2da from the asset hak")
+	top := filepath.Join(dir, "sow_top.hak")
+	core := filepath.Join(dir, "sow_core_01.hak")
+	writeHak(t, top, map[string][]byte{"appearance.2da": topBody})
+	writeHak(t, core, map[string][]byte{"appearance.2da": assetBody, "bloodstain1.tga": []byte("blood")})
+	tlkPath := filepath.Join(dir, "sow_tlk.tlk")
+	if err := os.WriteFile(tlkPath, []byte("TLK V3.0 payload"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	fixture.emit(t, top)
+	fixture.emit(t, core)
+	if _, err := Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, tlkPath),
+		ArtifactPath: tlkPath,
+		As:           "sow_tlk.tlk",
+		Sink:         fixture.sink,
+	}); err != nil {
+		t.Fatalf("emit tlk: %v", err)
+	}
+
+	result, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{artifactKey(t, top), artifactKey(t, core)},
+		TLKKey:       artifactKey(t, tlkPath),
+		GroupID:      2,
+		Sink:         fixture.sink,
+	})
+	if err != nil {
+		t.Fatalf("assemble: %v", err)
+	}
+	if result.Entries != 3 {
+		t.Fatalf("merged %d entries, want 3 (appearance.2da is shadowed, the TLK adds one)", result.Entries)
+	}
+
+	manifest, ok := fixture.zone.objects["manifests/"+result.SHA1]
+	if !ok {
+		t.Fatalf("no merged manifest in the zone; it holds %v", fixture.zone.puts)
+	}
+	entries, err := readManifest(manifest)
+	if err != nil {
+		t.Fatalf("parse merged manifest: %v", err)
+	}
+	for _, entry := range entries {
+		if entry.ResRef == "appearance" && entry.SHA1 != sha1Of(topBody) {
+			t.Errorf("appearance.2da resolved to the shadowed hak, not the first one given")
+		}
+	}
+}

internal/project/project.go

@@ -31,7 +31,7 @@ var SourceExtensions = []string{
 }
 
 var AssetExtensions = []string{
-	".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".gr2", ".itp", ".jpg", ".lod", ".lyt", ".mdb", ".mdl", ".mdx", ".mtr", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wlk", ".wok", ".xml",
+	".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".itp", ".jpg", ".lod", ".lyt", ".mdl", ".mdx", ".mtr", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wok",
 }
 
 var BuiltinScriptPrefixes = []string{

scripts/crucible-smoke.sh

@@ -18,7 +18,7 @@ bin=bin
 
 # Keep in sync with internal/dispatch.Registry (Wired flag).
 unwired=()
-wired=(assets depot hak module topdata wiki)
+wired=(assets depot hak module nwsync topdata wiki)
 
 exit_of() { set +e; "$@" >/dev/null 2>&1; local c=$?; set -e; echo "${c}"; }
 

scripts/prune-release-assets.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# Delete the binary assets of every release except the newest KEEP ones.
+#
+# Gitea has no asset retention, so 9 assets x ~57 MB per tag pile up forever on
+# the host disk (sow-tools #52). Nothing consumes an old asset: CI takes the
+# Crucible binary from the Nix flake, and every deleted file is reproducible
+# from its tag. Tags, source archives, release notes and the releases
+# themselves are never touched — only attachments.
+#
+# Nothing is excluded, SHA256SUMS and the wrappers included: the checksums are
+# only meaningful next to the binaries they cover, and the drift-check reads
+# the wrappers from the latest release, which always stays complete.
+#
+# Best-effort by design: a stale asset is cheaper than a blocked release, so
+# every API failure is a warning, never a non-zero exit.
+#
+# Env:
+#   API    repo API base, e.g. https://git.example/api/v1/repos/owner/repo  [required]
+#   TOKEN  Gitea token with releases:write                                  [required]
+#   KEEP   how many newest releases stay complete (default 2)
+set -uo pipefail
+
+: "${API:?set API}"
+: "${TOKEN:?set TOKEN}"
+keep="${KEEP:-2}"
+auth="Authorization: token ${TOKEN}"
+per_page=50
+
+warn() { echo "prune-release-assets: $*" >&2; }
+
+# Walk every page so an old backlog is cleared, not only the tag that fell out
+# of the window on this run. Each page already embeds its releases' assets, so
+# no follow-up request per release is needed.
+releases='[]'
+page=1
+while :; do
+  body="$(curl -fsS -H "$auth" "${API}/releases?limit=${per_page}&page=${page}&draft=false")" || {
+    warn "listing releases failed on page ${page}; pruning what was listed so far"
+    break
+  }
+  count="$(jq 'length' <<<"$body")" || { warn "unreadable release page ${page}"; break; }
+  releases="$(jq -s 'add' <(printf '%s' "$releases") <(printf '%s' "$body"))"
+  # A short page is the last one; this also stops a server that ignores `page`.
+  (( count < per_page )) && break
+  page=$(( page + 1 ))
+done
+
+# Sort here rather than trusting the response order, and skip drafts in case
+# the server ignored `draft=false` — a draft must not consume a keep slot and
+# strip the binaries off the newest real release.
+mapfile -t stale < <(jq -r --argjson keep "$keep" '
+  [.[] | select(.draft != true)]
+  | sort_by(.created_at) | reverse | .[$keep:]
+  | .[] | "\(.id) \(.assets // [] | map(.id) | join(","))"
+' <<<"$releases")
+
+(( ${#stale[@]} )) || { echo "prune-release-assets: nothing older than the newest ${keep} releases"; exit 0; }
+
+for line in "${stale[@]}"; do
+  id="${line%% *}"
+  assets="${line#* }"
+  for asset in ${assets//,/ }; do
+    echo "deleting asset ${asset} of release ${id}"
+    curl -fsS -X DELETE -H "$auth" "${API}/releases/${id}/assets/${asset}" >/dev/null \
+      || warn "deleting asset ${asset} of release ${id} failed"
+  done
+done

tests/prune-release-assets.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# Contract for scripts/prune-release-assets.sh, driven by a stub curl on PATH:
+# only assets of releases outside the keep window go, the window is decided by
+# release date rather than response order, drafts never consume a keep slot,
+# releases and tags stay, and an API failure never fails the run.
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+script="$repo_root/scripts/prune-release-assets.sh"
+tmp="$(mktemp -d)"
+trap 'rm -rf "$tmp"' EXIT
+
+# Stub curl: one short page of releases, deliberately out of date order and
+# with a draft that is newer than every published release. Each release embeds
+# two assets. Every request is appended to $CALLS for the assertions below.
+mkdir -p "$tmp/bin"
+cat >"$tmp/bin/curl" <<'STUB'
+#!/usr/bin/env bash
+url="${!#}"
+printf '%s\n' "$*" >>"$CALLS"
+rel() { printf '{"id":%s,"created_at":"%s","draft":%s,"assets":[{"id":%s01},{"id":%s02}]}' "$1" "$2" "$3" "$1" "$1"; }
+case "$url" in
+  *releases\?*)
+    [[ "${FAIL_LIST:-0}" == 1 ]] && exit 22
+    printf '[%s,%s,%s,%s,%s]\n' \
+      "$(rel 12 2024-01-02T00:00:00Z false)" \
+      "$(rel 10 2024-01-04T00:00:00Z false)" \
+      "$(rel 9  2024-06-01T00:00:00Z true)" \
+      "$(rel 13 2024-01-01T00:00:00Z false)" \
+      "$(rel 11 2024-01-03T00:00:00Z false)"
+    ;;
+  */assets/*) [[ "${FAIL_DELETE:-0}" == 1 ]] && exit 22 ;;
+esac
+exit 0
+STUB
+chmod +x "$tmp/bin/curl"
+export PATH="$tmp/bin:$PATH" API=https://git.example/api/v1/repos/o/r TOKEN=t
+
+run() { CALLS="$tmp/calls" bash "$script" >"$tmp/out" 2>"$tmp/err"; }
+
+# Default keep=2: the two newest published releases (10, 11) stay whole, the
+# older two (12, 13) lose their assets, and the newer draft is ignored.
+: >"$tmp/calls"; run
+deletes="$(grep -c -- '-X DELETE' "$tmp/calls" || true)"
+[[ "$deletes" == 4 ]] || { echo "expected 4 asset deletes, got $deletes" >&2; exit 1; }
+grep -q 'assets/1201' "$tmp/calls" && grep -q 'assets/1302' "$tmp/calls" || {
+  echo "expected assets of releases 12 and 13 to be deleted" >&2; exit 1; }
+if grep -q 'releases/10/assets/\|releases/11/assets/' "$tmp/calls"; then
+  echo "kept releases lost assets" >&2; exit 1
+fi
+if grep -q 'releases/9/assets/' "$tmp/calls"; then
+  echo "a draft release was pruned" >&2; exit 1
+fi
+# A release or tag must never be deleted, only attachments under /assets/.
+if grep -- '-X DELETE' "$tmp/calls" | grep -qv '/assets/'; then
+  echo "a non-asset DELETE was issued" >&2; exit 1
+fi
+# A short page ends the sweep: no second page, so a server ignoring `page`
+# cannot spin the job until the job timeout.
+[[ "$(grep -c 'releases?' "$tmp/calls")" == 1 ]] || {
+  echo "a short release page did not end the sweep" >&2; exit 1; }
+
+# KEEP=4 covers every published release: nothing to delete.
+: >"$tmp/calls"; KEEP=4 run
+if grep -q -- '-X DELETE' "$tmp/calls"; then echo "KEEP=4 still deleted" >&2; exit 1; fi
+
+# Failures are warnings, never a non-zero exit.
+: >"$tmp/calls"; FAIL_DELETE=1 run || { echo "delete failure failed the run" >&2; exit 1; }
+grep -q 'failed' "$tmp/err" || { echo "delete failure was not warned about" >&2; exit 1; }
+
+: >"$tmp/calls"; FAIL_LIST=1 run || { echo "listing failure failed the run" >&2; exit 1; }
+grep -q 'listing releases failed' "$tmp/err" || {
+  echo "listing failure was not warned about" >&2; exit 1; }
+if grep -q -- '-X DELETE' "$tmp/calls"; then
+  echo "deleted assets despite an unreadable release list" >&2; exit 1
+fi
+
+echo "prune-release-assets: prunes by release date outside the keep window, ignores drafts, never deletes a release, never fails the build"

tests/workflow-contract.sh

@@ -46,6 +46,8 @@ grep -Fqx '  releases: write' "$release"
 grep -Fqx '    timeout-minutes: 30' "$release"
 [[ "$(grep -Fc "$checkout" "$release")" -eq 1 ]]
 grep -Fq 'secrets.GITEA_TOKEN' "$release"
+grep -Fq 'scripts/prune-release-assets.sh' "$release"
+grep -Fq 'continue-on-error: true' "$release"
 if grep -Fq 'secrets.GITHUB_TOKEN' "$release"; then
   echo "workflow-contract: release uses the GitHub token alias" >&2
   exit 1