docs(nwsync): verify is what tells you which keys to purge

Running the repair for sow-tools#88 disproved the advice #90 had just
landed. "Purge the zone, then believe verify" assumed the stale set was
unknowable. It is not: verify reads the edge, so a run straight after a
repair names every key the edge is still serving stale — a survey, not a
verdict. Purge those, re-run, and the second run is the verdict.

The measured numbers are the argument. The repair rewrote 2,603 blobs at
the origin; 8 were stale at the edge, all of them ones a failed player
sync had pulled ninety minutes earlier. The edge only caches what someone
fetched, so purging the whole zone would have cooled 69,169 objects to
fix 8.

Refs #88, #89, #75.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.agents/skills/data-driven-design/SKILL.md

@@ -0,0 +1,55 @@
+---
+name: data-driven-design
+description: Use when adding a mode, switch case, config entry, special-cased name, or per-column/per-dataset handler to a tool that processes authored data — or when a tool "knows" a project's layout, dataset names, key prefixes, or format quirks and could not be open sourced as-is.
+---
+
+# Data-Driven Design
+
+The tool implements **general mechanisms**; the data declares **all specific knowledge**. Every time code names a thing that lives in data — a dataset, a column, a format quirk — the tool stops being a tool and becomes a private extension of one project's layout.
+
+**The test:** could this repo be open sourced and build a *different* project's data without code changes? Every hardcoded name, mode, and special case is a "no".
+
+## The two smells
+
+**1. The mode zoo.** A new data shape arrives and you add a named mode plus a switch case:
+
+```yaml
+- {column: AvailableHeadsMale, mode: external_hex_list, hex_width: 3}
+- {column: PreferredAlignments, mode: alignment_set_mask, hex_width: 3}
+- {column: ProficiencyFeats,   mode: id_hex_list,        hex_width: 4}
+```
+
+Three "modes" are one concept — *a set of ids with an engine-side encoding* — wearing three names. Each mode is code the data could have been. The fix is a smaller vocabulary of orthogonal primitives declared where the column is defined:
+
+```json
+"EquipSlots": {"shape": "id_set", "encode": "bitmask", "max": 31}
+```
+
+One parser for `id_set`, one encoder per encoding. Adding the next column is a data edit, zero code. The mode zoo *shrinks*: N special cases collapse into shape × encoding.
+
+**2. Names in code.** `datasetRows("skills")`, `case "skills":`, `TrimPrefix(key, "skills:")` — the tool assumes the project's layout. When data moves (`skills` → `skills/core`), the tool breaks even though the data is self-consistent. Names must flow *from* the data: the spec that references a dataset names it; a key's namespace is whatever precedes its own colon. If code needs a name, some piece of data already knows it — read it from there.
+
+## Deciding
+
+Ask of every constant, case, and config knob: **whose knowledge is this?**
+
+- *The mechanism's* (how to parse JSON, allocate rows, write a table) → code.
+- *The data's* (which columns exist, how the engine encodes them, what things are called) → data, even if code is faster to write today.
+
+Generalize only when it deletes: a primitive that replaces N modes is lazy; a framework "for future shapes" is not. One new shape may take its case *if* the case is spelled as a reusable primitive the next shape can declare.
+
+## Rationalizations
+
+| Excuse | Reality |
+|---|---|
+| "Just follow the existing pattern, it's proven" | The pattern *is* the defect. Each repetition raises the cost of the real fix. |
+| "N cases isn't a crisis yet" | The crisis is per-case: every one couples the tool to one project forever. |
+| "Release is tonight, generalize later" | Later never comes; tonight's mode is tomorrow's baseline. Declaring shape in data is usually the *same* hour of work. |
+| "This name never changes" | Today's rename broke exactly such a name. Data moves; mechanisms shouldn't care. |
+
+## Red flags — stop and re-shape
+
+- A config entry that names both a dataset and a column to select behavior
+- A new value in a `mode`/`kind`/`type` string enum with its own switch arm
+- A literal in code that also appears in the data tree (`"skills"`, `"skills:"`)
+- Per-project constants (row ranges, hex widths, id ceilings) living in Go instead of the dataset that owns them

.gitea/workflows/build-binaries.yml

@@ -1,25 +1,22 @@
-# Cross-platform Crucible binaries (D7 trigger standard):
+# Cross-platform Crucible release binaries: a v* tag builds all targets,
-#   PR / push main -> cross-build ALL targets to prove they compile (no publish)
+# then uploads them, SHA256SUMS, and the canonical wrappers to its Gitea release.
-#   tag v*         -> build all targets + SHA256SUMS, upload to the Gitea release
 # Crucible is pure Go (CGO_ENABLED=0), so cross-compiling is a fast loop.
 name: build-binaries
 
 on:
-  pull_request:
   push:
-    branches: [main]
     tags: ['v*']
-    paths-ignore:
+
-      - "docs/**"
+permissions:
-      - "README.md"
+  code: read
-      - "AGENTS.md"
+  releases: write
-      - "LICENSE"
 
 jobs:
   build-binaries:
     runs-on: nix-docker
+    timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
         with: { fetch-depth: 0 }
 
       - name: Cross-build all targets
@@ -45,9 +42,8 @@ jobs:
           '
 
       - name: Upload to Gitea release
-        if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
         env:
-          TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          TOKEN: ${{ secrets.GITEA_TOKEN }}
           TAG: ${{ github.ref_name }}
           REPO: ${{ github.repository }}
           SERVER: ${{ github.server_url }}
@@ -69,3 +65,17 @@ jobs:
                 "${api}/releases/${id}/assets?name=$(basename "$f")"
             done
           '
+
+      # Gitea has no asset retention (#52): without this every tag keeps its
+      # ~57 MB binary set forever. Best-effort — a stale asset is cheaper than
+      # a blocked publish, so a failure here never fails the release.
+      - name: Prune assets of older releases
+        continue-on-error: true
+        env:
+          TOKEN: ${{ secrets.GITEA_TOKEN }}
+          REPO: ${{ github.repository }}
+          SERVER: ${{ github.server_url }}
+        run: |
+          nix develop --command bash -c '
+            API="${SERVER}/api/v1/repos/${REPO}" scripts/prune-release-assets.sh
+          '

.gitea/workflows/build-image.yml

@@ -1,52 +0,0 @@
-# Publish the immutable crucible:<sha> image on version tags.
-# PR/main test builds live in test-image.yml and never publish, so registry
-# packages are only generated for releases we intend to use (not on every merge).
-#
-# Daemonless: the host-mode runner has no container runtime, so the image is
-# built by Nix (`nix build .#image`, see flake.nix) and pushed with skopeo
-# straight from the OCI tarball. No `docker build`/`docker login` involved.
-name: build-image
-
-on:
-  push:
-    tags: ['v*']
-
-env:
-  REGISTRY: registry.westgate.pw
-  IMAGE: deployment/crucible
-
-jobs:
-  publish:
-    runs-on: nix-docker
-    steps:
-      - uses: actions/checkout@v4
-        with: { fetch-depth: 0 }
-
-      - name: Resolve tag
-        id: tag
-        run: echo "sha=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT"
-
-      - name: Build OCI image (daemonless)
-        run: nix build .#image
-
-      # This workflow only runs on v* tags, so every run is a release publish.
-      - name: Publish image
-        env:
-          REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
-          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
-        run: |
-          nix shell nixpkgs#skopeo -c skopeo copy \
-            --dest-creds "${REGISTRY_USER}:${REGISTRY_PASSWORD}" \
-            docker-archive:result \
-            "docker://${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}"
-
-      - name: Emit release fragment
-        run: |
-          FRAG_REPO=sow-tools \
-          FRAG_SHA=${{ steps.tag.outputs.sha }} \
-          FRAG_ARTIFACT="${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}" \
-          FRAG_URL="${REGISTRY}/${IMAGE}" \
-          FRAG_RUN_ID=${{ gitea.run_id }} \
-          bash scripts/emit-release-fragment.sh
-      - uses: actions/upload-artifact@v3
-        with: { name: release-fragment, path: release-fragment.json }

.gitea/workflows/ci.yml

@@ -0,0 +1,47 @@
+# Pull-request validation for Crucible. Releases and wrapper synchronization
+# have their own narrow workflows because they need tag/main events.
+name: ci
+
+on:
+  pull_request:
+
+permissions: read-all
+
+jobs:
+  ci:
+    runs-on: nix-docker
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        with:
+          fetch-depth: 0
+
+      - name: vet + test + lint
+        run: |
+          nix develop --command bash -c '
+            set -euo pipefail
+            go vet ./...
+            go test ./...
+            shellcheck scripts/*.sh
+            yamllint .gitea
+          '
+
+      - name: binary smoke (fail-closed contract)
+        run: nix develop --command make smoke
+
+      - name: Cross-build all targets
+        run: |
+          nix develop --command bash -c '
+            set -euo pipefail
+            sha="$(git rev-parse --short=12 HEAD)"
+            ldflags="-s -w -X git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo.Version=${sha}"
+            rm -rf dist && mkdir -p dist
+            export CGO_ENABLED=0
+            for target in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64 windows/amd64 windows/arm64; do
+              os="${target%/*}"; arch="${target#*/}"
+              ext=""; [ "$os" = windows ] && ext=".exe"
+              echo "building crucible-${os}-${arch}${ext}"
+              GOOS="$os" GOARCH="$arch" go build -trimpath -ldflags "$ldflags" \
+                -o "dist/crucible-${os}-${arch}${ext}" ./cmd/crucible
+            done
+          '

.gitea/workflows/publish-image.yml

@@ -1,49 +0,0 @@
-# Manual compatibility publisher for the Crucible image.
-# Normal release publishing happens in build-image.yml on v* tags; this is the
-# break-glass / re-publish path, triggered by hand.
-#
-# Daemonless: built by Nix (`nix build .#image`) and pushed with skopeo from the
-# OCI tarball. No `docker build`/`docker login` involved.
-name: publish-image
-
-on:
-  workflow_dispatch:
-
-env:
-  REGISTRY: registry.westgate.pw
-  IMAGE: deployment/crucible
-
-jobs:
-  publish:
-    runs-on: nix-docker
-    steps:
-      - uses: actions/checkout@v4
-        with: { fetch-depth: 0 }
-
-      - name: Resolve tag
-        id: tag
-        run: echo "sha=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT"
-
-      - name: Build OCI image (daemonless)
-        run: nix build .#image
-
-      - name: Publish image
-        env:
-          REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
-          REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
-        run: |
-          nix shell nixpkgs#skopeo -c skopeo copy \
-            --dest-creds "${REGISTRY_USER}:${REGISTRY_PASSWORD}" \
-            docker-archive:result \
-            "docker://${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}"
-
-      - name: Emit release fragment
-        run: |
-          FRAG_REPO=sow-tools \
-          FRAG_SHA=${{ steps.tag.outputs.sha }} \
-          FRAG_ARTIFACT="${REGISTRY}/${IMAGE}:${{ steps.tag.outputs.sha }}" \
-          FRAG_URL="${REGISTRY}/${IMAGE}" \
-          FRAG_RUN_ID=${{ gitea.run_id }} \
-          bash scripts/emit-release-fragment.sh
-      - uses: actions/upload-artifact@v3
-        with: { name: release-fragment, path: release-fragment.json }

.gitea/workflows/sync-wrappers.yml

@@ -14,11 +14,14 @@ on:
       - 'wrappers/crucible.sh'
       - 'wrappers/crucible.ps1'
 
+permissions: read-all
+
 jobs:
   sync:
     runs-on: nix-docker
+    timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
 
       - name: Open sync PRs to consumers
         env:

.gitea/workflows/test-image.yml

@@ -1,19 +0,0 @@
-# Test-build the Crucible image on PRs. Proves `nix build .#image` still works
-# (daemonless, Nix-built OCI tarball) but does NOT publish — release publishing
-# happens in build-image.yml on v* tags. PR-only: with up-to-date-before-merge
-# protection, main == the tested PR head, so a throwaway post-merge rebuild that
-# publishes nothing is pure waste.
-name: test-image
-
-on:
-  pull_request:
-
-jobs:
-  build-image:
-    runs-on: nix-docker
-    steps:
-      - uses: actions/checkout@v4
-        with: { fetch-depth: 0 }
-
-      - name: Build OCI image (daemonless, no publish)
-        run: nix build .#image

.gitea/workflows/test.yml

@@ -1,30 +0,0 @@
-# Lint + unit tests for the Crucible suite. PR-first: PRs + push to main (D7).
-name: test
-
-on:
-  push:
-    branches: [main]
-    paths-ignore:
-      - "docs/**"
-      - "README.md"
-      - "AGENTS.md"
-      - "LICENSE"
-  pull_request:
-
-jobs:
-  test:
-    runs-on: nix-docker
-    steps:
-      - uses: actions/checkout@v4
-        with: { fetch-depth: 0 }
-      - name: vet + test + lint
-        run: |
-          nix develop --command bash -c '
-            set -euo pipefail
-            go vet ./...
-            go test ./...
-            shellcheck scripts/*.sh
-            yamllint .gitea
-          '
-      - name: binary smoke (fail-closed contract)
-        run: nix develop --command make smoke

AGENTS.md

@@ -41,13 +41,14 @@ section and `.gitea/workflows/`.
 - Dev loop: `nix develop`, then `make check` / `make build` / `make smoke`
   (see Commands below).
 - Release: push a `v*` tag. CI uploads cross-built binaries + wrappers to the
-  Gitea release and publishes the `crucible` container image.
+  Gitea release. There is no container image; Crucible ships as binaries,
+  wrappers, and the Nix input.
 - Consumers (they download released binaries via the wrapper; they never
   vendor a toolkit):
   - sow-module — https://git.westgate.pw/ShadowsOverWestgate/sow-module
   - sow-topdata — https://git.westgate.pw/ShadowsOverWestgate/sow-topdata
   - sow-assets-manifest — https://git.westgate.pw/ShadowsOverWestgate/sow-assets-manifest
-  - sow-platform (deploys the released image/pins) —
+  - sow-platform (infra/deploy authority) —
     https://git.westgate.pw/ShadowsOverWestgate/sow-platform
 
 ## What this repo owns / does not own
@@ -62,7 +63,7 @@ is `sow-platform`).
 
 1. **Fail closed, never fake.** A builder with no migrated logic yet (`depot`)
    exits `70`. Do not stub a builder to emit a placeholder artifact.
-2. **Binaries are not committed.** They are CI artifacts / image layers.
+2. **Binaries are not committed.** They are CI artifacts.
    `/bin/`, `*.exe`, `nwn-tool`, `sow-toolkit` are gitignored.
 3. **The registry is the command surface.** `internal/dispatch.Registry` is the
    single source of truth; keep it in sync with `cmd/` and
@@ -81,9 +82,26 @@ is `sow-platform`).
 nix develop && make check   # vet + test + shellcheck + yamllint
 make build                  # cmd/* -> ./bin
 make smoke                  # assert fail-closed contract
-make image                  # crucible:<sha>
 ```
 
 ## Tests
 
 Tests must survive harmless changes to constants, defaults, wording, ordering, fixture data, and internal implementation details. A test that fails merely because a basic value changed is usually a bad test. Only assert exact values when the value is part of a documented public contract, external protocol, compatibility requirement, security rule, migration, or business rule.
+
+## Agent skills
+
+### Issue tracker
+
+Issues live in Gitea at git.westgate.pw (`ShadowsOverWestgate/sow-tools`), managed with the `tea` CLI. Issues follow ownership — file work in the repo that owns it, not the one you happen to be standing in. See `docs/agents/issue-tracker.md`.
+
+### Triage labels
+
+Default label vocabulary (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
+
+### Domain docs
+
+Single-context: `CONTEXT.md` at the repo root plus `docs/adr/`. See `docs/agents/domain.md`.
+
+### Where work lives
+
+Markdown here is **reference, law, or an ADR — nothing else** (`sow-codebase` ADR-0001). Live work -> wayfinder maps + Gitea issues (closeable, assignable, queryable). Settled decisions -> ADR files in `docs/adr/` (immutable, never closed, only superseded). Standing law -> `DOCTRINE.md` / `AGENTS.md` / `CONTEXT.md`. Current-state reference -> docs describing what the code does now. Everything else — plans, specs, concepts, handoffs, trackers — is process: it belongs in a Gitea issue, not a file. Harvest unfinished intent to an issue before deleting a process doc. `sow-docs` is deprecated and read-only.

Makefile

@@ -1,11 +1,13 @@
 SHELL := bash
 .ONESHELL:
-.PHONY: check test vet build smoke fmt image
+.PHONY: check test vet build smoke fmt
 
 # Lint + unit tests. Mirrors the `test` CI job; runs green inside `nix develop`.
 check: vet test
 	shellcheck scripts/*.sh
 	yamllint .gitea
+	bash tests/workflow-contract.sh
+	bash tests/prune-release-assets.sh
 
 vet:
 	go vet ./...
@@ -22,9 +24,3 @@ smoke: build
 
 fmt:
 	gofmt -l -w cmd internal
-
-# Build the Crucible image locally. Requires docker; mirrors build-image CI.
-IMAGE ?= crucible
-GIT_SHA ?= $(shell git rev-parse --short=12 HEAD 2>/dev/null || echo unknown)
-image:
-	docker build --build-arg GIT_SHA=$(GIT_SHA) -f docker/Dockerfile -t $(IMAGE):$(GIT_SHA) .

README.md

@@ -19,6 +19,7 @@ Crucible is how the artifact repos turn source into artifacts.
 | `crucible-depot`   | `crucible depot`   | content-addressed depot blob verify/move        |
 | `crucible-hak`     | `crucible hak`     | ERF/HAK pack/unpack + hak manifests             |
 | `crucible-module`  | `crucible module`  | build/extract/validate/compare the `.mod`       |
+| `crucible-nwsync`  | `crucible nwsync`  | NWSync blob emit + manifest assemble + verify   |
 | `crucible-topdata` | `crucible topdata` | compile 2da/tlk topdata + packages              |
 | `crucible-wiki`    | `crucible wiki`    | render + deploy mechanical wiki pages           |
 
@@ -71,25 +72,23 @@ nix develop            # Go + shellcheck + yamllint + make
 make check             # go vet + go test + shellcheck + yamllint
 make build             # build every cmd/* into ./bin (gitignored)
 make smoke             # build + assert the fail-closed contract
-make image             # docker build -> crucible:<sha>
 ```
 
-Binaries are **never committed** — they are CI artifacts / image layers (D19).
+Binaries are **never committed** — they are CI artifacts (D19).
 This retires the old habit of checking in `nwn-tool` / `sow-toolkit`.
 
 ## CI
 
-PR-first (D7): checks run on pull requests and on push to `main`; the only
+PR-first (D7): checks run once on pull requests; the only publish event is a
-publish event is a `v*` tag (see `runbooks/ci-trigger-standard.md` in sow-docs,
+`v*` tag (see `runbooks/ci-trigger-standard.md` in sow-docs,
 https://git.westgate.pw/ShadowsOverWestgate/sow-docs).
 
-- `test.yml` — vet, test, shellcheck, yamllint, binary smoke (PR + main).
+- `ci.yml` — vet, test, shellcheck, yamllint, binary smoke, and cross-build all
-- `test-image.yml` — build the OCI image to prove it compiles (PR + main, no push).
+  targets once per pull request.
-- `build-binaries.yml` — cross-build all targets (PR + main); on a `v*` tag, upload
+- `build-binaries.yml` — on a `v*` tag, cross-build and upload the binaries,
-  the binaries, `SHA256SUMS`, and the wrappers to the Gitea release.
+  `SHA256SUMS`, and the wrappers to the Gitea release, then delete the assets
-- `build-image.yml` — on a `v*` tag, build and publish
+  of every release except the newest two — Gitea keeps them forever otherwise,
-  `registry.westgate.pw/deployment/crucible:<sha>`.
+  and every binary is reproducible from its tag.
-- `publish-image.yml` — manual `workflow_dispatch` break-glass republish.
 - `sync-wrappers.yml` — on a `main` push that touches `wrappers/`, auto-PR the
   canonical wrappers to the consumer repos in `wrappers/consumers.txt`.
   Consumer drift checks run after those PRs merge to `main`, not on the PRs

cmd/crucible-nwsync/main.go

@@ -0,0 +1,11 @@
+// Command crucible-nwsync is the standalone nwsync builder (equivalent to
+// `crucible nwsync`). A single-token binary keeps consumer wrapper scripts simple.
+package main
+
+import (
+	"os"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/dispatch"
+)
+
+func main() { os.Exit(dispatch.RunBuilder("nwsync", os.Args[1:])) }

docker/Dockerfile

@@ -1,38 +0,0 @@
-# syntax=docker/dockerfile:1
-#
-# Crucible toolchain image: registry.westgate.pw/deployment/crucible:<git-sha>
-#
-# Reproducible multi-stage build, no on-VPS build. Produces every cmd/* binary
-# and ships them on a static base. The `crucible` dispatcher is the entrypoint;
-# consumer CI can also call the standalone crucible-<name> binaries by path.
-#
-FROM golang:1.26-alpine AS build
-WORKDIR /src
-RUN apk add --no-cache git
-# go.sum is committed now that the migrated packages pull golang.org/x/text and
-# gopkg.in/yaml.v3; the glob keeps the build working if it is ever absent.
-COPY go.mod go.sum* ./
-RUN go mod download
-COPY . .
-ARG GIT_SHA=unknown
-ENV CGO_ENABLED=0
-RUN set -eux; \
-    mkdir -p /out; \
-    for dir in ./cmd/*/; do \
-      name="$(basename "${dir}")"; \
-      go build -trimpath \
-        -ldflags "-s -w -X git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo.Version=${GIT_SHA}" \
-        -o "/out/${name}" "${dir}"; \
-    done
-
-FROM debian:12-slim
-# ca-certificates: builders fetch published manifests over HTTPS.
-RUN set -eux; \
-    apt-get update; \
-    apt-get install -y --no-install-recommends ca-certificates; \
-    rm -rf /var/lib/apt/lists/*; \
-    useradd --system --create-home --uid 65532 nonroot
-COPY --from=build /out/ /usr/local/bin/
-USER nonroot
-ENTRYPOINT ["/usr/local/bin/crucible"]
-CMD ["help"]

docs/adr/0001-markdown-is-reference-or-law-only.md

@@ -0,0 +1,55 @@
+# ADR-0001: Markdown in this repo is reference, law, or an ADR — nothing else
+
+- **Status:** Accepted
+- **Date:** 2026-07-24
+- **Origin:** Adopted workspace-wide from `sow-codebase` ADR-0001, which records
+  the full context (a `docs/` tree that had grown to 434 files, ~27MB, most of
+  it dead process artifacts). This repo adopts the same law so the rule is
+  local, not a cross-repo reference.
+
+## Context
+
+Markdown that *claims to describe current state or a plan* rots, because
+reality diverges and nobody edits the file. Markdown that claims neither — a
+dated, immutable decision record, or a standing rule — does not rot.
+
+Two different things get conflated:
+
+- **Rationale** — why a system is shaped the way it is. Worth keeping.
+- **Process artifacts** — plans, specs, concepts, handoffs, trackers. A record
+  of how work happened, not what is true now.
+
+Left unchecked, every completed effort leaves its planning behind and the repo
+accumulates a "historical" pile that agents and humans must route around to
+find the few live docs.
+
+## Decision
+
+Markdown may exist in this repo only as one of three genres:
+
+1. **Current-state reference** — describes what the code does now, kept honest
+   by code review touching it.
+2. **Standing law** — `DOCTRINE.md`, root and folder-scoped `AGENTS.md`,
+   `CONTEXT.md`. States rules and vocabulary, not plans.
+3. **Immutable decision record** — ADRs under `docs/adr/`. Append-only; never
+   edited, only superseded by a later ADR that points back.
+
+Anything else — implementation plans, design specs, concepts, handoffs,
+progress trackers, scratch — is **process** and does not live in the repo. It
+lives in Gitea issues and wayfinder maps, where it can be assigned, closed, and
+superseded. Small tasks need no written plan at all.
+
+Deleting a process document is not destroying history: `git log -- <path>`
+recovers it. The exception is *unfinished intent* (a design never built, an
+open question still wanted) — that is live, not history, and must be harvested
+to a Gitea ticket before its file is deleted. A citation from a living doc or
+from code must be resolved before the cited file is deleted.
+
+## Consequences
+
+- The documentation map lists only current truth; there is no "historical" pile
+  to route around.
+- No agent, under any plugin or skill, writes process-markdown into the repo.
+  The rule is plugin-agnostic on purpose — it binds the agent, not a named tool.
+- History of deleted process docs is in git; unfinished intent is in the issue
+  tracker; rationale is in ADRs and law. Each thing has exactly one home.

docs/adr/0002-sow-tools-becomes-the-crucible-suite.md

@@ -0,0 +1,12 @@
+# ADR-0002: `sow-tools` becomes the Crucible suite
+
+- **Status:** Accepted
+- **Date:** 2026-06-11
+- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D11**, on the
+  retirement of `sow-docs`. Content is the original decision, unchanged.
+
+- **Decision:** Rename the future toolchain surface to Crucible: one Go module,
+  multiple `cmd/` binaries, plus a dispatcher so wrapper commands can stay
+  stable.
+- **Rationale:** Depot, HAK, module, topdata, and wiki tooling share internals
+  but have different command surfaces. One module avoids premature repo splits.

docs/adr/0003-crucible-binary-contract-standalone-shims-fail-closed-scaffold.md

@@ -0,0 +1,29 @@
+# ADR-0003: Crucible binary contract: standalone shims, fail-closed scaffold, no committed binaries
+
+- **Status:** Accepted
+- **Date:** 2026-06-11
+- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D19**, on the
+  retirement of `sow-docs`. Content is the original decision, unchanged.
+
+- **Decision:** Phase 5 builds `migration/sow-tools` (Crucible, D11) as a
+  multi-binary Go scaffold:
+  - One `crucible` dispatcher plus standalone `crucible-{depot,hak,module,topdata,wiki}`
+    binaries sharing one registry (`internal/dispatch`). The dispatcher is for
+    humans/CI; the **standalone shims are canonical for consumers** so wrapper
+    scripts resolve a single-token command and `"$builder" args` quoting stays
+    correct.
+  - Consumer resolution order: explicit env override
+    (`$SOW_MODULE_BUILD`/`$SOW_TOPDATA_BUILD`/`$CRUCIBLE`) → `crucible-<name>` →
+    legacy `sow-<name>-build`. CI runs inside the pinned `crucible:<sha>` image.
+  - Every builder is **unwired** in the scaffold and fails closed (exit 70);
+    it never fakes an artifact. The `internal/` logic is migrated from
+    `gitea/sow-tools` by the operator at cutover (hard rule: no source
+    transplant by tooling).
+  - **Binaries are never committed** — they are CI artifacts / image layers.
+    Retires the committed `nwn-tool` / `tools/sow-toolkit`.
+  - Builders take `NWN_ROOT` only via explicit env/flag; no `$HOME` defaulting.
+- **Rationale:** Locks the command surface, container, and CI shape so migrated
+  logic drops into a stable frame; aligns the three artifact repos onto one
+  Crucible contract while keeping back-compat with the pre-D11 skeletons.
+- **Image name:** `registry.westgate.pw/deployment/crucible:<git-sha>` (the bare
+  `crucible:<sha>` is the local/dev tag).

docs/adr/0004-nix-built-oci-images-start-server-side-with.md

@@ -0,0 +1,24 @@
+# ADR-0004: Nix-built OCI images start server-side with Crucible; local Anvil images deferred
+
+- **Status:** Accepted
+- **Date:** 2026-06-12
+- **Migrated from:** `sow-docs` `07-decisions-log.md` entry **D29**, on the
+  retirement of `sow-docs`. Content is the original decision, unchanged.
+
+- **Decision:** Introduce Nix-built OCI images as an artifact-production option,
+  starting with `sow-tools`/Crucible in server-side CI. `sow-tools` will grow a
+  Nix-built `crucible` package plus `crucible-image`; PR CI builds/smokes it and
+  `main` publishes the normal pinned OCI image
+  `registry.westgate.pw/deployment/crucible:<sha>`. `docker/Dockerfile` stays during
+  parity and as the client-compatible fallback.
+- **Local rule:** Nix users may get optional wrappers that provide tools and
+  call the existing Dockerfiles for local Anvil/NWServer testing. A true
+  `nix build .#nwserver-test-image` is explicitly deferred until real
+  `sow-codebase/src` exists and the Dockerfile image is proven.
+- **Rationale:** Server-side image build shape is harder to change once deploy
+  promotion and registry gates are active, so prove Nix image builds before
+  cutover. Crucible is the lowest-risk pilot because it is a Go toolchain image;
+  NodeBB and Anvil/NWServer depend more heavily on upstream container filesystem
+  semantics.
+- **Non-goals:** no source builds in `sow-platform`; no `nix-sidecar` or
+  Kubernetes-style runtime cache layer; no removal of client Dockerfiles.

docs/agents/domain.md

@@ -0,0 +1,51 @@
+# Domain Docs
+
+How the engineering skills should consume this repo's domain documentation when exploring the codebase.
+
+## Before exploring, read these
+
+- **`CONTEXT.md`** at the repo root, or
+- **`CONTEXT-MAP.md`** at the repo root if it exists — it points at one `CONTEXT.md` per context. Read each one relevant to the topic.
+- **`docs/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src/<context>/docs/adr/` for context-scoped decisions.
+
+If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
+
+## File structure
+
+Single-context repo (most repos):
+
+```
+/
+├── CONTEXT.md
+├── docs/adr/
+│   ├── 0001-event-sourced-orders.md
+│   └── 0002-postgres-for-write-model.md
+└── src/
+```
+
+Multi-context repo (presence of `CONTEXT-MAP.md` at the root):
+
+```
+/
+├── CONTEXT-MAP.md
+├── docs/adr/                          ← system-wide decisions
+└── src/
+    ├── ordering/
+    │   ├── CONTEXT.md
+    │   └── docs/adr/                  ← context-specific decisions
+    └── billing/
+        ├── CONTEXT.md
+        └── docs/adr/
+```
+
+## Use the glossary's vocabulary
+
+When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary explicitly avoids.
+
+If the concept you need isn't in the glossary yet, that's a signal — either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`).
+
+## Flag ADR conflicts
+
+If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
+
+> _Contradicts ADR-0007 (event-sourced orders) — but worth reopening because…_

docs/agents/issue-tracker.md

@@ -0,0 +1,88 @@
+# Issue tracker: Gitea (via tea)
+
+Issues for this repo live in Gitea at `git.westgate.pw`, repo
+`ShadowsOverWestgate/sow-tools`. Use the `tea` CLI for all operations —
+`gh` does not work here. For anything `tea` lacks a subcommand for, use
+`tea api <endpoint>` (Gitea's API mirrors GitHub's closely).
+
+Authenticate with your own `tea` login (`tea login add`); never commit tokens
+or tea config into this repo. Note Gitea blocks self-review, so approving a PR
+needs a different account than the one that opened it.
+
+## Where work lives
+
+Markdown in this repo is **reference, law, or an ADR — nothing else**
+(`sow-codebase` ADR-0001). Four homes, no overlap:
+
+- **Live work** → wayfinder maps + Gitea issues. Closeable, assignable,
+  queryable. Never a markdown file.
+- **Settled decisions** → ADR files in `docs/adr/`. Immutable, findable, never
+  closed, never edited — only superseded by a later ADR pointing back. When an
+  issue ends in a durable decision, write the ADR, then close the issue
+  pointing at it. Decisions spanning repos go to `sow-platform/docs/adr/`.
+- **Standing law** → `DOCTRINE.md`, `AGENTS.md`, `CONTEXT.md`. Rules that are
+  always true and vocabulary everyone shares — not the record of one decision.
+- **Current-state reference** → docs describing what the code does now, kept
+  honest by review touching them.
+
+Anything else — implementation plans, design specs, concepts, handoffs,
+progress trackers, scratch — is **process**. It does not live in this repo. It
+lives in a Gitea issue or a wayfinder map, where it can be assigned, closed,
+and superseded. Small tasks need no written plan at all.
+
+Deleting a process doc is not destroying history — `git log -- <path>` recovers
+it. But *unfinished intent* (a design never built, an open question still
+wanted) is live, not history: harvest it to a Gitea issue before deleting.
+
+`sow-docs` is deprecated and read-only. Never add to it, never send work there.
+
+## Which repo gets the issue
+
+Issues follow ownership. File the issue in the repo that **owns the work** —
+see the Repo/Owns/Produces table in the workspace root `AGENTS.md`. Standing
+in one repo is not a reason to file there.
+
+If work spans repos, file it in the repo that owns the *outcome* and reference
+the others from it. A wrong-repo issue is a routing bug, not a filing
+preference — move it.
+
+## Conventions
+
+- **Create an issue**: `tea issues create --title "..." --description "..."`
+- **Read an issue**: `tea issues <number>` and
+  `tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/comments` for comments.
+- **List issues**: `tea issues list --state open` (add `--labels ...` to filter).
+- **Comment**: `tea comment <number> "..." </dev/null`
+  Always redirect stdin. `tea` reads stdin to EOF and appends it to the body,
+  so any non-interactive shell (every agent) hangs forever without
+  `</dev/null`. Same trap on `tea issues create --description` and
+  `tea pr create`.
+- **Apply / remove labels**: `tea api --method PATCH` on the issue, or
+  `tea api repos/ShadowsOverWestgate/sow-tools/issues/<number>/labels` endpoints.
+- **Close**: `tea issues close <number>`
+
+`tea` infers the repo from the git remote when run inside the clone.
+Gitea shares one number space across issues and PRs.
+
+## Pull requests as a triage surface
+
+**PRs as a request surface: no.**
+
+## When a skill says "publish to the issue tracker"
+
+Create a Gitea issue with `tea issues create`.
+
+## When a skill says "fetch the relevant ticket"
+
+Run `tea issues <number>` plus the comments API call above.
+
+## Wayfinding operations
+
+Used by `/wayfinder`. The **map** is a single issue with **child** issues as tickets.
+
+- **Map**: a single issue labelled `wayfinder:map`, holding the Notes / Decisions-so-far / Fog body. `tea issues create --title "..." --description "..." --labels wayfinder:map`.
+- **Child ticket**: this Gitea instance (v1.27) has no native sub-issue hierarchy, so a child issue carries `Part of #<map>` at the top of its description, and is also added to a task list in the map body. Labels: `wayfinder:<type>` (`research`/`prototype`/`grilling`/`task`). Once claimed, the ticket is assigned to the driving dev.
+- **Blocking**: Gitea's **native dependencies API** — the canonical, UI-visible representation (shows as "Depends on" / "Blocks" on the issue page). Add an edge with `tea api -X POST repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies -f owner=ShadowsOverWestgate -f repo=sow-tools -F index=<blocker>`, where `<blocker>` is the blocker's issue **index** (its `#number` — Gitea's dependency API takes the index directly, unlike GitHub's numeric database id). Check status with `tea api repos/ShadowsOverWestgate/sow-tools/issues/<child>/dependencies` (GET) — a ticket is unblocked when every returned issue's `state` is `closed`.
+- **Frontier query**: list the map's open children (`tea issues list --state open`, keep the ones whose description contains `Part of #<map>`), drop any with an open dependency (per the GET above) or an assignee; first in map order wins.
+- **Claim**: `tea issues edit <n> --add-assignees <username>` — the session's first write.
+- **Resolve**: `tea comment <n> "<answer>" </dev/null`, then `tea issues close <n>`, then append a context pointer (gist + link) to the map's Decisions-so-far.

docs/agents/triage-labels.md

@@ -0,0 +1,15 @@
+# Triage Labels
+
+The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker (Gitea — see `issue-tracker.md` for how to apply labels with `tea`).
+
+| Label in mattpocock/skills | Label in our tracker | Meaning                                  |
+| -------------------------- | -------------------- | ---------------------------------------- |
+| `needs-triage`             | `needs-triage`       | Maintainer needs to evaluate this issue  |
+| `needs-info`               | `needs-info`         | Waiting on reporter for more information |
+| `ready-for-agent`          | `ready-for-agent`    | Fully specified, ready for an AFK agent  |
+| `ready-for-human`          | `ready-for-human`    | Requires human implementation            |
+| `wontfix`                  | `wontfix`            | Will not be actioned                     |
+
+When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
+
+Edit the right-hand column to match whatever vocabulary you actually use.

docs/command-surface.md

@@ -29,9 +29,136 @@ aliases.
 | `assets` | `fix-mdl` | Lowercase `.mdl` names and rewrite model identity to match. |
 | `assets` | `check-dupes` | Report runtime-name (basename) collisions across dirs. |
 | `assets` | `clean-dupes` | Delete clean-tree files whose basename collides with primary. |
+| `depot` | `status` | Report referenced-vs-present drift against a backend. |
+| `depot` | `push` | Upload referenced-but-absent blobs from a local depot. |
+| `depot` | `verify` | Existence sweep plus sampled download re-hash. |
+| `depot` | `get` | Fetch one blob with sha re-verify. |
+| `depot` | `pull` | Incremental verified pull of every referenced blob. |
+| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
+| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
+| `nwsync` | `verify` | Decompress and hash a published manifest's blobs through the pull zone. |
 
-`crucible-depot` remains registered but unwired. It fails closed with exit `70`
+`depot status` and `depot get` pick their backend either with `--out DIR`, a
-and never emits placeholder artifacts.
+depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
+flags are mutually exclusive.
+
+`nwsync emit` runs where an artifact is born (a `.hak`/`.erf`, or a loose file
+such as the TLK); `nwsync assemble` runs at module release and reads only the
+small per-artifact indexes. Both take **depot keys**: an artifact's index lives
+beside the artifact itself with the extension replaced, so `emit` and
+`assemble` agree on where it is without being told.
+
+```
+nwsync emit     [--as NAME] [--out DIR] [--jobs N] [--verify] <artifact-key> <file>
+nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
+nwsync verify   [--sample N] [--base URL] [--jobs N] <manifest-sha1>
+```
+
+`emit` is latency-bound, not CPU-bound: every blob costs an existence probe
+plus an upload, and a measured backfill spent 26 seconds of CPU across 9.5
+minutes of wall clock. `--jobs N` (default 16) sets how many resources are in
+flight at once. The manifest is byte-identical at any value — the number of
+workers is never observable in the output. Peak memory is `N` times the
+per-resource limit of 15 MB plus its compressed copy, so raising `N` far past
+the default costs real memory for little gain: the transport keeps 16 idle
+connections per host, and past that a worker pays a fresh TLS handshake.
+
+Both verbs upload by default; nothing bulky is ever written to the runner's
+disk. `--out DIR` writes a local repository tree instead, which is the
+conformance path against upstream `nwn_nwsync_write`. The zone comes from
+`NWSYNC_STORAGE_ZONE` and `NWSYNC_STORAGE_PASSWORD`, with the host from
+`BUNNY_STORAGE_HOST` — NWSync data is a separate zone from the asset depot.
+
+`assemble`'s artifact keys are in `Mod_HakList` order, highest priority first: a
+resref in more than one artifact resolves to the earliest one, the way the game
+resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
+`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
+out of the sidecar.
+
+`nwsync verify` is the only check on a published blob upstream of a player's
+client. It reads the **pull zone**, not the storage API, and needs no
+credential: what matters is the bytes a client is served, edge behaviour
+included. Every blob is decompressed and hashed, and the zstd frame is asserted
+to declare its content size. Neither half is optional — a `Content-Length` check
+passes a byte-correct-looking object whose contents are short, and a round-trip
+check alone passes a frame the game client cannot decode but Go's decoder can.
+Failures are reported per blob as missing, malformed framing, size mismatch or
+hash mismatch, and the exit code is 1.
+
+A full sweep of the live manifest is roughly 69,000 blobs and 15 GB, so
+`--sample N` exists to make verifying routine; the default is a full sweep.
+`--base URL` (or `NWSYNC_PULL_BASE`) overrides the public host.
+
+`emit --verify` applies the same check where `emit` would otherwise skip. `emit`
+normally reads a blob's presence as proof of its contents, decided by a 1-byte
+range GET, so an object written truncated — or written by an emitter since found
+broken — is skipped by every later run forever and no backfill repairs it. With
+`--verify` the stored copy is read back, unwrapped, hashed against its own name,
+and replaced when it does not match. It costs a full GET per existing blob, so
+it is a repair pass, not the default.
+
+**After a repair, `verify` is what tells you which keys to purge.** A repair is
+the one thing that makes a key serve different bytes than it did before, and the
+edge caches these objects for 30 days precisely because that normally cannot
+happen. The two commands look at different copies on purpose: `emit --verify`
+repairs the **origin**, `verify` reads the **edge**. So a `verify` run straight
+after a repair is not a verdict — it is a survey, and every blob it still calls
+bad is one the edge is serving stale. Purge exactly those, then re-run it; only
+that second run is the verdict.
+
+Purging the keys `verify` names beats purging the zone, because the edge only
+ever cached what somebody actually fetched: the 2026-08-01 repair rewrote 2,603
+blobs at the origin and left 8 stale at the edge. The purge belongs in the
+repair procedure rather than in `emit`, which reports how many blobs it wrote
+and never which ones — so it could not target one even with a CDN credential,
+which it deliberately does not hold (#89; the procedure itself is in
+sow-platform's NWSync runbook).
+
+`emit` uploads blobs first and the index last, so the presence of an index is
+the publication marker: an artifact whose emit died halfway leaves real blobs in
+the zone and no index. Blob names are content hashes, so re-running skips
+whatever already landed, and `assemble` fails closed on an artifact with no
+index rather than publishing a manifest that is missing a hak.
+
+### What an emitted tree looks like
+
+`--out DIR` produces the same tree `emit` would upload, which makes it the way
+to check a zone by hand without touching one:
+
+```
+<artifact-sha>.nsym          binary index
+<artifact-sha>.nsym.json     the same index, readable
+data/sha1/a7/4a/a74aa84a...  one blob per resource, two-level fanout
+```
+
+A blob's name is the SHA-1 of the resource's **original** bytes, but the file on
+disk is not those bytes: each blob is wrapped in NWCompressedBuffer framing, a
+24-byte `NSYC` header followed by a zstd frame. Hashing the file directly will
+not match its name, which is the obvious
+first thing to try and the obvious first thing to be confused by. Strip the
+header first:
+
+```
+tail -c +25 <blob> | zstd -dc | sha1sum   # == the blob's filename
+```
+
+The header carries the uncompressed length as a little-endian `uint32` at offset
+12, so the decompressed size is checkable without decompressing. Compression is
+worth roughly a 4:1
+saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
+against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
+`total_bytes`.
+
+The zstd frame always declares its `Frame_Content_Size`. The game client sizes
+its output buffer from that field and cannot decode a frame without one, but the
+Go encoder omits it below 256 bytes, so `emit` re-headers those frames into the
+shape reference libzstd emits: `Single_Segment_flag` set, `Window_Descriptor`
+dropped, and a one-byte content size in its place. `zstd -l <frame>` must print a
+decompressed size; a blank column there is the fault, and it is invisible to any
+check that only decompresses, because both `zstd -dc` and Go's decoder stream
+such a frame happily. This is what the sidecar's `emitter_version` counts:
+version 1 omitted the field and no client could sync past such a blob, version 2
+declares it. `assemble` refuses to merge indexes that disagree.
 
 ## Hidden compatibility aliases
 
@@ -49,6 +176,7 @@ but omitted from routine help and the interactive menu:
 | `topdata` | `build-top-package` | `build-top-package` |
 | `topdata` | `compare-topdata` | `compare-topdata` |
 | `topdata` | `convert-topdata` | `convert-topdata` |
+| `depot` | `--target local` | `--out $DEPOT_DIR` on `status`/`get` |
 | `wiki` | `build-wiki` | `build-wiki` |
 | `wiki` | `deploy-wiki` | `deploy-wiki` |
 

docs/consumer-contract.md

@@ -49,10 +49,10 @@ by `flake.lock`. CI runs the _same_ `crucible <build>` as local dev, inside the
 nix devshell (binary-cache fast). No build container, no token, no CI-only
 pre-steps.
 
-The `registry.westgate.pw/deployment/crucible:<sha>` image is **deployment-only**:
+The `registry.westgate.pw/deployment/crucible:<sha>` image is **retired**
-`prod.yml` pins it so tools travel with the runtime host (`nwn.enable`), a
+(2026-07-17): nothing consumed it, `prod.yml` no longer reserves a slot for
-disabled placeholder until the NWN stack lands. It is **not** a build tool and no
+it, and CI no longer builds or publishes it. Binaries, wrappers, and the Nix
-build/CI job consumes it.
+input are the only supported ways to run Crucible.
 
 ## Determinism
 

docs/superpowers/specs/2026-07-12-crucible-assets-builder-design.md

@@ -126,8 +126,9 @@ exactly as the reference does:
   - Beamdog install dirs.
   - `--nwn <install>` overrides (path to the install root or directly to the
     binary).
-- **Headless.** The engine is a GUI binary. If there is no `DISPLAY` and
+- **Headless.** The engine is a GUI binary. Require `xvfb-run` and wrap the call
-  `xvfb-run` is present, wrap the call:
+  regardless of the caller's `DISPLAY` so compilation never opens the client
+  UI; fail before invoking the engine when `xvfb-run` is unavailable:
   `xvfb-run -a --server-args=-screen 0 1024x768x24 nwmain-linux compilemodel <stem>`.
 - **Per model (one at a time — the engine's `development/` and `modelcompiler/`
   folders are flat and single-slot):**

flake.nix

@@ -22,12 +22,13 @@
             pname = "crucible";
             inherit version;
             src = ./.;
-            vendorHash = "sha256-hm6mrNAtXv0LidzHUfz4eukTFZouizGtxkZ8gKJFUVI=";
+            vendorHash = "sha256-0I8j7On9YGD2GK9xbj/KkgBrlkMJ6Y6XQv+KCLTgBBU=";
             subPackages = [
               "cmd/crucible"
               "cmd/crucible-depot"
               "cmd/crucible-hak"
               "cmd/crucible-module"
+              "cmd/crucible-nwsync"
               "cmd/crucible-topdata"
               "cmd/crucible-wiki"
             ];

go.mod

@@ -6,3 +6,5 @@ require (
 	golang.org/x/text v0.35.0
 	gopkg.in/yaml.v3 v3.0.1
 )
+
+require github.com/klauspost/compress v1.19.1

go.sum

@@ -1,3 +1,5 @@
+github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
+github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
 golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=

internal/app/app.go

@@ -385,7 +385,7 @@ func refreshBuildModuleManifest(ctx context, p *project.Project, progress func(s
 	}
 
 	progress("Refreshing hak list from the latest published sow-assets manifest...")
-	if err := runProjectScript(ctx, p, []string{"scripts", "fetch-hak-manifest"}, manifestPath); err != nil {
+	if err := runProjectScript(ctx, p, []string{"scripts", "fetch-upstream-manifests"}, manifestPath); err != nil {
 		return "", "", err
 	}
 	if _, err := pipeline.ApplyHAKManifest(p, manifestPath); err != nil {

internal/assets/compile.go

@@ -40,13 +40,13 @@ func runCompile(args []string, stdout, stderr io.Writer, getenv func(string) str
 	}
 	binDir := filepath.Dir(nwmain)
 
-	// Headless wrap: no DISPLAY + xvfb-run present -> run under a virtual X.
+	// Always use a virtual X so compilation never opens the client UI.
-	var wrap []string
+	xvfb := look("xvfb-run")
-	if getenv("DISPLAY") == "" {
+	if xvfb == "" {
-		if xvfb := look("xvfb-run"); xvfb != "" {
+		fmt.Fprintln(stderr, "assets compile: xvfb-run not found — install it to run the NWN model compiler headlessly")
-			wrap = []string{xvfb, "-a", "--server-args=-screen 0 1024x768x24"}
+		return exitTool
-		}
 	}
+	wrap := []string{xvfb, "-a", "--server-args=-screen 0 1024x768x24"}
 
 	files, err := walk(dirs, mdlExt, !*nonRecursive)
 	if err != nil {

internal/assets/compile_test.go

@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"os"
 	"path/filepath"
+	"strings"
 	"testing"
 )
 
@@ -23,13 +24,19 @@ func TestCompileDrivesEngineAndReplacesInPlace(t *testing.T) {
 	if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
 		t.Fatal(err)
 	}
+	xvfbDir := t.TempDir()
+	xvfb := filepath.Join(xvfbDir, "xvfb-run")
+	if err := os.WriteFile(xvfb, []byte("#!/bin/sh\n"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("PATH", xvfbDir+string(os.PathListSeparator)+os.Getenv("PATH"))
 
 	getenv := func(k string) string {
 		switch k {
 		case "HOME":
 			return home
 		case "DISPLAY":
-			return ":0" // pretend a display exists so no xvfb wrap is needed
+			return ":0" // a desktop display must not make compilation interactive
 		}
 		return ""
 	}
@@ -38,7 +45,11 @@ func TestCompileDrivesEngineAndReplacesInPlace(t *testing.T) {
 	orig := runner
 	defer func() { runner = orig }()
 	runner = func(dir string, env []string, name string, args ...string) ([]byte, error) {
-		// args: compilemodel <stem>
+		if name != xvfb || len(args) != 5 || args[0] != "-a" ||
+			args[1] != "--server-args=-screen 0 1024x768x24" || args[2] != nwmain ||
+			args[3] != "compilemodel" {
+			t.Fatalf("engine command = %q %q, want xvfb-run wrapping nwmain", name, args)
+		}
 		stem := args[len(args)-1]
 		compiled := filepath.Join(mc, stem+".mdl")
 		return nil, os.WriteFile(compiled, []byte("\x00\x00compiled"), 0o644)
@@ -77,6 +88,11 @@ func TestCompileAbortsOnNameMismatch(t *testing.T) {
 	if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
 		t.Fatal(err)
 	}
+	xvfbDir := t.TempDir()
+	if err := os.WriteFile(filepath.Join(xvfbDir, "xvfb-run"), []byte("#!/bin/sh\n"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("PATH", xvfbDir+string(os.PathListSeparator)+os.Getenv("PATH"))
 	getenv := func(k string) string {
 		if k == "HOME" {
 			return home
@@ -108,3 +124,48 @@ func TestCompileAbortsOnNameMismatch(t *testing.T) {
 		t.Fatal("engine should not run for a name-mismatched model")
 	}
 }
+
+func TestCompileFailsClosedWithoutXvfb(t *testing.T) {
+	home := t.TempDir()
+	userData := filepath.Join(home, ".local", "share", "Neverwinter Nights")
+	for _, d := range []string{filepath.Join(userData, "development"), filepath.Join(userData, "modelcompiler")} {
+		if err := os.MkdirAll(d, 0o755); err != nil {
+			t.Fatal(err)
+		}
+	}
+	nwmain := filepath.Join(t.TempDir(), "nwmain-linux")
+	if err := os.WriteFile(nwmain, []byte("#!/bin/sh\n"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("PATH", t.TempDir())
+
+	getenv := func(k string) string {
+		if k == "HOME" {
+			return home
+		}
+		return ""
+	}
+	orig := runner
+	defer func() { runner = orig }()
+	engineCalled := false
+	runner = func(string, []string, string, ...string) ([]byte, error) {
+		engineCalled = true
+		return nil, nil
+	}
+
+	srcDir := t.TempDir()
+	if err := os.WriteFile(filepath.Join(srcDir, "foo.mdl"),
+		[]byte("newmodel foo\nbeginmodelgeom foo\n  node dummy foo\n    parent null\n  endnode\nendmodelgeom foo\ndonemodel foo\n"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	var stdout, stderr bytes.Buffer
+	if code := runCompile([]string{"--nwn", nwmain, srcDir}, &stdout, &stderr, getenv); code != exitTool {
+		t.Fatalf("compile exit = %d, want %d\n%s", code, exitTool, stderr.String())
+	}
+	if engineCalled {
+		t.Fatal("engine must not run without xvfb-run")
+	}
+	if !strings.Contains(stderr.String(), "xvfb-run") {
+		t.Fatalf("missing actionable xvfb-run error: %s", stderr.String())
+	}
+}

internal/depot/keystore.go

@@ -0,0 +1,121 @@
+package depot
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+)
+
+// KeyStore is the zone addressed by object key rather than by depot sha. The
+// depot names every object after the sha256 of its contents; NWSync does not —
+// a blob is named after the sha1 of its *uncompressed* bytes while the body
+// uploaded is the compressed form, and a per-artifact index is named after its
+// artifact. Both addressing modes want the same transport, retry and probe
+// discipline, so the sha-addressed Backend rides on this rather than the other
+// way round.
+type KeyStore interface {
+	// ProbeKey returns the existence state of one key. transient=true means a
+	// retry might change the answer — never read it as "missing, re-upload".
+	ProbeKey(ctx context.Context, key string) (state ProbeState, transient bool, err error)
+	// PutReader uploads size bytes read from r to key. checksum is the
+	// uppercase hex sha256 of those bytes, which Bunny verifies server-side.
+	PutReader(ctx context.Context, key string, r io.Reader, size int64, checksum string) error
+	// GetKey fetches the whole object at key. Small objects only — it holds
+	// the body in memory and does no hash check, because a key is not always
+	// a content hash.
+	GetKey(ctx context.Context, key string) ([]byte, error)
+}
+
+// NewKeyStore returns a KeyStore for cfg's storage zone. Fails closed on a
+// missing host or read key, matching NewBackend.
+func NewKeyStore(cfg Config) (KeyStore, error) {
+	if cfg.StorageHost == "" {
+		return nil, errors.New("storage backend requires a storage host")
+	}
+	if cfg.StorageZone == "" {
+		return nil, errors.New("storage backend requires a storage zone")
+	}
+	if cfg.ReadKey == "" {
+		return nil, errors.New("storage backend requires a read key")
+	}
+	return &httpBackend{name: "bunny", client: newHTTPClient(cfg), cfg: cfg}, nil
+}
+
+// keyURL is the storage URL of one object key.
+func (b *httpBackend) keyURL(key string) string {
+	host := b.cfg.StorageHost
+	// StorageHost is normally a bare host ("storage.bunnycdn.com"); allow a
+	// full scheme (used by tests against httptest.NewServer) to pass through
+	// unchanged.
+	if !strings.Contains(host, "://") {
+		host = "https://" + host
+	}
+	return fmt.Sprintf("%s/%s/%s", strings.TrimSuffix(host, "/"), b.cfg.StorageZone, key)
+}
+
+func (b *httpBackend) ProbeKey(ctx context.Context, key string) (ProbeState, bool, error) {
+	return b.rangeProbe(ctx, b.keyURL(key), map[string]string{"AccessKey": b.cfg.ReadKey})
+}
+
+func (b *httpBackend) PutReader(ctx context.Context, key string, r io.Reader, size int64, checksum string) error {
+	if b.name == "cdn" {
+		return errors.New("cdn backend is read-only")
+	}
+	if b.cfg.WriteKey == "" {
+		return errors.New("storage backend requires a write key to write")
+	}
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodPut, b.keyURL(key), r)
+	if err != nil {
+		return err
+	}
+	req.ContentLength = size
+	req.Header.Set("AccessKey", b.cfg.WriteKey)
+	// Bunny defines Checksum as sha256 of the body and rejects a mismatch, so
+	// this is server-side integrity checking, not decoration.
+	req.Header.Set("Checksum", strings.ToUpper(checksum))
+
+	resp, err := b.client.Do(req)
+	if err != nil {
+		return err
+	}
+	defer resp.Body.Close()
+	_, _ = io.Copy(io.Discard, resp.Body)
+
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return fmt.Errorf("put %s: unexpected status %d", key, resp.StatusCode)
+	}
+	return nil
+}
+
+func (b *httpBackend) GetKey(ctx context.Context, key string) ([]byte, error) {
+	resp, err := b.get(ctx, b.keyURL(key), map[string]string{"AccessKey": b.cfg.ReadKey})
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		_, _ = io.Copy(io.Discard, resp.Body)
+		return nil, fmt.Errorf("get %s: unexpected status %d", key, resp.StatusCode)
+	}
+	return io.ReadAll(resp.Body)
+}
+
+// putFile uploads the file at src to key, streaming it. checksum is the
+// uppercase hex sha256 of the file's bytes.
+func (b *httpBackend) putFile(ctx context.Context, key, src, checksum string) error {
+	f, err := os.Open(src)
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+	info, err := f.Stat()
+	if err != nil {
+		return err
+	}
+	return b.PutReader(ctx, key, f, info.Size(), checksum)
+}

internal/depot/remote.go

@@ -10,7 +10,6 @@ import (
 	"net/http"
 	"os"
 	"path/filepath"
-	"strings"
 	"time"
 )
 
@@ -71,16 +70,7 @@ type httpBackend struct {
 
 func (b *httpBackend) Name() string { return b.name }
 
-func (b *httpBackend) storageURL(sha string) string {
+func (b *httpBackend) storageURL(sha string) string { return b.keyURL(BlobKey(sha)) }
-	host := b.cfg.StorageHost
-	// StorageHost is normally a bare host ("storage.bunnycdn.com"); allow a
-	// full scheme (used by tests against httptest.NewServer) to pass through
-	// unchanged.
-	if strings.Contains(host, "://") {
-		return fmt.Sprintf("%s/%s/%s", strings.TrimSuffix(host, "/"), b.cfg.StorageZone, BlobKey(sha))
-	}
-	return fmt.Sprintf("https://%s/%s/%s", host, b.cfg.StorageZone, BlobKey(sha))
-}
 
 func (b *httpBackend) cdnURL(sha string) string {
 	return fmt.Sprintf("%s/%s", b.cfg.CDNBase, BlobKey(sha))
@@ -140,7 +130,8 @@ func (b *httpBackend) Probe(ctx context.Context, sha string) (ProbeState, bool,
 	return b.rangeProbe(ctx, b.storageURL(sha), map[string]string{"AccessKey": b.cfg.ReadKey})
 }
 
-// Put uploads src for sha. cdn is read-only.
+// Put uploads src for sha. cdn is read-only. A depot object is named after the
+// sha256 of its own bytes, so the key's sha doubles as the Checksum header.
 func (b *httpBackend) Put(ctx context.Context, sha, src string) error {
 	if b.name == "cdn" {
 		return errors.New("cdn backend is read-only")
@@ -157,30 +148,7 @@ func (b *httpBackend) Put(ctx context.Context, sha, src string) error {
 		return nil
 	}
 
-	f, err := os.Open(src)
+	return b.putFile(ctx, BlobKey(sha), src, sha)
-	if err != nil {
-		return err
-	}
-	defer f.Close()
-
-	req, err := http.NewRequestWithContext(ctx, http.MethodPut, b.storageURL(sha), f)
-	if err != nil {
-		return err
-	}
-	req.Header.Set("AccessKey", b.cfg.WriteKey)
-	req.Header.Set("Checksum", strings.ToUpper(sha))
-
-	resp, err := b.client.Do(req)
-	if err != nil {
-		return err
-	}
-	defer resp.Body.Close()
-	_, _ = io.Copy(io.Discard, resp.Body)
-
-	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
-		return fmt.Errorf("bunny put %s: unexpected status %d", sha, resp.StatusCode)
-	}
-	return nil
 }
 
 // Get fetches sha into dest via temp file + rename, re-hashing and deleting

internal/depot/run.go

@@ -11,6 +11,7 @@ import (
 	"os"
 	"path/filepath"
 	"sort"
+	"strings"
 	"sync"
 	"time"
 )
@@ -51,13 +52,15 @@ func Run(args []string, stdout, stderr io.Writer, getenv func(string) string) in
 
 func printRunUsage(w io.Writer) {
 	fmt.Fprint(w, `usage:
-  depot status [--manifests DIR] [--source LOCAL_DEPOT] --target bunny|cdn|local
+  depot status [--manifests DIR] --target bunny|cdn | --out DIR
-  depot push   [--manifests DIR] --source LOCAL_DEPOT   --target bunny
+  depot push   [--manifests DIR] --source LOCAL_DEPOT --target bunny
-  depot verify [--manifests DIR] --target bunny|cdn     [--sample N]
+  depot verify [--manifests DIR] --target bunny|cdn   [--sample N]
-  depot get    <sha> <dest>       --target cdn|bunny|local
+  depot get    <sha> <dest>      --target cdn|bunny   | --out DIR
-  depot pull   [--manifests DIR] --dest DIR             --target cdn|bunny
+  depot pull   [--manifests DIR] --dest DIR           --target cdn|bunny
 
---target local uses the DEPOT_DIR environment variable as the local root.
+--out DIR reads and writes a depot tree on disk at DIR instead of a remote
+backend. --target names a remote backend only; the two are mutually exclusive.
+--target local is a deprecated alias for --out $DEPOT_DIR.
 `)
 }
 
@@ -65,16 +68,27 @@ func printRunUsage(w io.Writer) {
 // than internal/backend failures (exit 70).
 var errUsage = errors.New("usage error")
 
-// resolveBackend builds the named backend, resolving "local" against DEPOT_DIR.
+// resolveBackend picks the backend for a command that can work either against a
-func resolveBackend(target string, getenv func(string) string, cfg Config) (Backend, error) {
+// depot tree on disk (--out DIR) or a remote backend (--target bunny|cdn).
-	root := ""
+// "--target local" stays as a deprecated alias for --out $DEPOT_DIR so older
-	if target == "local" {
+// callers keep working.
-		root = getenv("DEPOT_DIR")
+func resolveBackend(out, target string, getenv func(string) string, cfg Config) (Backend, error) {
+	switch {
+	case out != "" && target != "":
+		return nil, fmt.Errorf("--out and --target are mutually exclusive: %w", errUsage)
+	case out != "":
+		return NewBackend("local", out, cfg)
+	case target == "local":
+		root := getenv("DEPOT_DIR")
 		if root == "" {
-			return nil, fmt.Errorf("--target local requires DEPOT_DIR to be set: %w", errUsage)
+			return nil, fmt.Errorf("--target local requires DEPOT_DIR to be set (prefer --out DIR): %w", errUsage)
 		}
+		return NewBackend("local", root, cfg)
+	case target == "":
+		return nil, fmt.Errorf("--out DIR or --target bunny|cdn is required: %w", errUsage)
+	default:
+		return NewBackend(target, "", cfg)
 	}
-	return NewBackend(target, root, cfg)
 }
 
 // backendErrExit maps a resolveBackend error onto the exit contract.
@@ -119,18 +133,18 @@ func runStatus(args []string, stdout, stderr io.Writer, getenv func(string) stri
 	fs := flag.NewFlagSet("status", flag.ContinueOnError)
 	fs.SetOutput(stderr)
 	manifests := fs.String("manifests", "assets", "directory of *.yml manifests")
-	_ = fs.String("source", "", "local depot root (unused for status target=local; see DEPOT_DIR)")
+	source := fs.String("source", "", "deprecated and ignored (use --out DIR for a depot tree on disk)")
-	target := fs.String("target", "", "bunny|cdn|local")
+	out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
+	target := fs.String("target", "", "bunny|cdn")
 	if err := fs.Parse(args); err != nil {
 		return exitUsage
 	}
-	if *target == "" {
+	if *source != "" {
-		fmt.Fprintln(stderr, "depot status: --target is required")
+		fmt.Fprintln(stderr, "depot status: --source is ignored; use --out DIR")
-		return exitUsage
 	}
 
 	cfg := LoadConfig(getenv)
-	backend, err := resolveBackend(*target, getenv, cfg)
+	backend, err := resolveBackend(*out, *target, getenv, cfg)
 	if err != nil {
 		fmt.Fprintln(stderr, "depot status:", err)
 		return backendErrExit(err)
@@ -309,13 +323,21 @@ func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) stri
 func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
 	fs := flag.NewFlagSet("get", flag.ContinueOnError)
 	fs.SetOutput(stderr)
-	target := fs.String("target", "", "cdn|bunny|local")
+	out := fs.String("out", "", "depot tree on disk to read instead of a remote backend")
-	if err := fs.Parse(args); err != nil {
+	target := fs.String("target", "", "cdn|bunny")
+	// Positionals come first in the documented usage, and Go's flag package
+	// stops parsing at the first one, so split them off by hand.
+	split := 0
+	for split < len(args) && !strings.HasPrefix(args[split], "-") {
+		split++
+	}
+	positional := args[:split]
+	if err := fs.Parse(args[split:]); err != nil {
 		return exitUsage
 	}
-	positional := fs.Args()
+	positional = append(positional, fs.Args()...)
 	if len(positional) != 2 {
-		fmt.Fprintln(stderr, "depot get: usage: depot get <sha> <dest> --target cdn|bunny|local")
+		fmt.Fprintln(stderr, "depot get: usage: depot get <sha> <dest> --target cdn|bunny | --out DIR")
 		return exitUsage
 	}
 	sha, dest := positional[0], positional[1]
@@ -323,13 +345,9 @@ func runGet(args []string, stdout, stderr io.Writer, getenv func(string) string)
 		fmt.Fprintf(stderr, "depot get: invalid sha %q\n", sha)
 		return exitUsage
 	}
-	if *target == "" {
-		fmt.Fprintln(stderr, "depot get: --target is required")
-		return exitUsage
-	}
 
 	cfg := LoadConfig(getenv)
-	backend, err := resolveBackend(*target, getenv, cfg)
+	backend, err := resolveBackend(*out, *target, getenv, cfg)
 	if err != nil {
 		fmt.Fprintln(stderr, "depot get:", err)
 		return backendErrExit(err)

internal/depot/run_test.go

@@ -54,6 +54,91 @@ func TestRunUsage(t *testing.T) {
 	})
 }
 
+func TestOutFlag(t *testing.T) {
+	t.Run("status --out reads the given tree", func(t *testing.T) {
+		sha := shaOf("out-blob")
+		manifests := t.TempDir()
+		writeManifest(t, manifests, sha, 8)
+		depotDir := t.TempDir()
+		writeBlob(t, depotDir, sha, "out-blob")
+
+		var out, errb bytes.Buffer
+		code := Run([]string{"status", "--manifests", manifests, "--out", depotDir}, &out, &errb, testGetenv(nil))
+		if code != 0 {
+			t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
+		}
+		if !bytesContains(out.String(), "present=1") {
+			t.Fatalf("expected present=1, got %s", out.String())
+		}
+	})
+
+	t.Run("get --out fetches from the given tree", func(t *testing.T) {
+		sha := shaOf("get-blob")
+		depotDir := t.TempDir()
+		writeBlob(t, depotDir, sha, "get-blob")
+		dest := filepath.Join(t.TempDir(), "fetched")
+
+		var out, errb bytes.Buffer
+		code := Run([]string{"get", sha, dest, "--out", depotDir}, &out, &errb, testGetenv(nil))
+		if code != 0 {
+			t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
+		}
+		got, err := os.ReadFile(dest)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if string(got) != "get-blob" {
+			t.Fatalf("got %q", got)
+		}
+	})
+
+	t.Run("--out with any --target is rejected", func(t *testing.T) {
+		for _, target := range []string{"bunny", "cdn", "local"} {
+			var out, errb bytes.Buffer
+			code := Run([]string{"status", "--manifests", t.TempDir(), "--out", t.TempDir(), "--target", target}, &out, &errb, testGetenv(nil))
+			if code != 64 {
+				t.Fatalf("--target %s: expected 64, got %d (stderr=%s)", target, code, errb.String())
+			}
+		}
+	})
+
+	t.Run("get accepts flags before the positionals", func(t *testing.T) {
+		sha := shaOf("flags-first-blob")
+		depotDir := t.TempDir()
+		writeBlob(t, depotDir, sha, "flags-first-blob")
+		dest := filepath.Join(t.TempDir(), "fetched")
+
+		var out, errb bytes.Buffer
+		code := Run([]string{"get", "--out", depotDir, sha, dest}, &out, &errb, testGetenv(nil))
+		if code != 0 {
+			t.Fatalf("expected 0, got %d (stderr=%s)", code, errb.String())
+		}
+	})
+
+	t.Run("neither --out nor --target is rejected", func(t *testing.T) {
+		var out, errb bytes.Buffer
+		code := Run([]string{"status", "--manifests", t.TempDir()}, &out, &errb, testGetenv(nil))
+		if code != 64 {
+			t.Fatalf("expected 64, got %d (stderr=%s)", code, errb.String())
+		}
+	})
+
+	t.Run("--target local still works as a hidden alias", func(t *testing.T) {
+		sha := shaOf("alias-blob")
+		manifests := t.TempDir()
+		writeManifest(t, manifests, sha, 11)
+		depotDir := t.TempDir()
+		writeBlob(t, depotDir, sha, "alias-blob")
+
+		var out, errb bytes.Buffer
+		code := Run([]string{"status", "--manifests", manifests, "--target", "local"}, &out, &errb,
+			testGetenv(map[string]string{"DEPOT_DIR": depotDir}))
+		if code != 0 {
+			t.Fatalf("expected 0, got %d (stdout=%s stderr=%s)", code, out.String(), errb.String())
+		}
+	})
+}
+
 func TestGetInvalidSHA(t *testing.T) {
 	var out, errb bytes.Buffer
 	dir := t.TempDir()
@@ -172,6 +257,17 @@ func bytesContains(s, substr string) bool {
 	return bytes.Contains([]byte(s), []byte(substr))
 }
 
+func writeBlob(t *testing.T, root, sha, content string) {
+	t.Helper()
+	path := filepath.Join(root, BlobKey(sha))
+	if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path, []byte(content), 0644); err != nil {
+		t.Fatal(err)
+	}
+}
+
 func writeManifest(t *testing.T, dir, sha string, size int64) {
 	t.Helper()
 	content := fmt.Sprintf("assets:\n  - path: foo\n    sha256: %s\n    size: %d\n", sha, size)

internal/dispatch/dispatch.go

@@ -21,6 +21,7 @@ import (
 	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
 	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/depot"
 	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/menu"
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/nwsync"
 )
 
 // Exit codes follow the sysexits(3) convention so CI can distinguish
@@ -94,16 +95,27 @@ var Registry = []Builder{
 	{
 		Name:    "depot",
 		Bin:     "crucible-depot",
-		Summary: "content-addressed asset depot (local/cdn/bunny)",
+		Summary: "content-addressed asset depot (disk/cdn/bunny)",
 		Commands: []Command{
-			{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] [--source DIR] --target bunny|cdn|local"},
+			{Name: "status", Summary: "report referenced-vs-present drift against a backend", Usage: "crucible depot status [--manifests DIR] --target bunny|cdn | --out DIR"},
 			{Name: "push", Summary: "upload referenced-but-absent blobs from a local depot", Usage: "crucible depot push [--manifests DIR] --source DIR --target bunny"},
 			{Name: "verify", Summary: "existence sweep plus sampled download re-hash", Usage: "crucible depot verify [--manifests DIR] --target bunny|cdn [--sample N]"},
-			{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny|local"},
+			{Name: "get", Summary: "fetch one blob with sha re-verify", Usage: "crucible depot get <sha> <dest> --target cdn|bunny | --out DIR"},
 			{Name: "pull", Summary: "incremental verified pull of every referenced blob", Usage: "crucible depot pull [--manifests DIR] --dest DIR --target cdn|bunny"},
 		},
 		Wired: true,
 	},
+	{
+		Name:    "nwsync",
+		Bin:     "crucible-nwsync",
+		Summary: "publish NWSync blobs and manifests (emit/assemble/verify)",
+		Commands: []Command{
+			{Name: "emit", Summary: "explode one artifact into blobs plus its own NSYM manifest", Usage: "crucible nwsync emit <artifact> --out DIR"},
+			{Name: "assemble", Summary: "merge per-artifact NSYM manifests into one", Usage: "crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]"},
+			{Name: "verify", Summary: "read a published manifest's blobs back through the pull zone and hash them", Usage: "crucible nwsync verify <manifest-sha1> [--sample N]"},
+		},
+		Wired: true,
+	},
 	{
 		Name:    "assets",
 		Bin:     "crucible-assets",
@@ -241,7 +253,7 @@ var Registry = []Builder{
 					"2da-to-module [flags] <input.2da> [output.json]",
 					"json-to-2da <input.json> <output.2da>",
 				},
-				Aliases:    []CommandAlias{{Name: "convert-topdata"}},
+				Aliases: []CommandAlias{{Name: "convert-topdata"}},
 			},
 		},
 		Wired: true,
@@ -392,7 +404,7 @@ func runBuilder(name string, args []string, out, errw io.Writer) int {
 		}
 	}
 	if b.Wired {
-		// depot and assets are self-contained builders: they parse their own
+		// depot, assets and nwsync are self-contained builders: they parse their own
 		// subcommands and own their exit contract, bypassing the
 		// b.Commands/delegateLegacy legacy routing entirely.
 		switch b.Name {
@@ -400,6 +412,8 @@ func runBuilder(name string, args []string, out, errw io.Writer) int {
 			return depot.Run(args, out, errw, os.Getenv)
 		case "assets":
 			return assets.Run(args, out, errw, os.Getenv)
+		case "nwsync":
+			return nwsync.Run(args, out, errw)
 		}
 	}
 	if !b.Wired {

internal/dispatch/dispatch_test.go

@@ -148,6 +148,10 @@ func TestBuilderHelpIsOK(t *testing.T) {
 	}
 }
 
+// selfContained builders parse their own subcommands instead of delegating to
+// the legacy internal/app surface.
+var selfContained = map[string]bool{"depot": true, "assets": true, "nwsync": true}
+
 func TestCanonicalCommandSurface(t *testing.T) {
 	want := map[string][]string{
 		"depot":   {"status", "push", "verify", "get", "pull"},
@@ -156,6 +160,7 @@ func TestCanonicalCommandSurface(t *testing.T) {
 		"module":  {"build", "extract", "validate", "compare", "manifest"},
 		"topdata": {"validate", "build", "package", "compare", "convert"},
 		"wiki":    {"build", "deploy"},
+		"nwsync":  {"emit", "assemble", "verify"},
 	}
 	for _, builder := range Registry {
 		got := builder.subcommands()
@@ -173,10 +178,10 @@ func TestRegistryCommandNamesAndAliasesAreUnambiguous(t *testing.T) {
 	for _, builder := range Registry {
 		seen := map[string]bool{}
 		for _, command := range builder.Commands {
-			// depot and assets parse their own subcommands and bypass AppCommand
+			// depot, assets and nwsync parse their own subcommands and bypass
-			// routing entirely (see the self-contained-builder branch in
+			// AppCommand routing entirely (see the self-contained-builder branch
-			// runBuilder), so their Commands carry no AppCommand.
+			// in runBuilder), so their Commands carry no AppCommand.
-			requireAppCommand := builder.Name != "depot" && builder.Name != "assets"
+			requireAppCommand := !selfContained[builder.Name]
 			if command.Name == "" || command.Summary == "" || command.Usage == "" || (requireAppCommand && command.AppCommand == "") {
 				t.Errorf("%s has incomplete command metadata: %#v", builder.Name, command)
 			}

internal/erf/erf.go

@@ -67,6 +67,19 @@ type resourceEntry struct {
 	Size   uint32
 }
 
+// extensionTypes and typeExtensions must stay exact inverses of each other;
+// init() panics if they drift apart.
+//
+// Numbers below 0x0BB8 follow upstream neverwinter.nim
+// (neverwinter/restype.nim). The 0x0BB8 and up entries are lyt/vis/mdx: real
+// Aurora archive types that NWN1 ships in its own data/*.bif but upstream
+// happens not to register. xoreos corroborates those three numbers
+// (src/aurora/types.h).
+//
+// This table is NWN:EE only. Do not add NWN2 formats (mdb, gr2, wlk, xml):
+// NWN:EE either gives that number to something else (2070 is xbc here and mdb
+// in NWN2) or has no number for it at all, so packing one into a HAK writes a
+// resource the game misreads. See the reserved list in erf_test.go.
 var extensionTypes = map[string]uint16{
 	"res": 0x0000,
 	"bmp": 0x0001,
@@ -96,12 +109,13 @@ var extensionTypes = map[string]uint16{
 	"utt": 0x07F0,
 	"dds": 0x07F1,
 	"uts": 0x07F3,
+	"ltr": 0x07F4,
+	"gff": 0x07F5,
 	"fac": 0x07F6,
-	"gff": 0x07F7,
 	"ute": 0x07F8,
 	"utd": 0x07FA,
 	"utp": 0x07FC,
-	"dfa": 0x07FD,
+	"dft": 0x07FD,
 	"gic": 0x07FE,
 	"gui": 0x07FF,
 	"utm": 0x0803,
@@ -117,20 +131,15 @@ var extensionTypes = map[string]uint16{
 	"ndb": 0x0810,
 	"ptm": 0x0811,
 	"ptt": 0x0812,
-	"ltr": 0x0813,
 	"shd": 0x0815,
-	"mdb": 0x0816,
 	"mtr": 0x0818,
-	"jpg": 0x081C,
 	"lod": 0x081E,
 	"gif": 0x081F,
 	"png": 0x0820,
+	"jpg": 0x0821,
 	"lyt": 0x0BB8,
 	"vis": 0x0BB9,
 	"mdx": 0x0BC0,
-	"wlk": 0x0BCC,
-	"xml": 0x0BCD,
-	"gr2": 0x0FA3,
 }
 
 var typeExtensions = map[uint16]string{
@@ -162,12 +171,13 @@ var typeExtensions = map[uint16]string{
 	0x07F0: "utt",
 	0x07F1: "dds",
 	0x07F3: "uts",
+	0x07F4: "ltr",
+	0x07F5: "gff",
 	0x07F6: "fac",
-	0x07F7: "gff",
 	0x07F8: "ute",
 	0x07FA: "utd",
 	0x07FC: "utp",
-	0x07FD: "dfa",
+	0x07FD: "dft",
 	0x07FE: "gic",
 	0x07FF: "gui",
 	0x0803: "utm",
@@ -183,20 +193,15 @@ var typeExtensions = map[uint16]string{
 	0x0810: "ndb",
 	0x0811: "ptm",
 	0x0812: "ptt",
-	0x0813: "ltr",
 	0x0815: "shd",
-	0x0816: "mdb",
 	0x0818: "mtr",
-	0x081C: "jpg",
 	0x081E: "lod",
 	0x081F: "gif",
 	0x0820: "png",
+	0x0821: "jpg",
 	0x0BB8: "lyt",
 	0x0BB9: "vis",
 	0x0BC0: "mdx",
-	0x0BCC: "wlk",
-	0x0BCD: "xml",
-	0x0FA3: "gr2",
 }
 
 func init() {
@@ -205,8 +210,13 @@ func init() {
 		if !ok {
 			panic(fmt.Sprintf("missing canonical extension for resource type 0x%04X", resourceType))
 		}
-		if canonicalExt == ext {
+		if canonicalExt != ext {
-			continue
+			panic(fmt.Sprintf("resource type 0x%04X is %q in extensionTypes but %q in typeExtensions", resourceType, ext, canonicalExt))
+		}
+	}
+	for resourceType, ext := range typeExtensions {
+		if registered, ok := extensionTypes[ext]; !ok || registered != resourceType {
+			panic(fmt.Sprintf("extension %q is 0x%04X in typeExtensions but 0x%04X in extensionTypes (registered=%v)", ext, resourceType, registered, ok))
 		}
 	}
 }
@@ -301,63 +311,110 @@ func Write(w io.Writer, archive Archive) error {
 	return nil
 }
 
+// IndexEntry locates one resource inside an archive without holding its
+// payload. Streaming callers read one payload at a time from these, so peak
+// memory tracks the largest resource instead of the whole archive.
+type IndexEntry struct {
+	Name   string
+	Type   uint16
+	Offset int64
+	Size   int64
+}
+
+// Index is the header plus the resource table of an ERF: everything except the
+// payloads.
+type Index struct {
+	FileType string
+	Version  string
+	Entries  []IndexEntry
+}
+
+// ReadIndex parses the tables of an ERF of the given size, reading only the
+// header, the key list and the resource list.
+func ReadIndex(r io.ReaderAt, size int64) (Index, error) {
+	if size < headerSize {
+		return Index{}, fmt.Errorf("erf file too small: %d bytes", size)
+	}
+
+	var hdr header
+	if err := binary.Read(io.NewSectionReader(r, 0, headerSize), binary.LittleEndian, &hdr); err != nil {
+		return Index{}, fmt.Errorf("decode erf header: %w", err)
+	}
+
+	if int64(hdr.KeyListOffset)+int64(hdr.EntryCount)*24 > size {
+		return Index{}, fmt.Errorf("erf key list exceeds file bounds")
+	}
+	keys := make([]keyEntry, hdr.EntryCount)
+	keyReader := io.NewSectionReader(r, int64(hdr.KeyListOffset), int64(hdr.EntryCount)*24)
+	if err := binary.Read(keyReader, binary.LittleEndian, &keys); err != nil {
+		return Index{}, fmt.Errorf("decode key list: %w", err)
+	}
+
+	if int64(hdr.ResourceListOffset)+int64(hdr.EntryCount)*8 > size {
+		return Index{}, fmt.Errorf("erf resource list exceeds file bounds")
+	}
+	entries := make([]resourceEntry, hdr.EntryCount)
+	entryReader := io.NewSectionReader(r, int64(hdr.ResourceListOffset), int64(hdr.EntryCount)*8)
+	if err := binary.Read(entryReader, binary.LittleEndian, &entries); err != nil {
+		return Index{}, fmt.Errorf("decode resource list: %w", err)
+	}
+
+	index := Index{
+		FileType: string(hdr.FileType[:]),
+		Version:  string(hdr.Version[:]),
+		Entries:  make([]IndexEntry, 0, hdr.EntryCount),
+	}
+	for position, key := range keys {
+		entry := entries[position]
+		if int64(entry.Offset)+int64(entry.Size) > size {
+			return Index{}, fmt.Errorf("resource %d exceeds file bounds", position)
+		}
+		index.Entries = append(index.Entries, IndexEntry{
+			Name:   string(bytes.TrimRight(key.ResRef[:], "\x00")),
+			Type:   key.ResourceType,
+			Offset: int64(entry.Offset),
+			Size:   int64(entry.Size),
+		})
+	}
+	return index, nil
+}
+
+// ReadPayload returns one resource's bytes.
+func ReadPayload(r io.ReaderAt, entry IndexEntry) ([]byte, error) {
+	payload := make([]byte, entry.Size)
+	if _, err := r.ReadAt(payload, entry.Offset); err != nil {
+		return nil, fmt.Errorf("read resource %q: %w", entry.Name, err)
+	}
+	return payload, nil
+}
+
+// Read materialises a whole archive. Payloads are subslices of the buffer the
+// archive was read into, so nothing is copied twice: a caller must not mutate
+// Data. Callers that only need one resource at a time should use ReadIndex
+// instead, which never holds the archive at all.
 func Read(r io.Reader) (Archive, error) {
 	data, err := io.ReadAll(r)
 	if err != nil {
 		return Archive{}, fmt.Errorf("read erf: %w", err)
 	}
-	if len(data) < headerSize {
+	index, err := ReadIndex(bytes.NewReader(data), int64(len(data)))
-		return Archive{}, fmt.Errorf("erf file too small: %d bytes", len(data))
+	if err != nil {
+		return Archive{}, err
 	}
 
-	var hdr header
+	resources := make([]Resource, 0, len(index.Entries))
-	if err := binary.Read(bytes.NewReader(data[:headerSize]), binary.LittleEndian, &hdr); err != nil {
+	for _, entry := range index.Entries {
-		return Archive{}, fmt.Errorf("decode erf header: %w", err)
-	}
-
-	keyStart := int(hdr.KeyListOffset)
-	keyEnd := keyStart + int(hdr.EntryCount)*24
-	if keyEnd > len(data) {
-		return Archive{}, fmt.Errorf("erf key list exceeds file bounds")
-	}
-	keys := make([]keyEntry, hdr.EntryCount)
-	if err := binary.Read(bytes.NewReader(data[keyStart:keyEnd]), binary.LittleEndian, &keys); err != nil {
-		return Archive{}, fmt.Errorf("decode key list: %w", err)
-	}
-
-	resourceStart := int(hdr.ResourceListOffset)
-	resourceEnd := resourceStart + int(hdr.EntryCount)*8
-	if resourceEnd > len(data) {
-		return Archive{}, fmt.Errorf("erf resource list exceeds file bounds")
-	}
-	entries := make([]resourceEntry, hdr.EntryCount)
-	if err := binary.Read(bytes.NewReader(data[resourceStart:resourceEnd]), binary.LittleEndian, &entries); err != nil {
-		return Archive{}, fmt.Errorf("decode resource list: %w", err)
-	}
-
-	resources := make([]Resource, 0, hdr.EntryCount)
-	for index, key := range keys {
-		entry := entries[index]
-		start := int(entry.Offset)
-		end := start + int(entry.Size)
-		if end > len(data) {
-			return Archive{}, fmt.Errorf("resource %d exceeds file bounds", index)
-		}
-
-		resref := string(bytes.TrimRight(key.ResRef[:], "\x00"))
-		payload := make([]byte, entry.Size)
-		copy(payload, data[start:end])
 		resources = append(resources, Resource{
-			Name: resref,
+			Name: entry.Name,
-			Type: key.ResourceType,
+			Type: entry.Type,
-			Data: payload,
+			Data: data[entry.Offset : entry.Offset+entry.Size],
-			Size: int64(entry.Size),
+			Size: entry.Size,
 		})
 	}
 
 	return Archive{
-		FileType:  string(hdr.FileType[:]),
+		FileType:  index.FileType,
-		Version:   string(hdr.Version[:]),
+		Version:   index.Version,
 		Resources: resources,
 	}, nil
 }

internal/erf/erf_test.go

@@ -134,14 +134,10 @@ func TestExtensionMappingsSupportModernAssetTypes(t *testing.T) {
 		"mtr": 0x0818,
 		"shd": 0x0815,
 		"txi": 0x07E6,
-		"jpg": 0x081C,
+		"jpg": 0x0821,
-		"mdb": 0x0816,
 		"lyt": 0x0BB8,
 		"vis": 0x0BB9,
 		"mdx": 0x0BC0,
-		"xml": 0x0BCD,
-		"wlk": 0x0BCC,
-		"gr2": 0x0FA3,
 	}
 	for ext, wantType := range cases {
 		gotType, ok := ResourceTypeForExtension(ext)
@@ -191,3 +187,71 @@ func TestExtensionMappingsSupportAllUTBlueprintTypes(t *testing.T) {
 		}
 	}
 }
+
+// TestRestypeTableMatchesUpstream pins the restype numbers Crucible shares with
+// neverwinter.nim's restype.nim. The values below are upstream's; a mismatch
+// means a HAK we write is mislabelled for the game.
+func TestRestypeTableMatchesUpstream(t *testing.T) {
+	upstream := map[string]uint16{
+		"res": 0, "bmp": 1, "mve": 2, "tga": 3, "wav": 4, "plt": 6, "ini": 7,
+		"bmu": 8, "txt": 10, "mdl": 2002, "nss": 2009, "ncs": 2010, "are": 2012,
+		"set": 2013, "ifo": 2014, "bic": 2015, "wok": 2016, "2da": 2017,
+		"tlk": 2018, "txi": 2022, "git": 2023, "uti": 2025, "utc": 2027,
+		"dlg": 2029, "itp": 2030, "utt": 2032, "dds": 2033, "uts": 2035,
+		"ltr": 2036, "gff": 2037, "fac": 2038, "ute": 2040, "utd": 2042,
+		"utp": 2044, "dft": 2045, "gic": 2046, "gui": 2047, "utm": 2051,
+		"dwk": 2052, "pwk": 2053, "utg": 2055, "jrl": 2056, "utw": 2058,
+		"ssf": 2060, "hak": 2061, "nwm": 2062, "bik": 2063, "ndb": 2064,
+		"ptm": 2065, "ptt": 2066, "shd": 2069, "mtr": 2072, "lod": 2078,
+		"gif": 2079, "png": 2080, "jpg": 2081,
+	}
+	for ext, want := range upstream {
+		got, ok := ResourceTypeForExtension(ext)
+		if !ok {
+			t.Errorf("%s: not registered, upstream has %d", ext, want)
+			continue
+		}
+		if got != want {
+			t.Errorf("%s: registered as %d, upstream has %d", ext, got, want)
+		}
+	}
+	// Extensions upstream registers that Crucible must not reuse for anything else.
+	reserved := map[uint16]string{2039: "bte", 2067: "bak", 2070: "xbc", 2076: "tml"}
+	for number, upstreamExt := range reserved {
+		if ext, ok := ExtensionForResourceType(number); ok {
+			t.Errorf("%d: registered as %s, upstream reserves it for %s", number, ext, upstreamExt)
+		}
+	}
+}
+
+// TestNWN2FormatsAreNotRegistered keeps NWN2 file formats out of an NWN:EE
+// table. mdb and gr2 have Aurora numbers that mean something else (or nothing)
+// in NWN:EE; wlk and xml have no archive number in any Aurora game, so the
+// values Crucible used for them were invented. Packing any of these into a HAK
+// writes a resource the game misreads.
+func TestNWN2FormatsAreNotRegistered(t *testing.T) {
+	for _, ext := range []string{"mdb", "gr2", "wlk", "xml"} {
+		if number, ok := ResourceTypeForExtension(ext); ok {
+			t.Errorf("%s is an NWN2 format but is registered as 0x%04X", ext, number)
+		}
+	}
+}
+
+// TestRestypeTablesAreMutualInverses is the check init() is meant to enforce.
+func TestRestypeTablesAreMutualInverses(t *testing.T) {
+	for ext, number := range extensionTypes {
+		canonical, ok := typeExtensions[number]
+		if !ok {
+			t.Errorf("%s: number 0x%04X missing from typeExtensions", ext, number)
+			continue
+		}
+		if canonical != ext {
+			t.Errorf("%s: maps to 0x%04X, which maps back to %s", ext, number, canonical)
+		}
+	}
+	for number, ext := range typeExtensions {
+		if got, ok := extensionTypes[ext]; !ok || got != number {
+			t.Errorf("0x%04X: maps to %s, which maps back to 0x%04X (ok=%v)", number, ext, got, ok)
+		}
+	}
+}

internal/nwsync/assemble.go

@@ -0,0 +1,128 @@
+package nwsync
+
+import (
+	"crypto/sha1"
+	"encoding/json"
+	"fmt"
+	"path"
+)
+
+// AssembleOptions describes one merged manifest.
+type AssembleOptions struct {
+	// ArtifactKeys are the depot keys of the artifacts to merge, in
+	// Mod_HakList order — highest priority first. Each one's index is read
+	// from the key beside it.
+	ArtifactKeys []string
+	TLKKey       string // the TLK's key, if the manifest carries one
+	OutDir       string // write locally instead of uploading — the conformance path
+	GroupID      int    // 1 = current, 2 = testing; 0 means absent
+	ModuleName   string
+	Description  string
+	Sink         sink // test seam; nil means OutDir or the zone
+}
+
+// AssembleResult reports what one assemble run produced.
+type AssembleResult struct {
+	SHA1         string
+	ManifestPath string
+	Entries      int
+}
+
+// Assemble merges the per-artifact NSYM manifests named by Order into one
+// manifest. It reads no bulk data at all — only the small index files.
+//
+// Merge rule is resref shadowing, not concatenation: a resref present in more
+// than one artifact resolves to the earliest artifact in Order, which is how
+// the game resolves it (upstream's resman adds haks in reverse and lets the
+// last one win). Get this backwards and the wrong texture ships silently.
+func Assemble(options AssembleOptions) (AssembleResult, error) {
+	if len(options.ArtifactKeys) == 0 {
+		return AssembleResult{}, fmt.Errorf("assemble: no artifact keys given")
+	}
+
+	target, err := openSink(options.OutDir, options.Sink)
+	if err != nil {
+		return AssembleResult{}, err
+	}
+
+	// The TLK carries no precedence — it is not a hak and shadows nothing —
+	// so it merges last, after every hak has had its say.
+	keys := append([]string{}, options.ArtifactKeys...)
+	if options.TLKKey != "" {
+		keys = append(keys, options.TLKKey)
+	}
+
+	merged := make([]Entry, 0, 1024)
+	winner := make(map[Identity]bool, 1024)
+	var onDiskBytes int64
+
+	for _, artifactKey := range keys {
+		key, err := resolveIndexKey(artifactKey, options.OutDir)
+		if err != nil {
+			return AssembleResult{}, err
+		}
+		data, sidecarBody, err := target.getIndex(key)
+		if err != nil {
+			// An artifact with no index is an artifact whose emit never
+			// finished. Publishing a manifest without it would ship a release
+			// missing a hak, so this fails closed.
+			return AssembleResult{}, fmt.Errorf("assemble: no index for %s: %w", artifactKey, err)
+		}
+		entries, err := readManifest(data)
+		if err != nil {
+			return AssembleResult{}, fmt.Errorf("%s: %w", target.describe(key), err)
+		}
+		sidecar, err := parseSidecar(target.describe(key), sidecarBody)
+		if err != nil {
+			return AssembleResult{}, err
+		}
+		// Two producers of blobs means a skewed emitter can write blobs the
+		// merged manifest quietly disagrees with. Refuse to merge across
+		// mismatched emitter versions.
+		if sidecar.EmitterVersion != emitterVersion {
+			return AssembleResult{}, fmt.Errorf(
+				"assemble: emitter version mismatch: %s was emitted by emitter %q, this is emitter %q",
+				artifactKey, sidecar.EmitterVersion, emitterVersion)
+		}
+		// on_disk_bytes overcounts by the handful of cross-artifact
+		// duplicates. It is a display statistic; no dedupe pass for it.
+		onDiskBytes += sidecar.OnDiskBytes
+
+		for _, entry := range entries {
+			identity := entry.identity()
+			if winner[identity] {
+				continue
+			}
+			winner[identity] = true
+			merged = append(merged, entry)
+		}
+	}
+
+	if len(merged) == 0 {
+		return AssembleResult{}, fmt.Errorf("assemble: merged manifest is empty")
+	}
+
+	data, err := writeManifest(merged)
+	if err != nil {
+		return AssembleResult{}, err
+	}
+	sha1Hex := fmt.Sprintf("%x", sha1.Sum(data))
+	manifestKey := path.Join("manifests", sha1Hex)
+	sidecar := Sidecar{
+		ModuleName:  options.ModuleName,
+		Description: options.Description,
+		GroupID:     options.GroupID,
+	}
+	if err := putManifestPair(target, manifestKey, data, merged, onDiskBytes, sidecar); err != nil {
+		return AssembleResult{}, err
+	}
+	return AssembleResult{SHA1: sha1Hex, ManifestPath: target.describe(manifestKey), Entries: len(merged)}, nil
+}
+
+func parseSidecar(where string, data []byte) (Sidecar, error) {
+	var sidecar Sidecar
+	if err := json.Unmarshal(data, &sidecar); err != nil {
+		return Sidecar{}, fmt.Errorf("%s: %w", where, err)
+	}
+	return sidecar, nil
+}

internal/nwsync/compressedbuf.go

@@ -0,0 +1,179 @@
+package nwsync
+
+import (
+	"bytes"
+	"encoding/binary"
+	"encoding/hex"
+	"fmt"
+
+	"github.com/klauspost/compress/zstd"
+)
+
+// NWCompressedBuffer framing, as upstream's neverwinter/compressedbuf.nim
+// writes it for NWSync blobs. All fields are little-endian uint32:
+//
+//	magic "NSYC", version 3, algorithm 2 (zstd), uncompressed size,
+//	zstd header version 1, dictionary 0, then the raw zstd frame.
+const (
+	blobMagic       = 0x4359534E // "NSYC" little-endian
+	blobVersion     = 3
+	algorithmZstd   = 2
+	zstdHeaderVer   = 1
+	zstdDictionary  = 0
+	blobHeaderBytes = 24
+)
+
+// EncodeAll/DecodeAll are single-threaded per call, so the default pool of one
+// encoder per CPU only buys idle memory: each holds a window-sized history, so
+// on a 24-core runner that is ~200 MB of live heap doing nothing. Concurrency 1
+// produces byte-identical output.
+var (
+	blobEncoder, _ = zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
+	blobDecoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(1))
+)
+
+// zstd frame header bits we care about. A frame starts with the magic, then a
+// one-byte Frame_Header_Descriptor: bits 7-6 size the Frame_Content_Size field,
+// bit 5 is Single_Segment_flag, bits 1-0 size the Dictionary_ID field.
+const (
+	zstdFrameMagic      = "\x28\xb5\x2f\xfd"
+	frameSingleSegment  = 1 << 5
+	frameDictionaryMask = 0x03
+	// oneByteContentSizeCeiling is the size above which a Frame_Content_Size no
+	// longer fits in one byte. Below it the field's size flag is 0, which is
+	// what lets klauspost/compress leave the field out entirely.
+	oneByteContentSizeCeiling = 256
+)
+
+// compressBlob wraps data in NWCompressedBuffer framing.
+func compressBlob(data []byte) []byte {
+	var out bytes.Buffer
+	header := []uint32{blobMagic, blobVersion, algorithmZstd, uint32(len(data)), zstdHeaderVer, zstdDictionary}
+	for _, field := range header {
+		_ = binary.Write(&out, binary.LittleEndian, field)
+	}
+	frame := declareFrameContentSize(blobEncoder.EncodeAll(data, nil), len(data))
+	// Fail closed rather than publish a blob no client can decode. An encoder
+	// upgrade that finds a new way to omit the field would otherwise reproduce
+	// #86 in silence, and a blob is skipped by every later emit once written.
+	if !frameDeclaresContentSize(frame) {
+		panic(fmt.Sprintf("nwsync: refusing to emit a %d-byte blob whose zstd frame declares no content size (descriptor %#x)",
+			len(data), frame[4]))
+	}
+	out.Write(frame)
+	return out.Bytes()
+}
+
+// inspectBlob unwraps a stored blob the way the game client reads it, and is the
+// only reader that should be trusted to judge a published blob.
+//
+// It asserts the frame property on top of the round trip. Go's decoder — like
+// the zstd CLI — streams a frame that declares no content size, so a check that
+// only decompresses and hashes is a *more* capable decoder than the client's: it
+// certifies exactly the blobs the client rejects, which is how #86 reached
+// production and survived an audit.
+func inspectBlob(blob []byte) ([]byte, error) {
+	data, err := decompressBlob(blob)
+	if err != nil {
+		return nil, fmt.Errorf("malformed framing: %w", err)
+	}
+	if len(blob) > blobHeaderBytes && !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
+		return nil, fmt.Errorf("malformed framing: the zstd frame declares no content size, which the game client cannot decode")
+	}
+	return data, nil
+}
+
+// blobMatchesName holds a stored blob to its own file name: a blob is named
+// after the sha1 of its uncompressed bytes, so the name is a complete statement
+// about the contents and nothing else is needed to check it.
+func blobMatchesName(blob []byte, sha1Hex string) error {
+	data, err := inspectBlob(blob)
+	if err != nil {
+		return err
+	}
+	if got := hex.EncodeToString(sha1Sum(data)); got != sha1Hex {
+		return fmt.Errorf("blob %s holds the contents of %s", sha1Hex, got)
+	}
+	return nil
+}
+
+// frameDeclaresContentSize reports whether a zstd frame states how many bytes it
+// decompresses to. A frame with a zero-sized Frame_Content_Size field declares
+// one only when Single_Segment_flag is set; otherwise the size is unknown.
+func frameDeclaresContentSize(frame []byte) bool {
+	if len(frame) < 5 || string(frame[:4]) != zstdFrameMagic {
+		return false
+	}
+	descriptor := frame[4]
+	return descriptor>>6 != 0 || descriptor&frameSingleSegment != 0
+}
+
+// declareFrameContentSize rewrites a frame that does not declare its
+// Frame_Content_Size so that it does, and returns any other frame unchanged.
+//
+// klauspost/compress omits the field for inputs under 256 bytes, which the spec
+// permits. Reference libzstd never does, so the NWN client — which sizes its
+// output buffer from ZSTD_getFrameContentSize and has therefore never met a
+// frame without one — rejects the blob outright with an empty "potential
+// compression error" (#86). No encoder option changes this, so the frame is
+// re-headered here.
+//
+// The result is the shape libzstd itself emits for the same input: setting
+// Single_Segment_flag drops the Window_Descriptor byte, and the freed byte pays
+// for a one-byte Frame_Content_Size. Window_Size then equals the content size,
+// which is sound because the content is under 256 bytes and every match in it
+// therefore falls inside that window. Same length in, same length out.
+func declareFrameContentSize(frame []byte, size int) []byte {
+	if size <= 0 || size >= oneByteContentSizeCeiling || len(frame) < 6 || string(frame[:4]) != zstdFrameMagic {
+		return frame
+	}
+	descriptor := frame[4]
+	// Rewrite only the exact shape a small input produces: no declared size, no
+	// single segment, no dictionary. Anything else either declares a size
+	// already or is not a frame this reinterpretation is safe on.
+	if descriptor>>6 != 0 || descriptor&frameSingleSegment != 0 || descriptor&frameDictionaryMask != 0 {
+		return frame
+	}
+	reframed := make([]byte, len(frame))
+	copy(reframed, frame)
+	reframed[4] = descriptor | frameSingleSegment
+	reframed[5] = byte(size) // replaces Window_Descriptor
+	return reframed
+}
+
+// decompressBlob unwraps NWCompressedBuffer framing. It exists so a blob this
+// package wrote — or one upstream wrote — can be compared by its uncompressed
+// bytes, which is the only comparison that is meaningful across zstd
+// implementations.
+func decompressBlob(blob []byte) ([]byte, error) {
+	if len(blob) < blobHeaderBytes {
+		return nil, fmt.Errorf("blob too small: %d bytes", len(blob))
+	}
+	header := make([]uint32, 6)
+	if err := binary.Read(bytes.NewReader(blob[:blobHeaderBytes]), binary.LittleEndian, header); err != nil {
+		return nil, fmt.Errorf("decode blob header: %w", err)
+	}
+	switch {
+	case header[0] != blobMagic:
+		return nil, fmt.Errorf("invalid blob magic: %#x", header[0])
+	case header[1] != blobVersion:
+		return nil, fmt.Errorf("unsupported blob version: %d", header[1])
+	case header[2] != algorithmZstd:
+		return nil, fmt.Errorf("unsupported compression algorithm: %d", header[2])
+	case header[4] != zstdHeaderVer:
+		return nil, fmt.Errorf("unsupported zstd header version: %d", header[4])
+	case header[5] != zstdDictionary:
+		return nil, fmt.Errorf("zstd dictionaries are not supported")
+	}
+	if header[3] == 0 {
+		return nil, nil
+	}
+	data, err := blobDecoder.DecodeAll(blob[blobHeaderBytes:], nil)
+	if err != nil {
+		return nil, fmt.Errorf("decompress blob: %w", err)
+	}
+	if uint32(len(data)) != header[3] {
+		return nil, fmt.Errorf("blob size mismatch: header says %d, got %d", header[3], len(data))
+	}
+	return data, nil
+}

internal/nwsync/emit.go

@@ -0,0 +1,365 @@
+package nwsync
+
+import (
+	"context"
+	"crypto/sha1"
+	"fmt"
+	"io"
+	"os"
+	"path"
+	"path/filepath"
+	"slices"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/buildinfo"
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+)
+
+// fileSizeLimit matches upstream's --limit-file-size default of 15 MB. A
+// resource over it is a hard failure, not a skip: upstream quit(1)s and so do
+// we. Our largest resource today is 13.66 MiB, so the headroom is thin.
+const fileSizeLimit = 15 * 1024 * 1024
+
+// skippedTypes are never published, matching upstream's GobalResTypeSkipList.
+var skippedTypes = resTypes("nss", "ndb", "gic")
+
+// emitterVersion identifies the blob/manifest byte format this package
+// produces. assemble refuses to merge indexes that disagree on it, because two
+// producers of blobs mean a skewed emitter can otherwise write blobs the merged
+// manifest quietly disagrees with. Bump it only when emitted bytes change — it
+// is deliberately not the build revision, which would invalidate every
+// published index on every unrelated commit.
+// Version 2 declares Frame_Content_Size on every blob (#86); version 1 omitted
+// it below 256 bytes and no client could sync past such a blob.
+const emitterVersion = "2"
+
+// serverTypes are loaded only server-side; a manifest holding nothing else
+// has no client contents. Mirrors upstream's GlobalResTypeServerList, whose
+// trailing 0 is RESTYPE_INVALID.
+var serverTypes = append(resTypes(
+	"are", "dlg", "fac", "gic", "git", "ifo", "itp", "jrl", "ncs", "ndb",
+	"nss", "ptm", "utc", "utd", "ute", "uti", "utm", "utp", "uts", "utt", "utw",
+), 0)
+
+func resTypes(extensions ...string) []uint16 {
+	types := make([]uint16, 0, len(extensions))
+	for _, extension := range extensions {
+		restype, ok := erf.ResourceTypeForExtension(extension)
+		if !ok {
+			panic("nwsync: unknown restype " + extension)
+		}
+		types = append(types, restype)
+	}
+	return types
+}
+
+// EmitResult reports what one emit run produced.
+type EmitResult struct {
+	Name         string // artifact name, without extension
+	ManifestPath string
+	Entries      int
+	BlobsWritten int
+}
+
+// defaultEmitJobs is how many resources are hashed, compressed and stored at
+// once. Emit is latency-bound, not CPU-bound: a blob costs a probe round-trip
+// plus an upload round-trip, and a measured backfill spent 26 s of CPU across
+// 9.5 minutes of wall clock. The figure matches depot's DEPOT_JOBS default and
+// the transport's MaxIdleConnsPerHost, so a worker per connection needs no new
+// TLS handshake.
+const defaultEmitJobs = 16
+
+// EmitOptions describes one emit run.
+type EmitOptions struct {
+	ArtifactKey  string // depot key of the artifact; the NSYM key is derived from it
+	ArtifactPath string // the file on disk
+	As           string // name override, for a TLK whose filename is not its published name
+	OutDir       string // write locally instead of uploading — the conformance path
+	Jobs         int    // resources in flight at once; 0 means defaultEmitJobs
+	Verify       bool   // hash what would be skipped instead of trusting presence
+	Sink         sink   // test seam; nil means OutDir or the zone
+}
+
+// Emit explodes one artifact — a .hak/.erf or a loose file such as the TLK —
+// into NWSync blobs plus a NSYM manifest describing only that artifact.
+//
+// Blobs go up as they are produced and the index lands last, so the presence of
+// an index is the publication marker: an artifact whose emit died halfway has
+// real blobs in the zone and no index, which is unambiguous. Blob names are
+// content hashes, so re-running skips whatever already landed.
+func Emit(options EmitOptions) (EmitResult, error) {
+	artifact, err := os.Open(options.ArtifactPath)
+	if err != nil {
+		return EmitResult{}, fmt.Errorf("read artifact: %w", err)
+	}
+	defer artifact.Close()
+	info, err := artifact.Stat()
+	if err != nil {
+		return EmitResult{}, fmt.Errorf("read artifact: %w", err)
+	}
+	// A section reader, not the file itself: hashing must not move the file
+	// offset out from under everything that reads the artifact afterwards.
+	if err := checkArtifactKey(options.ArtifactKey, io.NewSectionReader(artifact, 0, info.Size())); err != nil {
+		return EmitResult{}, err
+	}
+	name := options.As
+	if name == "" {
+		name = path.Base(options.ArtifactKey)
+	}
+	extension := path.Ext(name)
+	name = strings.TrimSuffix(name, extension)
+
+	key, err := resolveIndexKey(options.ArtifactKey, options.OutDir)
+	if err != nil {
+		return EmitResult{}, err
+	}
+
+	index, err := readArtifactIndex(options.ArtifactPath, artifact, info.Size(), name)
+	if err != nil {
+		return EmitResult{}, err
+	}
+
+	target, err := openSink(options.OutDir, options.Sink)
+	if err != nil {
+		return EmitResult{}, err
+	}
+
+	jobs := options.Jobs
+	if jobs < 1 {
+		jobs = defaultEmitJobs
+	}
+	entries, blobs, onDiskBytes, err := emitResources(artifact, index, target, jobs, options.Verify)
+	if err != nil {
+		return EmitResult{}, err
+	}
+	if len(entries) == 0 {
+		return EmitResult{}, fmt.Errorf("%s: nothing to index (no publishable resources)", options.ArtifactPath)
+	}
+
+	data, err := writeManifest(entries)
+	if err != nil {
+		return EmitResult{}, err
+	}
+	if err := putManifestPair(target, key, data, entries, onDiskBytes, Sidecar{ModuleName: name}); err != nil {
+		return EmitResult{}, err
+	}
+	return EmitResult{Name: name, ManifestPath: target.describe(key), Entries: len(entries), BlobsWritten: blobs}, nil
+}
+
+// openSink returns the zone sink, or a local directory when outDir is set.
+func openSink(outDir string, injected sink) (sink, error) {
+	if injected != nil {
+		return injected, nil
+	}
+	if outDir != "" {
+		return dirSink{root: outDir}, nil
+	}
+	return newZoneSink(context.Background(), os.Getenv)
+}
+
+// readArtifactIndex locates the resources of an ERF/HAK/MOD, or the single
+// resource a loose file represents, without reading any payload. Upstream's
+// resman does the same dispatch on the file's first three bytes. name is the
+// artifact's published name, which for a loose file is also its resref.
+func readArtifactIndex(path string, artifact io.ReaderAt, size int64, name string) ([]erf.IndexEntry, error) {
+	magic := make([]byte, 3)
+	if size >= 3 {
+		if _, err := artifact.ReadAt(magic, 0); err != nil {
+			return nil, fmt.Errorf("%s: %w", path, err)
+		}
+	}
+	switch string(magic) {
+	case "ERF", "HAK":
+		index, err := erf.ReadIndex(artifact, size)
+		if err != nil {
+			return nil, fmt.Errorf("%s: %w", path, err)
+		}
+		return index.Entries, nil
+	case "MOD":
+		// A persistent world never publishes module contents, so the .mod
+		// contributes no bytes to a manifest — it only says which haks and
+		// which TLK the manifest covers.
+		return nil, fmt.Errorf("%s: a module is never emitted; a manifest is haks plus the TLK", path)
+	}
+	extension := filepath.Ext(filepath.Base(path))
+	restype, ok := erf.ResourceTypeForExtension(extension)
+	if !ok {
+		return nil, fmt.Errorf("%s: unknown resource type %q", path, extension)
+	}
+	return []erf.IndexEntry{{Name: name, Type: restype, Offset: 0, Size: size}}, nil
+}
+
+// emitResources hashes, compresses and stores resources, reading each payload
+// from the artifact only when its turn comes. Peak memory tracks the resources
+// in flight, not the archive: a 2 GB hak must emit inside a runner's few spare
+// GB. jobs of them are in flight at once, so the ceiling is jobs multiplied by
+// fileSizeLimit and its compressed copy — bounded, and bounded by a constant
+// this package enforces itself.
+//
+// The returned entries are in artifact order whatever order the workers finish
+// in, because a manifest's bytes are promised deterministic by emitterVersion.
+func emitResources(artifact io.ReaderAt, index []erf.IndexEntry, target sink, jobs int, verify bool) ([]Entry, int, int64, error) {
+	// A resref appearing twice inside one artifact resolves to the last one,
+	// the way resman lets the last container added win.
+	order := make([]Identity, 0, len(index))
+	latest := make(map[Identity]erf.IndexEntry, len(index))
+	var tooBig []string
+	for _, entry := range index {
+		if _, ok := erf.ExtensionForResourceType(entry.Type); !ok {
+			return nil, 0, 0, fmt.Errorf("resref %s is not resolvable (unknown restype %d)", entry.Name, entry.Type)
+		}
+		if slices.Contains(skippedTypes, entry.Type) {
+			continue
+		}
+		if entry.Size > fileSizeLimit {
+			tooBig = append(tooBig, fmt.Sprintf("%s: %d bytes > %d", entry.Name, entry.Size, fileSizeLimit))
+			continue
+		}
+		identity := Identity{ResRef: strings.ToLower(entry.Name), ResType: entry.Type}
+		if _, seen := latest[identity]; !seen {
+			order = append(order, identity)
+		}
+		latest[identity] = entry
+	}
+	if len(tooBig) > 0 {
+		sort.Strings(tooBig)
+		return nil, 0, 0, fmt.Errorf("resources exceed the file size limit:\n  %s", strings.Join(tooBig, "\n  "))
+	}
+
+	// Index-addressed, never appended to: a worker owns entries[i] alone, so
+	// the slice comes back in artifact order and needs no lock.
+	entries := make([]Entry, len(order))
+	var blobs int
+	var onDiskBytes int64
+	var mu sync.Mutex
+	var firstErr error
+	// Two resrefs in one artifact can hold identical bytes, and therefore one
+	// blob. Serially the sink's existence check absorbed that; in parallel both
+	// workers would probe, both miss, and both upload. Claiming the sha1 here
+	// restores the dedupe and skips the probe round-trip as well.
+	claimed := make(map[[20]byte]bool, len(order))
+
+	failed := func() bool {
+		mu.Lock()
+		defer mu.Unlock()
+		return firstErr != nil
+	}
+
+	store := func(i int) {
+		identity := order[i]
+		payload, err := erf.ReadPayload(artifact, latest[identity])
+		if err != nil {
+			mu.Lock()
+			if firstErr == nil {
+				firstErr = err
+			}
+			mu.Unlock()
+			return
+		}
+		sum := sha1.Sum(payload)
+		entries[i] = Entry{
+			SHA1:    sum,
+			Size:    uint32(len(payload)),
+			ResRef:  identity.ResRef,
+			ResType: identity.ResType,
+		}
+		mu.Lock()
+		duplicate := claimed[sum]
+		claimed[sum] = true
+		mu.Unlock()
+		if duplicate {
+			return
+		}
+		written, err := target.putBlob(fmt.Sprintf("%x", sum), verify, func() []byte { return compressBlob(payload) })
+		mu.Lock()
+		defer mu.Unlock()
+		if err != nil {
+			if firstErr == nil {
+				firstErr = err
+			}
+			return
+		}
+		if written > 0 {
+			blobs++
+			onDiskBytes += written
+		}
+	}
+
+	if jobs < 1 {
+		jobs = 1
+	}
+	work := make(chan int)
+	var wg sync.WaitGroup
+	for range jobs {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			for i := range work {
+				// After a failure the run is over — the caller discards
+				// everything and no index is written. Draining the rest of the
+				// channel cheaply, rather than returning, keeps the feeder from
+				// blocking on workers that have gone away.
+				if failed() {
+					continue
+				}
+				store(i)
+			}
+		}()
+	}
+	for i := range order {
+		work <- i
+	}
+	close(work)
+	wg.Wait()
+
+	if firstErr != nil {
+		return nil, 0, 0, firstErr
+	}
+	return entries, blobs, onDiskBytes, nil
+}
+
+// created is the sidecar timestamp. SOURCE_DATE_EPOCH pins it so a build can
+// be reproduced byte for byte; the manifest itself is deterministic already.
+func created() int64 {
+	if raw := os.Getenv("SOURCE_DATE_EPOCH"); raw != "" {
+		if seconds, err := strconv.ParseInt(raw, 10, 64); err == nil {
+			return seconds
+		}
+	}
+	return time.Now().Unix()
+}
+
+// putManifestPair stores a NSYM manifest and its .json sidecar at key. The
+// caller supplies the sidecar fields it knows; the rest are derived from the
+// entries. data must be the serialised form of entries.
+func putManifestPair(target sink, key string, data []byte, entries []Entry, onDiskBytes int64, sidecar Sidecar) error {
+	var totalBytes int64
+	clientContents := false
+	for _, entry := range entries {
+		totalBytes += int64(entry.Size)
+		if !slices.Contains(serverTypes, entry.ResType) {
+			clientContents = true
+		}
+	}
+	sidecar.Version = manifestVersion
+	sidecar.SHA1 = fmt.Sprintf("%x", sha1.Sum(data))
+	sidecar.HashTreeDepth = hashTreeDepth
+	sidecar.IncludesModuleContents = false
+	sidecar.IncludesClientContents = clientContents
+	sidecar.TotalFiles = len(entries)
+	sidecar.TotalBytes = totalBytes
+	sidecar.OnDiskBytes = onDiskBytes
+	sidecar.Created = created()
+	sidecar.CreatedWith = buildinfo.String()
+	sidecar.EmitterVersion = emitterVersion
+
+	body, err := marshalSidecar(sidecar)
+	if err != nil {
+		return err
+	}
+	return target.putIndex(key, data, body)
+}

internal/nwsync/jobs_test.go

@@ -0,0 +1,134 @@
+package nwsync
+
+import (
+	"bytes"
+	"fmt"
+	"path/filepath"
+	"runtime/debug"
+	"strings"
+	"testing"
+)
+
+// manyResources is a hak body with enough distinct resources that a worker pool
+// actually interleaves. Payloads differ so nothing is deduplicated away.
+func manyResources(count int) map[string][]byte {
+	contents := make(map[string][]byte, count)
+	for i := range count {
+		contents[fmt.Sprintf("res%05d.tga", i)] = []byte(fmt.Sprintf("payload %d", i))
+	}
+	return contents
+}
+
+// TestEmitProducesTheSameIndexAtEveryJobCount is the contract that lets emit be
+// parallel at all: emitterVersion promises a manifest's bytes are a function of
+// its artifact, so the number of workers must not be observable in the output.
+func TestEmitProducesTheSameIndexAtEveryJobCount(t *testing.T) {
+	// The sidecar stamps a wall-clock time unless this is set, which would make
+	// two runs differ for a reason that has nothing to do with job count.
+	t.Setenv("SOURCE_DATE_EPOCH", "1700000000")
+
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, manyResources(64))
+	key := artifactKey(t, hak)
+
+	emit := func(jobs int) (manifest, sidecar []byte, result EmitResult) {
+		out := filepath.Join(t.TempDir(), "out")
+		result, err := Emit(EmitOptions{
+			ArtifactKey:  key,
+			ArtifactPath: hak,
+			OutDir:       out,
+			Jobs:         jobs,
+		})
+		if err != nil {
+			t.Fatalf("emit at -jobs %d: %v", jobs, err)
+		}
+		manifest, sidecar, err = dirSink{root: out}.getIndex(filepath.Base(result.ManifestPath))
+		if err != nil {
+			t.Fatalf("read index at -jobs %d: %v", jobs, err)
+		}
+		return manifest, sidecar, result
+	}
+
+	serialManifest, serialSidecar, serial := emit(1)
+	parallelManifest, parallelSidecar, parallel := emit(16)
+
+	if !bytes.Equal(serialManifest, parallelManifest) {
+		t.Errorf("manifest bytes differ between -jobs 1 and -jobs 16")
+	}
+	if !bytes.Equal(serialSidecar, parallelSidecar) {
+		t.Errorf("sidecar bytes differ between -jobs 1 and -jobs 16:\n %s\n %s", serialSidecar, parallelSidecar)
+	}
+	if serial.Entries != parallel.Entries || serial.BlobsWritten != parallel.BlobsWritten {
+		t.Errorf("-jobs 1 wrote %d entries/%d blobs, -jobs 16 wrote %d/%d",
+			serial.Entries, serial.BlobsWritten, parallel.Entries, parallel.BlobsWritten)
+	}
+}
+
+// TestEmitLeavesNoIndexWhenAParallelUploadFails is the fail-closed check with
+// workers in flight: several uploads are in the air when the first one fails,
+// and the index must still never appear. Run under -race this also covers the
+// shared counters.
+func TestEmitLeavesNoIndexWhenAParallelUploadFails(t *testing.T) {
+	fixture := newZoneFixture(t)
+	fixture.zone.failOn = func(key string) bool { return strings.HasPrefix(key, "data/sha1/") }
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, manyResources(64))
+
+	if _, err := Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, hak),
+		ArtifactPath: hak,
+		Sink:         fixture.sink,
+		Jobs:         16,
+	}); err == nil {
+		t.Fatal("emit reported success after an upload failed")
+	}
+	fixture.zone.mu.Lock()
+	defer fixture.zone.mu.Unlock()
+	for key := range fixture.zone.objects {
+		if strings.HasSuffix(key, ".nsym") {
+			t.Errorf("a half-emitted artifact published an index: %s", key)
+		}
+	}
+}
+
+// TestEmitPeakMemoryIsBoundedByJobCount pins the ceiling the parallel emit
+// rests on. Peak still must not track the archive — it tracks the resources in
+// flight, so a bigger hak at the same job count costs the same.
+func TestEmitPeakMemoryIsBoundedByJobCount(t *testing.T) {
+	if testing.Short() {
+		t.Skip("writes a 64 MB fixture")
+	}
+	defer debug.SetGCPercent(debug.SetGCPercent(10))
+
+	measure := func(count, jobs int) uint64 {
+		dir := t.TempDir()
+		hak := filepath.Join(dir, "big.hak")
+		writeStreamedHak(t, hak, count)
+		options := EmitOptions{
+			ArtifactKey:  artifactKey(t, hak),
+			ArtifactPath: hak,
+			As:           filepath.Base(hak),
+			OutDir:       filepath.Join(dir, "out"),
+			Jobs:         jobs,
+		}
+		return peakHeapDuring(func() {
+			if _, err := Emit(options); err != nil {
+				t.Fatalf("emit %d resources at -jobs %d: %v", count, jobs, err)
+			}
+		})
+	}
+
+	const jobs = 8
+	small := measure(8, jobs)  // 8 MB
+	large := measure(64, jobs) // 64 MB
+	// Each worker may hold one resourceSize payload plus its compressed copy,
+	// so the pool itself is the slack — not the archive.
+	const slack = 24 << 20
+
+	t.Logf("peak heap at -jobs %d: 8 MB hak %d bytes, 64 MB hak %d bytes", jobs, small, large)
+	if large > small+slack {
+		t.Fatalf("peak heap scaled with artifact size at -jobs %d: 8 MB hak peaked at %d bytes, 64 MB hak at %d", jobs, small, large)
+	}
+}

internal/nwsync/manifest.go

@@ -0,0 +1,208 @@
+package nwsync
+
+import (
+	"bytes"
+	"encoding/binary"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"io"
+	"path"
+	"path/filepath"
+	"sort"
+	"strings"
+)
+
+// NSYM manifest, version 3, exactly as upstream neverwinter/nwsync.nim writes
+// it. Everything is little-endian:
+//
+//	"NSYM", uint32 version, uint32 entry count, uint32 mapping count,
+//	entries:  byte[20] raw sha1, uint32 size, char[16] resref, uint16 restype
+//	mappings: uint32 entry index,             char[16] resref, uint16 restype
+//
+// Entries are sorted by lowercase sha1 hex then resref, and a resource whose
+// sha1 was already written becomes a mapping instead of a second entry.
+const (
+	manifestVersion = 3
+	hashTreeDepth   = 2
+	resRefBytes     = 16
+)
+
+// Entry is one resource in a manifest.
+type Entry struct {
+	SHA1    [20]byte
+	Size    uint32
+	ResRef  string // lowercase, no extension, at most 16 characters
+	ResType uint16
+}
+
+func (e Entry) sha1Hex() string { return hex.EncodeToString(e.SHA1[:]) }
+
+// Identity is what a resref resolves by: name plus type. It is the merge key
+// inside one artifact and across artifacts alike.
+type Identity struct {
+	ResRef  string
+	ResType uint16
+}
+
+func (e Entry) identity() Identity { return Identity{ResRef: e.ResRef, ResType: e.ResType} }
+
+// writeManifest serialises entries into NSYM v3 bytes.
+func writeManifest(entries []Entry) ([]byte, error) {
+	sorted := make([]Entry, len(entries))
+	copy(sorted, entries)
+	sort.SliceStable(sorted, func(i, j int) bool {
+		left, right := sorted[i].sha1Hex(), sorted[j].sha1Hex()
+		if left != right {
+			return left < right
+		}
+		return sorted[i].ResRef < sorted[j].ResRef
+	})
+
+	var body, mappings bytes.Buffer
+	seen := make(map[string]uint32, len(sorted))
+	var entryCount, mappingCount uint32
+	for _, entry := range sorted {
+		padded, err := padResRef(entry.ResRef)
+		if err != nil {
+			return nil, err
+		}
+		if index, ok := seen[entry.sha1Hex()]; ok {
+			_ = binary.Write(&mappings, binary.LittleEndian, index)
+			mappings.Write(padded)
+			_ = binary.Write(&mappings, binary.LittleEndian, entry.ResType)
+			mappingCount++
+			continue
+		}
+		seen[entry.sha1Hex()] = entryCount
+		entryCount++
+		body.Write(entry.SHA1[:])
+		_ = binary.Write(&body, binary.LittleEndian, entry.Size)
+		body.Write(padded)
+		_ = binary.Write(&body, binary.LittleEndian, entry.ResType)
+	}
+
+	var out bytes.Buffer
+	out.WriteString("NSYM")
+	for _, field := range []uint32{manifestVersion, entryCount, mappingCount} {
+		_ = binary.Write(&out, binary.LittleEndian, field)
+	}
+	out.Write(body.Bytes())
+	out.Write(mappings.Bytes())
+	return out.Bytes(), nil
+}
+
+// readManifest parses NSYM v3 bytes. Mappings are expanded back into entries,
+// the way upstream's reader does, so a caller sees one entry per resref.
+func readManifest(data []byte) ([]Entry, error) {
+	reader := bytes.NewReader(data)
+	magic := make([]byte, 4)
+	if _, err := io.ReadFull(reader, magic); err != nil || string(magic) != "NSYM" {
+		return nil, fmt.Errorf("not a manifest (invalid magic bytes)")
+	}
+	var version, entryCount, mappingCount uint32
+	for _, field := range []*uint32{&version, &entryCount, &mappingCount} {
+		if err := binary.Read(reader, binary.LittleEndian, field); err != nil {
+			return nil, fmt.Errorf("truncated manifest header: %w", err)
+		}
+	}
+	if version != manifestVersion {
+		return nil, fmt.Errorf("unsupported manifest version %d", version)
+	}
+
+	entries := make([]Entry, 0, entryCount+mappingCount)
+	for i := uint32(0); i < entryCount; i++ {
+		var entry Entry
+		if _, err := io.ReadFull(reader, entry.SHA1[:]); err != nil {
+			return nil, fmt.Errorf("truncated entry %d: %w", i, err)
+		}
+		if err := binary.Read(reader, binary.LittleEndian, &entry.Size); err != nil {
+			return nil, fmt.Errorf("truncated entry %d: %w", i, err)
+		}
+		resref, restype, err := readResRef(reader)
+		if err != nil {
+			return nil, fmt.Errorf("truncated entry %d: %w", i, err)
+		}
+		entry.ResRef, entry.ResType = resref, restype
+		entries = append(entries, entry)
+	}
+	for i := uint32(0); i < mappingCount; i++ {
+		var index uint32
+		if err := binary.Read(reader, binary.LittleEndian, &index); err != nil {
+			return nil, fmt.Errorf("truncated mapping %d: %w", i, err)
+		}
+		if index >= entryCount {
+			return nil, fmt.Errorf("mapping %d references non-existent entry %d", i, index)
+		}
+		resref, restype, err := readResRef(reader)
+		if err != nil {
+			return nil, fmt.Errorf("truncated mapping %d: %w", i, err)
+		}
+		target := entries[index]
+		entries = append(entries, Entry{SHA1: target.SHA1, Size: target.Size, ResRef: resref, ResType: restype})
+	}
+	return entries, nil
+}
+
+func readResRef(reader *bytes.Reader) (string, uint16, error) {
+	raw := make([]byte, resRefBytes)
+	if _, err := io.ReadFull(reader, raw); err != nil {
+		return "", 0, err
+	}
+	var restype uint16
+	if err := binary.Read(reader, binary.LittleEndian, &restype); err != nil {
+		return "", 0, err
+	}
+	return strings.ToLower(string(bytes.TrimRight(raw, "\x00"))), restype, nil
+}
+
+func padResRef(resref string) ([]byte, error) {
+	if len(resref) > resRefBytes {
+		return nil, fmt.Errorf("resref %q exceeds %d characters", resref, resRefBytes)
+	}
+	padded := make([]byte, resRefBytes)
+	copy(padded, strings.ToLower(resref))
+	return padded, nil
+}
+
+// Sidecar is the .json file written next to every manifest. Clients fetch it
+// (nwn_nwsync_fetch.nim), so the field names and order match upstream.
+// Upstream omits an integer meta field whose value is 0, hence group_id's
+// omitempty: 0 means absent, not "group zero".
+type Sidecar struct {
+	Version                int    `json:"version"`
+	SHA1                   string `json:"sha1"`
+	HashTreeDepth          int    `json:"hash_tree_depth"`
+	ModuleName             string `json:"module_name"`
+	Description            string `json:"description"`
+	IncludesModuleContents bool   `json:"includes_module_contents"`
+	IncludesClientContents bool   `json:"includes_client_contents"`
+	TotalFiles             int    `json:"total_files"`
+	TotalBytes             int64  `json:"total_bytes"`
+	OnDiskBytes            int64  `json:"on_disk_bytes"`
+	Created                int64  `json:"created"`
+	CreatedWith            string `json:"created_with"`
+	EmitterVersion         string `json:"emitter_version,omitempty"`
+	GroupID                int    `json:"group_id,omitempty"`
+}
+
+func marshalSidecar(sidecar Sidecar) ([]byte, error) {
+	body, err := json.MarshalIndent(sidecar, "", "  ")
+	if err != nil {
+		return nil, err
+	}
+	// Upstream terminates the file with CRLF; match it.
+	return append(body, '\r', '\n'), nil
+}
+
+// blobKey is where a blob lives in a zone, hash tree depth 2. emit writes it,
+// verify reads it and the game client requests it, so the rule lives here and
+// nowhere else.
+func blobKey(sha1Hex string) string {
+	return path.Join("data", "sha1", sha1Hex[0:2], sha1Hex[2:4], sha1Hex)
+}
+
+// blobPath is the same location inside a local repository tree.
+func blobPath(root, sha1Hex string) string {
+	return filepath.Join(root, filepath.FromSlash(blobKey(sha1Hex)))
+}

internal/nwsync/memory_test.go

@@ -0,0 +1,150 @@
+package nwsync
+
+import (
+	"bytes"
+	"fmt"
+	"math/rand"
+	"os"
+	"path/filepath"
+	"runtime"
+	"runtime/debug"
+	"testing"
+	"time"
+
+	"github.com/klauspost/compress/zstd"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+)
+
+// TestSingleThreadedEncoderMatchesDefault pins the claim the blob encoder's
+// concurrency setting rests on: it saves memory only, and a published blob is
+// the same bytes either way.
+func TestSingleThreadedEncoderMatchesDefault(t *testing.T) {
+	standard, err := zstd.NewWriter(nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer standard.Close()
+
+	body := make([]byte, 4<<20)
+	random := rand.New(rand.NewSource(1))
+	random.Read(body[:len(body)/2])
+	for _, size := range []int{0, 1, 4 << 10, len(body)} {
+		if !bytes.Equal(blobEncoder.EncodeAll(body[:size], nil), standard.EncodeAll(body[:size], nil)) {
+			t.Fatalf("%d bytes compress differently at concurrency 1", size)
+		}
+	}
+}
+
+// resourceSize is one payload in the memory fixtures. Real haks hold a few MB
+// per resource, and peak memory is meant to track that, not the archive.
+const resourceSize = 1 << 20
+
+// writeStreamedHak builds a hak of count resources without ever holding the
+// archive in memory, so the fixture itself does not decide the measurement.
+// Payloads are distinct, so no blob is deduplicated away.
+func writeStreamedHak(t *testing.T, path string, count int) {
+	t.Helper()
+	payload := filepath.Join(t.TempDir(), "payload.bin")
+	body := make([]byte, resourceSize)
+	for index := range body {
+		body[index] = byte(index)
+	}
+
+	resources := make([]erf.Resource, 0, count)
+	for index := range count {
+		// A distinct first byte per resource is enough to give every payload
+		// its own sha1 while still streaming from one file per resource.
+		unique := filepath.Join(filepath.Dir(payload), fmt.Sprintf("p%d.bin", index))
+		body[0] = byte(index)
+		body[1] = byte(index >> 8)
+		if err := os.WriteFile(unique, body, 0o644); err != nil {
+			t.Fatal(err)
+		}
+		resources = append(resources, erf.Resource{
+			Name:       fmt.Sprintf("res%05d", index),
+			Type:       restype(t, "tga"),
+			SourcePath: unique,
+			Size:       resourceSize,
+		})
+	}
+
+	file, err := os.Create(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer file.Close()
+	if err := erf.Write(file, erf.New("HAK", resources)); err != nil {
+		t.Fatalf("write hak: %v", err)
+	}
+}
+
+// peakHeapDuring runs work while sampling the heap, and returns the largest
+// live heap it saw.
+func peakHeapDuring(work func()) uint64 {
+	runtime.GC()
+	done := make(chan struct{})
+	peak := make(chan uint64, 1)
+	go func() {
+		var highest uint64
+		var stats runtime.MemStats
+		for {
+			select {
+			case <-done:
+				peak <- highest
+				return
+			default:
+			}
+			runtime.ReadMemStats(&stats)
+			if stats.HeapAlloc > highest {
+				highest = stats.HeapAlloc
+			}
+			time.Sleep(time.Millisecond)
+		}
+	}()
+	work()
+	close(done)
+	return <-peak
+}
+
+// TestEmitPeakMemoryDoesNotScaleWithArtifactSize is the regression check for
+// the OOM kills on large haks: emit used to hold the whole archive (twice), so
+// a 2 GB hak needed about 10 GB. Emitting an archive 8× bigger must not cost
+// meaningfully more memory.
+func TestEmitPeakMemoryDoesNotScaleWithArtifactSize(t *testing.T) {
+	if testing.Short() {
+		t.Skip("writes a 64 MB fixture")
+	}
+	// A lazy GC lets garbage pile up in proportion to the live heap, which
+	// hides the thing under test. Collecting eagerly makes the sampled heap
+	// track what emit actually holds.
+	defer debug.SetGCPercent(debug.SetGCPercent(10))
+
+	measure := func(count int) uint64 {
+		dir := t.TempDir()
+		hak := filepath.Join(dir, "big.hak")
+		writeStreamedHak(t, hak, count)
+		// The key is computed outside the measurement: the test helper reads
+		// the whole file to hash it, which emit itself no longer does.
+		options := EmitOptions{
+			ArtifactKey:  artifactKey(t, hak),
+			ArtifactPath: hak,
+			As:           filepath.Base(hak),
+			OutDir:       filepath.Join(dir, "out"),
+		}
+		return peakHeapDuring(func() {
+			if _, err := Emit(options); err != nil {
+				t.Fatalf("emit %d resources: %v", count, err)
+			}
+		})
+	}
+
+	small := measure(8)  // 8 MB
+	large := measure(64) // 64 MB
+	const slack = 24 << 20
+
+	t.Logf("peak heap: 8 MB hak %d bytes, 64 MB hak %d bytes", small, large)
+	if large > small+slack {
+		t.Fatalf("peak heap scaled with artifact size: 8 MB hak peaked at %d bytes, 64 MB hak at %d", small, large)
+	}
+}

internal/nwsync/nwsync_test.go

@@ -0,0 +1,531 @@
+package nwsync
+
+import (
+	"bytes"
+	"crypto/sha1"
+	"crypto/sha256"
+	"encoding/binary"
+	"encoding/hex"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+)
+
+func restype(t *testing.T, extension string) uint16 {
+	t.Helper()
+	value, ok := erf.ResourceTypeForExtension(extension)
+	if !ok {
+		t.Fatalf("unknown restype %q", extension)
+	}
+	return value
+}
+
+// writeHak builds a HAK fixture from resref.ext => body pairs.
+func writeHak(t *testing.T, path string, contents map[string][]byte) {
+	t.Helper()
+	resources := make([]erf.Resource, 0, len(contents))
+	for name, body := range contents {
+		stem, extension, _ := strings.Cut(name, ".")
+		resources = append(resources, erf.Resource{
+			Name: stem,
+			Type: restype(t, extension),
+			Data: body,
+			Size: int64(len(body)),
+		})
+	}
+	var out bytes.Buffer
+	if err := erf.Write(&out, erf.New("HAK", resources)); err != nil {
+		t.Fatalf("write hak: %v", err)
+	}
+	if err := os.WriteFile(path, out.Bytes(), 0o644); err != nil {
+		t.Fatalf("write hak file: %v", err)
+	}
+}
+
+// artifactKey is the depot key a file would be published under: the sha256 of
+// its bytes, hash-tree depth 2, keeping the extension.
+func artifactKey(t *testing.T, path string) string {
+	t.Helper()
+	body, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	sum := sha256.Sum256(body)
+	digest := hex.EncodeToString(sum[:])
+	return "artifacts/haks/sha256/" + digest[0:2] + "/" + digest[2:4] + "/" + digest + filepath.Ext(path)
+}
+
+// emitLocal emits one artifact into a local tree, the conformance path.
+func emitLocal(t *testing.T, path, out string) (EmitResult, error) {
+	t.Helper()
+	return Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, path),
+		ArtifactPath: path,
+		As:           filepath.Base(path),
+		OutDir:       out,
+	})
+}
+
+func TestBlobFramingRoundTrips(t *testing.T) {
+	data := []byte("the quick brown fox jumps over the lazy dog, repeatedly and at length")
+	blob := compressBlob(data)
+
+	header := make([]uint32, 6)
+	if err := binary.Read(bytes.NewReader(blob[:blobHeaderBytes]), binary.LittleEndian, header); err != nil {
+		t.Fatalf("read header: %v", err)
+	}
+	want := []uint32{blobMagic, 3, 2, uint32(len(data)), 1, 0}
+	for i := range want {
+		if header[i] != want[i] {
+			t.Errorf("header field %d = %d, want %d", i, header[i], want[i])
+		}
+	}
+	if string(blob[:4]) != "NSYC" {
+		t.Errorf("magic bytes = %q, want NSYC", blob[:4])
+	}
+
+	got, err := decompressBlob(blob)
+	if err != nil {
+		t.Fatalf("decompress: %v", err)
+	}
+	if !bytes.Equal(got, data) {
+		t.Errorf("round trip mismatch: %q", got)
+	}
+}
+
+func TestManifestBytesMatchUpstreamLayout(t *testing.T) {
+	shared := sha1.Sum([]byte("shared"))
+	other := sha1.Sum([]byte("other"))
+	// Deliberately out of order, and with two resrefs sharing one sha1: the
+	// second one must become a mapping, not a second entry.
+	entries := []Entry{
+		{SHA1: other, Size: 5, ResRef: "zzz", ResType: 1},
+		{SHA1: shared, Size: 6, ResRef: "bbb", ResType: 2},
+		{SHA1: shared, Size: 6, ResRef: "aaa", ResType: 3},
+	}
+	data, err := writeManifest(entries)
+	if err != nil {
+		t.Fatalf("write manifest: %v", err)
+	}
+
+	if string(data[:4]) != "NSYM" {
+		t.Fatalf("magic = %q", data[:4])
+	}
+	var version, entryCount, mappingCount uint32
+	reader := bytes.NewReader(data[4:16])
+	for _, field := range []*uint32{&version, &entryCount, &mappingCount} {
+		_ = binary.Read(reader, binary.LittleEndian, field)
+	}
+	if version != 3 || entryCount != 2 || mappingCount != 1 {
+		t.Fatalf("header = version %d, %d entries, %d mappings; want 3/2/1", version, entryCount, mappingCount)
+	}
+	wantSize := 16 + int(entryCount)*(20+4+16+2) + int(mappingCount)*(4+16+2)
+	if len(data) != wantSize {
+		t.Fatalf("manifest is %d bytes, want %d", len(data), wantSize)
+	}
+
+	// Sorted by sha1 hex then resref, so the shared hash's "aaa" is the entry
+	// and "bbb" is demoted to a mapping.
+	round, err := readManifest(data)
+	if err != nil {
+		t.Fatalf("read manifest: %v", err)
+	}
+	if len(round) != 3 {
+		t.Fatalf("round trip returned %d entries, want 3", len(round))
+	}
+	byResRef := map[string]Entry{}
+	for _, entry := range round {
+		byResRef[entry.ResRef] = entry
+	}
+	for _, entry := range entries {
+		got, ok := byResRef[entry.ResRef]
+		if !ok {
+			t.Fatalf("resref %q lost in round trip", entry.ResRef)
+		}
+		if got != entry {
+			t.Errorf("resref %q = %+v, want %+v", entry.ResRef, got, entry)
+		}
+	}
+	if round[0].ResRef != "aaa" && round[1].ResRef != "aaa" {
+		t.Errorf("entries are not sorted by sha1 then resref: %+v", round)
+	}
+}
+
+func TestEmitWritesBlobsAndManifest(t *testing.T) {
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	body := []byte("texture bytes")
+	writeHak(t, hak, map[string][]byte{
+		"bloodstain1.tga": body,
+		"copy1.txi":       body, // same content, different resref: one blob
+		"script1.nss":     []byte("void main() {}"),
+		"debug1.ndb":      []byte("debug"),
+		"comment1.gic":    []byte("comment"),
+	})
+
+	out := filepath.Join(dir, "out")
+	result, err := emitLocal(t, hak, out)
+	if err != nil {
+		t.Fatalf("emit: %v", err)
+	}
+	if result.Entries != 2 {
+		t.Errorf("emitted %d entries, want 2 (nss/ndb/gic are always skipped)", result.Entries)
+	}
+	if result.BlobsWritten != 1 {
+		t.Errorf("wrote %d blobs, want 1 (identical content shares a blob)", result.BlobsWritten)
+	}
+
+	// The blob is named by the sha1 of the uncompressed bytes, under a depth-2
+	// hash tree, and decompresses back to exactly those bytes.
+	sum := sha1.Sum(body)
+	name := hex.EncodeToString(sum[:])
+	path := filepath.Join(out, "data", "sha1", name[0:2], name[2:4], name)
+	blob, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatalf("blob missing at %s: %v", path, err)
+	}
+	got, err := decompressBlob(blob)
+	if err != nil {
+		t.Fatalf("decompress blob: %v", err)
+	}
+	if !bytes.Equal(got, body) {
+		t.Errorf("blob decompressed to %q, want %q", got, body)
+	}
+
+	entries := readEmitted(t, result.ManifestPath)
+	for _, entry := range entries {
+		if entry.ResType == restype(t, "nss") || entry.ResType == restype(t, "ndb") || entry.ResType == restype(t, "gic") {
+			t.Errorf("skipped restype leaked into the manifest: %+v", entry)
+		}
+	}
+
+	var sidecar Sidecar
+	body2, err := os.ReadFile(result.ManifestPath + ".json")
+	if err != nil {
+		t.Fatalf("sidecar missing: %v", err)
+	}
+	if err := json.Unmarshal(body2, &sidecar); err != nil {
+		t.Fatalf("sidecar json: %v", err)
+	}
+	if sidecar.TotalFiles != 2 || sidecar.HashTreeDepth != 2 || sidecar.Version != 3 {
+		t.Errorf("sidecar = %+v", sidecar)
+	}
+	if sidecar.IncludesModuleContents {
+		t.Error("sidecar claims module contents; a published manifest never has them")
+	}
+	if strings.Contains(string(body2), "group_id") {
+		t.Error("per-artifact sidecar should omit group_id (0 means absent)")
+	}
+	if _, err := os.Stat(filepath.Join(out, "latest")); err == nil {
+		t.Error("a latest file was written; there must never be one")
+	}
+}
+
+func readEmitted(t *testing.T, indexPath string) []Entry {
+	t.Helper()
+	data, err := os.ReadFile(indexPath)
+	if err != nil {
+		t.Fatalf("read emitted manifest: %v", err)
+	}
+	entries, err := readManifest(data)
+	if err != nil {
+		t.Fatalf("parse emitted manifest: %v", err)
+	}
+	return entries
+}
+
+func TestEmitFailsClosedOnOversizeResource(t *testing.T) {
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "big.hak")
+	writeHak(t, hak, map[string][]byte{"huge1.tga": make([]byte, fileSizeLimit+1)})
+
+	if _, err := emitLocal(t, hak, filepath.Join(dir, "out")); err == nil {
+		t.Fatal("emit accepted a resource over the 15 MB limit")
+	}
+}
+
+func TestEmitLooseFile(t *testing.T) {
+	dir := t.TempDir()
+	tlk := filepath.Join(dir, "sow_tlk.tlk")
+	if err := os.WriteFile(tlk, []byte("TLK V3.0 payload"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	out := filepath.Join(dir, "out")
+	result, err := emitLocal(t, tlk, out)
+	if err != nil {
+		t.Fatalf("emit tlk: %v", err)
+	}
+	entries := readEmitted(t, result.ManifestPath)
+	if len(entries) != 1 || entries[0].ResRef != "sow_tlk" || entries[0].ResType != restype(t, "tlk") {
+		t.Fatalf("tlk emitted as %+v", entries)
+	}
+	if result.BlobsWritten != 1 {
+		t.Errorf("wrote %d blobs, want 1", result.BlobsWritten)
+	}
+}
+
+// emitFixture emits two haks that share a resref, so the merge rule is
+// observable: "top" holds the winning body, "assets" the shadowed one.
+func emitFixture(t *testing.T) (out string, keys map[string]string, topBody, assetBody []byte) {
+	t.Helper()
+	dir := t.TempDir()
+	topBody = []byte("2da from sow_top")
+	assetBody = []byte("2da from the asset hak")
+	writeHak(t, filepath.Join(dir, "sow_top.hak"), map[string][]byte{"appearance.2da": topBody})
+	writeHak(t, filepath.Join(dir, "sow_core_01.hak"), map[string][]byte{
+		"appearance.2da":  assetBody,
+		"bloodstain1.tga": []byte("blood"),
+	})
+	out = filepath.Join(dir, "out")
+	keys = map[string]string{}
+	for _, name := range []string{"sow_top", "sow_core_01"} {
+		path := filepath.Join(dir, name+".hak")
+		keys[name] = artifactKey(t, path)
+		if _, err := emitLocal(t, path, out); err != nil {
+			t.Fatalf("emit %s: %v", name, err)
+		}
+	}
+	return out, keys, topBody, assetBody
+}
+
+func TestAssembleShadowsByOrder(t *testing.T) {
+	entriesDir, keys, topBody, assetBody := emitFixture(t)
+
+	result, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{keys["sow_top"], keys["sow_core_01"]},
+		OutDir:       entriesDir,
+		GroupID:      2,
+	})
+	if err != nil {
+		t.Fatalf("assemble: %v", err)
+	}
+	if result.Entries != 2 {
+		t.Fatalf("merged %d entries, want 2 (appearance.2da is shadowed, not duplicated)", result.Entries)
+	}
+
+	data, err := os.ReadFile(result.ManifestPath)
+	if err != nil {
+		t.Fatalf("read merged manifest: %v", err)
+	}
+	merged := sha1.Sum(data)
+	if hex.EncodeToString(merged[:]) != result.SHA1 || filepath.Base(result.ManifestPath) != result.SHA1 {
+		t.Errorf("manifest is not named by its own sha1: %s", result.ManifestPath)
+	}
+	entries, err := readManifest(data)
+	if err != nil {
+		t.Fatalf("parse merged manifest: %v", err)
+	}
+	for _, entry := range entries {
+		if entry.ResRef != "appearance" {
+			continue
+		}
+		if entry.SHA1 != sha1.Sum(topBody) {
+			t.Errorf("appearance.2da resolved to the wrong hak; want the earliest in --order")
+		}
+		if entry.SHA1 == sha1.Sum(assetBody) {
+			t.Error("appearance.2da resolved to the shadowed hak")
+		}
+	}
+
+	sidecar := Sidecar{}
+	body, err := os.ReadFile(result.ManifestPath + ".json")
+	if err != nil {
+		t.Fatalf("merged sidecar missing: %v", err)
+	}
+	if err := json.Unmarshal(body, &sidecar); err != nil {
+		t.Fatalf("merged sidecar json: %v", err)
+	}
+	if sidecar.GroupID != 2 {
+		t.Errorf("group_id = %d, want 2 (testing)", sidecar.GroupID)
+	}
+	if sidecar.SHA1 != result.SHA1 {
+		t.Errorf("sidecar sha1 = %s, want %s", sidecar.SHA1, result.SHA1)
+	}
+	if sidecar.TotalFiles != 2 {
+		t.Errorf("total_files = %d, want 2", sidecar.TotalFiles)
+	}
+}
+
+func TestAssembleReversedOrderPicksTheOtherHak(t *testing.T) {
+	entriesDir, keys, topBody, assetBody := emitFixture(t)
+
+	result, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{keys["sow_core_01"], keys["sow_top"]},
+		OutDir:       entriesDir,
+	})
+	if err != nil {
+		t.Fatalf("assemble: %v", err)
+	}
+	data, _ := os.ReadFile(result.ManifestPath)
+	entries, err := readManifest(data)
+	if err != nil {
+		t.Fatalf("parse merged manifest: %v", err)
+	}
+	for _, entry := range entries {
+		if entry.ResRef == "appearance" && entry.SHA1 != sha1.Sum(assetBody) {
+			t.Errorf("appearance.2da did not follow --order; still resolves to %x", entry.SHA1)
+		}
+	}
+	_ = topBody
+}
+
+func TestAssembleRefusesMismatchedEmitterVersions(t *testing.T) {
+	entriesDir, keys, _, _ := emitFixture(t)
+
+	index, err := resolveIndexKey(keys["sow_core_01"], entriesDir)
+	if err != nil {
+		t.Fatal(err)
+	}
+	path := filepath.Join(entriesDir, index+".json")
+	var sidecar Sidecar
+	body, err := os.ReadFile(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := json.Unmarshal(body, &sidecar); err != nil {
+		t.Fatal(err)
+	}
+	sidecar.EmitterVersion = "0"
+	patched, err := marshalSidecar(sidecar)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path, patched, 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	_, err = Assemble(AssembleOptions{
+		ArtifactKeys: []string{keys["sow_top"], keys["sow_core_01"]},
+		OutDir:       entriesDir,
+	})
+	if err == nil || !strings.Contains(err.Error(), "emitter version mismatch") {
+		t.Fatalf("assemble merged across emitter versions: %v", err)
+	}
+}
+
+func TestEmitHonoursSourceDateEpoch(t *testing.T) {
+	t.Setenv("SOURCE_DATE_EPOCH", "1700000000")
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "pinned.hak")
+	writeHak(t, hak, map[string][]byte{"one1.tga": []byte("body")})
+	result, err := emitLocal(t, hak, filepath.Join(dir, "out"))
+	if err != nil {
+		t.Fatalf("emit: %v", err)
+	}
+	body, err := os.ReadFile(result.ManifestPath + ".json")
+	if err != nil {
+		t.Fatal(err)
+	}
+	var sidecar Sidecar
+	if err := json.Unmarshal(body, &sidecar); err != nil {
+		t.Fatal(err)
+	}
+	if sidecar.Created != 1700000000 {
+		t.Errorf("created = %d, want the pinned SOURCE_DATE_EPOCH", sidecar.Created)
+	}
+}
+
+func TestEmitRejectsAModule(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "sow.mod")
+	var out bytes.Buffer
+	if err := erf.Write(&out, erf.New("MOD", []erf.Resource{
+		{Name: "module", Type: restype(t, "ifo"), Data: []byte("ifo"), Size: 3},
+	})); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(path, out.Bytes(), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := emitLocal(t, path, filepath.Join(dir, "out")); err == nil {
+		t.Fatal("emit accepted a .mod; a manifest never carries module contents")
+	}
+}
+
+func TestAssembleFailsClosedOnMissingIndex(t *testing.T) {
+	entriesDir, keys, _, _ := emitFixture(t)
+	missing := "artifacts/haks/sha256/00/11/" + strings.Repeat("0", 64) + ".hak"
+	_, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{keys["sow_top"], missing},
+		OutDir:       entriesDir,
+	})
+	if err == nil || !strings.Contains(err.Error(), missing) {
+		t.Fatalf("assemble did not fail closed and name the missing artifact: %v", err)
+	}
+}
+
+// Callers capture a script's stdout as a value: `dir="$(pack-haks.sh)"`. A
+// summary line on stdout gets glued onto that value, so both summaries belong
+// on stderr.
+func TestRunKeepsSummariesOffStdout(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "sow_top.hak")
+	writeHak(t, path, map[string][]byte{"appearance.2da": []byte("2da from sow_top")})
+	key := artifactKey(t, path)
+	out := filepath.Join(dir, "out")
+
+	for _, args := range [][]string{
+		{"emit", "--out", out, "--as", "sow_top.hak", key, path},
+		{"assemble", "--out", out, key},
+	} {
+		var stdout, stderr bytes.Buffer
+		if code := Run(args, &stdout, &stderr); code != exitOK {
+			t.Fatalf("Run(%v) exit=%d: %s", args, code, stderr.String())
+		}
+		if stdout.Len() != 0 {
+			t.Errorf("Run(%v) wrote to stdout: %q", args, stdout.String())
+		}
+		if stderr.Len() == 0 {
+			t.Errorf("Run(%v) reported no summary on stderr", args)
+		}
+	}
+}
+
+func TestRunUsageErrors(t *testing.T) {
+	cases := [][]string{
+		nil,
+		{"nope"},
+		{"emit"},
+		{"emit", "artifact-key.hak"},
+		{"emit", "a", "b", "c"},
+		{"assemble", "--out", "y"},
+	}
+	for _, args := range cases {
+		var out, errw bytes.Buffer
+		if code := Run(args, &out, &errw); code != exitUsage {
+			t.Errorf("Run(%v) exit=%d, want %d", args, code, exitUsage)
+		}
+	}
+}
+
+// TestEveryBlobDeclaresItsFrameContentSize guards the fault that stopped every
+// client sync (#86): klauspost/compress omits Frame_Content_Size for inputs
+// under 256 bytes, and the game client cannot decode a frame without it. This
+// asserts a frame property, not a round trip — the zstd CLI and Go's decoder
+// both stream such a frame happily, so round-tripping cannot see the defect.
+func TestEveryBlobDeclaresItsFrameContentSize(t *testing.T) {
+	// 230 and 175 are real sizes from the manifest that failed to sync; 255/256
+	// straddle the encoder's threshold.
+	for _, size := range []int{1, 32, 175, 230, 255, 256, 257, 1024, 5000} {
+		payload := make([]byte, size)
+		for i := range payload {
+			payload[i] = byte('a' + i%26)
+		}
+		blob := compressBlob(payload)
+		if !frameDeclaresContentSize(blob[blobHeaderBytes:]) {
+			t.Errorf("blob of %d bytes declares no frame content size (descriptor %#x)",
+				size, blob[blobHeaderBytes+4])
+		}
+		got, err := decompressBlob(blob)
+		if err != nil {
+			t.Fatalf("decompress %d-byte blob: %v", size, err)
+		}
+		if !bytes.Equal(got, payload) {
+			t.Errorf("%d-byte blob did not round trip", size)
+		}
+	}
+}

internal/nwsync/run.go

@@ -0,0 +1,202 @@
+// Package nwsync publishes NWSync repository data: blobs and a per-artifact
+// NSYM manifest at each artifact's birth (emit), and one merged manifest at
+// module release (assemble).
+//
+// The split exists because upstream nwn_nwsync_write wants every hak, the TLK
+// and the module present in one run on one disk, which our build hosts cannot
+// hold. Upstream stays the conformance oracle: manifests compare byte for
+// byte, blobs compare after decompression.
+package nwsync
+
+import (
+	"flag"
+	"fmt"
+	"io"
+	"os"
+)
+
+const (
+	exitOK       = 0
+	exitUsage    = 64
+	exitInternal = 70
+	// exitDrift says the command worked and the zone is wrong, which is a
+	// different thing for CI to act on than the command failing. It matches
+	// depot's code for the same meaning.
+	exitDrift = 1
+)
+
+// Run executes an nwsync subcommand. args[0] is the subcommand (emit|assemble);
+// returns the process exit code.
+func Run(args []string, stdout, stderr io.Writer) int {
+	if len(args) == 0 {
+		printRunUsage(stderr)
+		return exitUsage
+	}
+	switch args[0] {
+	case "emit":
+		return runEmit(args[1:], stderr)
+	case "assemble":
+		return runAssemble(args[1:], stderr)
+	case "verify":
+		return runVerify(args[1:], stdout, stderr, os.Getenv)
+	case "-h", "--help", "help":
+		printRunUsage(stdout)
+		return exitOK
+	default:
+		fmt.Fprintf(stderr, "nwsync: unknown subcommand %q\n\n", args[0])
+		printRunUsage(stderr)
+		return exitUsage
+	}
+}
+
+func printRunUsage(w io.Writer) {
+	fmt.Fprint(w, `usage:
+  nwsync emit     [--as NAME] [--out DIR] [--verify] <artifact-key> <file>
+  nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
+  nwsync verify   [--sample N] [--base URL] <manifest-sha1>
+
+emit explodes one .hak/.erf or one loose file (the TLK) into NWSync blobs plus
+a NSYM index covering only that artifact, and uploads both. assemble merges
+those indexes into one manifest, reading no bulk data. Artifact keys are depot
+keys; an index lives beside its artifact, with the extension replaced.
+
+verify reads a published manifest and its blobs back through the public pull
+zone, with no credential, and decompresses and hashes every one. It is the only
+check on a published blob upstream of a player's client.
+
+--verify makes emit hash what it would otherwise skip. emit normally treats a
+blob's presence as proof of its contents, so without this an object written
+truncated, or written by an emitter since found broken, is skipped forever.
+--verify repairs the storage zone, while verify reads the edge in front of it.
+So a verify run right after a repair is a survey, not a verdict: it names the
+keys the edge still serves stale. Purge those, then run it again.
+
+--out DIR writes to a local repository tree instead of uploading, which is the
+conformance path against upstream nwn_nwsync_write. Without it, the zone comes
+from NWSYNC_STORAGE_ZONE, NWSYNC_STORAGE_PASSWORD and BUNNY_STORAGE_HOST.
+verify needs none of those; its base comes from --base or NWSYNC_PULL_BASE.
+`)
+}
+
+// parseArgs parses flags that may appear before, after or between positionals.
+// Go's flag package stops at the first non-flag argument, which turns
+// `emit <key> <file> --out DIR` into a confusing arity error.
+func parseArgs(fs *flag.FlagSet, args []string) ([]string, error) {
+	var positional []string
+	for {
+		if err := fs.Parse(args); err != nil {
+			return nil, err
+		}
+		rest := fs.Args()
+		if len(rest) == 0 {
+			return positional, nil
+		}
+		positional = append(positional, rest[0])
+		args = rest[1:]
+	}
+}
+
+func runEmit(args []string, stderr io.Writer) int {
+	fs := flag.NewFlagSet("emit", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	as := fs.String("as", "", "published name of the artifact, when it differs from the key")
+	out := fs.String("out", "", "write to a local repository tree instead of uploading")
+	jobs := fs.Int("jobs", defaultEmitJobs, "resources to hash, compress and store at once")
+	verify := fs.Bool("verify", false, "read back and hash blobs that already exist instead of trusting their presence")
+	positional, err := parseArgs(fs, args)
+	if err != nil {
+		return exitUsage
+	}
+	if len(positional) != 2 {
+		fmt.Fprintf(stderr, "nwsync emit: <artifact-key> and <file> are both required\n")
+		return exitUsage
+	}
+	if *jobs < 1 {
+		fmt.Fprintf(stderr, "nwsync emit: -jobs must be at least 1, got %d\n", *jobs)
+		return exitUsage
+	}
+
+	result, err := Emit(EmitOptions{
+		ArtifactKey:  positional[0],
+		ArtifactPath: positional[1],
+		As:           *as,
+		OutDir:       *out,
+		Jobs:         *jobs,
+		Verify:       *verify,
+	})
+	if err != nil {
+		fmt.Fprintf(stderr, "nwsync emit: %v\n", err)
+		return exitInternal
+	}
+	fmt.Fprintf(stderr, "emitted %s: %d resources, %d new blobs, index %s\n",
+		result.Name, result.Entries, result.BlobsWritten, result.ManifestPath)
+	return exitOK
+}
+
+func runVerify(args []string, stdout, stderr io.Writer, getenv func(string) string) int {
+	fs := flag.NewFlagSet("verify", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	base := fs.String("base", getenv("NWSYNC_PULL_BASE"), "pull zone base URL to read through")
+	sample := fs.Int("sample", 0, "check this many random blobs instead of all of them")
+	jobs := fs.Int("jobs", defaultEmitJobs, "blobs to fetch and hash at once")
+	positional, err := parseArgs(fs, args)
+	if err != nil {
+		return exitUsage
+	}
+	if len(positional) != 1 {
+		fmt.Fprintf(stderr, "nwsync verify: exactly one <manifest-sha1> is required\n")
+		return exitUsage
+	}
+
+	result, err := Verify(VerifyOptions{
+		ManifestSHA1: positional[0],
+		Base:         *base,
+		Sample:       *sample,
+		Jobs:         *jobs,
+		Log:          stderr,
+	})
+	if err != nil {
+		fmt.Fprintf(stderr, "nwsync verify: %v\n", err)
+		return exitInternal
+	}
+	fmt.Fprintf(stdout, "verified %d of %d blobs behind %d resources: %d failures, %d bytes checked\n",
+		result.Checked, result.Blobs, result.Entries, result.Failures, result.Bytes)
+	if result.Failures > 0 {
+		return exitDrift
+	}
+	return exitOK
+}
+
+func runAssemble(args []string, stderr io.Writer) int {
+	fs := flag.NewFlagSet("assemble", flag.ContinueOnError)
+	fs.SetOutput(stderr)
+	tlkKey := fs.String("tlk-key", "", "depot key of the TLK, which shadows nothing and merges last")
+	out := fs.String("out", "", "write to a local repository tree instead of uploading")
+	groupID := fs.Int("group-id", 0, "NWSync group id (1 = current, 2 = testing; 0 omits it)")
+	moduleName := fs.String("module-name", "", "module name recorded in the sidecar")
+	description := fs.String("description", "", "description recorded in the sidecar")
+	positional, err := parseArgs(fs, args)
+	if err != nil {
+		return exitUsage
+	}
+	if len(positional) == 0 {
+		fmt.Fprintf(stderr, "nwsync assemble: at least one artifact key is required\n")
+		return exitUsage
+	}
+
+	result, err := Assemble(AssembleOptions{
+		ArtifactKeys: positional,
+		TLKKey:       *tlkKey,
+		OutDir:       *out,
+		GroupID:      *groupID,
+		ModuleName:   *moduleName,
+		Description:  *description,
+	})
+	if err != nil {
+		fmt.Fprintf(stderr, "nwsync assemble: %v\n", err)
+		return exitInternal
+	}
+	fmt.Fprintf(stderr, "assembled manifest %s: %d resources, %s\n",
+		result.SHA1, result.Entries, result.ManifestPath)
+	return exitOK
+}

internal/nwsync/sink.go

@@ -0,0 +1,241 @@
+package nwsync
+
+import (
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"os"
+	"path"
+	"path/filepath"
+	"strings"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/depot"
+)
+
+// sink is where an emit or assemble run puts what it produces. The zone is the
+// production sink; a local directory exists only as the conformance path, so
+// upstream's output and ours can be diffed on a developer machine.
+type sink interface {
+	// putBlob stores one NWCompressedBuffer blob under its sha1 name and
+	// returns the bytes stored, or 0 if a good copy was already there. Blob
+	// names are content hashes, so an existing name is normally taken as
+	// existing content — which is why body is a thunk: compression is the
+	// expensive part of emit and a blob that is already stored must not pay
+	// for it.
+	//
+	// verify stops trusting presence: the stored copy is read back, unwrapped
+	// and hashed, and replaced when it is not what its name claims. Without it
+	// an object written truncated, or written by an emitter since found broken,
+	// is skipped by every later emit forever and no backfill can repair it.
+	putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error)
+	// putIndex stores a NSYM manifest and its sidecar under key, which is
+	// either an artifact-derived object key or a local path.
+	putIndex(key string, manifest, sidecar []byte) error
+	// getIndex reads back a NSYM manifest and its sidecar.
+	getIndex(key string) (manifest, sidecar []byte, err error)
+	// describe names the sink for messages.
+	describe(key string) string
+}
+
+// dirSink writes a local NWSync repository tree.
+type dirSink struct{ root string }
+
+func (s dirSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) {
+	blob := blobPath(s.root, sha1Hex)
+	if !verify {
+		// Stat, not read: the common path must not pay to open every blob that
+		// is already there.
+		if _, err := os.Stat(blob); err == nil {
+			return 0, nil
+		}
+	} else if stored, err := os.ReadFile(blob); err == nil && blobMatchesName(stored, sha1Hex) == nil {
+		return 0, nil
+	}
+	if err := os.MkdirAll(filepath.Dir(blob), 0o755); err != nil {
+		return 0, fmt.Errorf("create blob directory: %w", err)
+	}
+	data := body()
+	if err := os.WriteFile(blob, data, 0o644); err != nil {
+		return 0, fmt.Errorf("write blob: %w", err)
+	}
+	return int64(len(data)), nil
+}
+
+func (s dirSink) putIndex(key string, manifest, sidecar []byte) error {
+	target := filepath.Join(s.root, filepath.FromSlash(key))
+	if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
+		return fmt.Errorf("create manifest directory: %w", err)
+	}
+	if err := os.WriteFile(target, manifest, 0o644); err != nil {
+		return fmt.Errorf("write manifest: %w", err)
+	}
+	if err := os.WriteFile(target+".json", sidecar, 0o644); err != nil {
+		return fmt.Errorf("write sidecar: %w", err)
+	}
+	return nil
+}
+
+func (s dirSink) getIndex(key string) ([]byte, []byte, error) {
+	target := filepath.Join(s.root, filepath.FromSlash(key))
+	manifest, err := os.ReadFile(target)
+	if err != nil {
+		return nil, nil, fmt.Errorf("read index: %w", err)
+	}
+	sidecar, err := os.ReadFile(target + ".json")
+	if err != nil {
+		return nil, nil, fmt.Errorf("read sidecar: %w", err)
+	}
+	return manifest, sidecar, nil
+}
+
+func (s dirSink) describe(key string) string {
+	return filepath.Join(s.root, filepath.FromSlash(key))
+}
+
+// zoneSink uploads straight to the NWSync storage zone. Nothing bulky is ever
+// written to the runner's disk: the working set is one resource at a time.
+type zoneSink struct {
+	store depot.KeyStore
+	ctx   context.Context
+	zone  string
+}
+
+func (s zoneSink) putBlob(sha1Hex string, verify bool, body func() []byte) (int64, error) {
+	key := blobKey(sha1Hex)
+	// A throttled probe must never be read as "missing, re-upload" or as
+	// "present, skip", so only a confirmed Present skips the upload.
+	state, _, err := s.store.ProbeKey(s.ctx, key)
+	if err != nil {
+		return 0, fmt.Errorf("probe blob %s: %w", sha1Hex, err)
+	}
+	if state == depot.Present {
+		if !verify {
+			return 0, nil
+		}
+		// The probe only proved the object exists. Read it back and hold it to
+		// its own name.
+		//
+		// This reads the storage API rather than the pull zone: emit holds the
+		// write credential, and a repair decision has to be made against the
+		// copy it is about to overwrite, not against an edge cache of it. A
+		// read that fails outright is a fault, not a verdict — treating it as
+		// "bad, re-upload" would turn a throttled zone into a full backfill.
+		stored, err := s.store.GetKey(s.ctx, key)
+		if err != nil {
+			return 0, fmt.Errorf("read back blob %s: %w", sha1Hex, err)
+		}
+		if blobMatchesName(stored, sha1Hex) == nil {
+			return 0, nil
+		}
+	}
+	data := body()
+	if err := s.put(key, data); err != nil {
+		return 0, fmt.Errorf("upload blob %s: %w", sha1Hex, err)
+	}
+	return int64(len(data)), nil
+}
+
+func (s zoneSink) putIndex(key string, manifest, sidecar []byte) error {
+	// The manifest lands last: its presence is the publication marker, so it
+	// must never appear before the blobs it names.
+	if err := s.put(key+".json", sidecar); err != nil {
+		return fmt.Errorf("upload sidecar %s: %w", key, err)
+	}
+	if err := s.put(key, manifest); err != nil {
+		return fmt.Errorf("upload index %s: %w", key, err)
+	}
+	return nil
+}
+
+func (s zoneSink) getIndex(key string) ([]byte, []byte, error) {
+	manifest, err := s.store.GetKey(s.ctx, key)
+	if err != nil {
+		return nil, nil, fmt.Errorf("read index: %w", err)
+	}
+	sidecar, err := s.store.GetKey(s.ctx, key+".json")
+	if err != nil {
+		return nil, nil, fmt.Errorf("read sidecar: %w", err)
+	}
+	return manifest, sidecar, nil
+}
+
+func (s zoneSink) describe(key string) string { return s.zone + "/" + key }
+
+func (s zoneSink) put(key string, body []byte) error {
+	sum := sha256.Sum256(body)
+	return s.store.PutReader(s.ctx, key, bytes.NewReader(body), int64(len(body)), hex.EncodeToString(sum[:]))
+}
+
+// newZoneSink builds the upload sink from the environment. NWSync data lives
+// in its own zone, separate from the asset depot, so it has its own zone and
+// credential; only the host is shared, and Crucible has no default host.
+func newZoneSink(ctx context.Context, getenv func(string) string) (sink, error) {
+	cfg := depot.LoadConfig(getenv)
+	cfg.StorageZone = getenv("NWSYNC_STORAGE_ZONE")
+	cfg.WriteKey = getenv("NWSYNC_STORAGE_PASSWORD")
+	cfg.ReadKey = cfg.WriteKey
+	if cfg.StorageZone == "" {
+		return nil, fmt.Errorf("NWSYNC_STORAGE_ZONE is unset (or pass --out DIR to write locally)")
+	}
+	if cfg.WriteKey == "" {
+		return nil, fmt.Errorf("NWSYNC_STORAGE_PASSWORD is unset (or pass --out DIR to write locally)")
+	}
+	if cfg.StorageHost == "" {
+		return nil, fmt.Errorf("BUNNY_STORAGE_HOST is unset")
+	}
+	store, err := depot.NewKeyStore(cfg)
+	if err != nil {
+		return nil, err
+	}
+	return zoneSink{store: store, ctx: ctx, zone: cfg.StorageZone}, nil
+}
+
+// indexKey is where an artifact's NSYM lives: beside the artifact itself, with
+// the final extension replaced. emit and assemble must agree on this one rule,
+// so it lives here and nowhere else.
+//
+// artifacts/haks/sha256/30/46/3046….hak -> artifacts/haks/sha256/30/46/3046….nsym
+func indexKey(artifactKey string) (string, error) {
+	extension := path.Ext(artifactKey)
+	if extension == "" {
+		return "", fmt.Errorf("artifact key %q has no extension", artifactKey)
+	}
+	return strings.TrimSuffix(artifactKey, extension) + ".nsym", nil
+}
+
+// resolveIndexKey is where emit writes an artifact's index and where assemble
+// reads it from. On the zone that is beside the artifact; locally the indexes
+// sit flat beside the data tree, so upstream's output and ours diff directly.
+func resolveIndexKey(artifactKey, outDir string) (string, error) {
+	key, err := indexKey(artifactKey)
+	if err != nil {
+		return "", err
+	}
+	if outDir != "" {
+		return path.Base(key), nil
+	}
+	return key, nil
+}
+
+// checkArtifactKey fails closed when the key's embedded digest is not the
+// digest of the bytes being emitted. Publishing an index under the wrong key
+// silently pairs a manifest with the wrong artifact.
+// artifact is hashed by streaming, so a multi-gigabyte hak is never resident.
+func checkArtifactKey(artifactKey string, artifact io.Reader) error {
+	base := path.Base(artifactKey)
+	digest := strings.TrimSuffix(base, path.Ext(base))
+	if len(digest) != 64 {
+		return fmt.Errorf("artifact key %q does not name a sha256", artifactKey)
+	}
+	hash := sha256.New()
+	if _, err := io.Copy(hash, artifact); err != nil {
+		return fmt.Errorf("hash artifact: %w", err)
+	}
+	if got := hex.EncodeToString(hash.Sum(nil)); got != digest {
+		return fmt.Errorf("artifact key %q names digest %s but the file hashes to %s", artifactKey, digest, got)
+	}
+	return nil
+}

internal/nwsync/verify.go

@@ -0,0 +1,233 @@
+package nwsync
+
+import (
+	"crypto/sha1"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"io"
+	"math/rand/v2"
+	"net/http"
+	"path"
+	"sort"
+	"strconv"
+	"sync"
+	"time"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+)
+
+// defaultPullBase is the public NWSync host, which is a Bunny pull zone fronting
+// the storage zone. Verify reads through it rather than through the storage API
+// on purpose: what matters is the bytes a client is served, edge behaviour
+// included, not what the origin believes it holds.
+const defaultPullBase = "https://nwsync.westgate.pw"
+
+// errBlobMissing marks an object the zone does not serve at all, as distinct
+// from one it serves badly.
+var errBlobMissing = errors.New("missing")
+
+// blobSource reads one object out of the zone by key. Verify never writes and
+// never authenticates, so this is deliberately narrower than sink.
+type blobSource interface {
+	get(key string) ([]byte, error)
+	describe(key string) string
+}
+
+// pullZone reads the zone over plain HTTP, with no credential.
+type pullZone struct {
+	base   string
+	client *http.Client
+}
+
+func newPullZone(base string) blobSource {
+	if base == "" {
+		base = defaultPullBase
+	}
+	return pullZone{
+		base: base,
+		// A full sweep is tens of thousands of small requests, so connections
+		// have to be reused; the default transport does that already.
+		client: &http.Client{Timeout: 60 * time.Second},
+	}
+}
+
+func (z pullZone) describe(key string) string { return z.base + "/" + key }
+
+func (z pullZone) get(key string) ([]byte, error) {
+	resp, err := z.client.Get(z.describe(key))
+	if err != nil {
+		return nil, err
+	}
+	defer resp.Body.Close()
+	if resp.StatusCode == http.StatusNotFound || resp.StatusCode == http.StatusGone {
+		_, _ = io.Copy(io.Discard, resp.Body)
+		return nil, errBlobMissing
+	}
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		_, _ = io.Copy(io.Discard, resp.Body)
+		return nil, fmt.Errorf("unexpected status %d", resp.StatusCode)
+	}
+	return io.ReadAll(resp.Body)
+}
+
+// VerifyOptions describes one verify run.
+type VerifyOptions struct {
+	ManifestSHA1 string     // the merged manifest to verify
+	Base         string     // pull zone base URL; empty means defaultPullBase
+	Sample       int        // check this many random blobs; 0 means all of them
+	Jobs         int        // blobs in flight at once; 0 means defaultEmitJobs
+	Source       blobSource // test seam; nil means the pull zone at Base
+	Log          io.Writer  // per-blob failures land here; nil discards them
+}
+
+// VerifyResult reports what one verify run found.
+type VerifyResult struct {
+	Entries  int   // resources the manifest names
+	Blobs    int   // distinct blobs behind those resources
+	Checked  int   // blobs actually fetched
+	Failures int   // blobs that failed a check
+	Bytes    int64 // uncompressed bytes verified
+}
+
+// Verify reads a published manifest and its blobs the way a client reads them,
+// and reports every blob that is not what the manifest says it is.
+//
+// Presence is not correctness. emit skips an object that already exists on the
+// strength of a 1-byte range GET, so a truncated or wrongly framed object is
+// skipped by every later emit forever and the backfill cannot repair it. This is
+// the only thing upstream of a player's client that can tell that has happened.
+//
+// Every blob is decompressed and hashed. A Content-Length check would pass the
+// exact failure mode being hunted — a byte-correct-looking object whose contents
+// are wrong — and a round-trip check alone would pass a frame that omits its
+// content size, because Go's decoder is more capable than the client's (#86).
+func Verify(options VerifyOptions) (VerifyResult, error) {
+	// The argument is interpolated straight into a URL path, so it is checked
+	// rather than trusted: exactly 20 bytes of hex, nothing else.
+	if sum, err := hex.DecodeString(options.ManifestSHA1); err != nil || len(sum) != sha1.Size {
+		return VerifyResult{}, fmt.Errorf("%q is not a manifest sha1", options.ManifestSHA1)
+	}
+	source := options.Source
+	if source == nil {
+		source = newPullZone(options.Base)
+	}
+	log := options.Log
+	if log == nil {
+		log = io.Discard
+	}
+
+	manifestKey := path.Join("manifests", options.ManifestSHA1)
+	data, err := source.get(manifestKey)
+	if err != nil {
+		return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
+	}
+	// A manifest is named after its own sha1, so this catches the zone serving
+	// a different manifest — or a truncated one — before any blob is fetched.
+	if got := hex.EncodeToString(sha1Sum(data)); got != options.ManifestSHA1 {
+		return VerifyResult{}, fmt.Errorf("%s hashes to %s, not the manifest asked for",
+			source.describe(manifestKey), got)
+	}
+	entries, err := readManifest(data)
+	if err != nil {
+		return VerifyResult{}, fmt.Errorf("%s: %w", source.describe(manifestKey), err)
+	}
+
+	// A manifest names one blob many times over: mappings share a sha1, and so
+	// do resrefs with identical contents. Fetch each blob once.
+	blobs := make([]Entry, 0, len(entries))
+	seen := make(map[[20]byte]bool, len(entries))
+	for _, entry := range entries {
+		if seen[entry.SHA1] {
+			continue
+		}
+		seen[entry.SHA1] = true
+		blobs = append(blobs, entry)
+	}
+
+	result := VerifyResult{Entries: len(entries), Blobs: len(blobs)}
+	checking := blobs
+	if options.Sample > 0 && options.Sample < len(blobs) {
+		// A full sweep of the live manifest is ~69,000 objects and ~15 GB, so
+		// sampling is what makes verifying a routine act rather than an event.
+		picks := rand.Perm(len(blobs))[:options.Sample]
+		checking = make([]Entry, 0, options.Sample)
+		for _, i := range picks {
+			checking = append(checking, blobs[i])
+		}
+	}
+	result.Checked = len(checking)
+
+	jobs := options.Jobs
+	if jobs < 1 {
+		jobs = defaultEmitJobs
+	}
+	var (
+		mu       sync.Mutex
+		failures []string
+	)
+	work := make(chan Entry)
+	var wg sync.WaitGroup
+	for range jobs {
+		wg.Go(func() {
+			for entry := range work {
+				fault := checkEntry(source, entry)
+				mu.Lock()
+				if fault != "" {
+					failures = append(failures, fault)
+				} else {
+					result.Bytes += int64(entry.Size)
+				}
+				mu.Unlock()
+			}
+		})
+	}
+	for _, entry := range checking {
+		work <- entry
+	}
+	close(work)
+	wg.Wait()
+
+	// Workers finish in any order; a report an operator can diff must not.
+	sort.Strings(failures)
+	for _, fault := range failures {
+		fmt.Fprintln(log, fault)
+	}
+	result.Failures = len(failures)
+	return result, nil
+}
+
+// checkEntry fetches one blob and returns a one-line fault, or "" if it is
+// exactly what the manifest entry says it is.
+func checkEntry(source blobSource, entry Entry) string {
+	// Name the resource, not just the hash: an operator has to find the thing
+	// in a hak, and a bare sha1 says nothing about where to look.
+	extension, ok := erf.ExtensionForResourceType(entry.ResType)
+	if !ok {
+		extension = strconv.Itoa(int(entry.ResType))
+	}
+	where := fmt.Sprintf("%s (%s.%s)", entry.sha1Hex(), entry.ResRef, extension)
+	blob, err := source.get(blobKey(entry.sha1Hex()))
+	if err != nil {
+		if errors.Is(err, errBlobMissing) {
+			return where + ": missing"
+		}
+		return where + ": unreadable: " + err.Error()
+	}
+	data, err := inspectBlob(blob)
+	if err != nil {
+		return where + ": " + err.Error()
+	}
+	if uint32(len(data)) != entry.Size {
+		return fmt.Sprintf("%s: size mismatch: %d bytes, manifest says %d", where, len(data), entry.Size)
+	}
+	if sha1.Sum(data) != entry.SHA1 {
+		return fmt.Sprintf("%s: hash mismatch: contents hash to %x", where, sha1.Sum(data))
+	}
+	return ""
+}
+
+func sha1Sum(data []byte) []byte {
+	sum := sha1.Sum(data)
+	return sum[:]
+}

internal/nwsync/verify_test.go

@@ -0,0 +1,285 @@
+package nwsync
+
+import (
+	"bytes"
+	"crypto/sha1"
+	"encoding/hex"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+
+	"github.com/klauspost/compress/zstd"
+)
+
+// verifyFixture emits two haks and a TLK into a fake zone, assembles them, and
+// hands back a verifier reading that zone the way a client would.
+type verifyFixture struct {
+	*zoneSinkFixture
+	manifestSHA1 string
+}
+
+func newVerifyFixture(t *testing.T) *verifyFixture {
+	t.Helper()
+	zone := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	// A payload under 256 bytes is the one the frame-header check exists for.
+	writeHak(t, hak, map[string][]byte{
+		"bloodstain1.tga": []byte("small"),
+		"appearance.2da":  bytes.Repeat([]byte("2DA V2.0\n"), 200),
+	})
+	tlk := filepath.Join(dir, "sow_tlk.tlk")
+	if err := os.WriteFile(tlk, []byte("TLK V3.0 payload"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	zone.emit(t, hak)
+	if _, err := Emit(EmitOptions{
+		ArtifactKey: artifactKey(t, tlk), ArtifactPath: tlk, As: "sow_tlk.tlk", Sink: zone.sink,
+	}); err != nil {
+		t.Fatalf("emit tlk: %v", err)
+	}
+	assembled, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{artifactKey(t, hak)}, TLKKey: artifactKey(t, tlk), Sink: zone.sink,
+	})
+	if err != nil {
+		t.Fatalf("assemble: %v", err)
+	}
+	return &verifyFixture{zoneSinkFixture: zone, manifestSHA1: assembled.SHA1}
+}
+
+func (f *verifyFixture) verify(t *testing.T, sample int) (VerifyResult, string, error) {
+	t.Helper()
+	var log bytes.Buffer
+	result, err := Verify(VerifyOptions{
+		ManifestSHA1: f.manifestSHA1,
+		Sample:       sample,
+		Source:       f.zone.pullZone(),
+		Log:          &log,
+	})
+	return result, log.String(), err
+}
+
+// keyOf is where a resource's blob lives, addressed by the sha1 of its
+// uncompressed bytes — the same path the client requests.
+func keyOf(body []byte) string {
+	sum := sha1.Sum(body)
+	return blobKey(hex.EncodeToString(sum[:]))
+}
+
+func TestVerifyPassesACleanZone(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 0 {
+		t.Errorf("verify reported %d failures on a clean zone: %s", result.Failures, log)
+	}
+	// A default run is a full sweep, so it must reach every blob the manifest
+	// names — not some of them.
+	if result.Checked != result.Blobs || result.Blobs == 0 {
+		t.Errorf("checked %d of %d blobs; a full sweep must check all of them", result.Checked, result.Blobs)
+	}
+}
+
+func TestVerifyReportsAMissingBlob(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	key := keyOf([]byte("small"))
+	fixture.zone.mu.Lock()
+	delete(fixture.zone.objects, key)
+	fixture.zone.mu.Unlock()
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "missing") {
+		t.Errorf("a deleted blob was not reported as missing: %s", log)
+	}
+}
+
+func TestVerifyReportsATruncatedBlob(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	key := keyOf([]byte("small"))
+	fixture.zone.mu.Lock()
+	fixture.zone.objects[key] = fixture.zone.objects[key][:blobHeaderBytes+4]
+	fixture.zone.mu.Unlock()
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "framing") {
+		t.Errorf("a truncated blob was not reported as malformed framing: %s", log)
+	}
+}
+
+// TestVerifyRejectsABlobWithNoDeclaredFrameContentSize is the check that #86
+// slipped past: the blob decompresses to exactly the right bytes, so a verifier
+// that only round-trips certifies it, yet the client cannot decode it.
+func TestVerifyRejectsABlobWithNoDeclaredFrameContentSize(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	body := []byte("small")
+	key := keyOf(body)
+
+	fixture.zone.mu.Lock()
+	good := fixture.zone.objects[key]
+	encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
+	if err != nil {
+		t.Fatal(err)
+	}
+	bad := append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
+	fixture.zone.objects[key] = bad
+	fixture.zone.mu.Unlock()
+
+	if frameDeclaresContentSize(bad[blobHeaderBytes:]) {
+		t.Fatal("the fixture blob declares a content size; it cannot exercise the check")
+	}
+	if got, err := decompressBlob(bad); err != nil || !bytes.Equal(got, body) {
+		t.Fatalf("the fixture blob must round trip, or it proves nothing: %v", err)
+	}
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "content size") {
+		t.Errorf("undeclared frame content size was not the reported reason: %s", log)
+	}
+}
+
+func TestVerifyReportsWrongContents(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	key := keyOf([]byte("small"))
+	fixture.zone.mu.Lock()
+	// Valid framing, valid zstd, wrong bytes: only decompressing and hashing
+	// can see this, which is why Content-Length is not enough.
+	fixture.zone.objects[key] = compressBlob([]byte("wrong"))
+	fixture.zone.mu.Unlock()
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "hash mismatch") {
+		t.Errorf("wrong contents were not reported as a hash mismatch: %s", log)
+	}
+}
+
+func TestVerifyReportsAShortBlob(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	key := keyOf([]byte("small"))
+	fixture.zone.mu.Lock()
+	// Well-formed all the way down and simply too short — the shape a killed
+	// upload leaves behind, and the one a Content-Length check would pass.
+	fixture.zone.objects[key] = compressBlob([]byte("sma"))
+	fixture.zone.mu.Unlock()
+
+	result, log, err := fixture.verify(t, 0)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Failures != 1 {
+		t.Fatalf("reported %d failures, want 1: %s", result.Failures, log)
+	}
+	if !strings.Contains(log, "size mismatch") {
+		t.Errorf("a short blob was not reported as a size mismatch: %s", log)
+	}
+}
+
+func TestVerifyFailsWhenTheManifestIsNotTheOneAsked(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	fixture.zone.mu.Lock()
+	fixture.zone.objects["manifests/"+fixture.manifestSHA1] = []byte("NSYM garbage")
+	fixture.zone.mu.Unlock()
+
+	if _, _, err := fixture.verify(t, 0); err == nil {
+		t.Fatal("verify accepted a manifest that is not the one requested")
+	}
+}
+
+func TestVerifySampleChecksFewerBlobs(t *testing.T) {
+	fixture := newVerifyFixture(t)
+	result, log, err := fixture.verify(t, 1)
+	if err != nil {
+		t.Fatalf("verify: %v", err)
+	}
+	if result.Checked != 1 {
+		t.Errorf("--sample 1 checked %d blobs, want 1: %s", result.Checked, log)
+	}
+	if result.Blobs <= result.Checked {
+		t.Errorf("sampling %d of %d blobs is not a sample", result.Checked, result.Blobs)
+	}
+}
+
+// TestEmitVerifyReplacesABlobThatIsNotItsName covers the reason #86 could not be
+// fixed by the encoder alone: emit skips whatever is already present, so every
+// blob published by the broken encoder stays broken until emit stops trusting
+// presence.
+func TestEmitVerifyReplacesABlobThatIsNotItsName(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	body := []byte("blood")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": body})
+
+	emit := func(verify bool) EmitResult {
+		t.Helper()
+		result, err := Emit(EmitOptions{
+			ArtifactKey:  artifactKey(t, hak),
+			ArtifactPath: hak,
+			Sink:         fixture.sink,
+			Verify:       verify,
+		})
+		if err != nil {
+			t.Fatalf("emit (verify=%v): %v", verify, err)
+		}
+		return result
+	}
+
+	emit(false)
+	key := keyOf(body)
+	encoder, err := zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
+	if err != nil {
+		t.Fatal(err)
+	}
+	fixture.zone.mu.Lock()
+	good := fixture.zone.objects[key]
+	fixture.zone.objects[key] = append(append([]byte{}, good[:blobHeaderBytes]...), encoder.EncodeAll(body, nil)...)
+	fixture.zone.mu.Unlock()
+
+	if plain := emit(false); plain.BlobsWritten != 0 {
+		t.Fatalf("a plain re-emit wrote %d blobs; it is supposed to trust presence", plain.BlobsWritten)
+	}
+	if verified := emit(true); verified.BlobsWritten != 1 {
+		t.Fatalf("--verify wrote %d blobs, want 1 (the bad copy must be replaced)", verified.BlobsWritten)
+	}
+
+	fixture.zone.mu.Lock()
+	repaired := fixture.zone.objects[key]
+	fixture.zone.mu.Unlock()
+	if !bytes.Equal(repaired, good) {
+		t.Error("the replaced blob is not what the current encoder produces")
+	}
+	if _, err := inspectBlob(repaired); err != nil {
+		t.Errorf("the replaced blob still fails inspection: %v", err)
+	}
+
+	// A second verifying run has nothing left to repair.
+	if again := emit(true); again.BlobsWritten != 0 {
+		t.Errorf("--verify rewrote %d good blobs", again.BlobsWritten)
+	}
+}

internal/nwsync/zone_test.go

@@ -0,0 +1,257 @@
+package nwsync
+
+import (
+	"crypto/sha1"
+	"crypto/sha256"
+	"encoding/hex"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+)
+
+// fakeZone is a Bunny-shaped object store: PUT stores, GET reads, and the
+// Checksum header is verified the way Bunny verifies it.
+type fakeZone struct {
+	mu      sync.Mutex
+	objects map[string][]byte
+	puts    []string
+	failOn  func(key string) bool // when true, the PUT fails
+	url     string                // base the same objects are readable at
+}
+
+// pullZone reads the fake zone the way the public pull zone is read: plain
+// unauthenticated GETs, no storage API.
+func (z *fakeZone) pullZone() blobSource {
+	return newPullZone(z.url)
+}
+
+func newFakeZone(t *testing.T) (*fakeZone, func(string) string) {
+	t.Helper()
+	zone := &fakeZone{objects: map[string][]byte{}}
+	server := httptest.NewServer(zone)
+	t.Cleanup(server.Close)
+	zone.url = server.URL + "/sow-nwsync"
+	getenv := func(name string) string {
+		switch name {
+		case "NWSYNC_STORAGE_ZONE":
+			return "sow-nwsync"
+		case "NWSYNC_STORAGE_PASSWORD":
+			return "write-key"
+		case "BUNNY_STORAGE_HOST":
+			return server.URL
+		}
+		return ""
+	}
+	return zone, getenv
+}
+
+func (z *fakeZone) ServeHTTP(w http.ResponseWriter, r *http.Request) {
+	key := strings.TrimPrefix(r.URL.Path, "/sow-nwsync/")
+	switch r.Method {
+	case http.MethodPut:
+		if z.failOn != nil && z.failOn(key) {
+			http.Error(w, "boom", http.StatusInternalServerError)
+			return
+		}
+		body, err := io.ReadAll(r.Body)
+		if err != nil {
+			http.Error(w, err.Error(), http.StatusBadRequest)
+			return
+		}
+		sum := sha256.Sum256(body)
+		if want := strings.ToUpper(hex.EncodeToString(sum[:])); r.Header.Get("Checksum") != want {
+			http.Error(w, "checksum mismatch", http.StatusBadRequest)
+			return
+		}
+		z.mu.Lock()
+		z.objects[key] = body
+		z.puts = append(z.puts, key)
+		z.mu.Unlock()
+		w.WriteHeader(http.StatusCreated)
+	case http.MethodGet:
+		z.mu.Lock()
+		body, ok := z.objects[key]
+		z.mu.Unlock()
+		if !ok {
+			http.Error(w, "not found", http.StatusNotFound)
+			return
+		}
+		_, _ = w.Write(body)
+	default:
+		http.Error(w, "unsupported", http.StatusMethodNotAllowed)
+	}
+}
+
+func (z *zoneSinkFixture) emit(t *testing.T, path string) EmitResult {
+	t.Helper()
+	result, err := Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, path),
+		ArtifactPath: path,
+		Sink:         z.sink,
+	})
+	if err != nil {
+		t.Fatalf("emit %s: %v", path, err)
+	}
+	return result
+}
+
+type zoneSinkFixture struct {
+	zone *fakeZone
+	sink sink
+}
+
+func newZoneFixture(t *testing.T) *zoneSinkFixture {
+	t.Helper()
+	zone, getenv := newFakeZone(t)
+	target, err := newZoneSink(t.Context(), getenv)
+	if err != nil {
+		t.Fatalf("zone sink: %v", err)
+	}
+	return &zoneSinkFixture{zone: zone, sink: target}
+}
+
+func sha1Of(body []byte) [20]byte { return sha1.Sum(body) }
+
+func TestEmitUploadsBlobsThenIndex(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	body := []byte("texture bytes")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": body, "copy1.txi": body})
+
+	key := artifactKey(t, hak)
+	result := fixture.emit(t, hak)
+
+	index, err := indexKey(key)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, ok := fixture.zone.objects[index]; !ok {
+		t.Fatalf("no index at %s; zone holds %v", index, fixture.zone.puts)
+	}
+	if result.BlobsWritten != 1 {
+		t.Errorf("uploaded %d blobs, want 1 (identical content shares a blob)", result.BlobsWritten)
+	}
+
+	// The index is the publication marker, so it must land after every blob it
+	// names — including its own sidecar.
+	last := fixture.zone.puts[len(fixture.zone.puts)-1]
+	if last != index {
+		t.Errorf("index landed at position %d of %d; it must be last", len(fixture.zone.puts), len(fixture.zone.puts))
+	}
+	for _, key := range fixture.zone.puts[:len(fixture.zone.puts)-1] {
+		if strings.HasPrefix(key, "data/sha1/") || key == index+".json" {
+			continue
+		}
+		t.Errorf("unexpected object uploaded before the index: %s", key)
+	}
+}
+
+func TestEmitSkipsBlobsAlreadyInTheZone(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": []byte("blood")})
+
+	first := fixture.emit(t, hak)
+	if first.BlobsWritten != 1 {
+		t.Fatalf("first emit uploaded %d blobs, want 1", first.BlobsWritten)
+	}
+	second := fixture.emit(t, hak)
+	if second.BlobsWritten != 0 {
+		t.Errorf("re-emit uploaded %d blobs, want 0 (a blob name is its content)", second.BlobsWritten)
+	}
+}
+
+func TestEmitLeavesNoIndexWhenAnUploadFails(t *testing.T) {
+	fixture := newZoneFixture(t)
+	fixture.zone.failOn = func(key string) bool { return strings.HasPrefix(key, "data/sha1/") }
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": []byte("blood")})
+
+	_, err := Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, hak),
+		ArtifactPath: hak,
+		Sink:         fixture.sink,
+	})
+	if err == nil {
+		t.Fatal("emit reported success after an upload failed")
+	}
+	for key := range fixture.zone.objects {
+		if strings.HasSuffix(key, ".nsym") {
+			t.Errorf("a half-emitted artifact published an index: %s", key)
+		}
+	}
+}
+
+func TestEmitRejectsAKeyThatDoesNotMatchTheFile(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	hak := filepath.Join(dir, "sow_test_01.hak")
+	writeHak(t, hak, map[string][]byte{"bloodstain1.tga": []byte("blood")})
+
+	wrong := "artifacts/haks/sha256/00/11/" + strings.Repeat("0", 64) + ".hak"
+	_, err := Emit(EmitOptions{ArtifactKey: wrong, ArtifactPath: hak, Sink: fixture.sink})
+	if err == nil || !strings.Contains(err.Error(), "hashes to") {
+		t.Fatalf("emit published under a key that names another artifact: %v", err)
+	}
+}
+
+func TestAssembleReadsIndexesFromTheZone(t *testing.T) {
+	fixture := newZoneFixture(t)
+	dir := t.TempDir()
+	topBody := []byte("2da from sow_top")
+	assetBody := []byte("2da from the asset hak")
+	top := filepath.Join(dir, "sow_top.hak")
+	core := filepath.Join(dir, "sow_core_01.hak")
+	writeHak(t, top, map[string][]byte{"appearance.2da": topBody})
+	writeHak(t, core, map[string][]byte{"appearance.2da": assetBody, "bloodstain1.tga": []byte("blood")})
+	tlkPath := filepath.Join(dir, "sow_tlk.tlk")
+	if err := os.WriteFile(tlkPath, []byte("TLK V3.0 payload"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+
+	fixture.emit(t, top)
+	fixture.emit(t, core)
+	if _, err := Emit(EmitOptions{
+		ArtifactKey:  artifactKey(t, tlkPath),
+		ArtifactPath: tlkPath,
+		As:           "sow_tlk.tlk",
+		Sink:         fixture.sink,
+	}); err != nil {
+		t.Fatalf("emit tlk: %v", err)
+	}
+
+	result, err := Assemble(AssembleOptions{
+		ArtifactKeys: []string{artifactKey(t, top), artifactKey(t, core)},
+		TLKKey:       artifactKey(t, tlkPath),
+		GroupID:      2,
+		Sink:         fixture.sink,
+	})
+	if err != nil {
+		t.Fatalf("assemble: %v", err)
+	}
+	if result.Entries != 3 {
+		t.Fatalf("merged %d entries, want 3 (appearance.2da is shadowed, the TLK adds one)", result.Entries)
+	}
+
+	manifest, ok := fixture.zone.objects["manifests/"+result.SHA1]
+	if !ok {
+		t.Fatalf("no merged manifest in the zone; it holds %v", fixture.zone.puts)
+	}
+	entries, err := readManifest(manifest)
+	if err != nil {
+		t.Fatalf("parse merged manifest: %v", err)
+	}
+	for _, entry := range entries {
+		if entry.ResRef == "appearance" && entry.SHA1 != sha1Of(topBody) {
+			t.Errorf("appearance.2da resolved to the shadowed hak, not the first one given")
+		}
+	}
+}

internal/pipeline/build.go

@@ -593,9 +593,14 @@ func envBool(name string) bool {
 func collectModuleResources(p *project.Project, moduleHakOrder []string) ([]erf.Resource, error) {
 	var moduleResources []erf.Resource
 
+	palettes, err := collectPaletteDescriptors(p)
+	if err != nil {
+		return nil, err
+	}
+
 	for _, rel := range p.Inventory.SourceFiles {
 		abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
-		resource, err := resourceFromJSON(abs, moduleHakOrder)
+		resource, err := resourceFromJSON(abs, moduleHakOrder, palettes)
 		if err != nil {
 			return nil, err
 		}
@@ -1030,7 +1035,7 @@ func compareResourceKeys(a, b erf.Resource) int {
 	return 0
 }
 
-func resourceFromJSON(path string, moduleHakOrder []string) (erf.Resource, error) {
+func resourceFromJSON(path string, moduleHakOrder []string, palettes paletteProjection) (erf.Resource, error) {
 	name, extension, err := splitSourceName(path)
 	if err != nil {
 		return erf.Resource{}, err
@@ -1059,6 +1064,9 @@ func resourceFromJSON(path string, moduleHakOrder []string) (erf.Resource, error
 	if extension == ".ifo" && name == "module" && len(moduleHakOrder) > 0 {
 		setModuleHAKList(&document, moduleHakOrder)
 	}
+	if extension == ".itp" && isPaletteProjectionResref(name) {
+		projectPaletteDocument(&document, palettes[strings.ToLower(name)])
+	}
 
 	var buf bytes.Buffer
 	if err := gff.Write(&buf, document); err != nil {

internal/pipeline/compare.go

@@ -117,6 +117,10 @@ func expectedResources(p *project.Project) (map[string]resourceExpectation, erro
 	if err != nil {
 		return nil, err
 	}
+	palettes, err := collectPaletteDescriptors(p)
+	if err != nil {
+		return nil, err
+	}
 
 	for _, rel := range p.Inventory.SourceFiles {
 		abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
@@ -136,6 +140,9 @@ func expectedResources(p *project.Project) (map[string]resourceExpectation, erro
 		if extension == ".ifo" && strings.EqualFold(name, "module") && len(moduleHakOrder) > 0 {
 			setModuleHAKList(&document, moduleHakOrder)
 		}
+		if extension == ".itp" && isPaletteProjectionResref(name) {
+			projectPaletteDocument(&document, palettes[strings.ToLower(name)])
+		}
 
 		canonical, err := json.Marshal(document)
 		if err != nil {

internal/pipeline/extract.go

@@ -124,6 +124,13 @@ func extractArchiveResources(p *project.Project, archive erf.Archive, desired ma
 			skippedCount++
 			continue
 		}
+		// *palcus.itp are Toolset-generated palette projections; the module
+		// build regenerates them from source blueprints, so extraction never
+		// writes them back into source.
+		if ext == "itp" && isPaletteProjectionResref(resource.Name) {
+			skippedCount++
+			continue
+		}
 
 		target, data, err := extractedFile(p, resource, ext)
 		if err != nil {

internal/pipeline/palette.go

@@ -0,0 +1,298 @@
+package pipeline
+
+// Custom palette projections (*palcus.itp) are generated artifacts: the Toolset
+// derives them from the category frameworks plus the blueprints present in the
+// module. Crucible reproduces that projection deterministically at build time so
+// the module source only carries descriptor-free category skeletons and the
+// blueprint files themselves. Extract never writes palcus files back.
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
+)
+
+const noStrref = 0xFFFFFFFF
+
+// hiddenPaletteID suppresses a blueprint from the Custom palette (engine
+// convention; see docs in sow-module's palette research notes).
+const hiddenPaletteID = 255
+
+var paletteFamilyByExtension = map[string]string{
+	".utc": "creaturepalcus",
+	".utd": "doorpalcus",
+	".ute": "encounterpalcus",
+	".uti": "itempalcus",
+	".utm": "storepalcus",
+	".utp": "placeablepalcus",
+	".uts": "soundpalcus",
+	".utt": "triggerpalcus",
+	".utw": "waypointpalcus",
+}
+
+type paletteDescriptor struct {
+	name     string
+	strref   uint32
+	resref   string
+	creature bool
+	cr       float32
+	faction  string
+}
+
+func (d paletteDescriptor) sortKey() string {
+	if d.strref != noStrref || d.name == "" {
+		return strings.ToLower(d.resref)
+	}
+	return strings.ToLower(d.name)
+}
+
+// palette resref (e.g. "itempalcus") -> terminal category ID -> descriptors.
+type paletteProjection map[string]map[uint8][]paletteDescriptor
+
+func isPaletteProjectionResref(name string) bool {
+	return strings.HasSuffix(strings.ToLower(name), "palcus")
+}
+
+func collectPaletteDescriptors(p *project.Project) (paletteProjection, error) {
+	projection := paletteProjection{}
+	factions, err := loadFactionNames(p)
+	if err != nil {
+		return nil, err
+	}
+
+	for _, rel := range p.Inventory.SourceFiles {
+		abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
+		name, extension, err := splitSourceName(abs)
+		if err != nil {
+			return nil, err
+		}
+		family, ok := paletteFamilyByExtension[extension]
+		if !ok {
+			continue
+		}
+
+		raw, err := os.ReadFile(abs)
+		if err != nil {
+			return nil, fmt.Errorf("read %s: %w", abs, err)
+		}
+		var document gff.Document
+		if err := json.Unmarshal(raw, &document); err != nil {
+			return nil, fmt.Errorf("parse gff json %s: %w", abs, err)
+		}
+
+		descriptor, paletteID, ok := blueprintDescriptor(document.Root, name, extension, factions)
+		if !ok {
+			continue
+		}
+		if projection[family] == nil {
+			projection[family] = map[uint8][]paletteDescriptor{}
+		}
+		projection[family][paletteID] = append(projection[family][paletteID], descriptor)
+	}
+
+	for _, byID := range projection {
+		for _, descriptors := range byID {
+			sort.SliceStable(descriptors, func(i, j int) bool {
+				a, b := descriptors[i], descriptors[j]
+				if a.sortKey() != b.sortKey() {
+					return a.sortKey() < b.sortKey()
+				}
+				return a.resref < b.resref
+			})
+		}
+	}
+	return projection, nil
+}
+
+func blueprintDescriptor(root gff.Struct, fileName, extension string, factions []string) (paletteDescriptor, uint8, bool) {
+	descriptor := paletteDescriptor{
+		strref:   noStrref,
+		resref:   strings.ToLower(fileName),
+		creature: extension == ".utc",
+	}
+	paletteID := -1
+
+	for _, field := range root.Fields {
+		switch field.Label {
+		case "PaletteID":
+			if v, ok := field.Value.(gff.ByteValue); ok {
+				paletteID = int(v)
+			}
+		case "TemplateResRef":
+			if v, ok := field.Value.(gff.ResRefValue); ok && v != "" {
+				descriptor.resref = strings.ToLower(string(v))
+			}
+		case "LocalizedName", "LocName", "FirstName":
+			if v, ok := field.Value.(gff.LocString); ok {
+				name, strref := locStringLabel(v)
+				if field.Label == "FirstName" {
+					descriptor.name = strings.TrimSpace(descriptor.name + " " + name)
+					if descriptor.strref == noStrref {
+						descriptor.strref = strref
+					}
+				} else {
+					descriptor.name = name
+					descriptor.strref = strref
+				}
+			}
+		case "LastName":
+			if v, ok := field.Value.(gff.LocString); ok {
+				name, _ := locStringLabel(v)
+				descriptor.name = strings.TrimSpace(descriptor.name + " " + name)
+			}
+		case "ChallengeRating":
+			if v, ok := field.Value.(gff.FloatValue); ok {
+				descriptor.cr = float32(v)
+			}
+		case "FactionID":
+			if v, ok := field.Value.(gff.WordValue); ok && int(v) < len(factions) {
+				descriptor.faction = factions[v]
+			}
+		}
+	}
+
+	if paletteID < 0 || paletteID == hiddenPaletteID {
+		return paletteDescriptor{}, 0, false
+	}
+	return descriptor, uint8(paletteID), true
+}
+
+func locStringLabel(value gff.LocString) (string, uint32) {
+	if value.StringRef != noStrref {
+		return "", value.StringRef
+	}
+	for _, entry := range value.Entries {
+		if entry.Value != "" {
+			return entry.Value, noStrref
+		}
+	}
+	return "", noStrref
+}
+
+func loadFactionNames(p *project.Project) ([]string, error) {
+	for _, rel := range p.Inventory.SourceFiles {
+		if !strings.HasSuffix(strings.ToLower(rel), "repute.fac.json") {
+			continue
+		}
+		abs := filepath.Join(p.SourceDir(), filepath.FromSlash(rel))
+		raw, err := os.ReadFile(abs)
+		if err != nil {
+			return nil, fmt.Errorf("read %s: %w", abs, err)
+		}
+		var document gff.Document
+		if err := json.Unmarshal(raw, &document); err != nil {
+			return nil, fmt.Errorf("parse gff json %s: %w", abs, err)
+		}
+		var names []string
+		for _, field := range document.Root.Fields {
+			if field.Label != "FactionList" {
+				continue
+			}
+			list, ok := field.Value.(gff.ListValue)
+			if !ok {
+				continue
+			}
+			for _, faction := range list {
+				name := ""
+				for _, f := range faction.Fields {
+					if f.Label == "FactionName" {
+						if v, ok := f.Value.(gff.StringValue); ok {
+							name = string(v)
+						}
+					}
+				}
+				names = append(names, name)
+			}
+		}
+		return names, nil
+	}
+	return nil, nil
+}
+
+// projectPaletteDocument strips every blueprint descriptor from the palette
+// tree and re-inserts the descriptors derived from module source. Category
+// structure (branches, terminal IDs, labels) passes through untouched.
+func projectPaletteDocument(document *gff.Document, byID map[uint8][]paletteDescriptor) {
+	for i, field := range document.Root.Fields {
+		if field.Label != "MAIN" {
+			continue
+		}
+		if list, ok := field.Value.(gff.ListValue); ok {
+			document.Root.Fields[i] = gff.NewField("MAIN", projectPaletteNodes(list, byID))
+		}
+	}
+}
+
+func projectPaletteNodes(nodes gff.ListValue, byID map[uint8][]paletteDescriptor) gff.ListValue {
+	out := make(gff.ListValue, 0, len(nodes))
+	for _, node := range nodes {
+		if structHasField(node, "RESREF") {
+			continue // blueprint descriptor — regenerated below
+		}
+
+		terminalID := -1
+		fields := make([]gff.Field, 0, len(node.Fields))
+		for _, field := range node.Fields {
+			if field.Label == "LIST" {
+				continue // rebuilt for terminals, recursed for branches
+			}
+			if field.Label == "ID" {
+				if v, ok := field.Value.(gff.ByteValue); ok {
+					terminalID = int(v)
+				}
+			}
+			fields = append(fields, field)
+		}
+
+		switch {
+		case terminalID >= 0:
+			if descriptors := byID[uint8(terminalID)]; len(descriptors) > 0 {
+				fields = append(fields, gff.NewField("LIST", descriptorList(descriptors)))
+			}
+		default:
+			for _, field := range node.Fields {
+				if field.Label == "LIST" {
+					if list, ok := field.Value.(gff.ListValue); ok {
+						fields = append(fields, gff.NewField("LIST", projectPaletteNodes(list, byID)))
+					}
+				}
+			}
+		}
+		out = append(out, gff.Struct{Type: node.Type, Fields: fields})
+	}
+	return out
+}
+
+func descriptorList(descriptors []paletteDescriptor) gff.ListValue {
+	list := make(gff.ListValue, 0, len(descriptors))
+	for _, d := range descriptors {
+		fields := make([]gff.Field, 0, 4)
+		if d.strref != noStrref {
+			fields = append(fields, gff.NewField("STRREF", gff.DWordValue(d.strref)))
+		} else {
+			fields = append(fields, gff.NewField("NAME", gff.StringValue(d.name)))
+		}
+		fields = append(fields, gff.NewField("RESREF", gff.ResRefValue(d.resref)))
+		if d.creature {
+			fields = append(fields, gff.NewField("CR", gff.FloatValue(d.cr)))
+			fields = append(fields, gff.NewField("FACTION", gff.StringValue(d.faction)))
+		}
+		list = append(list, gff.Struct{Fields: fields})
+	}
+	return list
+}
+
+func structHasField(s gff.Struct, label string) bool {
+	for _, field := range s.Fields {
+		if field.Label == label {
+			return true
+		}
+	}
+	return false
+}

internal/pipeline/palette_test.go

@@ -0,0 +1,191 @@
+package pipeline
+
+import (
+	"bytes"
+	"encoding/json"
+	"os"
+	"path/filepath"
+	"testing"
+
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
+)
+
+const paletteTestSkeleton = `{
+  "file_type": "ITP ",
+  "file_version": "V3.2",
+  "root": {
+    "struct_type": 4294967295,
+    "fields": [
+      {
+        "label": "MAIN",
+        "type": "List",
+        "value": [
+          {
+            "struct_type": 0,
+            "fields": [
+              {"label": "STRREF", "type": "DWord", "value": 500},
+              {
+                "label": "LIST",
+                "type": "List",
+                "value": [
+                  {
+                    "struct_type": 0,
+                    "fields": [
+                      {"label": "STRREF", "type": "DWord", "value": 6699},
+                      {"label": "ID", "type": "Byte", "value": 23},
+                      {
+                        "label": "LIST",
+                        "type": "List",
+                        "value": [
+                          {
+                            "struct_type": 0,
+                            "fields": [
+                              {"label": "NAME", "type": "CExoString", "value": "Stale Junk"},
+                              {"label": "RESREF", "type": "ResRef", "value": "stalejunk"}
+                            ]
+                          }
+                        ]
+                      }
+                    ]
+                  },
+                  {
+                    "struct_type": 0,
+                    "fields": [
+                      {"label": "STRREF", "type": "DWord", "value": 6753},
+                      {"label": "ID", "type": "Byte", "value": 24}
+                    ]
+                  }
+                ]
+              }
+            ]
+          }
+        ]
+      }
+    ]
+  }
+}
+`
+
+func paletteTestItem(resref, name string, paletteID int) string {
+	return `{
+  "file_type": "UTI ",
+  "file_version": "V3.2",
+  "root": {
+    "struct_type": 4294967295,
+    "fields": [
+      {"label": "TemplateResRef", "type": "ResRef", "value": "` + resref + `"},
+      {
+        "label": "LocalizedName",
+        "type": "CExoLocString",
+        "value": {"string_ref": 4294967295, "entries": [{"id": 0, "value": "` + name + `"}]}
+      },
+      {"label": "PaletteID", "type": "Byte", "value": ` + itoa(paletteID) + `}
+    ]
+  }
+}
+`
+}
+
+func itoa(v int) string {
+	data, _ := json.Marshal(v)
+	return string(data)
+}
+
+func TestBuildProjectsPaletteDescriptorsAndExtractSkipsThem(t *testing.T) {
+	root := t.TempDir()
+	mustMkdir(t, filepath.Join(root, "src", "module"))
+	mustMkdir(t, filepath.Join(root, "src", "palettes"))
+	mustMkdir(t, filepath.Join(root, "src", "blueprints", "items"))
+	mustMkdir(t, filepath.Join(root, "build"))
+
+	mustWriteFile(t, filepath.Join(root, "nwn-tool.json"), `{
+  "module": {"name": "Test Module", "resref": "testmod"},
+  "paths": {"source": "src", "build": "build"}
+}
+`)
+	mustWriteFile(t, filepath.Join(root, "src", "module", "module.ifo.json"), `{
+  "file_type": "IFO ",
+  "file_version": "V3.2",
+  "root": {"struct_type": 4294967295, "fields": [{"label": "Mod_Name", "type": "CExoString", "value": "Test Module"}]}
+}
+`)
+	mustWriteFile(t, filepath.Join(root, "src", "palettes", "itempalcus.itp.json"), paletteTestSkeleton)
+	mustWriteFile(t, filepath.Join(root, "src", "blueprints", "items", "i_b.uti.json"), paletteTestItem("i_b", "Bravo Item", 23))
+	mustWriteFile(t, filepath.Join(root, "src", "blueprints", "items", "i_a.uti.json"), paletteTestItem("i_a", "Alpha Item", 23))
+	mustWriteFile(t, filepath.Join(root, "src", "blueprints", "items", "i_hidden.uti.json"), paletteTestItem("i_hidden", "Hidden Item", 255))
+
+	p, err := project.Load(root)
+	if err != nil {
+		t.Fatalf("load project: %v", err)
+	}
+	if err := p.Scan(); err != nil {
+		t.Fatalf("scan: %v", err)
+	}
+	if _, err := BuildModule(p); err != nil {
+		t.Fatalf("build: %v", err)
+	}
+
+	archiveFile, err := os.Open(p.ModuleArchivePath())
+	if err != nil {
+		t.Fatalf("open module archive: %v", err)
+	}
+	defer archiveFile.Close()
+	archive, err := erf.Read(archiveFile)
+	if err != nil {
+		t.Fatalf("read module archive: %v", err)
+	}
+
+	var palette *gff.Document
+	for _, resource := range archive.Resources {
+		if resource.Name == "itempalcus" {
+			document, err := gff.Read(bytes.NewReader(resource.Data))
+			if err != nil {
+				t.Fatalf("decode itempalcus: %v", err)
+			}
+			palette = &document
+		}
+	}
+	if palette == nil {
+		t.Fatal("itempalcus missing from built module")
+	}
+
+	canonical, err := json.Marshal(palette)
+	if err != nil {
+		t.Fatalf("marshal palette: %v", err)
+	}
+	text := string(canonical)
+	if bytes.Contains(canonical, []byte("stalejunk")) {
+		t.Fatalf("stale descriptor survived projection: %s", text)
+	}
+	for _, resref := range []string{"i_a", "i_b"} {
+		if !bytes.Contains(canonical, []byte(resref)) {
+			t.Fatalf("descriptor %s missing from projection: %s", resref, text)
+		}
+	}
+	if bytes.Contains(canonical, []byte("i_hidden")) {
+		t.Fatalf("PaletteID 255 blueprint leaked into projection: %s", text)
+	}
+	if a, b := bytes.Index(canonical, []byte("i_a")), bytes.Index(canonical, []byte("i_b")); a > b {
+		t.Fatalf("descriptors not name-sorted: %s", text)
+	}
+
+	if _, err := Compare(p); err != nil {
+		t.Fatalf("compare after build: %v", err)
+	}
+
+	// Extraction must never write palcus files back into source.
+	if err := os.Remove(filepath.Join(root, "src", "palettes", "itempalcus.itp.json")); err != nil {
+		t.Fatalf("remove skeleton: %v", err)
+	}
+	if err := p.Scan(); err != nil {
+		t.Fatalf("rescan: %v", err)
+	}
+	if _, err := Extract(p); err != nil {
+		t.Fatalf("extract: %v", err)
+	}
+	if _, err := os.Stat(filepath.Join(root, "src", "palettes", "itempalcus.itp.json")); !os.IsNotExist(err) {
+		t.Fatalf("extract wrote palcus file back (stat err: %v)", err)
+	}
+}

internal/project/project.go

@@ -31,7 +31,7 @@ var SourceExtensions = []string{
 }
 
 var AssetExtensions = []string{
-	".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".gr2", ".itp", ".jpg", ".lod", ".lyt", ".mdb", ".mdl", ".mdx", ".mtr", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wlk", ".wok", ".xml",
+	".2da", ".bik", ".bmp", ".bmu", ".dds", ".dwk", ".itp", ".jpg", ".lod", ".lyt", ".mdl", ".mdx", ".mtr", ".plt", ".png", ".pwk", ".set", ".shd", ".tga", ".txi", ".uti", ".vis", ".wav", ".wok",
 }
 
 var BuiltinScriptPrefixes = []string{

internal/topdata/appearance_migrate.go

@@ -1,96 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacyAppearance(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "appearance")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "appearance")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	targetLockPath := filepath.Join(targetDir, "lock.json")
-	targetModulesDir := filepath.Join(targetDir, "modules")
-	moduleNames := []string{
-		"cotblreaver.json",
-		"crawlingclaw.json",
-		"halfogre.json",
-		"zombieknight.json",
-	}
-	targetModulePaths := make([]string, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
-	}
-	if fileExists(targetBasePath) && fileExists(targetLockPath) {
-		allModulesPresent := true
-		for _, path := range targetModulePaths {
-			if !fileExists(path) {
-				allModulesPresent = false
-				break
-			}
-		}
-		if allModulesPresent {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	baseObj["output"] = "appearance.2da"
-
-	lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-
-	moduleObjs := make([]map[string]any, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
-		if err != nil {
-			return 0, err
-		}
-		moduleObjs = append(moduleObjs, obj)
-	}
-
-	if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: targetBasePath, obj: baseObj},
-		{path: targetLockPath, obj: lockObj},
-	}
-	for i, path := range targetModulePaths {
-		writes = append(writes, struct {
-			path string
-			obj  map[string]any
-		}{path: path, obj: moduleObjs[i]})
-	}
-
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}

internal/topdata/armor_migrate.go

@@ -1,160 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-	"sort"
-)
-
-func importLegacyArmor(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "armor")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "armor")
-	targetPath := filepath.Join(targetDir, "armor.json")
-	if _, err := os.Stat(targetPath); err == nil {
-		obj, err := loadJSONObject(targetPath)
-		if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	rows, err := mergeArmorRows(baseObj, filepath.Join(legacyDir, "modules"))
-	if err != nil {
-		return 0, err
-	}
-
-	if err := os.MkdirAll(targetDir, 0o755); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
-		return 0, err
-	}
-
-	out := map[string]any{
-		"output":  "armor.2da",
-		"columns": baseObj["columns"],
-		"rows":    rows,
-	}
-	raw, err := json.MarshalIndent(out, "", "  ")
-	if err != nil {
-		return 0, err
-	}
-	raw = append(raw, '\n')
-	if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
-		return 0, err
-	}
-	return 1, nil
-}
-
-func mergeArmorRows(baseObj map[string]any, modulesDir string) ([]any, error) {
-	rawRows, ok := baseObj["rows"].([]any)
-	if !ok {
-		return nil, nil
-	}
-
-	rows := make([]map[string]any, 0, len(rawRows))
-	byID := map[int]map[string]any{}
-	for _, raw := range rawRows {
-		row, ok := deepCopyValue(raw).(map[string]any)
-		if !ok {
-			continue
-		}
-		delete(row, "key")
-		rawID, ok := row["id"]
-		if !ok {
-			continue
-		}
-		id, err := asInt(rawID)
-		if err != nil {
-			return nil, err
-		}
-		row["id"] = id
-		rows = append(rows, row)
-		byID[id] = row
-	}
-
-	modulePaths, err := collectModulePaths(modulesDir)
-	if err != nil {
-		return nil, err
-	}
-	for _, path := range modulePaths {
-		obj, err := loadJSONObject(path)
-		if err != nil {
-			return nil, err
-		}
-		if entries, ok := obj["entries"].(map[string]any); ok {
-			for _, raw := range entries {
-				row, ok := deepCopyValue(raw).(map[string]any)
-				if !ok {
-					continue
-				}
-				delete(row, "key")
-				rawID, ok := row["id"]
-				if !ok {
-					continue
-				}
-				id, err := asInt(rawID)
-				if err != nil {
-					return nil, err
-				}
-				row["id"] = id
-				rows = append(rows, row)
-				byID[id] = row
-			}
-		}
-		if overrides, ok := obj["overrides"].([]any); ok {
-			for _, raw := range overrides {
-				override, ok := raw.(map[string]any)
-				if !ok {
-					continue
-				}
-				rawID, ok := override["id"]
-				if !ok {
-					continue
-				}
-				id, err := asInt(rawID)
-				if err != nil {
-					return nil, err
-				}
-				row := byID[id]
-				if row == nil {
-					continue
-				}
-				for key, value := range override {
-					if key == "id" || key == "key" || key == "_tlk" || isMetadataField(key) {
-						continue
-					}
-					row[key] = deepCopyValue(value)
-				}
-			}
-		}
-	}
-
-	sort.Slice(rows, func(i, j int) bool {
-		return rows[i]["id"].(int) < rows[j]["id"].(int)
-	})
-	out := make([]any, 0, len(rows))
-	for _, row := range rows {
-		out = append(out, row)
-	}
-	return out, nil
-}

internal/topdata/base_dialog_migrate.go

@@ -1,41 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacyBaseDialog(referenceBuilderDir, sourceDir string) (int, error) {
-	legacyPath := filepath.Join(referenceBuilderDir, "tlk", "base.json")
-	if _, err := os.Stat(legacyPath); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetPath := filepath.Join(sourceDir, "base_dialog.json")
-	if fileExists(targetPath) {
-		current, err := loadJSONObject(targetPath)
-		if err == nil {
-			if entries, ok := current["entries"].(map[string]any); ok && len(entries) > 0 {
-				return 0, nil
-			}
-		}
-	}
-
-	obj, err := loadJSONObject(legacyPath)
-	if err != nil {
-		return 0, err
-	}
-	raw, err := json.MarshalIndent(obj, "", "  ")
-	if err != nil {
-		return 0, err
-	}
-	raw = append(raw, '\n')
-	if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
-		return 0, err
-	}
-	return 1, nil
-}

internal/topdata/baseitems_migrate.go

@@ -1,105 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacyBaseitems(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "baseitems")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "baseitems")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	targetLockPath := filepath.Join(targetDir, "lock.json")
-	targetModulesDir := filepath.Join(targetDir, "modules")
-	moduleNames := []string{
-		"blunderbuss.json",
-		"coin.json",
-		"estoc.json",
-		"heavymace.json",
-		"maulandfalchion.json",
-		"ovr_baseitems.json",
-		"ovr_cloak255.json",
-		"ovr_helmet255.json",
-		"shortspear.json",
-	}
-	targetModulePaths := make([]string, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
-	}
-	if fileExists(targetBasePath) && fileExists(targetLockPath) {
-		allModulesPresent := true
-		for _, path := range targetModulePaths {
-			if !fileExists(path) {
-				allModulesPresent = false
-				break
-			}
-		}
-		if allModulesPresent {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	baseObj["output"] = "baseitems.2da"
-	canonicalizeWikiMetadataDocument(baseObj)
-
-	lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-
-	moduleObjs := make([]map[string]any, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
-		if err != nil {
-			return 0, err
-		}
-		canonicalizeWikiMetadataDocument(obj)
-		moduleObjs = append(moduleObjs, obj)
-	}
-
-	if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: targetBasePath, obj: baseObj},
-		{path: targetLockPath, obj: lockObj},
-	}
-	for index, path := range targetModulePaths {
-		writes = append(writes, struct {
-			path string
-			obj  map[string]any
-		}{
-			path: path,
-			obj:  moduleObjs[index],
-		})
-	}
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}

internal/topdata/classes_migrate.go

@@ -1,226 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-	"slices"
-	"strings"
-)
-
-var legacyClassesPlainFamilies = []string{"feats", "skills", "savthr", "bfeat", "pres"}
-
-func importLegacyClasses(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	legacyRoot := filepath.Join(referenceBuilderDir, "data", "classes")
-	if _, err := os.Stat(legacyRoot); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetRoot := filepath.Join(dataDir, "classes")
-	if canonicalClassesPresent(targetRoot) {
-		return 0, nil
-	}
-
-	coreCollected, plainCollected, err := collectLegacyClassesDatasets(legacyRoot)
-	if err != nil {
-		return 0, err
-	}
-
-	tableKeyByOutput := map[string]string{}
-	for _, dataset := range plainCollected {
-		if dataset.TableKey == "" {
-			continue
-		}
-		registerLegacyClassesTableKey(tableKeyByOutput, dataset.TableKey, dataset.Dataset.OutputName)
-	}
-
-	coreRows := make([]map[string]any, 0, len(coreCollected.Rows))
-	for _, rawRow := range coreCollected.Rows {
-		row, ok := deepCopyValue(rawRow).(map[string]any)
-		if !ok {
-			continue
-		}
-		if legacyTLK != nil {
-			if _, _, err := inlineLegacyTLKValue(row, legacyTLK); err != nil {
-				return 0, err
-			}
-		}
-		rewriteLegacyClassesCoreRow(row, tableKeyByOutput)
-		coreRows = append(coreRows, row)
-	}
-
-	if err := removePathIfExists(targetRoot); err != nil {
-		return 0, err
-	}
-	if err := os.MkdirAll(filepath.Join(targetRoot, "core"), 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{
-			path: filepath.Join(targetRoot, "core", "base.json"),
-			obj: map[string]any{
-				"output":  coreCollected.Dataset.OutputName,
-				"columns": stringSliceToAny(coreCollected.Columns),
-				"rows":    rowsToAny(coreRows),
-			},
-		},
-		{
-			path: filepath.Join(targetRoot, "core", "lock.json"),
-			obj:  anyMapInt(coreCollected.LockData),
-		},
-	}
-
-	for _, collected := range plainCollected {
-		obj := map[string]any{
-			"output":  collected.Dataset.OutputName,
-			"columns": stringSliceToAny(collected.Columns),
-			"rows":    rowsToAny(collected.Rows),
-		}
-		if strings.TrimSpace(collected.TableKey) != "" {
-			obj["key"] = collected.TableKey
-		}
-		writes = append(writes, struct {
-			path string
-			obj  map[string]any
-		}{
-			path: filepath.Join(dataDir, collected.Dataset.Name+".json"),
-			obj:  obj,
-		})
-	}
-
-	for _, write := range writes {
-		if err := os.MkdirAll(filepath.Dir(write.path), 0o755); err != nil {
-			return 0, err
-		}
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-
-	return len(writes), nil
-}
-
-func canonicalClassesPresent(targetRoot string) bool {
-	if !fileExists(filepath.Join(targetRoot, "core", "base.json")) || !fileExists(filepath.Join(targetRoot, "core", "lock.json")) {
-		return false
-	}
-	for _, family := range legacyClassesPlainFamilies {
-		ok, err := hasJSONFiles(filepath.Join(targetRoot, family))
-		if err != nil || !ok {
-			return false
-		}
-	}
-	return true
-}
-
-func collectLegacyClassesDatasets(legacyRoot string) (nativeCollectedDataset, []nativeCollectedDataset, error) {
-	coreCollected, err := collectBaseDataset(nativeDataset{
-		Name:       "classes/core",
-		BasePath:   filepath.Join(legacyRoot, "core", "base.json"),
-		LockPath:   filepath.Join(legacyRoot, "core", "lock.json"),
-		ModulesDir: filepath.Join(legacyRoot, "core", "modules"),
-		OutputName: "classes.2da",
-		Spec:       specForDataset("classes"),
-	})
-	if err != nil {
-		return nativeCollectedDataset{}, nil, err
-	}
-	coreCollected.Dataset.OutputName = "classes.2da"
-
-	plainCollected := make([]nativeCollectedDataset, 0)
-	for _, family := range legacyClassesPlainFamilies {
-		familyDir := filepath.Join(legacyRoot, family)
-		entries, err := os.ReadDir(familyDir)
-		if err != nil {
-			if os.IsNotExist(err) {
-				continue
-			}
-			return nativeCollectedDataset{}, nil, err
-		}
-		for _, entry := range entries {
-			if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" || strings.HasPrefix(entry.Name(), ".") || entry.Name() == "lock.json" {
-				continue
-			}
-			filePath := filepath.Join(familyDir, entry.Name())
-			tableData, err := loadJSONObject(filePath)
-			if err != nil {
-				return nativeCollectedDataset{}, nil, err
-			}
-			outputName, _ := tableData["output"].(string)
-			if strings.TrimSpace(outputName) == "" {
-				outputName = strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name())) + ".2da"
-			}
-			collected, err := collectPlainDataset(nativeDataset{
-				Name:       filepath.ToSlash(filepath.Join("classes", family, strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name())))),
-				BasePath:   filePath,
-				LockPath:   filepath.Join(familyDir, "lock.json"),
-				OutputName: outputName,
-				Spec:       specForDataset(filepath.ToSlash(filepath.Join("classes", family, strings.TrimSuffix(entry.Name(), filepath.Ext(entry.Name()))))),
-			})
-			if err != nil {
-				return nativeCollectedDataset{}, nil, err
-			}
-			plainCollected = append(plainCollected, collected)
-		}
-	}
-
-	slices.SortFunc(plainCollected, func(a, b nativeCollectedDataset) int {
-		return strings.Compare(a.Dataset.Name, b.Dataset.Name)
-	})
-	return coreCollected, plainCollected, nil
-}
-
-func registerLegacyClassesTableKey(tableKeyByOutput map[string]string, tableKey, outputName string) {
-	trimmedOutput := strings.TrimSpace(outputName)
-	if trimmedOutput != "" {
-		tableKeyByOutput[trimmedOutput] = tableKey
-	}
-	trimmedStem := strings.TrimSpace(outputStem(outputName))
-	if trimmedStem != "" {
-		tableKeyByOutput[trimmedStem] = tableKey
-	}
-}
-
-func rewriteLegacyClassesCoreRow(row map[string]any, tableKeyByOutput map[string]string) {
-	for _, field := range []string{"FeatsTable", "SavingThrowTable", "SkillsTable", "BonusFeatsTable", "PreReqTable"} {
-		tableKey := legacyClassesTableKeyForValue(row[field], tableKeyByOutput)
-		if tableKey == "" {
-			continue
-		}
-		row[field] = map[string]any{"table": tableKey}
-	}
-}
-
-func legacyClassesTableKeyForValue(value any, tableKeyByOutput map[string]string) string {
-	switch typed := value.(type) {
-	case map[string]any:
-		tableKey, _ := typed["table"].(string)
-		return strings.TrimSpace(tableKey)
-	case string:
-		trimmed := strings.TrimSpace(typed)
-		if trimmed == "" || trimmed == nullValue {
-			return ""
-		}
-		if trimmed != strings.ToLower(trimmed) {
-			return ""
-		}
-		if tableKey, ok := tableKeyByOutput[trimmed]; ok {
-			return tableKey
-		}
-		return ""
-	default:
-		return ""
-	}
-}

internal/topdata/cloakmodel_migrate.go

@@ -1,6 +0,0 @@
-package topdata
-
-func importLegacyCloakmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-	return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "cloakmodel", "cloakmodel.2da", nil)
-}

internal/topdata/creaturespeed_migrate.go

@@ -1,145 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacyCreaturespeed(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "creaturespeed")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "creaturespeed")
-	targetPath := filepath.Join(targetDir, "creaturespeed.json")
-	if _, err := os.Stat(targetPath); err == nil {
-		obj, err := loadJSONObject(targetPath)
-		if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	rows, err := mergeCreaturespeedRows(baseObj, filepath.Join(legacyDir, "modules"))
-	if err != nil {
-		return 0, err
-	}
-	if err := os.MkdirAll(targetDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
-		return 0, err
-	}
-
-	out := map[string]any{
-		"output":  "creaturespeed.2da",
-		"columns": baseObj["columns"],
-		"rows":    rows,
-	}
-
-	raw, err := json.MarshalIndent(out, "", "  ")
-	if err != nil {
-		return 0, err
-	}
-	raw = append(raw, '\n')
-	if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
-		return 0, err
-	}
-	return 1, nil
-}
-
-func mergeCreaturespeedRows(baseObj map[string]any, modulesDir string) ([]any, error) {
-	rawRows, ok := baseObj["rows"].([]any)
-	if !ok {
-		return nil, nil
-	}
-	rows := make([]map[string]any, 0, len(rawRows))
-	byID := map[int]map[string]any{}
-	for _, raw := range rawRows {
-		row, ok := deepCopyValue(raw).(map[string]any)
-		if !ok {
-			continue
-		}
-		delete(row, "key")
-		rows = append(rows, row)
-		if rawID, ok := row["id"]; ok {
-			id, err := asInt(rawID)
-			if err != nil {
-				return nil, err
-			}
-			byID[id] = row
-		}
-	}
-
-	modulePaths, err := collectModulePaths(modulesDir)
-	if err != nil {
-		return nil, err
-	}
-	for _, path := range modulePaths {
-		obj, err := loadJSONObject(path)
-		if err != nil {
-			return nil, err
-		}
-		overrideList, ok := obj["overrides"].([]any)
-		if !ok {
-			continue
-		}
-		for _, raw := range overrideList {
-			override, ok := raw.(map[string]any)
-			if !ok {
-				continue
-			}
-			rawID, ok := override["id"]
-			if !ok {
-				continue
-			}
-			id, err := asInt(rawID)
-			if err != nil {
-				return nil, err
-			}
-			row := byID[id]
-			if row == nil {
-				continue
-			}
-			for key, value := range override {
-				if key == "id" || key == "key" || key == "_tlk" {
-					continue
-				}
-				row[key] = deepCopyValue(value)
-			}
-		}
-	}
-
-	out := make([]any, 0, len(rows))
-	for _, row := range rows {
-		out = append(out, row)
-	}
-	return out, nil
-}
-
-func removePathIfExists(path string) error {
-	if _, err := os.Stat(path); err != nil {
-		if os.IsNotExist(err) {
-			return nil
-		}
-		return err
-	}
-	return os.RemoveAll(path)
-}

internal/topdata/damagetypes_migrate.go

@@ -1,288 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"fmt"
-	"os"
-	"path/filepath"
-	"sort"
-	"strings"
-)
-
-func importLegacyDamagetypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyRoot := filepath.Join(referenceBuilderDir, "data", "damagetypes")
-	if _, err := os.Stat(legacyRoot); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "damagetypes", "registry")
-	targetPath := filepath.Join(targetDir, "types.json")
-	if _, err := os.Stat(targetPath); err == nil {
-		obj, err := loadJSONObject(targetPath)
-		if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
-			return 0, nil
-		}
-	}
-
-	coreBase, err := loadJSONObject(filepath.Join(legacyRoot, "core", "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	groupBase, err := loadJSONObject(filepath.Join(legacyRoot, "groups", "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	hitvisualBase, err := loadJSONObject(filepath.Join(legacyRoot, "hitvisual", "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	coreModule, err := loadJSONObject(filepath.Join(legacyRoot, "core", "modules", "add_eos.json"))
-	if err != nil {
-		return 0, err
-	}
-	coreLock, err := loadLockfile(filepath.Join(legacyRoot, "core", "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-	groupModule, err := loadJSONObject(filepath.Join(legacyRoot, "groups", "modules", "add_eos.json"))
-	if err != nil {
-		return 0, err
-	}
-	groupLock, err := loadLockfile(filepath.Join(legacyRoot, "groups", "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-	hitvisualModule, err := loadJSONObject(filepath.Join(legacyRoot, "hitvisual", "modules", "add_eos.json"))
-	if err != nil {
-		return 0, err
-	}
-	hitvisualLock, err := loadLockfile(filepath.Join(legacyRoot, "hitvisual", "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-
-	rows, lockData, err := mergeLegacyDamagetypes(coreBase, groupBase, hitvisualBase, coreModule, coreLock, groupModule, groupLock, hitvisualModule, hitvisualLock)
-	if err != nil {
-		return 0, err
-	}
-
-	if err := os.MkdirAll(targetDir, 0o755); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "core")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "groups")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(dataDir, "damagetypes", "hitvisual")); err != nil {
-		return 0, err
-	}
-
-	typesOut := map[string]any{"rows": rows}
-	raw, err := json.MarshalIndent(typesOut, "", "  ")
-	if err != nil {
-		return 0, err
-	}
-	raw = append(raw, '\n')
-	if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
-		return 0, err
-	}
-	if err := saveLockfile(filepath.Join(targetDir, damagetypesRegistryLock), lockData); err != nil {
-		return 0, err
-	}
-	return 2, nil
-}
-
-func mergeLegacyDamagetypes(coreBase, groupBase, hitvisualBase, coreModule map[string]any, coreLock map[string]int, groupModule map[string]any, groupLock map[string]int, hitvisualModule map[string]any, hitvisualLock map[string]int) ([]map[string]any, map[string]int, error) {
-	coreRows, err := coerceRowMapSlice(coreBase["rows"])
-	if err != nil {
-		return nil, nil, err
-	}
-	groupRows, err := coerceRowMapSlice(groupBase["rows"])
-	if err != nil {
-		return nil, nil, err
-	}
-	hitRows, err := coerceRowMapSlice(hitvisualBase["rows"])
-	if err != nil {
-		return nil, nil, err
-	}
-	groupByID := map[int]map[string]any{}
-	for _, row := range groupRows {
-		id, err := asInt(row["id"])
-		if err != nil {
-			return nil, nil, err
-		}
-		groupByID[id] = row
-	}
-	hitByID := map[int]map[string]any{}
-	for _, row := range hitRows {
-		id, err := asInt(row["id"])
-		if err != nil {
-			return nil, nil, err
-		}
-		hitByID[id] = row
-	}
-
-	outRows := make([]map[string]any, 0, len(coreRows))
-	lockData := map[string]int{}
-	for _, core := range coreRows {
-		id, err := asInt(core["id"])
-		if err != nil {
-			return nil, nil, err
-		}
-		groupID, err := asInt(core["DamageTypeGroup"])
-		if err != nil {
-			return nil, nil, err
-		}
-		group := groupByID[groupID]
-		if group == nil {
-			return nil, nil, fmt.Errorf("core id %d references missing group %d", id, groupID)
-		}
-		hit := hitByID[id]
-		if hit == nil {
-			return nil, nil, fmt.Errorf("core id %d is missing hitvisual row", id)
-		}
-		key := "damagetype:" + normalizeDamagetypeKey(core["Label"])
-		lockData[key] = id
-		outRows = append(outRows, map[string]any{
-			"key":                      key,
-			"id":                       id,
-			"label":                    deepCopyValue(core["Label"]),
-			"charsheet_strref":         deepCopyValue(core["CharsheetStrref"]),
-			"damage_type_group":        strconvString(groupID),
-			"damage_ranged_projectile": deepCopyValue(core["DamageRangedProjectile"]),
-			"group_label":              deepCopyValue(group["Label"]),
-			"feedback_strref":          deepCopyValue(group["FeedbackStrref"]),
-			"color_r":                  deepCopyValue(group["ColorR"]),
-			"color_g":                  deepCopyValue(group["ColorG"]),
-			"color_b":                  deepCopyValue(group["ColorB"]),
-			"visual_effect_id":         deepCopyValue(hit["VisualEffectID"]),
-			"ranged_effect_id":         deepCopyValue(hit["RangedEffectID"]),
-		})
-	}
-
-	coreEntries, err := coerceEntriesMap(coreModule["entries"])
-	if err != nil {
-		return nil, nil, err
-	}
-	groupEntries, err := coerceEntriesMap(groupModule["entries"])
-	if err != nil {
-		return nil, nil, err
-	}
-	hitEntries, err := coerceEntriesMap(hitvisualModule["entries"])
-	if err != nil {
-		return nil, nil, err
-	}
-
-	keys := make([]string, 0, len(coreEntries))
-	for key := range coreEntries {
-		keys = append(keys, key)
-	}
-	sort.Slice(keys, func(i, j int) bool {
-		a := strings.TrimPrefix(keys[i], "damagetypes:")
-		b := strings.TrimPrefix(keys[j], "damagetypes:")
-		return a < b
-	})
-	for _, coreKey := range keys {
-		core := coreEntries[coreKey]
-		suffix := strings.TrimPrefix(coreKey, "damagetypes:")
-		group := groupEntries["damagetypegroups:"+suffix]
-		if group == nil {
-			return nil, nil, fmt.Errorf("%s: missing matching group entry", coreKey)
-		}
-		hit := hitEntries["damagehitvisual:"+suffix]
-		if hit == nil {
-			return nil, nil, fmt.Errorf("%s: missing matching hitvisual entry", coreKey)
-		}
-		id, ok := coreLock[coreKey]
-		if !ok {
-			return nil, nil, fmt.Errorf("%s: missing core lock id", coreKey)
-		}
-		groupKey := "damagetypegroups:" + suffix
-		groupID, ok := groupLock[groupKey]
-		if !ok {
-			return nil, nil, fmt.Errorf("%s: missing group lock id", groupKey)
-		}
-		hitKey := "damagehitvisual:" + suffix
-		hitID, ok := hitvisualLock[hitKey]
-		if !ok {
-			return nil, nil, fmt.Errorf("%s: missing hitvisual lock id", hitKey)
-		}
-		if hitID != id {
-			return nil, nil, fmt.Errorf("%s: core id %d does not match hitvisual id %d", suffix, id, hitID)
-		}
-		key := "damagetype:" + suffix
-		lockData[key] = id
-		outRows = append(outRows, map[string]any{
-			"key":                      key,
-			"id":                       id,
-			"label":                    deepCopyValue(core["Label"]),
-			"charsheet_strref":         deepCopyValue(core["CharsheetStrref"]),
-			"damage_type_group":        strconvString(groupID),
-			"damage_ranged_projectile": deepCopyValue(core["DamageRangedProjectile"]),
-			"group_label":              deepCopyValue(group["Label"]),
-			"feedback_strref":          deepCopyValue(group["FeedbackStrref"]),
-			"color_r":                  deepCopyValue(group["ColorR"]),
-			"color_g":                  deepCopyValue(group["ColorG"]),
-			"color_b":                  deepCopyValue(group["ColorB"]),
-			"visual_effect_id":         deepCopyValue(hit["VisualEffectID"]),
-			"ranged_effect_id":         deepCopyValue(hit["RangedEffectID"]),
-		})
-	}
-
-	sort.Slice(outRows, func(i, j int) bool {
-		return outRows[i]["id"].(int) < outRows[j]["id"].(int)
-	})
-	return outRows, lockData, nil
-}
-
-func coerceRowMapSlice(value any) ([]map[string]any, error) {
-	rawRows, ok := value.([]any)
-	if !ok {
-		return nil, fmt.Errorf("rows must be an array")
-	}
-	rows := make([]map[string]any, 0, len(rawRows))
-	for _, raw := range rawRows {
-		row, ok := raw.(map[string]any)
-		if !ok {
-			return nil, fmt.Errorf("row must be an object")
-		}
-		rows = append(rows, row)
-	}
-	return rows, nil
-}
-
-func coerceEntriesMap(value any) (map[string]map[string]any, error) {
-	rawEntries, ok := value.(map[string]any)
-	if !ok {
-		return nil, fmt.Errorf("entries must be an object")
-	}
-	entries := make(map[string]map[string]any, len(rawEntries))
-	for key, raw := range rawEntries {
-		row, ok := raw.(map[string]any)
-		if !ok {
-			return nil, fmt.Errorf("entry %s must be an object", key)
-		}
-		entries[key] = row
-	}
-	return entries, nil
-}
-
-func normalizeDamagetypeKey(value any) string {
-	text := strings.TrimSpace(strings.ToLower(format2DAValue(value)))
-	text = strings.TrimSuffix(text, "damage")
-	text = strings.ReplaceAll(text, " ", "")
-	text = strings.ReplaceAll(text, "_", "")
-	text = strings.ReplaceAll(text, "-", "")
-	return text
-}
-
-func strconvString(v int) string {
-	return fmt.Sprintf("%d", v)
-}

internal/topdata/doortypes_migrate.go

@@ -1,99 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacyDoortypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "doortypes")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "doortypes")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	targetLockPath := filepath.Join(targetDir, "lock.json")
-	targetModulesDir := filepath.Join(targetDir, "modules")
-	moduleNames := []string{
-		"add_pld01.json",
-		"add_sic11.json",
-		"add_tapr.json",
-		"add_tdm01.json",
-		"add_tdx01.json",
-		"add_tei01.json",
-		"add_tfm01.json",
-	}
-	targetModulePaths := make([]string, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
-	}
-	if fileExists(targetBasePath) && fileExists(targetLockPath) {
-		allModulesPresent := true
-		for _, path := range targetModulePaths {
-			if !fileExists(path) {
-				allModulesPresent = false
-				break
-			}
-		}
-		if allModulesPresent {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	baseObj["output"] = "doortypes.2da"
-
-	lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-
-	moduleObjs := make([]map[string]any, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
-		if err != nil {
-			return 0, err
-		}
-		moduleObjs = append(moduleObjs, obj)
-	}
-
-	if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: targetBasePath, obj: baseObj},
-		{path: targetLockPath, obj: lockObj},
-	}
-	for i, path := range targetModulePaths {
-		writes = append(writes, struct {
-			path string
-			obj  map[string]any
-		}{path: path, obj: moduleObjs[i]})
-	}
-
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}

internal/topdata/feat_migrate.go

@@ -1,64 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacyFeat(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "feat")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-	legacyBasePath := filepath.Join(legacyDir, "base.json")
-	if _, err := os.Stat(legacyBasePath); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "feat")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	var targetObj map[string]any
-	if _, err := os.Stat(targetBasePath); err == nil {
-		targetObj, err = loadJSONObject(targetBasePath)
-		if err != nil {
-			return 0, err
-		}
-	} else if !os.IsNotExist(err) {
-		return 0, err
-	}
-	if targetObj != nil {
-		if rows, ok := targetObj["rows"].([]any); ok && len(rows) > 0 {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(legacyBasePath)
-	if err != nil {
-		return 0, err
-	}
-	baseObj["output"] = "feat.2da"
-	baseObj["compare_reference"] = false
-	canonicalizeWikiMetadataDocument(baseObj)
-
-	if err := os.MkdirAll(targetDir, 0o755); err != nil {
-		return 0, err
-	}
-	raw, err := json.MarshalIndent(baseObj, "", "  ")
-	if err != nil {
-		return 0, err
-	}
-	raw = append(raw, '\n')
-	if err := os.WriteFile(targetBasePath, raw, 0o644); err != nil {
-		return 0, err
-	}
-	return 1, nil
-}

internal/topdata/genericdoors_migrate.go

@@ -1,6 +0,0 @@
-package topdata
-
-func importLegacyGenericdoors(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-	return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "genericdoors", "genericdoors.2da", nil)
-}

internal/topdata/itemprops_registry.go

@@ -601,397 +601,6 @@ func deepCopyValue(value any) any {
 	return out
 }
 
-func importLegacyItempropsRegistry(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	legacyDataDir := filepath.Join(referenceBuilderDir, "data")
-	registryDir := filepath.Join(dataDir, filepath.FromSlash(itempropsRegistryDirName))
-	if _, err := os.Stat(filepath.Join(registryDir, "properties.json")); err == nil {
-		if refs, err := countLegacyTLKRefsInRegistry(registryDir); err == nil && refs == 0 {
-			return 0, nil
-		}
-	}
-	if _, err := os.Stat(filepath.Join(legacyDataDir, "itemprops")); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	defs, err := collectBaseDataset(nativeDataset{
-		Name:       "itemprops/defs",
-		BasePath:   filepath.Join(legacyDataDir, "itemprops", "defs", "base.json"),
-		LockPath:   filepath.Join(legacyDataDir, "itemprops", "defs", "lock.json"),
-		ModulesDir: filepath.Join(legacyDataDir, "itemprops", "defs", "modules"),
-		Spec:       specForDataset("itemprops"),
-	})
-	if err != nil {
-		return 0, err
-	}
-	sets, err := collectBaseDataset(nativeDataset{
-		Name:       "itemprops/sets",
-		BasePath:   filepath.Join(legacyDataDir, "itemprops", "sets", "base.json"),
-		LockPath:   filepath.Join(legacyDataDir, "itemprops", "sets", "lock.json"),
-		ModulesDir: filepath.Join(legacyDataDir, "itemprops", "sets", "modules"),
-		Spec:       specForDataset("itemprops"),
-	})
-	if err != nil {
-		return 0, err
-	}
-	costIndex, err := collectBaseDataset(nativeDataset{
-		Name:       "itemprops/costtables/index",
-		BasePath:   filepath.Join(legacyDataDir, "itemprops", "costtables", "index", "base.json"),
-		LockPath:   filepath.Join(legacyDataDir, "itemprops", "costtables", "index", "lock.json"),
-		ModulesDir: filepath.Join(legacyDataDir, "itemprops", "costtables", "index", "modules"),
-		Spec:       specForDataset("itemprops"),
-	})
-	if err != nil {
-		return 0, err
-	}
-	paramIndex, err := collectBaseDataset(nativeDataset{
-		Name:       "itemprops/paramtables/index",
-		BasePath:   filepath.Join(legacyDataDir, "itemprops", "paramtables", "index", "base.json"),
-		LockPath:   filepath.Join(legacyDataDir, "itemprops", "paramtables", "index", "lock.json"),
-		ModulesDir: filepath.Join(legacyDataDir, "itemprops", "paramtables", "index", "modules"),
-		Spec:       specForDataset("itemprops"),
-	})
-	if err != nil {
-		return 0, err
-	}
-
-	ownedCostTables, err := collectLegacyOwnedItempropTables(filepath.Join(legacyDataDir, "itemprops", "costtables"), "index")
-	if err != nil {
-		return 0, err
-	}
-	ownedParamTables, err := collectLegacyOwnedItempropTables(filepath.Join(legacyDataDir, "itemprops", "paramtables"), "index")
-	if err != nil {
-		return 0, err
-	}
-	ownedSubtypeTables, err := collectLegacyOwnedItempropTables(filepath.Join(legacyDataDir, "itemprops", "subtypes"), "")
-	if err != nil && !os.IsNotExist(err) {
-		return 0, err
-	}
-
-	setsByID := map[int]map[string]any{}
-	for _, row := range sets.Rows {
-		setsByID[row["id"].(int)] = row
-	}
-	costByID := map[int]map[string]any{}
-	for _, row := range costIndex.Rows {
-		costByID[row["id"].(int)] = row
-	}
-	paramByID := map[int]map[string]any{}
-	for _, row := range paramIndex.Rows {
-		paramByID[row["id"].(int)] = row
-	}
-
-	registryLock := map[string]int{}
-	legacyKeyByID := map[string]string{}
-	if legacyTLK != nil {
-		for key, id := range legacyTLK.Lock {
-			legacyKeyByID[strconv.Itoa(id)] = key
-		}
-	}
-	properties := make([]map[string]any, 0, len(defs.Rows))
-	subtypeRegistryByKey := map[string]map[string]any{}
-	for _, row := range defs.Rows {
-		rowID := row["id"].(int)
-		key := canonicalItempropPropertyKey(row)
-		registryLock[key] = rowID
-
-		property := map[string]any{
-			"key":           key,
-			"label":         row["Label"],
-			"name":          deepCopyValue(row["Name"]),
-			"property_text": deepCopyValue(row["GameStrRef"]),
-		}
-		if description := nullableRegistryField(row, "Description"); description != nullValue {
-			property["description"] = description
-		}
-		if setRow, ok := setsByID[rowID]; ok {
-			property["availability"] = importLegacyAvailability(setRow)
-			if setName := nullableRegistryField(setRow, "StringRef"); setName != nullValue && setName != stringField(row, "Name") {
-				property["set_name"] = setName
-			}
-			if setLabel := stringField(setRow, "Label"); setLabel != "" && setLabel != stringField(row, "Label") {
-				property["set_label"] = setLabel
-			}
-		} else {
-			property["availability"] = []any{}
-		}
-
-		if subtypeResRef := stringField(row, "SubTypeResRef"); subtypeResRef != "" && subtypeResRef != nullValue {
-			subtypeKey := canonicalModelKey("subtype_models", subtypeResRef)
-			property["subtype"] = map[string]any{"ref": subtypeKey}
-			if _, ok := subtypeRegistryByKey[subtypeKey]; !ok {
-				subtypeRegistryByKey[subtypeKey] = buildLegacySubtypeModel(subtypeKey, subtypeResRef, ownedSubtypeTables)
-			}
-		}
-		costValue := stringField(row, "Cost")
-		costID := stringField(row, "CostTableResRef")
-		if costValue != "" && costValue != nullValue || costID != "" && costID != nullValue {
-			cost := map[string]any{}
-			if costValue != "" && costValue != nullValue {
-				cost["value"] = costValue
-			}
-			if costID != "" && costID != nullValue {
-				id, err := strconv.Atoi(costID)
-				if err != nil {
-					return 0, err
-				}
-				costRow, ok := costByID[id]
-				if !ok {
-					return 0, fmt.Errorf("%s references missing cost table index %s", key, costID)
-				}
-				cost["ref"] = canonicalModelKey("cost_models", stringField(costRow, "Name"))
-			}
-			property["cost"] = cost
-		}
-		if paramID := stringField(row, "Param1ResRef"); paramID != "" && paramID != nullValue {
-			id, err := strconv.Atoi(paramID)
-			if err != nil {
-				return 0, err
-			}
-			paramRow, ok := paramByID[id]
-			if !ok {
-				return 0, fmt.Errorf("%s references missing param table index %s", key, paramID)
-			}
-			property["param"] = map[string]any{"ref": canonicalModelKey("param_models", stringField(paramRow, "TableResRef"))}
-		}
-		if legacyTLK != nil {
-			property["name"] = importLegacyStrrefValue(property["name"], key+".name", legacyTLK, legacyKeyByID)
-			property["property_text"] = importLegacyStrrefValue(property["property_text"], key+".property_text", legacyTLK, legacyKeyByID)
-			if description, ok := property["description"]; ok {
-				property["description"] = importLegacyStrrefValue(description, key+".description", legacyTLK, legacyKeyByID)
-			}
-			if updated, _, err := inlineLegacyTLKValue(property, legacyTLK); err != nil {
-				return 0, err
-			} else if updated {
-				// property updated in place
-			}
-		}
-		properties = append(properties, property)
-	}
-
-	costModels := make([]map[string]any, 0, len(costIndex.Rows))
-	for _, row := range costIndex.Rows {
-		key := canonicalModelKey("cost_models", stringField(row, "Name"))
-		registryLock[key] = row["id"].(int)
-		model := map[string]any{
-			"key":         key,
-			"index_name":  row["Name"],
-			"label":       row["Label"],
-			"client_load": row["ClientLoad"],
-			"table":       buildLegacyOwnedModelTable(stringField(row, "Name"), ownedCostTables),
-		}
-		costModels = append(costModels, model)
-	}
-
-	paramModels := make([]map[string]any, 0, len(paramIndex.Rows))
-	for _, row := range paramIndex.Rows {
-		key := canonicalModelKey("param_models", stringField(row, "TableResRef"))
-		registryLock[key] = row["id"].(int)
-		model := map[string]any{
-			"key":   key,
-			"name":  deepCopyValue(row["Name"]),
-			"label": row["Lable"],
-			"table": buildLegacyOwnedModelTable(stringField(row, "TableResRef"), ownedParamTables),
-		}
-		paramModels = append(paramModels, model)
-	}
-
-	subtypeModels := make([]map[string]any, 0, len(subtypeRegistryByKey))
-	for _, key := range sortedKeysAny(subtypeRegistryByKey) {
-		subtypeModels = append(subtypeModels, subtypeRegistryByKey[key])
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{filepath.Join(registryDir, "properties.json"), map[string]any{"rows": properties}},
-		{filepath.Join(registryDir, "cost_models.json"), map[string]any{"rows": costModels}},
-		{filepath.Join(registryDir, "param_models.json"), map[string]any{"rows": paramModels}},
-		{filepath.Join(registryDir, "subtype_models.json"), map[string]any{"rows": subtypeModels}},
-		{filepath.Join(registryDir, itempropsRegistryLock), anyMapInt(registryLock)},
-	}
-	if err := os.MkdirAll(registryDir, 0o755); err != nil {
-		return 0, err
-	}
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}
-
-func countLegacyTLKRefsInRegistry(registryDir string) (int, error) {
-	paths, err := collectDataJSONPaths(registryDir)
-	if err != nil {
-		return 0, err
-	}
-	refs := 0
-	for _, path := range paths {
-		obj, err := loadJSONObject(path)
-		if err != nil {
-			return 0, err
-		}
-		refs += countLegacyTLKRefsInValue(obj)
-	}
-	return refs, nil
-}
-
-func collectLegacyOwnedItempropTables(root, skipDir string) (map[string]map[string]any, error) {
-	result := map[string]map[string]any{}
-	entries, err := os.ReadDir(root)
-	if err != nil {
-		return nil, err
-	}
-	for _, entry := range entries {
-		if !entry.IsDir() {
-			continue
-		}
-		if skipDir != "" && entry.Name() == skipDir {
-			continue
-		}
-		dir := filepath.Join(root, entry.Name())
-		basePath := filepath.Join(dir, "base.json")
-		if _, err := os.Stat(basePath); err != nil {
-			continue
-		}
-		collected, err := collectBaseDataset(nativeDataset{
-			Name:       filepath.ToSlash(filepath.Join("itemprops", filepath.Base(root), entry.Name())),
-			BasePath:   basePath,
-			LockPath:   filepath.Join(dir, "lock.json"),
-			ModulesDir: filepath.Join(dir, "modules"),
-			Spec:       specForDataset("itemprops"),
-		})
-		if err != nil {
-			return nil, err
-		}
-		result[strings.ToLower(outputStem(collected.Dataset.OutputName))] = map[string]any{
-			"ownership": "owned",
-			"resref":    outputStem(collected.Dataset.OutputName),
-			"output":    collected.Dataset.OutputName,
-			"columns":   stringSliceToAny(collected.Columns),
-			"rows":      rowsToAny(collected.Rows),
-		}
-	}
-	return result, nil
-}
-
-func buildLegacySubtypeModel(key, resref string, owned map[string]map[string]any) map[string]any {
-	return map[string]any{
-		"key":          key,
-		"table_resref": resref,
-		"table":        buildLegacyOwnedModelTable(resref, owned),
-	}
-}
-
-func buildLegacyOwnedModelTable(resref string, owned map[string]map[string]any) map[string]any {
-	if table, ok := owned[strings.ToLower(resref)]; ok {
-		return table
-	}
-	return map[string]any{
-		"ownership": "external",
-		"resref":    resref,
-	}
-}
-
-func canonicalItempropPropertyKey(row map[string]any) string {
-	if key := stringField(row, "key"); key != "" {
-		return "itemprop:" + strings.TrimPrefix(key, "itempropdef:")
-	}
-	return canonicalModelKey("itemprop", stringField(row, "Label"))
-}
-
-func canonicalModelKey(prefix, source string) string {
-	normalized := strings.ToLower(strings.TrimSpace(source))
-	normalized = strings.ReplaceAll(normalized, " ", "")
-	normalized = strings.ReplaceAll(normalized, "-", "")
-	normalized = strings.ReplaceAll(normalized, "_", "")
-	return prefix + ":" + normalized
-}
-
-func importLegacyStrrefValue(value any, fallbackKey string, legacy *legacyTLKData, keyByID map[string]string) any {
-	if legacy == nil {
-		return value
-	}
-	if _, ok, _ := parseTLKPayload(value, true); ok {
-		return value
-	}
-	_ = keyByID
-	key := fallbackKey
-	entry, ok := legacy.Entries[key]
-	if !ok {
-		return value
-	}
-	payload := map[string]any{
-		"key":  key,
-		"text": entry.Text,
-	}
-	if entry.SoundResRef != "" {
-		payload["sound_resref"] = entry.SoundResRef
-	}
-	if entry.SoundLength != 0 {
-		payload["sound_length"] = entry.SoundLength
-	}
-	return map[string]any{"tlk": payload}
-}
-
-func importLegacyAvailability(row map[string]any) []any {
-	out := make([]any, 0)
-	for _, column := range itempropsAvailabilityColumns {
-		value, ok := row[column]
-		if !ok {
-			continue
-		}
-		text := formatRegistryScalar(value)
-		if text != "1" {
-			continue
-		}
-		out = append(out, normalizeAvailabilityName(column))
-	}
-	return out
-}
-
-func anyMapInt(values map[string]int) map[string]any {
-	out := make(map[string]any, len(values))
-	for key, value := range values {
-		out[key] = value
-	}
-	return out
-}
-
-func stringSliceToAny(values []string) []any {
-	out := make([]any, 0, len(values))
-	for _, value := range values {
-		out = append(out, value)
-	}
-	return out
-}
-
-func rowsToAny(rows []map[string]any) []any {
-	out := make([]any, 0, len(rows))
-	for _, row := range rows {
-		out = append(out, deepCopyValue(row))
-	}
-	return out
-}
-
-func sortedKeysAny(values map[string]map[string]any) []string {
-	out := make([]string, 0, len(values))
-	for key := range values {
-		out = append(out, key)
-	}
-	slices.Sort(out)
-	return out
-}
-
 func validateItempropsRegistryGraph(dataDir string, report *ValidationReport) {
 	registry, err := loadItempropsRegistry(dataDir)
 	if err != nil {

internal/topdata/legacy_dataset_migrate.go

@@ -1,176 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"io/fs"
-	"os"
-	"path/filepath"
-	"sort"
-	"strings"
-)
-
-type legacyDatasetTransform func(relativePath string, obj map[string]any)
-
-func importLegacyDatasetMirror(referenceBuilderDir, dataDir, datasetName, outputName string, transform legacyDatasetTransform) (int, error) {
-	legacyDir := filepath.Join(referenceBuilderDir, "data", datasetName)
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, datasetName)
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	if outputName != "" {
-		baseObj["output"] = outputName
-	}
-	if transform != nil {
-		transform("base.json", baseObj)
-	}
-
-	lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-	if transform != nil {
-		transform("lock.json", lockObj)
-	}
-
-	moduleRelPaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
-	if err != nil {
-		return 0, err
-	}
-
-	moduleObjs := make(map[string]map[string]any, len(moduleRelPaths))
-	for _, relPath := range moduleRelPaths {
-		obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", relPath))
-		if err != nil {
-			return 0, err
-		}
-		normalizedRel := filepath.ToSlash(relPath)
-		if transform != nil {
-			transform(filepath.ToSlash(filepath.Join("modules", normalizedRel)), obj)
-		}
-		moduleObjs[normalizedRel] = obj
-	}
-
-	if err := os.MkdirAll(targetDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	updated := 0
-	for _, write := range []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: filepath.Join(targetDir, "base.json"), obj: baseObj},
-		{path: filepath.Join(targetDir, "lock.json"), obj: lockObj},
-	} {
-		changed, err := writeJSONObjectIfChanged(write.path, write.obj)
-		if err != nil {
-			return 0, err
-		}
-		if changed {
-			updated++
-		}
-	}
-
-	targetModulesDir := filepath.Join(targetDir, "modules")
-	if len(moduleObjs) > 0 {
-		if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
-			return 0, err
-		}
-	}
-	for relPath, obj := range moduleObjs {
-		targetPath := filepath.Join(targetModulesDir, filepath.FromSlash(relPath))
-		if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
-			return 0, err
-		}
-		changed, err := writeJSONObjectIfChanged(targetPath, obj)
-		if err != nil {
-			return 0, err
-		}
-		if changed {
-			updated++
-		}
-	}
-
-	staleModules, err := collectJSONRelativePaths(targetModulesDir)
-	if err != nil {
-		return 0, err
-	}
-	for _, relPath := range staleModules {
-		normalizedRel := filepath.ToSlash(relPath)
-		if _, ok := moduleObjs[normalizedRel]; ok {
-			continue
-		}
-		if err := os.Remove(filepath.Join(targetModulesDir, filepath.FromSlash(normalizedRel))); err != nil && !os.IsNotExist(err) {
-			return 0, err
-		}
-		updated++
-	}
-
-	return updated, nil
-}
-
-func collectJSONRelativePaths(root string) ([]string, error) {
-	if _, err := os.Stat(root); err != nil {
-		if os.IsNotExist(err) {
-			return nil, nil
-		}
-		return nil, err
-	}
-
-	var relPaths []string
-	err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
-		if err != nil {
-			return err
-		}
-		if d.IsDir() {
-			return nil
-		}
-		if !strings.EqualFold(filepath.Ext(d.Name()), ".json") {
-			return nil
-		}
-		relPath, err := filepath.Rel(root, path)
-		if err != nil {
-			return err
-		}
-		relPaths = append(relPaths, filepath.ToSlash(relPath))
-		return nil
-	})
-	if err != nil {
-		return nil, err
-	}
-	sort.Strings(relPaths)
-	return relPaths, nil
-}
-
-func writeJSONObjectIfChanged(path string, obj map[string]any) (bool, error) {
-	raw, err := json.MarshalIndent(obj, "", "  ")
-	if err != nil {
-		return false, err
-	}
-	raw = append(raw, '\n')
-
-	current, err := os.ReadFile(path)
-	if err == nil && string(current) == string(raw) {
-		return false, nil
-	}
-	if err != nil && !os.IsNotExist(err) {
-		return false, err
-	}
-	if err := os.WriteFile(path, raw, 0o644); err != nil {
-		return false, err
-	}
-	return true, nil
-}
-
-func fileExists(path string) bool {
-	_, err := os.Stat(path)
-	return err == nil
-}

internal/topdata/loadscreens_migrate.go

@@ -1,6 +0,0 @@
-package topdata
-
-func importLegacyLoadscreens(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-	return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "loadscreens", "loadscreens.2da", nil)
-}

internal/topdata/masterfeats_migrate.go

@@ -1,237 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-	"strconv"
-)
-
-func importLegacyMasterfeats(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "feat", "masterfeats")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "masterfeats")
-	targetPath := filepath.Join(targetDir, "base.json")
-	legacyPlainPath := filepath.Join(targetDir, "masterfeats.json")
-	if _, err := os.Stat(targetPath); err == nil {
-		obj, err := loadJSONObject(targetPath)
-		if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
-			return 0, nil
-		}
-	}
-
-	collected, err := importLegacyMasterfeatsDataset(legacyDir)
-	if err != nil {
-		return 0, err
-	}
-	referenceIDs, err := collectReferenceLockIDs(filepath.Join(referenceBuilderDir, "data"))
-	if err != nil {
-		return 0, err
-	}
-
-	rows := make([]any, 0, len(collected.Rows))
-	for _, row := range collected.Rows {
-		copyRow, ok := deepCopyValue(row).(map[string]any)
-		if !ok {
-			continue
-		}
-		if legacyTLK != nil {
-			if _, _, err := inlineLegacyTLKValue(copyRow, legacyTLK); err != nil {
-				return 0, err
-			}
-		}
-		copyRow = rewriteImportedIDRefs(copyRow, referenceIDs)
-		rows = append(rows, copyRow)
-	}
-
-	if err := os.MkdirAll(targetDir, 0o755); err != nil {
-		return 0, err
-	}
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{
-			path: targetPath,
-			obj: map[string]any{
-				"output":  collected.Dataset.OutputName,
-				"columns": stringSliceToAny(collected.Columns),
-				"rows":    rows,
-			},
-		},
-		{
-			path: filepath.Join(targetDir, "lock.json"),
-			obj:  anyMapInt(collected.LockData),
-		},
-	}
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	updatedFiles := len(writes)
-	if err := os.Remove(legacyPlainPath); err == nil {
-		updatedFiles++
-	} else if err != nil && !os.IsNotExist(err) {
-		return 0, err
-	}
-	return updatedFiles, nil
-}
-
-func importLegacyMasterfeatsDataset(legacyDir string) (nativeCollectedDataset, error) {
-	dataset := nativeDataset{
-		Name:     "masterfeats",
-		BasePath: filepath.Join(legacyDir, "masterfeats.json"),
-		LockPath: filepath.Join(legacyDir, "lock.json"),
-		Spec:     specForDataset("masterfeats"),
-	}
-	tableData, err := loadJSONObject(dataset.BasePath)
-	if err != nil {
-		return nativeCollectedDataset{}, err
-	}
-	columns, err := parseColumns(tableData, dataset.Name)
-	if err != nil {
-		return nativeCollectedDataset{}, err
-	}
-	rawRows, ok := tableData["rows"].([]any)
-	if !ok {
-		return nativeCollectedDataset{}, nil
-	}
-	lockData, err := loadLockfile(dataset.LockPath)
-	if err != nil {
-		return nativeCollectedDataset{}, err
-	}
-	usedIDs := map[int]struct{}{}
-	for _, rowID := range lockData {
-		usedIDs[rowID] = struct{}{}
-	}
-
-	rows := make([]map[string]any, 0, len(rawRows))
-	explicitID := make([]bool, 0, len(rawRows))
-	for index, raw := range rawRows {
-		rowObj, ok := raw.(map[string]any)
-		if !ok {
-			continue
-		}
-		rowID := index
-		if value, ok := rowObj["id"]; ok {
-			parsed, err := asInt(value)
-			if err != nil {
-				return nativeCollectedDataset{}, err
-			}
-			rowID = parsed
-		}
-		row := map[string]any{"id": rowID}
-		for _, column := range columns {
-			row[column] = nullValue
-		}
-		for key, value := range rowObj {
-			switch key {
-			case "id":
-			case "key":
-				if keyText, ok := value.(string); ok && keyText != "" {
-					row["key"] = keyText
-				}
-			default:
-				columnName, ok := canonicalColumn(columns, key)
-				if !ok {
-					continue
-				}
-				row[columnName] = value
-			}
-		}
-		rows = append(rows, row)
-		_, hasExplicitID := rowObj["id"]
-		explicitID = append(explicitID, hasExplicitID)
-		if hasExplicitID {
-			usedIDs[rowID] = struct{}{}
-		}
-	}
-
-	nextID := nextAvailableID(usedIDs)
-	lockModified := false
-	for index, row := range rows {
-		key, hasKey := row["key"].(string)
-		if hasKey && key != "" {
-			if lockedID, ok := lockData[key]; ok {
-				row["id"] = lockedID
-				continue
-			}
-			if explicitID[index] {
-				lockData[key] = row["id"].(int)
-			} else {
-				row["id"] = nextID
-				lockData[key] = nextID
-				usedIDs[nextID] = struct{}{}
-				nextID = nextAvailableID(usedIDs)
-			}
-			lockModified = true
-			continue
-		}
-		if !explicitID[index] {
-			row["id"] = index
-		}
-	}
-	if lockModified {
-		_ = lockModified
-	}
-
-	return nativeCollectedDataset{
-		Dataset: nativeDataset{
-			Name:       dataset.Name,
-			OutputName: "masterfeats.2da",
-			Columns:    columns,
-			Spec:       dataset.Spec,
-		},
-		Columns:  columns,
-		Rows:     rows,
-		LockData: lockData,
-	}, nil
-}
-
-func rewriteImportedIDRefs(value any, ids map[string]int) map[string]any {
-	row, ok := rewriteImportedIDRefsValue(value, ids).(map[string]any)
-	if !ok {
-		return map[string]any{}
-	}
-	return row
-}
-
-func rewriteImportedIDRefsValue(value any, ids map[string]int) any {
-	switch typed := value.(type) {
-	case map[string]any:
-		if len(typed) == 1 {
-			if rawID, ok := typed["id"]; ok {
-				if key, ok := rawID.(string); ok {
-					if resolved, ok := ids[key]; ok {
-						return strconv.Itoa(resolved)
-					}
-				}
-			}
-		}
-		out := make(map[string]any, len(typed))
-		for key, child := range typed {
-			out[key] = rewriteImportedIDRefsValue(child, ids)
-		}
-		return out
-	case []any:
-		out := make([]any, 0, len(typed))
-		for _, child := range typed {
-			out = append(out, rewriteImportedIDRefsValue(child, ids))
-		}
-		return out
-	default:
-		return value
-	}
-}

internal/topdata/migrate.go

@@ -1,415 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"fmt"
-	"os"
-	"path/filepath"
-	"strings"
-
-	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
-)
-
-type NormalizeResult struct {
-	StatePath           string
-	ScannedFiles        int
-	UpdatedFiles        int
-	InlineValues        int
-	RemainingFiles      int
-	RemainingLegacyRefs int
-}
-
-func NormalizeProject(p *project.Project) (NormalizeResult, error) {
-	if !p.HasTopData() {
-		return NormalizeResult{}, fmt.Errorf("topdata is not configured for this project")
-	}
-
-	sourceDir := p.TopDataSourceDir()
-	dataDir := filepath.Join(sourceDir, "data")
-	migrationRoot := resolveMigrationInputRoot(sourceDir, p.TopDataReferenceBuilderDir())
-	legacyTLK, err := loadLegacyTLK(filepath.Join(sourceDir, "tlk"))
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	if migrationRoot != "" {
-		migrationTLK, err := loadLegacyTLK(filepath.Join(migrationRoot, "tlk"))
-		if err != nil {
-			return NormalizeResult{}, err
-		}
-		legacyTLK = mergeLegacyTLK(migrationTLK, legacyTLK)
-	}
-	baseDialogImported, err := importLegacyBaseDialog(migrationRoot, sourceDir)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-
-	itempropsImported, err := importLegacyItempropsRegistry(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	masterfeatsImported, err := importLegacyMasterfeats(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	featImported, err := importLegacyFeat(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	creaturespeedImported, err := importLegacyCreaturespeed(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	armorImported, err := importLegacyArmor(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	baseitemsImported, err := importLegacyBaseitems(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	damagetypesImported, err := importLegacyDamagetypes(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	skyboxesImported, err := importLegacySkyboxes(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	vfxPersistentImported, err := importLegacyVFXPersistent(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	loadscreensImported, err := importLegacyLoadscreens(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	progfxImported, err := importLegacyProgfx(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	genericdoorsImported, err := importLegacyGenericdoors(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	cloakmodelImported, err := importLegacyCloakmodel(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	repadjustImported, err := importLegacyRepadjust(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	wingmodelImported, err := importLegacyWingmodel(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	tailmodelImported, err := importLegacyTailmodel(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	doortypesImported, err := importLegacyDoortypes(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	visualeffectsImported, err := importLegacyVisualeffects(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	appearanceImported, err := importLegacyAppearance(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	placeablesImported, err := importLegacyPlaceables(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	rulesetImported, err := importLegacyRuleset(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	skillsImported, err := importLegacySkills(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	spellsImported, err := importLegacySpells(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	racialtypesImported, err := importLegacyRacialtypes(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	classesImported, err := importLegacyClasses(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-	portraitsImported, err := importLegacyPortraits(migrationRoot, dataDir, legacyTLK)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-
-	paths, err := collectDataJSONPaths(dataDir)
-	if err != nil {
-		return NormalizeResult{}, err
-	}
-
-	result := NormalizeResult{
-		StatePath:    filepath.Join(sourceDir, tlkStateFile),
-		ScannedFiles: len(paths),
-		UpdatedFiles: baseDialogImported + itempropsImported + masterfeatsImported + featImported + creaturespeedImported + armorImported + baseitemsImported + damagetypesImported + skyboxesImported + vfxPersistentImported + loadscreensImported + progfxImported + genericdoorsImported + cloakmodelImported + repadjustImported + wingmodelImported + tailmodelImported + doortypesImported + visualeffectsImported + appearanceImported + placeablesImported + rulesetImported + skillsImported + spellsImported + racialtypesImported + classesImported + portraitsImported,
-	}
-	state, err := loadTLKState(result.StatePath)
-	if err != nil {
-		return result, err
-	}
-	if state.Entries == nil {
-		state.Entries = map[string]tlkStateMapping{}
-	}
-	if legacyTLK != nil && state.Language == "" {
-		state.Language = legacyTLK.Language
-	}
-	if legacyTLK != nil {
-		for key, id := range legacyTLK.Lock {
-			if _, ok := state.Entries[key]; !ok {
-				state.Entries[key] = tlkStateMapping{ID: id}
-			}
-		}
-	}
-
-	if legacyTLK != nil {
-		for _, path := range paths {
-			updated, count, err := inlineLegacyTLKFile(path, legacyTLK)
-			if err != nil {
-				return result, err
-			}
-			if updated {
-				result.UpdatedFiles++
-				result.InlineValues += count
-			}
-		}
-	}
-
-	if err := saveTLKState(result.StatePath, state); err != nil {
-		return result, err
-	}
-	if err := removeLegacyTLKAuthoredInput(filepath.Join(sourceDir, "tlk")); err != nil {
-		return result, err
-	}
-	remainingFiles, remainingRefs, err := countLegacyTLKRefs(paths)
-	if err != nil {
-		return result, err
-	}
-	result.RemainingFiles = remainingFiles
-	result.RemainingLegacyRefs = remainingRefs
-	return result, nil
-}
-
-func mergeLegacyTLK(base, override *legacyTLKData) *legacyTLKData {
-	if base == nil && override == nil {
-		return nil
-	}
-	merged := &legacyTLKData{
-		Language: "en",
-		Entries:  map[string]tlkEntryData{},
-		Lock:     map[string]int{},
-	}
-	if base != nil {
-		if base.Language != "" {
-			merged.Language = base.Language
-		}
-		for key, entry := range base.Entries {
-			merged.Entries[key] = entry
-		}
-		for key, id := range base.Lock {
-			merged.Lock[key] = id
-		}
-	}
-	if override != nil {
-		if override.Language != "" {
-			merged.Language = override.Language
-		}
-		for key, entry := range override.Entries {
-			merged.Entries[key] = entry
-		}
-		for key, id := range override.Lock {
-			merged.Lock[key] = id
-		}
-	}
-	return merged
-}
-
-func removeLegacyTLKAuthoredInput(tlkDir string) error {
-	modulesDir := filepath.Join(tlkDir, "modules")
-	if err := os.RemoveAll(modulesDir); err != nil {
-		return fmt.Errorf("remove %s: %w", modulesDir, err)
-	}
-	lockPath := filepath.Join(tlkDir, "lock.json")
-	if err := os.Remove(lockPath); err != nil && !os.IsNotExist(err) {
-		return fmt.Errorf("remove %s: %w", lockPath, err)
-	}
-	gitkeepPath := filepath.Join(tlkDir, ".gitkeep")
-	if err := os.Remove(gitkeepPath); err != nil && !os.IsNotExist(err) {
-		return fmt.Errorf("remove %s: %w", gitkeepPath, err)
-	}
-	if err := os.Remove(tlkDir); err != nil && !os.IsNotExist(err) {
-		if !strings.Contains(err.Error(), "directory not empty") {
-			return fmt.Errorf("remove %s: %w", tlkDir, err)
-		}
-	}
-	return nil
-}
-
-func resolveMigrationInputRoot(sourceDir, referenceBuilderDir string) string {
-	if strings.TrimSpace(referenceBuilderDir) != "" {
-		return referenceBuilderDir
-	}
-	snapshotRoot := filepath.Join(sourceDir, "migration_snapshot")
-	if info, err := os.Stat(snapshotRoot); err == nil && info.IsDir() {
-		return snapshotRoot
-	}
-	return ""
-}
-
-func inlineLegacyTLKFile(path string, legacy *legacyTLKData) (bool, int, error) {
-	obj, err := loadJSONObject(path)
-	if err != nil {
-		return false, 0, err
-	}
-	updated, count, err := inlineLegacyTLKValue(obj, legacy)
-	if err != nil {
-		return false, 0, err
-	}
-	if !updated {
-		return false, 0, nil
-	}
-	raw, err := json.MarshalIndent(obj, "", "  ")
-	if err != nil {
-		return false, 0, err
-	}
-	raw = append(raw, '\n')
-	if err := os.WriteFile(path, raw, 0o644); err != nil {
-		return false, 0, err
-	}
-	return true, count, nil
-}
-
-func inlineLegacyTLKValue(value any, legacy *legacyTLKData) (bool, int, error) {
-	switch typed := value.(type) {
-	case map[string]any:
-		if rawTLK, ok := typed["tlk"]; ok {
-			key, ok := rawTLK.(string)
-			if ok {
-				entry, ok := legacy.Entries[key]
-				if !ok {
-					return false, 0, nil
-				}
-				payload := map[string]any{
-					"key":  key,
-					"text": entry.Text,
-				}
-				if entry.SoundResRef != "" {
-					payload["sound_resref"] = entry.SoundResRef
-				}
-				if entry.SoundLength != 0 {
-					payload["sound_length"] = entry.SoundLength
-				}
-				typed["tlk"] = payload
-				return true, 1, nil
-			}
-		}
-
-		updated := false
-		total := 0
-		for key, child := range typed {
-			childUpdated, childCount, err := inlineLegacyTLKValue(child, legacy)
-			if err != nil {
-				return false, 0, err
-			}
-			if childUpdated {
-				typed[key] = child
-				updated = true
-				total += childCount
-			}
-		}
-		return updated, total, nil
-	case []any:
-		updated := false
-		total := 0
-		for index, child := range typed {
-			childUpdated, childCount, err := inlineLegacyTLKValue(child, legacy)
-			if err != nil {
-				return false, 0, err
-			}
-			if childUpdated {
-				typed[index] = child
-				updated = true
-				total += childCount
-			}
-		}
-		return updated, total, nil
-	default:
-		return false, 0, nil
-	}
-}
-
-func collectDataJSONPaths(dataDir string) ([]string, error) {
-	paths := make([]string, 0)
-	err := filepath.WalkDir(dataDir, func(path string, d os.DirEntry, err error) error {
-		if err != nil {
-			return err
-		}
-		if d.IsDir() {
-			return nil
-		}
-		if strings.HasSuffix(strings.ToLower(d.Name()), ".json") && d.Name() != "lock.json" {
-			paths = append(paths, path)
-		}
-		return nil
-	})
-	if err != nil {
-		return nil, err
-	}
-	return paths, nil
-}
-
-func countLegacyTLKRefs(paths []string) (int, int, error) {
-	files := 0
-	refs := 0
-	for _, path := range paths {
-		obj, err := loadJSONObject(path)
-		if err != nil {
-			return 0, 0, err
-		}
-		count := countLegacyTLKRefsInValue(obj)
-		if count == 0 {
-			continue
-		}
-		files++
-		refs += count
-	}
-	return files, refs, nil
-}
-
-func countLegacyTLKRefsInValue(value any) int {
-	switch typed := value.(type) {
-	case map[string]any:
-		total := 0
-		if rawTLK, ok := typed["tlk"]; ok {
-			if _, ok := rawTLK.(string); ok {
-				total++
-			}
-		}
-		for _, child := range typed {
-			total += countLegacyTLKRefsInValue(child)
-		}
-		return total
-	case []any:
-		total := 0
-		for _, child := range typed {
-			total += countLegacyTLKRefsInValue(child)
-		}
-		return total
-	default:
-		return 0
-	}
-}

internal/topdata/native.go

@@ -410,6 +410,9 @@ func buildNativeUnchecked(p *project.Project, opts NativeBuildOptions, progress
 	if err != nil {
 		return BuildResult{}, err
 	}
+	if err := loadStandaloneTLKStrings(sourceDir, compiler); err != nil {
+		return BuildResult{}, err
+	}
 
 	output2DA := compiled2DAOutputDir(p)
 	outputTLK := compiledTLKOutputDir(p)
@@ -2317,6 +2320,18 @@ func (c *featGeneratedContext) familyHasExistingRows(familyKey string) bool {
 	return false
 }
 
+// skillsDatasetName returns the dataset the skill_focus family sources its
+// children from. The skills dataset location is data-driven via that spec;
+// "skills" is only the fallback when no spec is loaded.
+func (c *featGeneratedContext) skillsDatasetName() string {
+	for key, spec := range c.familySpecs {
+		if normalizeKeyIdentity(key) == "skillfocus" && spec.ChildSource.Dataset != "" {
+			return spec.ChildSource.Dataset
+		}
+	}
+	return "skills"
+}
+
 func (c *featGeneratedContext) datasetRows(name string) (map[string]map[string]any, error) {
 	if rows, ok := c.rowsByDataset[name]; ok {
 		return rows, nil
@@ -2555,7 +2570,10 @@ func buildFamilyExpansionGeneratedModule(path string, obj map[string]any, ctx *f
 		if !include {
 			continue
 		}
-		slug := strings.TrimPrefix(sourceKey, spec.ChildSource.Dataset+":")
+		slug := sourceKey
+		if idx := strings.Index(slug, ":"); idx >= 0 {
+			slug = slug[idx+1:]
+		}
 		featKey, rowID, hasID, err := ctx.resolveGeneratedFeatIdentity(spec, slug, row)
 		if err != nil {
 			return nil, fmt.Errorf("generated feat file %s: %w", path, err)
@@ -2655,7 +2673,7 @@ func buildRacialtypesSkillAffinityModule(ctx *featGeneratedContext) (map[string]
 	if len(grants) == 0 {
 		return nil, nil
 	}
-	skillRows, err := ctx.datasetRows("skills")
+	skillRows, err := ctx.datasetRows(ctx.skillsDatasetName())
 	if err != nil {
 		return nil, err
 	}
@@ -2854,7 +2872,7 @@ func (c *featGeneratedContext) displayNameForGeneratedSource(dataset string, row
 		return text
 	}
 	switch dataset {
-	case "skills":
+	case c.skillsDatasetName():
 		return displayNameForSkill(row)
 	case "baseitems":
 		return displayNameForBaseitem(row)
@@ -3457,10 +3475,20 @@ func legacyFamilyChildAliases(familyKey, sourceSlug string) []string {
 		aliases = append(aliases, "craftweapon")
 	case "craftwoodworking":
 		aliases = append(aliases, "crafttrap")
+	case "diplomacy":
+		aliases = append(aliases, "influence", "persuade")
 	case "disabledevice":
 		aliases = append(aliases, "disabletrap")
 	case "influence":
 		aliases = append(aliases, "persuade")
+	case "intimidate":
+		aliases = append(aliases, "taunt")
+	case "investigation":
+		aliases = append(aliases, "search")
+	case "medicine":
+		aliases = append(aliases, "heal")
+	case "security":
+		aliases = append(aliases, "openlock")
 	case "sleightofhand":
 		aliases = append(aliases, "pickpocket")
 	}
@@ -6617,81 +6645,6 @@ func marshalOrderedLockfile(keys []string, lockData map[string]int, formatting j
 	return buffer.Bytes(), nil
 }
 
-func loadLegacyTLK(root string) (*legacyTLKData, error) {
-	info, err := os.Stat(root)
-	if err != nil {
-		if os.IsNotExist(err) {
-			return nil, nil
-		}
-		return nil, err
-	}
-	if !info.IsDir() {
-		return nil, fmt.Errorf("legacy tlk root must be a directory: %s", root)
-	}
-
-	result := &legacyTLKData{
-		Language: "en",
-		Entries:  map[string]tlkEntryData{},
-		Lock:     map[string]int{},
-	}
-
-	lockPath := filepath.Join(root, "lock.json")
-	lockData, err := loadLockfile(lockPath)
-	if err != nil && !os.IsNotExist(err) {
-		return nil, err
-	}
-	for key, id := range lockData {
-		result.Lock[key] = id
-	}
-
-	basePath := filepath.Join(root, "base.json")
-	if _, err := os.Stat(basePath); err == nil {
-		base, err := loadJSONObject(basePath)
-		if err != nil {
-			return nil, err
-		}
-		if language, ok := base["language"].(string); ok && language != "" {
-			result.Language = language
-		}
-		if entries, ok := base["entries"].(map[string]any); ok {
-			normalized, err := normalizeLegacyTLKEntries(entries)
-			if err != nil {
-				return nil, err
-			}
-			for key, entry := range normalized {
-				result.Entries[key] = entry
-			}
-		}
-	}
-
-	modulesDir := filepath.Join(root, "modules")
-	modulePaths, err := collectModulePaths(modulesDir)
-	if err != nil {
-		if os.IsNotExist(err) {
-			return result, nil
-		}
-		return nil, err
-	}
-	for _, path := range modulePaths {
-		obj, err := loadJSONObject(path)
-		if err != nil {
-			return nil, err
-		}
-		entries, ok := obj["entries"].(map[string]any)
-		if !ok {
-			continue
-		}
-		normalized, err := normalizeLegacyTLKEntries(entries)
-		if err != nil {
-			return nil, fmt.Errorf("%s: %w", path, err)
-		}
-		for key, entry := range normalized {
-			result.Entries[key] = entry
-		}
-	}
-	return result, nil
-}
-
 func normalizeLegacyTLKEntries(entries map[string]any) (map[string]tlkEntryData, error) {
 	normalized := map[string]tlkEntryData{}
 	for _, key := range sortedKeys(entries) {

internal/topdata/placeables_migrate.go

@@ -1,6 +0,0 @@
-package topdata
-
-func importLegacyPlaceables(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-	return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "placeables", "placeables.2da", nil)
-}

internal/topdata/portraits_migrate.go

@@ -1,8 +0,0 @@
-package topdata
-
-func importLegacyPortraits(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	if legacyTLK == nil {
-		return 0, nil
-	}
-	return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "portraits", "portraits.2da", nil)
-}

internal/topdata/progfx_migrate.go

@@ -1,6 +0,0 @@
-package topdata
-
-func importLegacyProgfx(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-	return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "progfx", "progfx.2da", nil)
-}

internal/topdata/racialtypes_migrate.go

@@ -1,282 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"fmt"
-	"os"
-	"path/filepath"
-	"slices"
-	"strings"
-)
-
-type legacyRacialtypesFeatTable struct {
-	Output string
-	Rows   []map[string]any
-}
-
-func importLegacyRacialtypes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	if strings.TrimSpace(referenceBuilderDir) == "" {
-		return 0, nil
-	}
-
-	legacyCoreDir := filepath.Join(referenceBuilderDir, "data", "racialtypes", "core")
-	legacyFeatsDir := filepath.Join(referenceBuilderDir, "data", "racialtypes", "feats")
-	if _, err := os.Stat(legacyCoreDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-	if _, err := os.Stat(legacyFeatsDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetRoot := filepath.Join(dataDir, "racialtypes", "registry")
-	targetBasePath := filepath.Join(targetRoot, "base.json")
-	targetLockPath := filepath.Join(targetRoot, "lock.json")
-	targetRacesDir := filepath.Join(targetRoot, "races")
-	if fileExists(targetBasePath) && fileExists(targetLockPath) {
-		entries, err := os.ReadDir(targetRacesDir)
-		if err == nil {
-			raceFiles := 0
-			for _, entry := range entries {
-				if !entry.IsDir() && strings.HasSuffix(strings.ToLower(entry.Name()), ".json") {
-					raceFiles++
-				}
-			}
-			if raceFiles == 16 {
-				return 0, nil
-			}
-		}
-	}
-
-	baseColumns, baseRows, lockData, raceRows, err := collectLegacyRacialtypesRegistry(legacyCoreDir, legacyFeatsDir, legacyTLK)
-	if err != nil {
-		return 0, err
-	}
-
-	if err := removePathIfExists(filepath.Join(dataDir, "racialtypes", "core")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(dataDir, "racialtypes", "feats")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(targetRoot, "base_rows.json")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(targetRacesDir); err != nil {
-		return 0, err
-	}
-	if err := os.MkdirAll(targetRacesDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  any
-	}{
-		{
-			path: targetBasePath,
-			obj: map[string]any{
-				"columns": stringSliceToAny(baseColumns),
-				"rows":    rowsToAny(baseRows),
-			},
-		},
-		{
-			path: targetLockPath,
-			obj:  anyMapInt(lockData),
-		},
-	}
-
-	raceKeys := make([]string, 0, len(raceRows))
-	for key := range raceRows {
-		raceKeys = append(raceKeys, key)
-	}
-	slices.Sort(raceKeys)
-	for _, key := range raceKeys {
-		fileName := strings.TrimPrefix(key, "racialtypes:") + ".json"
-		writes = append(writes, struct {
-			path string
-			obj  any
-		}{
-			path: filepath.Join(targetRacesDir, fileName),
-			obj:  raceRows[key],
-		})
-	}
-
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}
-
-func collectLegacyRacialtypesRegistry(coreDir, featsDir string, legacyTLK *legacyTLKData) ([]string, []map[string]any, map[string]int, map[string]map[string]any, error) {
-	coreCollected, err := collectBaseDataset(nativeDataset{
-		Name:       "racialtypes",
-		BasePath:   filepath.Join(coreDir, "base.json"),
-		LockPath:   filepath.Join(coreDir, "lock.json"),
-		ModulesDir: filepath.Join(coreDir, "modules"),
-		Spec:       specForDataset("racialtypes"),
-	})
-	if err != nil {
-		return nil, nil, nil, nil, err
-	}
-
-	featTables, err := loadLegacyRacialtypesFeatTables(featsDir)
-	if err != nil {
-		return nil, nil, nil, nil, err
-	}
-
-	baseRows := make([]map[string]any, 0)
-	raceRows := map[string]map[string]any{}
-	lockData := map[string]int{}
-	for key, id := range coreCollected.LockData {
-		if strings.HasPrefix(key, "racialtypes:") {
-			lockData[key] = id
-		}
-	}
-
-	for _, collectedRow := range coreCollected.Rows {
-		row, ok := deepCopyValue(collectedRow).(map[string]any)
-		if !ok {
-			continue
-		}
-		if legacyTLK != nil {
-			if _, _, err := inlineLegacyTLKValue(row, legacyTLK); err != nil {
-				return nil, nil, nil, nil, err
-			}
-		}
-		key, _ := row["key"].(string)
-		if !strings.HasPrefix(key, "racialtypes:") {
-			assignRacialtypesBaseRowKey(row)
-			if key, _ := row["key"].(string); key != "" {
-				rowID, err := asInt(row["id"])
-				if err != nil {
-					return nil, nil, nil, nil, err
-				}
-				lockData[key] = rowID
-			}
-			baseRows = append(baseRows, row)
-			continue
-		}
-
-		delete(row, "id")
-		delete(row, "key")
-
-		raceFile := map[string]any{
-			"key":  key,
-			"core": row,
-		}
-		if tableKey := extractRacialtypesFeatTableKey(row["FeatsTable"]); tableKey != "" {
-			table, ok := featTables[tableKey]
-			if !ok {
-				return nil, nil, nil, nil, fmt.Errorf("%s: unknown feat table %s", key, tableKey)
-			}
-			delete(row, "FeatsTable")
-			raceFile["feat_output"] = table.Output
-			raceFile["feats"] = rowsToAny(table.Rows)
-		}
-		raceRows[key] = raceFile
-	}
-
-	slices.SortFunc(baseRows, func(a, b map[string]any) int {
-		left, _ := asInt(a["id"])
-		right, _ := asInt(b["id"])
-		return left - right
-	})
-	return coreCollected.Columns, baseRows, lockData, raceRows, nil
-}
-
-func loadLegacyRacialtypesFeatTables(featsDir string) (map[string]legacyRacialtypesFeatTable, error) {
-	entries, err := os.ReadDir(featsDir)
-	if err != nil {
-		return nil, err
-	}
-
-	tables := map[string]legacyRacialtypesFeatTable{}
-	for _, entry := range entries {
-		if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" {
-			continue
-		}
-		obj, err := loadJSONObject(filepath.Join(featsDir, entry.Name()))
-		if err != nil {
-			return nil, err
-		}
-		tableKey, _ := obj["key"].(string)
-		outputName, _ := obj["output"].(string)
-		rawRows, ok := obj["rows"].([]any)
-		if tableKey == "" || outputName == "" || !ok {
-			return nil, fmt.Errorf("%s: racialtypes feat table must define key, output, and rows", entry.Name())
-		}
-		rows := make([]map[string]any, 0, len(rawRows))
-		for index, raw := range rawRows {
-			row, ok := raw.(map[string]any)
-			if !ok {
-				return nil, fmt.Errorf("%s: row %d must be an object", entry.Name(), index)
-			}
-			rows = append(rows, row)
-		}
-		tables[tableKey] = legacyRacialtypesFeatTable{
-			Output: outputName,
-			Rows:   rows,
-		}
-	}
-	return tables, nil
-}
-
-func extractRacialtypesFeatTableKey(value any) string {
-	obj, ok := value.(map[string]any)
-	if !ok {
-		return ""
-	}
-	tableKey, _ := obj["table"].(string)
-	return tableKey
-}
-
-func assignRacialtypesBaseRowKey(row map[string]any) {
-	name, _ := row["Name"].(string)
-	if strings.TrimSpace(name) == "" || name == nullValue {
-		delete(row, "key")
-		return
-	}
-	label, _ := row["Label"].(string)
-	keySuffix := normalizeRacialtypesKeySuffix(label)
-	if keySuffix == "" {
-		delete(row, "key")
-		return
-	}
-	row["key"] = "racialtypes:" + keySuffix
-}
-
-func normalizeRacialtypesKeySuffix(label string) string {
-	normalized := strings.ToLower(strings.TrimSpace(label))
-	if normalized == "" {
-		return ""
-	}
-	var b strings.Builder
-	lastUnderscore := false
-	for _, ch := range normalized {
-		isAlphaNum := (ch >= 'a' && ch <= 'z') || (ch >= '0' && ch <= '9')
-		if isAlphaNum {
-			b.WriteRune(ch)
-			lastUnderscore = false
-			continue
-		}
-		if lastUnderscore {
-			continue
-		}
-		b.WriteByte('_')
-		lastUnderscore = true
-	}
-	return strings.Trim(b.String(), "_")
-}

internal/topdata/repadjust_migrate.go

@@ -1,63 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacyRepadjust(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "repadjust")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "repadjust")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	targetLockPath := filepath.Join(targetDir, "lock.json")
-	if fileExists(targetBasePath) && fileExists(targetLockPath) {
-		obj, err := loadJSONObject(targetBasePath)
-		if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	baseObj["output"] = "repadjust.2da"
-
-	lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-
-	if err := os.MkdirAll(targetDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: targetBasePath, obj: baseObj},
-		{path: targetLockPath, obj: lockObj},
-	}
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}

internal/topdata/ruleset_migrate.go

@@ -1,171 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"fmt"
-	"os"
-	"path/filepath"
-	"sort"
-)
-
-func importLegacyRuleset(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "ruleset")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "ruleset")
-	targetPath := filepath.Join(targetDir, "ruleset.json")
-	if _, err := os.Stat(targetPath); err == nil {
-		obj, err := loadJSONObject(targetPath)
-		if err == nil && countLegacyTLKRefsInValue(obj) == 0 {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	rows, err := mergeRulesetRows(baseObj, filepath.Join(legacyDir, "modules"))
-	if err != nil {
-		return 0, err
-	}
-
-	if err := os.MkdirAll(targetDir, 0o755); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(targetDir, "base.json")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(targetDir, "lock.json")); err != nil {
-		return 0, err
-	}
-	if err := removePathIfExists(filepath.Join(targetDir, "modules")); err != nil {
-		return 0, err
-	}
-
-	out := map[string]any{
-		"output":  "ruleset.2da",
-		"columns": baseObj["columns"],
-		"rows":    rows,
-	}
-	raw, err := json.MarshalIndent(out, "", "  ")
-	if err != nil {
-		return 0, err
-	}
-	raw = append(raw, '\n')
-	if err := os.WriteFile(targetPath, raw, 0o644); err != nil {
-		return 0, err
-	}
-	return 1, nil
-}
-
-func mergeRulesetRows(baseObj map[string]any, modulesDir string) ([]any, error) {
-	rawRows, ok := baseObj["rows"].([]any)
-	if !ok {
-		return nil, nil
-	}
-
-	rows := make([]map[string]any, 0, len(rawRows))
-	byLabel := map[string]map[string]any{}
-	for _, raw := range rawRows {
-		row, ok := deepCopyValue(raw).(map[string]any)
-		if !ok {
-			continue
-		}
-		delete(row, "key")
-		if rawID, ok := row["id"]; ok {
-			id, err := asInt(rawID)
-			if err != nil {
-				return nil, err
-			}
-			row["id"] = id
-		}
-		if label, ok := row["Label"].(string); ok && label != "" && label != "****" {
-			if _, exists := byLabel[label]; exists {
-				return nil, fmt.Errorf("duplicate ruleset label %q", label)
-			}
-			byLabel[label] = row
-		}
-		rows = append(rows, row)
-	}
-
-	modulePaths, err := collectModulePaths(modulesDir)
-	if err != nil {
-		return nil, err
-	}
-	for _, path := range modulePaths {
-		obj, err := loadJSONObject(path)
-		if err != nil {
-			return nil, err
-		}
-		overrideList, ok := obj["overrides"].([]any)
-		if !ok {
-			continue
-		}
-		for _, raw := range overrideList {
-			override, ok := raw.(map[string]any)
-			if !ok {
-				continue
-			}
-			matchObj, ok := override["match"].(map[string]any)
-			if !ok {
-				return nil, fmt.Errorf("%s: ruleset override is missing match object", path)
-			}
-			rawLabel, ok := matchObj["Label"]
-			if !ok {
-				return nil, fmt.Errorf("%s: ruleset override match is missing Label", path)
-			}
-			labels, err := normalizeRulesetMatchLabels(rawLabel)
-			if err != nil {
-				return nil, fmt.Errorf("%s: %w", path, err)
-			}
-			for _, label := range labels {
-				row, ok := byLabel[label]
-				if !ok {
-					return nil, fmt.Errorf("%s: ruleset override target label %q was not found", path, label)
-				}
-				for key, value := range override {
-					if key == "id" || key == "key" || key == "_tlk" || isMetadataField(key) || key == "match" {
-						continue
-					}
-					row[key] = deepCopyValue(value)
-				}
-			}
-		}
-	}
-
-	sort.Slice(rows, func(i, j int) bool {
-		return rows[i]["id"].(int) < rows[j]["id"].(int)
-	})
-	out := make([]any, 0, len(rows))
-	for _, row := range rows {
-		out = append(out, row)
-	}
-	return out, nil
-}
-
-func normalizeRulesetMatchLabels(raw any) ([]string, error) {
-	switch typed := raw.(type) {
-	case string:
-		return []string{typed}, nil
-	case []any:
-		out := make([]string, 0, len(typed))
-		for _, item := range typed {
-			label, ok := item.(string)
-			if !ok {
-				return nil, fmt.Errorf("ruleset match.Label entries must be strings")
-			}
-			out = append(out, label)
-		}
-		return out, nil
-	default:
-		return nil, fmt.Errorf("ruleset match.Label must be a string or string array")
-	}
-}

internal/topdata/skills_migrate.go

@@ -1,140 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacySkills(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "skills")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "skills")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	targetLockPath := filepath.Join(targetDir, "lock.json")
-	targetModulesDir := filepath.Join(targetDir, "modules")
-	moduleNames := []string{
-		"add_athletics.json",
-		"add_disguise.json",
-		"add_linguistics.json",
-		"add_perception.json",
-		"add_scriptcraft.json",
-		"add_sensemotive.json",
-		"add_stealth.json",
-		"add_survival.json",
-		"add_userope.json",
-		filepath.Join("craft", "add_craftalchemy.json"),
-		filepath.Join("craft", "add_craftcooking.json"),
-		filepath.Join("craft", "add_craftjewelry.json"),
-		filepath.Join("craft", "add_craftleatherworking.json"),
-		filepath.Join("craft", "add_craftstonework.json"),
-		filepath.Join("craft", "add_crafttextiles.json"),
-		filepath.Join("craft", "ovr_craftarmorsmithing.json"),
-		filepath.Join("craft", "ovr_crafttinkering.json"),
-		filepath.Join("craft", "ovr_craftweaponsmithing.json"),
-		filepath.Join("craft", "ovr_craftwoodworking.json"),
-		filepath.Join("knowledge", "add_knowledgearcana.json"),
-		filepath.Join("knowledge", "add_knowledgearchitecture.json"),
-		filepath.Join("knowledge", "add_knowledgedungeoneering.json"),
-		filepath.Join("knowledge", "add_knowledgegeography.json"),
-		filepath.Join("knowledge", "add_knowledgehistory.json"),
-		filepath.Join("knowledge", "add_knowledgelocal.json"),
-		filepath.Join("knowledge", "add_knowledgenature.json"),
-		filepath.Join("knowledge", "add_knowledgenobility.json"),
-		filepath.Join("knowledge", "add_knowledgeplanar.json"),
-		filepath.Join("knowledge", "add_knowledgereligion.json"),
-		"ovr_acrobatics.json",
-		"ovr_animalhandling.json",
-		"ovr_appraise.json",
-		"ovr_concentration.json",
-		"ovr_disabledevice.json",
-		"ovr_heal.json",
-		"ovr_hiddenskills.json",
-		"ovr_influence.json",
-		"ovr_openlock.json",
-		"ovr_parry.json",
-		"ovr_searchtowis.json",
-		"ovr_sleightofhand.json",
-		"ovr_taunttointimidate.json",
-		"rmv_removedskills.json",
-	}
-	targetModulePaths := make([]string, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
-	}
-	if fileExists(targetBasePath) && fileExists(targetLockPath) {
-		allModulesPresent := true
-		for _, path := range targetModulePaths {
-			if !fileExists(path) {
-				allModulesPresent = false
-				break
-			}
-		}
-		if allModulesPresent {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	baseObj["output"] = "skills.2da"
-	canonicalizeWikiMetadataDocument(baseObj)
-
-	lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-
-	moduleObjs := make([]map[string]any, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
-		if err != nil {
-			return 0, err
-		}
-		canonicalizeWikiMetadataDocument(obj)
-		moduleObjs = append(moduleObjs, obj)
-	}
-
-	if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: targetBasePath, obj: baseObj},
-		{path: targetLockPath, obj: lockObj},
-	}
-	for i, path := range targetModulePaths {
-		if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
-			return 0, err
-		}
-		writes = append(writes, struct {
-			path string
-			obj  map[string]any
-		}{path: path, obj: moduleObjs[i]})
-	}
-
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}

internal/topdata/skyboxes_migrate.go

@@ -1,6 +0,0 @@
-package topdata
-
-func importLegacySkyboxes(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-	return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "skyboxes", "skyboxes.2da", nil)
-}

internal/topdata/spells_migrate.go

@@ -1,103 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacySpells(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "spells")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "spells")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	targetLockPath := filepath.Join(targetDir, "lock.json")
-	targetModulesDir := filepath.Join(targetDir, "modules")
-	moduleNames := []string{
-		"specialattacks.json",
-		"ovr_fixduplicates.json",
-		filepath.Join("bardicmusic", "fascinate.json"),
-		filepath.Join("bardicmusic", "inspirecompetence.json"),
-		filepath.Join("bardicmusic", "inspirecourage.json"),
-		filepath.Join("bardicmusic", "inspiregreatness.json"),
-		filepath.Join("bardicmusic", "inspireheroics.json"),
-		filepath.Join("bardicmusic", "songoffreedom.json"),
-	}
-	targetModulePaths := make([]string, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		targetModulePaths = append(targetModulePaths, filepath.Join(targetModulesDir, name))
-	}
-	if fileExists(targetBasePath) && fileExists(targetLockPath) {
-		allModulesPresent := true
-		for _, path := range targetModulePaths {
-			if !fileExists(path) {
-				allModulesPresent = false
-				break
-			}
-		}
-		if allModulesPresent {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	baseObj["output"] = "spells.2da"
-
-	lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-
-	moduleObjs := make([]map[string]any, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
-		if err != nil {
-			return 0, err
-		}
-		moduleObjs = append(moduleObjs, obj)
-	}
-
-	if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: targetBasePath, obj: baseObj},
-		{path: targetLockPath, obj: lockObj},
-	}
-	for i, path := range targetModulePaths {
-		if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
-			return 0, err
-		}
-		writes = append(writes, struct {
-			path string
-			obj  map[string]any
-		}{path: path, obj: moduleObjs[i]})
-	}
-
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}

internal/topdata/tailmodel_migrate.go

@@ -1,86 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacyTailmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "tailmodel")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "tailmodel")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	targetLockPath := filepath.Join(targetDir, "lock.json")
-	targetModulesDir := filepath.Join(targetDir, "modules")
-	targetModulePaths := []string{
-		filepath.Join(targetModulesDir, "add.json"),
-		filepath.Join(targetModulesDir, "ovr_tailprefixes.json"),
-	}
-	if fileExists(targetBasePath) && fileExists(targetLockPath) {
-		allModulesPresent := true
-		for _, path := range targetModulePaths {
-			if !fileExists(path) {
-				allModulesPresent = false
-				break
-			}
-		}
-		if allModulesPresent {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	baseObj["output"] = "tailmodel.2da"
-
-	lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-
-	moduleNames := []string{"add.json", "ovr_tailprefixes.json"}
-	moduleObjs := make([]map[string]any, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
-		if err != nil {
-			return 0, err
-		}
-		moduleObjs = append(moduleObjs, obj)
-	}
-
-	if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: targetBasePath, obj: baseObj},
-		{path: targetLockPath, obj: lockObj},
-		{path: targetModulePaths[0], obj: moduleObjs[0]},
-		{path: targetModulePaths[1], obj: moduleObjs[1]},
-	}
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}

internal/topdata/tlk_native.go

@@ -13,6 +13,7 @@ import (
 	"strings"
 
 	"golang.org/x/text/encoding/charmap"
+	"gopkg.in/yaml.v3"
 )
 
 const (
@@ -74,6 +75,16 @@ type tlkEntryData struct {
 	SoundLength float32
 }
 
+type standaloneTLKDocument struct {
+	Schema     string `yaml:"schema"`
+	BaseStrref int    `yaml:"base_strref"`
+	Strings    []struct {
+		Key  string `yaml:"key"`
+		Text string `yaml:"text"`
+		ID   *int   `yaml:"id"`
+	} `yaml:"strings"`
+}
+
 type tlkStateDocument struct {
 	Version  int                        `json:"version"`
 	Language string                     `json:"language"`
@@ -98,6 +109,7 @@ type tlkCompiler struct {
 	active       map[string]tlkEntryData
 	activeKeys   map[string]struct{}
 	reservedByID map[int]string
+	pinnedByID   map[int]string
 	nextID       int
 }
 
@@ -147,6 +159,7 @@ func newTLKCompiler(sourceDir string, legacy *legacyTLKData) (*tlkCompiler, erro
 		active:       map[string]tlkEntryData{},
 		activeKeys:   map[string]struct{}{},
 		reservedByID: reserved,
+		pinnedByID:   map[int]string{},
 		nextID:       nextID,
 	}
 	if legacy != nil {
@@ -157,6 +170,42 @@ func newTLKCompiler(sourceDir string, legacy *legacyTLKData) (*tlkCompiler, erro
 	return compiler, nil
 }
 
+func loadStandaloneTLKStrings(sourceDir string, compiler *tlkCompiler) error {
+	path := filepath.Join(sourceDir, "tlk", "custom.tlk.yml")
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil
+		}
+		return fmt.Errorf("read %s: %w", path, err)
+	}
+	var document standaloneTLKDocument
+	if err := yaml.Unmarshal(raw, &document); err != nil {
+		return fmt.Errorf("parse %s: %w", path, err)
+	}
+	if document.Schema != "sow-topdata/tlk/v1" {
+		return fmt.Errorf("%s: unsupported schema %q", path, document.Schema)
+	}
+	if document.BaseStrref != customTLKBase {
+		return fmt.Errorf("%s: base_strref must be %d", path, customTLKBase)
+	}
+	seen := map[string]struct{}{}
+	for index, entry := range document.Strings {
+		entry.Key = strings.TrimSpace(entry.Key)
+		if entry.Key == "" || entry.Text == "" || entry.ID == nil {
+			return fmt.Errorf("%s: strings[%d] requires key, text, and id", path, index)
+		}
+		if _, ok := seen[entry.Key]; ok {
+			return fmt.Errorf("%s: duplicate string key %q", path, entry.Key)
+		}
+		seen[entry.Key] = struct{}{}
+		if err := compiler.registerInlineAtID(entry.Key, *entry.ID, tlkEntryData{Text: entry.Text}); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
 func loadBaseDialogData(path string) (*legacyTLKData, error) {
 	if _, err := os.Stat(path); err != nil {
 		if os.IsNotExist(err) {
@@ -329,6 +378,33 @@ func (c *tlkCompiler) registerInline(key string, entry tlkEntryData) (tlkCompile
 	}, nil
 }
 
+func (c *tlkCompiler) registerInlineAtID(key string, id int, entry tlkEntryData) error {
+	if id < 0 {
+		return fmt.Errorf("TLK key %q has negative id %d", key, id)
+	}
+	// The pin is authoritative over the per-machine .tlk_state.json cache: a
+	// stale mapping that dynamically grabbed this id on an older build must
+	// yield so the pinned key can take it. Only a genuine clash between two
+	// pins in custom.tlk.yml is an author error.
+	if owner, ok := c.pinnedByID[id]; ok && owner != key {
+		return fmt.Errorf("TLK id %d is pinned by both %q and %q", id, owner, key)
+	}
+	if mapping, ok := c.state.Entries[key]; ok && mapping.ID != id {
+		// This key held a different cached id; release it so the pin wins.
+		if c.reservedByID[mapping.ID] == key {
+			delete(c.reservedByID, mapping.ID)
+		}
+	}
+	if owner, ok := c.reservedByID[id]; ok && owner != key {
+		// Evict the stale owner; it gets a fresh id when next made active.
+		delete(c.state.Entries, owner)
+	}
+	c.pinnedByID[id] = key
+	c.state.Entries[key] = tlkStateMapping{ID: id}
+	c.reservedByID[id] = key
+	return c.markActive(key, entry)
+}
+
 func (c *tlkCompiler) customStrrefForKey(key string) int {
 	return customTLKBase + c.state.Entries[key].ID
 }
@@ -338,6 +414,7 @@ func (c *tlkCompiler) markActive(key string, entry tlkEntryData) error {
 	if !ok {
 		mapping = tlkStateMapping{ID: c.allocateID(), Retired: false}
 		c.state.Entries[key] = mapping
+		c.reservedByID[mapping.ID] = key
 	}
 	existing, ok := c.active[key]
 	if ok && existing != entry {
@@ -351,9 +428,13 @@ func (c *tlkCompiler) markActive(key string, entry tlkEntryData) error {
 }
 
 func (c *tlkCompiler) allocateID() int {
-	id := c.nextID
+	for {
-	c.nextID++
+		id := c.nextID
-	return id
+		c.nextID++
+		if _, reserved := c.reservedByID[id]; !reserved {
+			return id
+		}
+	}
 }
 
 func (c *tlkCompiler) finish(outputDir, tlkName string) (int, error) {
@@ -727,3 +808,36 @@ func sortedKeys[M ~map[string]V, V any](input M) []string {
 func almostEqualFloat32(a, b float32) bool {
 	return math.Abs(float64(a-b)) < 0.0001
 }
+func mergeLegacyTLK(base, override *legacyTLKData) *legacyTLKData {
+	if base == nil && override == nil {
+		return nil
+	}
+	merged := &legacyTLKData{
+		Language: "en",
+		Entries:  map[string]tlkEntryData{},
+		Lock:     map[string]int{},
+	}
+	if base != nil {
+		if base.Language != "" {
+			merged.Language = base.Language
+		}
+		for key, entry := range base.Entries {
+			merged.Entries[key] = entry
+		}
+		for key, id := range base.Lock {
+			merged.Lock[key] = id
+		}
+	}
+	if override != nil {
+		if override.Language != "" {
+			merged.Language = override.Language
+		}
+		for key, entry := range override.Entries {
+			merged.Entries[key] = entry
+		}
+		for key, id := range override.Lock {
+			merged.Lock[key] = id
+		}
+	}
+	return merged
+}

internal/topdata/top_package.go

@@ -2,6 +2,7 @@ package topdata
 
 import (
 	"bytes"
+	"encoding/json"
 	"errors"
 	"fmt"
 	"io"
@@ -13,6 +14,7 @@ import (
 	"time"
 
 	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/erf"
+	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/gff"
 	"git.westgate.pw/ShadowsOverWestgate/sow-tools/internal/project"
 )
 
@@ -215,10 +217,15 @@ func collectTopPackageResources(p *project.Project, compiled2DADir string) ([]er
 			if err != nil {
 				return err
 			}
-			if skipTopPackageAsset(rel) {
+			var resource erf.Resource
-				return nil
+			if strings.HasSuffix(strings.ToLower(rel), ".itp.json") {
+				resource, err = topPackageITPResourceFromJSON(path)
+			} else {
+				if skipTopPackageAsset(rel) {
+					return nil
+				}
+				resource, err = topPackageResourceFromPath(path)
 			}
-			resource, err := topPackageResourceFromPath(path)
 			if err != nil {
 				return err
 			}
@@ -251,6 +258,27 @@ func collectTopPackageResources(p *project.Project, compiled2DADir string) ([]er
 	return resources, assetFiles, nil
 }
 
+func topPackageITPResourceFromJSON(path string) (erf.Resource, error) {
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		return erf.Resource{}, fmt.Errorf("read %s: %w", path, err)
+	}
+	var document gff.Document
+	if err := json.Unmarshal(raw, &document); err != nil {
+		return erf.Resource{}, fmt.Errorf("parse %s: %w", path, err)
+	}
+	if document.FileType != "ITP " {
+		return erf.Resource{}, fmt.Errorf("%s: file_type must be ITP", path)
+	}
+	var payload bytes.Buffer
+	if err := gff.Write(&payload, document); err != nil {
+		return erf.Resource{}, fmt.Errorf("compile %s: %w", path, err)
+	}
+	resourceType, _ := erf.HAKResourceTypeForExtension("itp")
+	name := strings.TrimSuffix(filepath.Base(path), ".itp.json")
+	return erf.Resource{Name: strings.ToLower(name), Type: resourceType, Data: payload.Bytes(), Size: int64(payload.Len())}, nil
+}
+
 func shouldSkipTopDataSourceDir(path string, skipDirs map[string]struct{}) bool {
 	_, ok := skipDirs[filepath.Clean(path)]
 	return ok

internal/topdata/topdata.go

@@ -1867,8 +1867,10 @@ func validateGeneratedFeatFamilies(dataDir string, report *ValidationReport) {
 	}
 
 	required := []requiredFeatFamily{
-		{FamilyKey: "skillfocus", Dataset: "skills", Predicate: "accessible"},
+		// skill families: the source dataset is data-driven (family spec), only
-		{FamilyKey: "greaterskillfocus", Dataset: "skills", Predicate: "accessible"},
+		// the accessibility predicate is required
+		{FamilyKey: "skillfocus", Predicate: "accessible"},
+		{FamilyKey: "greaterskillfocus", Predicate: "accessible"},
 		{FamilyKey: "weaponfocus", Dataset: "baseitems", Column: "WeaponFocusFeat"},
 		{FamilyKey: "weaponspecialization", Dataset: "baseitems", Column: "WeaponSpecializationFeat"},
 		{FamilyKey: "improvedcritical", Dataset: "baseitems", Column: "WeaponImprovedCriticalFeat"},
@@ -1882,7 +1884,7 @@ func validateGeneratedFeatFamilies(dataDir string, report *ValidationReport) {
 		if !ok {
 			continue
 		}
-		if spec.ChildSource.Dataset != requirement.Dataset ||
+		if (requirement.Dataset != "" && spec.ChildSource.Dataset != requirement.Dataset) ||
 			spec.ChildSource.Column != requirement.Column ||
 			spec.ChildSource.Predicate != requirement.Predicate {
 			report.Diagnostics = append(report.Diagnostics, Diagnostic{
@@ -2081,7 +2083,10 @@ func validateGeneratedFeatFamilyCompleteness(path string, spec familyExpansionSp
 		}
 		if include {
 			if familyAllowlist {
-				slug := strings.TrimPrefix(sourceKey, spec.ChildSource.Dataset+":")
+				slug := sourceKey
+				if idx := strings.Index(slug, ":"); idx >= 0 {
+					slug = slug[idx+1:]
+				}
 				featKey, _, _, err := ctx.resolveGeneratedFeatIdentity(spec, slug, row)
 				if err != nil {
 					report.Diagnostics = append(report.Diagnostics, Diagnostic{

internal/topdata/vfx_persistent_migrate.go

@@ -1,198 +0,0 @@
-package topdata
-
-import (
-	"os"
-	"path/filepath"
-	"strings"
-)
-
-func importLegacyVFXPersistent(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-	if strings.TrimSpace(referenceBuilderDir) == "" {
-		return 0, nil
-	}
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "vfx_persistent")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "vfx_persistent")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	targetLockPath := filepath.Join(targetDir, "lock.json")
-	targetModulesDir := filepath.Join(targetDir, "modules")
-	targetModulePath := filepath.Join(targetModulesDir, "freeformaoes.json")
-	if fileExists(targetBasePath) && fileExists(targetLockPath) && fileExists(targetModulePath) {
-		baseObj, baseErr := loadJSONObject(targetBasePath)
-		lockObj, lockErr := loadJSONObject(targetLockPath)
-		if baseErr == nil && lockErr == nil && vfxPersistentImportComplete(baseObj, lockObj) {
-			legacyModulePaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
-			if err == nil {
-				targetModulePaths, err := collectJSONRelativePaths(targetModulesDir)
-				if err == nil && equalStringSlices(legacyModulePaths, targetModulePaths) {
-					return 0, nil
-				}
-			}
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	lockObj, err := synthesizeVFXPersistentKeys(baseObj)
-	if err != nil {
-		return 0, err
-	}
-	legacyLockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-	for key, value := range legacyLockObj {
-		lockObj[key] = value
-	}
-	if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	updated := 0
-	for _, write := range []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: targetBasePath, obj: baseObj},
-		{path: targetLockPath, obj: lockObj},
-	} {
-		changed, err := writeJSONObjectIfChanged(write.path, write.obj)
-		if err != nil {
-			return 0, err
-		}
-		if changed {
-			updated++
-		}
-	}
-
-	moduleRelPaths, err := collectJSONRelativePaths(filepath.Join(legacyDir, "modules"))
-	if err != nil {
-		return 0, err
-	}
-	for _, relPath := range moduleRelPaths {
-		moduleObj, err := loadJSONObject(filepath.Join(legacyDir, "modules", filepath.FromSlash(relPath)))
-		if err != nil {
-			return 0, err
-		}
-		targetPath := filepath.Join(targetModulesDir, filepath.FromSlash(relPath))
-		if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
-			return 0, err
-		}
-		changed, err := writeJSONObjectIfChanged(targetPath, moduleObj)
-		if err != nil {
-			return 0, err
-		}
-		if changed {
-			updated++
-		}
-	}
-
-	targetModulePaths, err := collectJSONRelativePaths(targetModulesDir)
-	if err != nil {
-		return 0, err
-	}
-	legacyModules := make(map[string]struct{}, len(moduleRelPaths))
-	for _, relPath := range moduleRelPaths {
-		legacyModules[relPath] = struct{}{}
-	}
-	for _, relPath := range targetModulePaths {
-		if _, ok := legacyModules[relPath]; ok {
-			continue
-		}
-		if err := os.Remove(filepath.Join(targetModulesDir, filepath.FromSlash(relPath))); err != nil && !os.IsNotExist(err) {
-			return 0, err
-		}
-		updated++
-	}
-
-	return updated, nil
-}
-
-func vfxPersistentImportComplete(baseObj, lockObj map[string]any) bool {
-	if !vfxPersistentRowsHaveKeys(baseObj) {
-		return false
-	}
-	expected := map[string]int{
-		"vfx_persistent:blankradius5":  47,
-		"vfx_persistent:blankradius10": 48,
-		"vfx_persistent:blankradius15": 49,
-		"vfx_persistent:blankradius20": 50,
-		"vfx_persistent:blankradius25": 51,
-		"vfx_persistent:blankradius30": 52,
-	}
-	for key, want := range expected {
-		raw, ok := lockObj[key]
-		if !ok {
-			return false
-		}
-		got, err := asInt(raw)
-		if err != nil || got != want {
-			return false
-		}
-	}
-	return true
-}
-
-func synthesizeVFXPersistentKeys(baseObj map[string]any) (map[string]any, error) {
-	rawRows, ok := baseObj["rows"].([]any)
-	if !ok {
-		return map[string]any{}, nil
-	}
-	lockObj := map[string]any{}
-	for _, raw := range rawRows {
-		row, ok := raw.(map[string]any)
-		if !ok {
-			continue
-		}
-		label, _ := row["LABEL"].(string)
-		if label == "" {
-			continue
-		}
-		key := "vfx_persistent:" + strings.ToLower(label)
-		row["key"] = key
-		if rawID, ok := row["id"]; ok {
-			lockObj[key] = rawID
-		}
-	}
-	return lockObj, nil
-}
-
-func vfxPersistentRowsHaveKeys(baseObj map[string]any) bool {
-	rawRows, ok := baseObj["rows"].([]any)
-	if !ok {
-		return false
-	}
-	for _, raw := range rawRows {
-		row, ok := raw.(map[string]any)
-		if !ok {
-			return false
-		}
-		key, _ := row["key"].(string)
-		if key == "" {
-			return false
-		}
-	}
-	return true
-}
-
-func equalStringSlices(left, right []string) bool {
-	if len(left) != len(right) {
-		return false
-	}
-	for i := range left {
-		if left[i] != right[i] {
-			return false
-		}
-	}
-	return true
-}

internal/topdata/visualeffects_migrate.go

@@ -1,6 +0,0 @@
-package topdata
-
-func importLegacyVisualeffects(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-	return importLegacyDatasetMirror(referenceBuilderDir, dataDir, "visualeffects", "visualeffects.2da", nil)
-}

internal/topdata/wiki_native.go

@@ -569,11 +569,13 @@ func loadWikiContext(dataDir, sourceDir string) (*wikiContext, error) {
 	if err != nil {
 		return nil, err
 	}
-	loadBase := func(name string) (nativeCollectedDataset, bool, error) {
+	loadBase := func(names ...string) (nativeCollectedDataset, bool, error) {
-		for _, dataset := range datasets {
+		for _, name := range names {
-			if dataset.Name == name {
+			for _, dataset := range datasets {
-				collected, err := collectNativeDataset(dataset)
+				if dataset.Name == name {
-				return collected, true, err
+					collected, err := collectNativeDataset(dataset)
+					return collected, true, err
+				}
 			}
 		}
 		return nativeCollectedDataset{}, false, nil
@@ -583,7 +585,7 @@ func loadWikiContext(dataDir, sourceDir string) (*wikiContext, error) {
 	if err != nil {
 		return nil, err
 	}
-	skillDataset, skillOK, err := loadBase("skills")
+	skillDataset, skillOK, err := loadBase("skills", "skills/core")
 	if err != nil {
 		return nil, err
 	}

internal/topdata/wingmodel_migrate.go

@@ -1,86 +0,0 @@
-package topdata
-
-import (
-	"encoding/json"
-	"os"
-	"path/filepath"
-)
-
-func importLegacyWingmodel(referenceBuilderDir, dataDir string, legacyTLK *legacyTLKData) (int, error) {
-	_ = legacyTLK
-
-	legacyDir := filepath.Join(referenceBuilderDir, "data", "wingmodel")
-	if _, err := os.Stat(legacyDir); err != nil {
-		if os.IsNotExist(err) {
-			return 0, nil
-		}
-		return 0, err
-	}
-
-	targetDir := filepath.Join(dataDir, "wingmodel")
-	targetBasePath := filepath.Join(targetDir, "base.json")
-	targetLockPath := filepath.Join(targetDir, "lock.json")
-	targetModulesDir := filepath.Join(targetDir, "modules")
-	targetModulePaths := []string{
-		filepath.Join(targetModulesDir, "add.json"),
-		filepath.Join(targetModulesDir, "ovr_wingprefixes.json"),
-	}
-	if fileExists(targetBasePath) && fileExists(targetLockPath) {
-		allModulesPresent := true
-		for _, path := range targetModulePaths {
-			if !fileExists(path) {
-				allModulesPresent = false
-				break
-			}
-		}
-		if allModulesPresent {
-			return 0, nil
-		}
-	}
-
-	baseObj, err := loadJSONObject(filepath.Join(legacyDir, "base.json"))
-	if err != nil {
-		return 0, err
-	}
-	baseObj["output"] = "wingmodel.2da"
-
-	lockObj, err := loadJSONObject(filepath.Join(legacyDir, "lock.json"))
-	if err != nil {
-		return 0, err
-	}
-
-	moduleNames := []string{"add.json", "ovr_wingprefixes.json"}
-	moduleObjs := make([]map[string]any, 0, len(moduleNames))
-	for _, name := range moduleNames {
-		obj, err := loadJSONObject(filepath.Join(legacyDir, "modules", name))
-		if err != nil {
-			return 0, err
-		}
-		moduleObjs = append(moduleObjs, obj)
-	}
-
-	if err := os.MkdirAll(targetModulesDir, 0o755); err != nil {
-		return 0, err
-	}
-
-	writes := []struct {
-		path string
-		obj  map[string]any
-	}{
-		{path: targetBasePath, obj: baseObj},
-		{path: targetLockPath, obj: lockObj},
-		{path: targetModulePaths[0], obj: moduleObjs[0]},
-		{path: targetModulePaths[1], obj: moduleObjs[1]},
-	}
-	for _, write := range writes {
-		raw, err := json.MarshalIndent(write.obj, "", "  ")
-		if err != nil {
-			return 0, err
-		}
-		raw = append(raw, '\n')
-		if err := os.WriteFile(write.path, raw, 0o644); err != nil {
-			return 0, err
-		}
-	}
-	return len(writes), nil
-}

scripts/crucible-smoke.sh

@@ -18,7 +18,7 @@ bin=bin
 
 # Keep in sync with internal/dispatch.Registry (Wired flag).
 unwired=()
-wired=(assets depot hak module topdata wiki)
+wired=(assets depot hak module nwsync topdata wiki)
 
 exit_of() { set +e; "$@" >/dev/null 2>&1; local c=$?; set -e; echo "${c}"; }
 

scripts/prune-release-assets.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# Delete the binary assets of every release except the newest KEEP ones.
+#
+# Gitea has no asset retention, so 9 assets x ~57 MB per tag pile up forever on
+# the host disk (sow-tools #52). Nothing consumes an old asset: CI takes the
+# Crucible binary from the Nix flake, and every deleted file is reproducible
+# from its tag. Tags, source archives, release notes and the releases
+# themselves are never touched — only attachments.
+#
+# Nothing is excluded, SHA256SUMS and the wrappers included: the checksums are
+# only meaningful next to the binaries they cover, and the drift-check reads
+# the wrappers from the latest release, which always stays complete.
+#
+# Best-effort by design: a stale asset is cheaper than a blocked release, so
+# every API failure is a warning, never a non-zero exit.
+#
+# Env:
+#   API    repo API base, e.g. https://git.example/api/v1/repos/owner/repo  [required]
+#   TOKEN  Gitea token with releases:write                                  [required]
+#   KEEP   how many newest releases stay complete (default 2)
+set -uo pipefail
+
+: "${API:?set API}"
+: "${TOKEN:?set TOKEN}"
+keep="${KEEP:-2}"
+auth="Authorization: token ${TOKEN}"
+per_page=50
+
+warn() { echo "prune-release-assets: $*" >&2; }
+
+# Walk every page so an old backlog is cleared, not only the tag that fell out
+# of the window on this run. Each page already embeds its releases' assets, so
+# no follow-up request per release is needed.
+releases='[]'
+page=1
+while :; do
+  body="$(curl -fsS -H "$auth" "${API}/releases?limit=${per_page}&page=${page}&draft=false")" || {
+    warn "listing releases failed on page ${page}; pruning what was listed so far"
+    break
+  }
+  count="$(jq 'length' <<<"$body")" || { warn "unreadable release page ${page}"; break; }
+  releases="$(jq -s 'add' <(printf '%s' "$releases") <(printf '%s' "$body"))"
+  # A short page is the last one; this also stops a server that ignores `page`.
+  (( count < per_page )) && break
+  page=$(( page + 1 ))
+done
+
+# Sort here rather than trusting the response order, and skip drafts in case
+# the server ignored `draft=false` — a draft must not consume a keep slot and
+# strip the binaries off the newest real release.
+mapfile -t stale < <(jq -r --argjson keep "$keep" '
+  [.[] | select(.draft != true)]
+  | sort_by(.created_at) | reverse | .[$keep:]
+  | .[] | "\(.id) \(.assets // [] | map(.id) | join(","))"
+' <<<"$releases")
+
+(( ${#stale[@]} )) || { echo "prune-release-assets: nothing older than the newest ${keep} releases"; exit 0; }
+
+for line in "${stale[@]}"; do
+  id="${line%% *}"
+  assets="${line#* }"
+  for asset in ${assets//,/ }; do
+    echo "deleting asset ${asset} of release ${id}"
+    curl -fsS -X DELETE -H "$auth" "${API}/releases/${id}/assets/${asset}" >/dev/null \
+      || warn "deleting asset ${asset} of release ${id} failed"
+  done
+done

tests/prune-release-assets.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# Contract for scripts/prune-release-assets.sh, driven by a stub curl on PATH:
+# only assets of releases outside the keep window go, the window is decided by
+# release date rather than response order, drafts never consume a keep slot,
+# releases and tags stay, and an API failure never fails the run.
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+script="$repo_root/scripts/prune-release-assets.sh"
+tmp="$(mktemp -d)"
+trap 'rm -rf "$tmp"' EXIT
+
+# Stub curl: one short page of releases, deliberately out of date order and
+# with a draft that is newer than every published release. Each release embeds
+# two assets. Every request is appended to $CALLS for the assertions below.
+mkdir -p "$tmp/bin"
+cat >"$tmp/bin/curl" <<'STUB'
+#!/usr/bin/env bash
+url="${!#}"
+printf '%s\n' "$*" >>"$CALLS"
+rel() { printf '{"id":%s,"created_at":"%s","draft":%s,"assets":[{"id":%s01},{"id":%s02}]}' "$1" "$2" "$3" "$1" "$1"; }
+case "$url" in
+  *releases\?*)
+    [[ "${FAIL_LIST:-0}" == 1 ]] && exit 22
+    printf '[%s,%s,%s,%s,%s]\n' \
+      "$(rel 12 2024-01-02T00:00:00Z false)" \
+      "$(rel 10 2024-01-04T00:00:00Z false)" \
+      "$(rel 9  2024-06-01T00:00:00Z true)" \
+      "$(rel 13 2024-01-01T00:00:00Z false)" \
+      "$(rel 11 2024-01-03T00:00:00Z false)"
+    ;;
+  */assets/*) [[ "${FAIL_DELETE:-0}" == 1 ]] && exit 22 ;;
+esac
+exit 0
+STUB
+chmod +x "$tmp/bin/curl"
+export PATH="$tmp/bin:$PATH" API=https://git.example/api/v1/repos/o/r TOKEN=t
+
+run() { CALLS="$tmp/calls" bash "$script" >"$tmp/out" 2>"$tmp/err"; }
+
+# Default keep=2: the two newest published releases (10, 11) stay whole, the
+# older two (12, 13) lose their assets, and the newer draft is ignored.
+: >"$tmp/calls"; run
+deletes="$(grep -c -- '-X DELETE' "$tmp/calls" || true)"
+[[ "$deletes" == 4 ]] || { echo "expected 4 asset deletes, got $deletes" >&2; exit 1; }
+grep -q 'assets/1201' "$tmp/calls" && grep -q 'assets/1302' "$tmp/calls" || {
+  echo "expected assets of releases 12 and 13 to be deleted" >&2; exit 1; }
+if grep -q 'releases/10/assets/\|releases/11/assets/' "$tmp/calls"; then
+  echo "kept releases lost assets" >&2; exit 1
+fi
+if grep -q 'releases/9/assets/' "$tmp/calls"; then
+  echo "a draft release was pruned" >&2; exit 1
+fi
+# A release or tag must never be deleted, only attachments under /assets/.
+if grep -- '-X DELETE' "$tmp/calls" | grep -qv '/assets/'; then
+  echo "a non-asset DELETE was issued" >&2; exit 1
+fi
+# A short page ends the sweep: no second page, so a server ignoring `page`
+# cannot spin the job until the job timeout.
+[[ "$(grep -c 'releases?' "$tmp/calls")" == 1 ]] || {
+  echo "a short release page did not end the sweep" >&2; exit 1; }
+
+# KEEP=4 covers every published release: nothing to delete.
+: >"$tmp/calls"; KEEP=4 run
+if grep -q -- '-X DELETE' "$tmp/calls"; then echo "KEEP=4 still deleted" >&2; exit 1; fi
+
+# Failures are warnings, never a non-zero exit.
+: >"$tmp/calls"; FAIL_DELETE=1 run || { echo "delete failure failed the run" >&2; exit 1; }
+grep -q 'failed' "$tmp/err" || { echo "delete failure was not warned about" >&2; exit 1; }
+
+: >"$tmp/calls"; FAIL_LIST=1 run || { echo "listing failure failed the run" >&2; exit 1; }
+grep -q 'listing releases failed' "$tmp/err" || {
+  echo "listing failure was not warned about" >&2; exit 1; }
+if grep -q -- '-X DELETE' "$tmp/calls"; then
+  echo "deleted assets despite an unreadable release list" >&2; exit 1
+fi
+
+echo "prune-release-assets: prunes by release date outside the keep window, ignores drafts, never deletes a release, never fails the build"

tests/workflow-contract.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+contract_script="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
+default_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+repo_root="${WORKFLOW_ROOT:-$default_root}"
+cd "$repo_root"
+
+ci=.gitea/workflows/ci.yml
+release=.gitea/workflows/build-binaries.yml
+sync=.gitea/workflows/sync-wrappers.yml
+checkout='actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683'
+
+assert_exact_events() {
+  local workflow="$1" expected="$2" actual
+  actual="$(awk '
+    /^on:$/ { in_on = 1 }
+    in_on && /^$/ { next }
+    in_on && $0 != "on:" && $0 !~ /^[[:space:]]/ { exit }
+    in_on { print }
+  ' "$workflow")"
+  [[ "$actual" == "$expected" ]] || {
+    echo "workflow-contract: unexpected event scope in $workflow" >&2
+    return 1
+  }
+}
+
+[[ -f "$ci" ]] || { echo "missing consolidated CI workflow" >&2; exit 1; }
+[[ ! -e .gitea/workflows/test.yml ]] || { echo "legacy test workflow still present" >&2; exit 1; }
+
+assert_exact_events "$ci" $'on:\n  pull_request:'
+grep -Fqx 'permissions: read-all' "$ci"
+grep -Fqx '    timeout-minutes: 60' "$ci"
+[[ "$(grep -Fc "$checkout" "$ci")" -eq 1 ]]
+grep -Fqx '          fetch-depth: 0' "$ci"
+grep -Fq 'go vet ./...' "$ci"
+grep -Fq 'go test ./...' "$ci"
+grep -Fq 'shellcheck scripts/*.sh' "$ci"
+grep -Fq 'yamllint .gitea' "$ci"
+grep -Fq 'make smoke' "$ci"
+grep -Fq 'for target in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64 windows/amd64 windows/arm64; do' "$ci"
+
+assert_exact_events "$release" $'on:\n  push:\n    tags: [\'v*\']'
+grep -Fqx '  code: read' "$release"
+grep -Fqx '  releases: write' "$release"
+grep -Fqx '    timeout-minutes: 30' "$release"
+[[ "$(grep -Fc "$checkout" "$release")" -eq 1 ]]
+grep -Fq 'secrets.GITEA_TOKEN' "$release"
+grep -Fq 'scripts/prune-release-assets.sh' "$release"
+grep -Fq 'continue-on-error: true' "$release"
+if grep -Fq 'secrets.GITHUB_TOKEN' "$release"; then
+  echo "workflow-contract: release uses the GitHub token alias" >&2
+  exit 1
+fi
+
+grep -Fqx 'permissions: read-all' "$sync"
+grep -Fqx '    timeout-minutes: 30' "$sync"
+[[ "$(grep -Fc "$checkout" "$sync")" -eq 1 ]]
+assert_exact_events "$sync" $'on:\n  push:\n    branches: [main]\n    paths:\n      - \'wrappers/crucible.sh\'\n      - \'wrappers/crucible.ps1\''
+
+expect_mutation_rejected() {
+  local name="$1" workflow="$2" expression="$3" tmp
+  tmp="$(mktemp -d)"
+  mkdir -p "$tmp/.gitea"
+  cp -R "$default_root/.gitea/workflows" "$tmp/.gitea/workflows"
+  sed -i "$expression" "$tmp/$workflow"
+  if WORKFLOW_ROOT="$tmp" WORKFLOW_CONTRACT_MUTATION=1 bash "$contract_script" >/dev/null 2>&1; then
+    echo "workflow-contract: accepted forbidden mutation: $name" >&2
+    rm -rf "$tmp"
+    return 1
+  fi
+  rm -rf "$tmp"
+}
+
+if [[ "${WORKFLOW_CONTRACT_MUTATION:-0}" != 1 ]]; then
+  mutations_ok=0
+  expect_mutation_rejected 'CI workflow_dispatch event' "$ci" '/^  pull_request:$/a\  workflow_dispatch:' || mutations_ok=1
+  expect_mutation_rejected 'release develop branch' "$release" "/^    tags:/a\\    branches: [develop]" || mutations_ok=1
+  expect_mutation_rejected 'release schedule event' "$release" "/^    tags:/a\\  schedule:\n    - cron: '0 0 * * *'" || mutations_ok=1
+  expect_mutation_rejected 'release extra tag pattern' "$release" "s/tags: \['v\*'\]/tags: ['v*', 'release-*']/" || mutations_ok=1
+  expect_mutation_rejected 'wrapper sync pull_request event' "$sync" '/^  push:$/i\  pull_request:' || mutations_ok=1
+  expect_mutation_rejected 'wrapper sync broad path' "$sync" "/^      - 'wrappers\/crucible.ps1'$/a\\      - 'wrappers/**'" || mutations_ok=1
+  (( mutations_ok == 0 )) || exit 1
+fi
+
+echo "workflow-contract: CI is PR-only; release and wrapper sync retain their narrow triggers"