Files

T

archvillainette a131b25e5b feat(nwsync): emit blobs and per-artifact NSYM, assemble merged manifests (#71 )

Builds sow-tools#53. Format spec followed is the resolution comment of sow-platform#94, checked line by line against niv/neverwinter.nim at HEAD (`nwsync.nim`, `compressedbuf.nim`, `nwsync/private/libupdate.nim`).

## What lands

`crucible nwsync emit <artifact> --out DIR` — explodes one `.hak`/`.erf`, or one loose file such as the TLK, into NWSync blobs plus a NSYM v3 manifest covering only that artifact, with the same `.json` sidecar upstream writes. Blob path `data/sha1/<h0h1>/<h2h3>/<sha1>`, body in NWCompressedBuffer framing (magic `NSYC`, version 3, algorithm 2, uncompressed size, zstd header version 1, dictionary 0, raw zstd frame). The sha1 that names a blob is over the uncompressed bytes.

`crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]` — merges the per-artifact manifests into one, reading no bulk data at all. Merge rule is resref shadowing, not concatenation: a resref in more than one artifact resolves to the earliest artifact in `--order`, which is how the game resolves it. `--group-id` stays caller-supplied (1 current, 2 testing; 0 is absent, matching upstream omitting a zero integer meta field).

Rules taken from upstream and not re-invented: `nss`/`ndb`/`gic` always skipped; an unresolvable restype is a hard error, not a skip; a resource over 15 MB fails closed; no `latest` file and no `.origin` file, ever. A `.mod` is refused outright — a persistent world publishes no module contents, so the module contributes no bytes.

## Two deliberate departures

- **Emitter version is its own field, not the build revision.** `emitter_version` is a constant bumped only when emitted bytes change. Keying the refuse-to-merge check on `created_with` would invalidate every published index on every unrelated crucible commit and force a re-emit of the whole 15 GB corpus — the opposite of "nothing downstream ever needs the hak again".
- **`SOURCE_DATE_EPOCH` pins the sidecar timestamp.** The manifest itself was already deterministic; the sidecar's `created` was not, against the determinism rule in `docs/consumer-contract.md`.

## Not in this PR, and why

- **Direct upload.** Only the local `--out` sink exists, which is the conformance path. The upload sink and the mid-hak-failure question are sow-tools#60, and the consumer wiring is #65.
- **The conformance run against upstream.** sow-tools#59 owns getting `nwn_nwsync_write` running and capturing reference output. The format here was read from upstream source rather than from its output, so the byte-for-byte manifest comparison and the after-decompression blob comparison still have to happen — that is what #59 is for. The tests in this PR check the layout against the spec, so a shared misreading would pass them.
- **The `artifacts/haks/sha256/<a>/<b>/<sha256>.nsym` location.** emit writes `<out>/<name>.nsym`; where a publisher puts it is the publisher's business (#65).
- **The acceptance gate** — a real client syncing from an assembled manifest — is unchanged and still open.

## Checks

`make check` green (vet, unit tests, shellcheck, yamllint, workflow contract), `make smoke` green with the new builder, `nix build .#crucible` produces `crucible-nwsync`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #71

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>

2026-07-29 10:50:05 +00:00

4.8 KiB

Raw Blame History

Crucible command surface

internal/dispatch.Registry is the single source of truth for Crucible's builders, visible commands, descriptions, help, and hidden compatibility aliases.

Visible commands

Builder	Command	Purpose
`hak`	`build`	Build configured HAK archives and their manifest.
`hak`	`manifest`	Apply a generated HAK list to module source.
`module`	`build`	Build the module and configured project outputs.
`module`	`extract`	Extract built archives into configured source trees.
`module`	`validate`	Validate project layout, config, and source inventory.
`module`	`compare`	Compare source content with built archives.
`module`	`manifest`	Apply a generated HAK list to module source.
`topdata`	`validate`	Validate topdata source and canonical JSON compatibility.
`topdata`	`build`	Compile topdata and refresh the configured package.
`topdata`	`package`	Package cached outputs into HAK and TLK files.
`topdata`	`compare`	Compare current output with a fresh native build.
`topdata`	`convert`	Convert between 2DA and native JSON/module formats.
`wiki`	`build`	Render wiki page drafts from compiled topdata.
`wiki`	`deploy`	Deploy generated wiki pages to NodeBB.
`assets`	`compile`	Compile ASCII `.mdl` models to binary in place.
`assets`	`convert`	Convert textures to/from NWN DDS (flips vertically).
`assets`	`upscale`	Upscale textures through an installed backend.
`assets`	`check-mdl`	Report uncompiled ASCII `.mdl` and model-name mismatches.
`assets`	`fix-mdl`	Lowercase `.mdl` names and rewrite model identity to match.
`assets`	`check-dupes`	Report runtime-name (basename) collisions across dirs.
`assets`	`clean-dupes`	Delete clean-tree files whose basename collides with primary.
`depot`	`status`	Report referenced-vs-present drift against a backend.
`depot`	`push`	Upload referenced-but-absent blobs from a local depot.
`depot`	`verify`	Existence sweep plus sampled download re-hash.
`depot`	`get`	Fetch one blob with sha re-verify.
`depot`	`pull`	Incremental verified pull of every referenced blob.
`nwsync`	`emit`	Explode one artifact into NWSync blobs plus its own NSYM manifest.
`nwsync`	`assemble`	Merge per-artifact NSYM manifests into one merged manifest.

depot status and depot get pick their backend either with --out DIR, a depot tree on disk, or with --target bunny|cdn, a remote backend. The two flags are mutually exclusive.

nwsync emit runs where an artifact is born (a .hak/.erf, or a loose file such as the TLK); nwsync assemble runs at module release and reads only the small per-artifact manifests. --order lists artifact names highest priority first: a resref in more than one artifact resolves to the earliest one, the way the game resolves it. --group-id is per channel — 1 is current, 2 is testing, and 0 leaves the field out of the sidecar.

Hidden compatibility aliases

Existing scripts may continue using these names indefinitely. They are accepted but omitted from routine help and the interactive menu:

Builder	Hidden alias	Implementation
`hak`	`build-haks`	`build-haks`
`hak`	`apply-hak-manifest`	`apply-hak-manifest`
`module`	`build-module`	module-only `build-module`
`module`	`apply-hak-manifest`	`apply-hak-manifest`
`topdata`	`validate-topdata`	`validate-topdata`
`topdata`	`build-topdata`	`build-topdata`
`topdata`	`build-top-package`	`build-top-package`
`topdata`	`compare-topdata`	`compare-topdata`
`topdata`	`convert-topdata`	`convert-topdata`
`depot`	`--target local`	`--out $DEPOT_DIR` on `status`/`get`
`wiki`	`build-wiki`	`build-wiki`
`wiki`	`deploy-wiki`	`deploy-wiki`

module build-module deliberately retains its narrower module-only behavior; it is not redirected to the broader module build.

Global commands

crucible help | -h          usage + builder list
crucible version | -V       build version
crucible list               name<TAB>binary<TAB>summary
crucible config [args]      inspect and validate effective configuration
crucible changelog [args]   generate a release changelog

HAK build options

crucible hak build
  [--hak <hak-name> ...]
  [--archive <archive-name> ...]
  [--source-manifest <path>]
  [--content-addressed-root <path>]
  [--plan-only]
  [--quiet|--verbose|--debug]

When a source manifest contains asset_sources, pass --content-addressed-root <directory>. Crucible resolves each declared SHA-256 under sha256/<first-two>/<next-two>/<full-sha256> and verifies streamed bytes while writing selected HAK archives. Authored .bmu files are ordinary assets and require no conversion pipeline.

4.8 KiB Raw Blame History