Files

T

build-binaries / build-binaries (push) Successful in 2m21s

Details

feat(nwsync): emit blobs in parallel with a bounded worker pool (#80 )

Closes #79.

Emit is latency-bound, not CPU-bound. Every blob costs two serial round-trips to the zone — a `ProbeKey` HEAD, then a `PutReader` PUT — so a hak with a few thousand resources pays a few thousand serialised latencies. Measured on the live sow-assets-manifest backfill: 26 s of CPU across 9.5 minutes of wall clock, on a 4-core host with 5 GB free and peak RSS of 51 MB.

`emit` now hashes, compresses and stores `--jobs N` resources at once, default 16 — matching `DEPOT_JOBS` and the transport's `MaxIdleConnsPerHost`, so a worker per connection needs no fresh TLS handshake. `--jobs 1` is exactly the old behaviour.

### Three properties had to survive

Each has a test in `internal/nwsync/jobs_test.go`:

- **Deterministic manifest bytes.** `emitterVersion` promises a manifest is a function of its artifact, so `entries` is index-addressed rather than appended to — a worker owns `entries[i]` alone and the slice comes back in artifact order whatever order uploads finish in. `TestEmitProducesTheSameIndexAtEveryJobCount` diffs the `.nsym` and its sidecar between `-jobs 1` and `-jobs 16`.
- **Index still lands last.** Any worker's failure aborts before a manifest is written. `TestEmitLeavesNoIndexWhenAParallelUploadFails` fails every blob PUT with 16 workers in flight and asserts no `.nsym` appears. Under `-race` it also covers the shared counters.
- **Identical content still shares one blob.** This one bit during development and is the reason to read the diff carefully: serially, the sink's existence check absorbed two resrefs with identical bytes. In parallel both workers probe, both miss, and both upload — `TestEmitWritesBlobsAndManifest` caught it as "wrote 2 blobs, want 1". Claiming the sha1 in-process restores the dedupe and skips a probe round-trip as well.

### Memory

Peak now tracks the resources in flight rather than one resource. The ceiling is `N` × the 15 MB `fileSizeLimit` plus its compressed copy — bounded by a constant this package enforces itself, and still not tracking the archive. `TestEmitPeakMemoryIsBoundedByJobCount` re-runs the #76 regression check at `-jobs 8`: a hak 8× bigger still costs the same.

This is only cheap because of #78. Before streaming emit, N workers would have meant N whole archives resident.

### Not done

Skipping the `ProbeKey` HEAD on a first-time emit would halve round-trips, but doubles uploaded bytes on a re-run — which is exactly what a backfill is. Noted in #79 so it is not rediscovered; parallelism is the better lever and this PR takes it.

Worker compression still serialises on `blobEncoder`, which is `WithEncoderConcurrency(1)` for the memory reason in `compressedbuf.go`. At 26 s of CPU per hak that is not worth trading memory for, but it is where to look if the numbers ever say otherwise.

🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #80

Co-authored-by: vickydotbat <vickydotbat@tutamail.com>

2026-07-31 15:25:33 +00:00

7.6 KiB

Raw Blame History

Crucible command surface

internal/dispatch.Registry is the single source of truth for Crucible's builders, visible commands, descriptions, help, and hidden compatibility aliases.

Visible commands

Builder	Command	Purpose
`hak`	`build`	Build configured HAK archives and their manifest.
`hak`	`manifest`	Apply a generated HAK list to module source.
`module`	`build`	Build the module and configured project outputs.
`module`	`extract`	Extract built archives into configured source trees.
`module`	`validate`	Validate project layout, config, and source inventory.
`module`	`compare`	Compare source content with built archives.
`module`	`manifest`	Apply a generated HAK list to module source.
`topdata`	`validate`	Validate topdata source and canonical JSON compatibility.
`topdata`	`build`	Compile topdata and refresh the configured package.
`topdata`	`package`	Package cached outputs into HAK and TLK files.
`topdata`	`compare`	Compare current output with a fresh native build.
`topdata`	`convert`	Convert between 2DA and native JSON/module formats.
`wiki`	`build`	Render wiki page drafts from compiled topdata.
`wiki`	`deploy`	Deploy generated wiki pages to NodeBB.
`assets`	`compile`	Compile ASCII `.mdl` models to binary in place.
`assets`	`convert`	Convert textures to/from NWN DDS (flips vertically).
`assets`	`upscale`	Upscale textures through an installed backend.
`assets`	`check-mdl`	Report uncompiled ASCII `.mdl` and model-name mismatches.
`assets`	`fix-mdl`	Lowercase `.mdl` names and rewrite model identity to match.
`assets`	`check-dupes`	Report runtime-name (basename) collisions across dirs.
`assets`	`clean-dupes`	Delete clean-tree files whose basename collides with primary.
`depot`	`status`	Report referenced-vs-present drift against a backend.
`depot`	`push`	Upload referenced-but-absent blobs from a local depot.
`depot`	`verify`	Existence sweep plus sampled download re-hash.
`depot`	`get`	Fetch one blob with sha re-verify.
`depot`	`pull`	Incremental verified pull of every referenced blob.
`nwsync`	`emit`	Explode one artifact into NWSync blobs plus its own NSYM manifest.
`nwsync`	`assemble`	Merge per-artifact NSYM manifests into one merged manifest.

depot status and depot get pick their backend either with --out DIR, a depot tree on disk, or with --target bunny|cdn, a remote backend. The two flags are mutually exclusive.

nwsync emit runs where an artifact is born (a .hak/.erf, or a loose file such as the TLK); nwsync assemble runs at module release and reads only the small per-artifact indexes. Both take depot keys: an artifact's index lives beside the artifact itself with the extension replaced, so emit and assemble agree on where it is without being told.

nwsync emit     [--as NAME] [--out DIR] [--jobs N] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...

emit is latency-bound, not CPU-bound: every blob costs an existence probe plus an upload, and a measured backfill spent 26 seconds of CPU across 9.5 minutes of wall clock. --jobs N (default 16) sets how many resources are in flight at once. The manifest is byte-identical at any value — the number of workers is never observable in the output. Peak memory is N times the per-resource limit of 15 MB plus its compressed copy, so raising N far past the default costs real memory for little gain: the transport keeps 16 idle connections per host, and past that a worker pays a fresh TLS handshake.

Both verbs upload by default; nothing bulky is ever written to the runner's disk. --out DIR writes a local repository tree instead, which is the conformance path against upstream nwn_nwsync_write. The zone comes from NWSYNC_STORAGE_ZONE and NWSYNC_STORAGE_PASSWORD, with the host from BUNNY_STORAGE_HOST — NWSync data is a separate zone from the asset depot.

assemble's artifact keys are in Mod_HakList order, highest priority first: a resref in more than one artifact resolves to the earliest one, the way the game resolves it. --tlk-key has its own slot because the TLK shadows nothing. --group-id is per channel — 1 is current, 2 is testing, and 0 leaves the field out of the sidecar.

emit uploads blobs first and the index last, so the presence of an index is the publication marker: an artifact whose emit died halfway leaves real blobs in the zone and no index. Blob names are content hashes, so re-running skips whatever already landed, and assemble fails closed on an artifact with no index rather than publishing a manifest that is missing a hak.

What an emitted tree looks like

--out DIR produces the same tree emit would upload, which makes it the way to check a zone by hand without touching one:

<artifact-sha>.nsym          binary index
<artifact-sha>.nsym.json     the same index, readable
data/sha1/a7/4a/a74aa84a...  one blob per resource, two-level fanout

A blob's name is the SHA-1 of the resource's original bytes, but the file on disk is not those bytes: each blob is wrapped in NWCompressedBuffer framing, a 24-byte NSYC header followed by a zstd frame. Hashing the file directly will not match its name, which is the obvious first thing to try and the obvious first thing to be confused by. Strip the header first:

tail -c +25 <blob> | zstd -dc | sha1sum   # == the blob's filename

The header carries the uncompressed length as a little-endian uint32 at offset 12, so the decompressed size is checkable without decompressing. Compression is worth roughly a 4:1 saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk against 249 MB of resources, as recorded in the sidecar's on_disk_bytes and total_bytes.

Hidden compatibility aliases

Existing scripts may continue using these names indefinitely. They are accepted but omitted from routine help and the interactive menu:

Builder	Hidden alias	Implementation
`hak`	`build-haks`	`build-haks`
`hak`	`apply-hak-manifest`	`apply-hak-manifest`
`module`	`build-module`	module-only `build-module`
`module`	`apply-hak-manifest`	`apply-hak-manifest`
`topdata`	`validate-topdata`	`validate-topdata`
`topdata`	`build-topdata`	`build-topdata`
`topdata`	`build-top-package`	`build-top-package`
`topdata`	`compare-topdata`	`compare-topdata`
`topdata`	`convert-topdata`	`convert-topdata`
`depot`	`--target local`	`--out $DEPOT_DIR` on `status`/`get`
`wiki`	`build-wiki`	`build-wiki`
`wiki`	`deploy-wiki`	`deploy-wiki`

module build-module deliberately retains its narrower module-only behavior; it is not redirected to the broader module build.

Global commands

crucible help | -h          usage + builder list
crucible version | -V       build version
crucible list               name<TAB>binary<TAB>summary
crucible config [args]      inspect and validate effective configuration
crucible changelog [args]   generate a release changelog

HAK build options

crucible hak build
  [--hak <hak-name> ...]
  [--archive <archive-name> ...]
  [--source-manifest <path>]
  [--content-addressed-root <path>]
  [--plan-only]
  [--quiet|--verbose|--debug]

When a source manifest contains asset_sources, pass --content-addressed-root <directory>. Crucible resolves each declared SHA-256 under sha256/<first-two>/<next-two>/<full-sha256> and verifies streamed bytes while writing selected HAK archives. Authored .bmu files are ordinary assets and require no conversion pipeline.

7.6 KiB Raw Blame History