archvillainetteandClaude Opus 5
f69121a9e8
ci / ci (pull_request) Successful in 3m32s
Emit is latency-bound, not CPU-bound. Every blob costs two serial HTTP round-trips to the zone — an existence probe, then an upload — so a hak with a few thousand resources pays a few thousand serialised latencies. A measured backfill spent 26 seconds of CPU across 9.5 minutes of wall clock, on a host with three of four cores idle and 5 GB free. Emit now hashes, compresses and stores `--jobs N` resources at once (default 16, matching DEPOT_JOBS and the transport's idle connections per host). Three properties had to survive, and each has a test: - The manifest's bytes are promised deterministic by emitterVersion, so entries is index-addressed rather than appended to: a worker owns entries[i] alone and the slice comes back in artifact order whatever order the uploads finish in. - The index is still the publication marker, so any worker's failure aborts the run before a manifest is written. Workers drain the rest of the channel instead of returning, which keeps the feeder from blocking on workers that have gone away. - Two resrefs holding identical bytes still share one blob. Serially the sink's existence check absorbed that; in parallel both workers would probe, both miss, and both upload. Claiming the sha1 in-process restores the dedupe and skips a probe round-trip as well. Peak memory is now the resources in flight rather than one resource, so the ceiling is N times the 15 MB per-resource limit and its compressed copy — bounded by a constant this package enforces itself, and still nowhere near tracking the archive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
161 lines
7.6 KiB
Markdown
161 lines
7.6 KiB
Markdown
# Crucible command surface
|
|
|
|
`internal/dispatch.Registry` is the single source of truth for Crucible's
|
|
builders, visible commands, descriptions, help, and hidden compatibility
|
|
aliases.
|
|
|
|
## Visible commands
|
|
|
|
| Builder | Command | Purpose |
|
|
| --- | --- | --- |
|
|
| `hak` | `build` | Build configured HAK archives and their manifest. |
|
|
| `hak` | `manifest` | Apply a generated HAK list to module source. |
|
|
| `module` | `build` | Build the module and configured project outputs. |
|
|
| `module` | `extract` | Extract built archives into configured source trees. |
|
|
| `module` | `validate` | Validate project layout, config, and source inventory. |
|
|
| `module` | `compare` | Compare source content with built archives. |
|
|
| `module` | `manifest` | Apply a generated HAK list to module source. |
|
|
| `topdata` | `validate` | Validate topdata source and canonical JSON compatibility. |
|
|
| `topdata` | `build` | Compile topdata and refresh the configured package. |
|
|
| `topdata` | `package` | Package cached outputs into HAK and TLK files. |
|
|
| `topdata` | `compare` | Compare current output with a fresh native build. |
|
|
| `topdata` | `convert` | Convert between 2DA and native JSON/module formats. |
|
|
| `wiki` | `build` | Render wiki page drafts from compiled topdata. |
|
|
| `wiki` | `deploy` | Deploy generated wiki pages to NodeBB. |
|
|
| `assets` | `compile` | Compile ASCII `.mdl` models to binary in place. |
|
|
| `assets` | `convert` | Convert textures to/from NWN DDS (flips vertically). |
|
|
| `assets` | `upscale` | Upscale textures through an installed backend. |
|
|
| `assets` | `check-mdl` | Report uncompiled ASCII `.mdl` and model-name mismatches. |
|
|
| `assets` | `fix-mdl` | Lowercase `.mdl` names and rewrite model identity to match. |
|
|
| `assets` | `check-dupes` | Report runtime-name (basename) collisions across dirs. |
|
|
| `assets` | `clean-dupes` | Delete clean-tree files whose basename collides with primary. |
|
|
| `depot` | `status` | Report referenced-vs-present drift against a backend. |
|
|
| `depot` | `push` | Upload referenced-but-absent blobs from a local depot. |
|
|
| `depot` | `verify` | Existence sweep plus sampled download re-hash. |
|
|
| `depot` | `get` | Fetch one blob with sha re-verify. |
|
|
| `depot` | `pull` | Incremental verified pull of every referenced blob. |
|
|
| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
|
|
| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
|
|
|
|
`depot status` and `depot get` pick their backend either with `--out DIR`, a
|
|
depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
|
|
flags are mutually exclusive.
|
|
|
|
`nwsync emit` runs where an artifact is born (a `.hak`/`.erf`, or a loose file
|
|
such as the TLK); `nwsync assemble` runs at module release and reads only the
|
|
small per-artifact indexes. Both take **depot keys**: an artifact's index lives
|
|
beside the artifact itself with the extension replaced, so `emit` and
|
|
`assemble` agree on where it is without being told.
|
|
|
|
```
|
|
nwsync emit [--as NAME] [--out DIR] [--jobs N] <artifact-key> <file>
|
|
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
|
|
```
|
|
|
|
`emit` is latency-bound, not CPU-bound: every blob costs an existence probe
|
|
plus an upload, and a measured backfill spent 26 seconds of CPU across 9.5
|
|
minutes of wall clock. `--jobs N` (default 16) sets how many resources are in
|
|
flight at once. The manifest is byte-identical at any value — the number of
|
|
workers is never observable in the output. Peak memory is `N` times the
|
|
per-resource limit of 15 MB plus its compressed copy, so raising `N` far past
|
|
the default costs real memory for little gain: the transport keeps 16 idle
|
|
connections per host, and past that a worker pays a fresh TLS handshake.
|
|
|
|
Both verbs upload by default; nothing bulky is ever written to the runner's
|
|
disk. `--out DIR` writes a local repository tree instead, which is the
|
|
conformance path against upstream `nwn_nwsync_write`. The zone comes from
|
|
`NWSYNC_STORAGE_ZONE` and `NWSYNC_STORAGE_PASSWORD`, with the host from
|
|
`BUNNY_STORAGE_HOST` — NWSync data is a separate zone from the asset depot.
|
|
|
|
`assemble`'s artifact keys are in `Mod_HakList` order, highest priority first: a
|
|
resref in more than one artifact resolves to the earliest one, the way the game
|
|
resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
|
|
`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
|
|
out of the sidecar.
|
|
|
|
`emit` uploads blobs first and the index last, so the presence of an index is
|
|
the publication marker: an artifact whose emit died halfway leaves real blobs in
|
|
the zone and no index. Blob names are content hashes, so re-running skips
|
|
whatever already landed, and `assemble` fails closed on an artifact with no
|
|
index rather than publishing a manifest that is missing a hak.
|
|
|
|
### What an emitted tree looks like
|
|
|
|
`--out DIR` produces the same tree `emit` would upload, which makes it the way
|
|
to check a zone by hand without touching one:
|
|
|
|
```
|
|
<artifact-sha>.nsym binary index
|
|
<artifact-sha>.nsym.json the same index, readable
|
|
data/sha1/a7/4a/a74aa84a... one blob per resource, two-level fanout
|
|
```
|
|
|
|
A blob's name is the SHA-1 of the resource's **original** bytes, but the file on
|
|
disk is not those bytes: each blob is wrapped in NWCompressedBuffer framing, a
|
|
24-byte `NSYC` header followed by a zstd frame. Hashing the file directly will
|
|
not match its name, which is the obvious
|
|
first thing to try and the obvious first thing to be confused by. Strip the
|
|
header first:
|
|
|
|
```
|
|
tail -c +25 <blob> | zstd -dc | sha1sum # == the blob's filename
|
|
```
|
|
|
|
The header carries the uncompressed length as a little-endian `uint32` at offset
|
|
12, so the decompressed size is checkable without decompressing. Compression is
|
|
worth roughly a 4:1
|
|
saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
|
|
against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
|
|
`total_bytes`.
|
|
|
|
## Hidden compatibility aliases
|
|
|
|
Existing scripts may continue using these names indefinitely. They are accepted
|
|
but omitted from routine help and the interactive menu:
|
|
|
|
| Builder | Hidden alias | Implementation |
|
|
| --- | --- | --- |
|
|
| `hak` | `build-haks` | `build-haks` |
|
|
| `hak` | `apply-hak-manifest` | `apply-hak-manifest` |
|
|
| `module` | `build-module` | module-only `build-module` |
|
|
| `module` | `apply-hak-manifest` | `apply-hak-manifest` |
|
|
| `topdata` | `validate-topdata` | `validate-topdata` |
|
|
| `topdata` | `build-topdata` | `build-topdata` |
|
|
| `topdata` | `build-top-package` | `build-top-package` |
|
|
| `topdata` | `compare-topdata` | `compare-topdata` |
|
|
| `topdata` | `convert-topdata` | `convert-topdata` |
|
|
| `depot` | `--target local` | `--out $DEPOT_DIR` on `status`/`get` |
|
|
| `wiki` | `build-wiki` | `build-wiki` |
|
|
| `wiki` | `deploy-wiki` | `deploy-wiki` |
|
|
|
|
`module build-module` deliberately retains its narrower module-only behavior;
|
|
it is not redirected to the broader `module build`.
|
|
|
|
## Global commands
|
|
|
|
```text
|
|
crucible help | -h usage + builder list
|
|
crucible version | -V build version
|
|
crucible list name<TAB>binary<TAB>summary
|
|
crucible config [args] inspect and validate effective configuration
|
|
crucible changelog [args] generate a release changelog
|
|
```
|
|
|
|
## HAK build options
|
|
|
|
```text
|
|
crucible hak build
|
|
[--hak <hak-name> ...]
|
|
[--archive <archive-name> ...]
|
|
[--source-manifest <path>]
|
|
[--content-addressed-root <path>]
|
|
[--plan-only]
|
|
[--quiet|--verbose|--debug]
|
|
```
|
|
|
|
When a source manifest contains `asset_sources`, pass
|
|
`--content-addressed-root <directory>`. Crucible resolves each declared SHA-256
|
|
under `sha256/<first-two>/<next-two>/<full-sha256>` and verifies streamed bytes
|
|
while writing selected HAK archives. Authored `.bmu` files are ordinary assets
|
|
and require no conversion pipeline.
|