archvillainette
7cc53aeb68
build-binaries / build-binaries (push) Successful in 2m40s
Closes #86. Closes #85. These land together on purpose. Fixing the encoder alone changes nothing for the blobs already in the zone, because `emit` skips whatever is already present. ## #86 — the framing fix `klauspost/compress` omits the zstd `Frame_Content_Size` field for inputs under 256 bytes, which the format permits. Reference libzstd never does, so the NWN client — which sizes its output buffer from `ZSTD_getFrameContentSize` and has therefore never met a frame without one — rejected roughly 6% of our blobs outright. Any single one stops a sync dead, so no client could complete a sync of the live manifest. No encoder option changes this, so `compressBlob` re-headers the affected frames into the shape libzstd itself emits: `Single_Segment_flag` set, `Window_Descriptor` dropped, and the freed byte spent on a one-byte `Frame_Content_Size`. Same length in, same length out, and the same descriptor byte (`0x24`) the issue recorded from libzstd. `compressBlob` then asserts its own output. An encoder upgrade that finds another way to omit the field would otherwise reproduce #86 in silence, and a blob is skipped by every later emit once written. `emitter_version` goes to `2`, so `assemble` refuses to merge an index written by the encoder that omitted the field. **Proved against the reference decoder, not just a round trip.** A real emitted 175-byte blob: ``` Frames Skips Compressed Uncompressed Ratio Check Filename 1 0 48 B 175 B 3.646 XXH64 frame.zst c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 - <- zstd -dc | sha1sum c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 <- the blob's own name ``` Before the fix that `Uncompressed` column was blank. ## #85 — `nwsync verify` `crucible nwsync verify <manifest-sha1>` reads a manifest and its blobs back through the **public pull zone**, with no credential, because what matters is the bytes a client is served, edge behaviour included. Every distinct blob is decompressed and hashed; failures are reported per blob as missing / malformed framing / size mismatch / hash mismatch, and the exit code is 1. - `--sample N` makes a routine check cheap against a manifest that is ~69,000 blobs and 15 GB; the default is a full sweep. - `--base URL` / `NWSYNC_PULL_BASE` overrides the public host. - The manifest is checked against its own sha1 before a single blob is fetched. - `emit --verify` applies the same check where `emit` would otherwise trust presence, and replaces a stored blob that is not what its name claims. This is what makes the #86 blobs repairable. ## Why the existing checks missed this Both new checks assert the **frame property**, not just a round trip. The conformance suite (#59) compares decompressed bytes, so a frame that decodes correctly passes regardless of its header; and the earlier zone audit decompressed 68 blobs with the `zstd` CLI, a *more* capable decoder than the client's, which certified exactly the blobs the client rejects. ## Checks `make check` and `make smoke` green. Second commit is the fixes from a two-axis review of the first. ## Not in this PR Three follow-ups, filed separately: the backfill has not been run, replacing a blob does not purge the pull-zone edge cache, and #85's runbook line belongs to `sow-platform`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #87 Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
196 lines
10 KiB
Markdown
196 lines
10 KiB
Markdown
# Crucible command surface
|
|
|
|
`internal/dispatch.Registry` is the single source of truth for Crucible's
|
|
builders, visible commands, descriptions, help, and hidden compatibility
|
|
aliases.
|
|
|
|
## Visible commands
|
|
|
|
| Builder | Command | Purpose |
|
|
| --- | --- | --- |
|
|
| `hak` | `build` | Build configured HAK archives and their manifest. |
|
|
| `hak` | `manifest` | Apply a generated HAK list to module source. |
|
|
| `module` | `build` | Build the module and configured project outputs. |
|
|
| `module` | `extract` | Extract built archives into configured source trees. |
|
|
| `module` | `validate` | Validate project layout, config, and source inventory. |
|
|
| `module` | `compare` | Compare source content with built archives. |
|
|
| `module` | `manifest` | Apply a generated HAK list to module source. |
|
|
| `topdata` | `validate` | Validate topdata source and canonical JSON compatibility. |
|
|
| `topdata` | `build` | Compile topdata and refresh the configured package. |
|
|
| `topdata` | `package` | Package cached outputs into HAK and TLK files. |
|
|
| `topdata` | `compare` | Compare current output with a fresh native build. |
|
|
| `topdata` | `convert` | Convert between 2DA and native JSON/module formats. |
|
|
| `wiki` | `build` | Render wiki page drafts from compiled topdata. |
|
|
| `wiki` | `deploy` | Deploy generated wiki pages to NodeBB. |
|
|
| `assets` | `compile` | Compile ASCII `.mdl` models to binary in place. |
|
|
| `assets` | `convert` | Convert textures to/from NWN DDS (flips vertically). |
|
|
| `assets` | `upscale` | Upscale textures through an installed backend. |
|
|
| `assets` | `check-mdl` | Report uncompiled ASCII `.mdl` and model-name mismatches. |
|
|
| `assets` | `fix-mdl` | Lowercase `.mdl` names and rewrite model identity to match. |
|
|
| `assets` | `check-dupes` | Report runtime-name (basename) collisions across dirs. |
|
|
| `assets` | `clean-dupes` | Delete clean-tree files whose basename collides with primary. |
|
|
| `depot` | `status` | Report referenced-vs-present drift against a backend. |
|
|
| `depot` | `push` | Upload referenced-but-absent blobs from a local depot. |
|
|
| `depot` | `verify` | Existence sweep plus sampled download re-hash. |
|
|
| `depot` | `get` | Fetch one blob with sha re-verify. |
|
|
| `depot` | `pull` | Incremental verified pull of every referenced blob. |
|
|
| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
|
|
| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
|
|
| `nwsync` | `verify` | Decompress and hash a published manifest's blobs through the pull zone. |
|
|
|
|
`depot status` and `depot get` pick their backend either with `--out DIR`, a
|
|
depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
|
|
flags are mutually exclusive.
|
|
|
|
`nwsync emit` runs where an artifact is born (a `.hak`/`.erf`, or a loose file
|
|
such as the TLK); `nwsync assemble` runs at module release and reads only the
|
|
small per-artifact indexes. Both take **depot keys**: an artifact's index lives
|
|
beside the artifact itself with the extension replaced, so `emit` and
|
|
`assemble` agree on where it is without being told.
|
|
|
|
```
|
|
nwsync emit [--as NAME] [--out DIR] [--jobs N] [--verify] <artifact-key> <file>
|
|
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
|
|
nwsync verify [--sample N] [--base URL] [--jobs N] <manifest-sha1>
|
|
```
|
|
|
|
`emit` is latency-bound, not CPU-bound: every blob costs an existence probe
|
|
plus an upload, and a measured backfill spent 26 seconds of CPU across 9.5
|
|
minutes of wall clock. `--jobs N` (default 16) sets how many resources are in
|
|
flight at once. The manifest is byte-identical at any value — the number of
|
|
workers is never observable in the output. Peak memory is `N` times the
|
|
per-resource limit of 15 MB plus its compressed copy, so raising `N` far past
|
|
the default costs real memory for little gain: the transport keeps 16 idle
|
|
connections per host, and past that a worker pays a fresh TLS handshake.
|
|
|
|
Both verbs upload by default; nothing bulky is ever written to the runner's
|
|
disk. `--out DIR` writes a local repository tree instead, which is the
|
|
conformance path against upstream `nwn_nwsync_write`. The zone comes from
|
|
`NWSYNC_STORAGE_ZONE` and `NWSYNC_STORAGE_PASSWORD`, with the host from
|
|
`BUNNY_STORAGE_HOST` — NWSync data is a separate zone from the asset depot.
|
|
|
|
`assemble`'s artifact keys are in `Mod_HakList` order, highest priority first: a
|
|
resref in more than one artifact resolves to the earliest one, the way the game
|
|
resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
|
|
`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
|
|
out of the sidecar.
|
|
|
|
`nwsync verify` is the only check on a published blob upstream of a player's
|
|
client. It reads the **pull zone**, not the storage API, and needs no
|
|
credential: what matters is the bytes a client is served, edge behaviour
|
|
included. Every blob is decompressed and hashed, and the zstd frame is asserted
|
|
to declare its content size. Neither half is optional — a `Content-Length` check
|
|
passes a byte-correct-looking object whose contents are short, and a round-trip
|
|
check alone passes a frame the game client cannot decode but Go's decoder can.
|
|
Failures are reported per blob as missing, malformed framing, size mismatch or
|
|
hash mismatch, and the exit code is 1.
|
|
|
|
A full sweep of the live manifest is roughly 69,000 blobs and 15 GB, so
|
|
`--sample N` exists to make verifying routine; the default is a full sweep.
|
|
`--base URL` (or `NWSYNC_PULL_BASE`) overrides the public host.
|
|
|
|
`emit --verify` applies the same check where `emit` would otherwise skip. `emit`
|
|
normally reads a blob's presence as proof of its contents, decided by a 1-byte
|
|
range GET, so an object written truncated — or written by an emitter since found
|
|
broken — is skipped by every later run forever and no backfill repairs it. With
|
|
`--verify` the stored copy is read back, unwrapped, hashed against its own name,
|
|
and replaced when it does not match. It costs a full GET per existing blob, so
|
|
it is a repair pass, not the default.
|
|
|
|
`emit` uploads blobs first and the index last, so the presence of an index is
|
|
the publication marker: an artifact whose emit died halfway leaves real blobs in
|
|
the zone and no index. Blob names are content hashes, so re-running skips
|
|
whatever already landed, and `assemble` fails closed on an artifact with no
|
|
index rather than publishing a manifest that is missing a hak.
|
|
|
|
### What an emitted tree looks like
|
|
|
|
`--out DIR` produces the same tree `emit` would upload, which makes it the way
|
|
to check a zone by hand without touching one:
|
|
|
|
```
|
|
<artifact-sha>.nsym binary index
|
|
<artifact-sha>.nsym.json the same index, readable
|
|
data/sha1/a7/4a/a74aa84a... one blob per resource, two-level fanout
|
|
```
|
|
|
|
A blob's name is the SHA-1 of the resource's **original** bytes, but the file on
|
|
disk is not those bytes: each blob is wrapped in NWCompressedBuffer framing, a
|
|
24-byte `NSYC` header followed by a zstd frame. Hashing the file directly will
|
|
not match its name, which is the obvious
|
|
first thing to try and the obvious first thing to be confused by. Strip the
|
|
header first:
|
|
|
|
```
|
|
tail -c +25 <blob> | zstd -dc | sha1sum # == the blob's filename
|
|
```
|
|
|
|
The header carries the uncompressed length as a little-endian `uint32` at offset
|
|
12, so the decompressed size is checkable without decompressing. Compression is
|
|
worth roughly a 4:1
|
|
saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
|
|
against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
|
|
`total_bytes`.
|
|
|
|
The zstd frame always declares its `Frame_Content_Size`. The game client sizes
|
|
its output buffer from that field and cannot decode a frame without one, but the
|
|
Go encoder omits it below 256 bytes, so `emit` re-headers those frames into the
|
|
shape reference libzstd emits: `Single_Segment_flag` set, `Window_Descriptor`
|
|
dropped, and a one-byte content size in its place. `zstd -l <frame>` must print a
|
|
decompressed size; a blank column there is the fault, and it is invisible to any
|
|
check that only decompresses, because both `zstd -dc` and Go's decoder stream
|
|
such a frame happily. This is what the sidecar's `emitter_version` counts:
|
|
version 1 omitted the field and no client could sync past such a blob, version 2
|
|
declares it. `assemble` refuses to merge indexes that disagree.
|
|
|
|
## Hidden compatibility aliases
|
|
|
|
Existing scripts may continue using these names indefinitely. They are accepted
|
|
but omitted from routine help and the interactive menu:
|
|
|
|
| Builder | Hidden alias | Implementation |
|
|
| --- | --- | --- |
|
|
| `hak` | `build-haks` | `build-haks` |
|
|
| `hak` | `apply-hak-manifest` | `apply-hak-manifest` |
|
|
| `module` | `build-module` | module-only `build-module` |
|
|
| `module` | `apply-hak-manifest` | `apply-hak-manifest` |
|
|
| `topdata` | `validate-topdata` | `validate-topdata` |
|
|
| `topdata` | `build-topdata` | `build-topdata` |
|
|
| `topdata` | `build-top-package` | `build-top-package` |
|
|
| `topdata` | `compare-topdata` | `compare-topdata` |
|
|
| `topdata` | `convert-topdata` | `convert-topdata` |
|
|
| `depot` | `--target local` | `--out $DEPOT_DIR` on `status`/`get` |
|
|
| `wiki` | `build-wiki` | `build-wiki` |
|
|
| `wiki` | `deploy-wiki` | `deploy-wiki` |
|
|
|
|
`module build-module` deliberately retains its narrower module-only behavior;
|
|
it is not redirected to the broader `module build`.
|
|
|
|
## Global commands
|
|
|
|
```text
|
|
crucible help | -h usage + builder list
|
|
crucible version | -V build version
|
|
crucible list name<TAB>binary<TAB>summary
|
|
crucible config [args] inspect and validate effective configuration
|
|
crucible changelog [args] generate a release changelog
|
|
```
|
|
|
|
## HAK build options
|
|
|
|
```text
|
|
crucible hak build
|
|
[--hak <hak-name> ...]
|
|
[--archive <archive-name> ...]
|
|
[--source-manifest <path>]
|
|
[--content-addressed-root <path>]
|
|
[--plan-only]
|
|
[--quiet|--verbose|--debug]
|
|
```
|
|
|
|
When a source manifest contains `asset_sources`, pass
|
|
`--content-addressed-root <directory>`. Crucible resolves each declared SHA-256
|
|
under `sha256/<first-two>/<next-two>/<full-sha256>` and verifies streamed bytes
|
|
while writing selected HAK archives. Authored `.bmu` files are ordinary assets
|
|
and require no conversion pipeline.
|