ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

Run upstream nwn_nwsync_write and capture reference output for one real hak #59

New Issue
Closed
opened 2026-07-27 18:51:20 +00:00 by archvillainette · 2 comments
archvillainette commented 2026-07-27 18:51:20 +00:00
Owner
Copy Link
Copy Source

Question

Get upstream nwn_nwsync_write running on this machine and capture reference output for one real hak.

Part of #54. This is the conformance oracle for the whole effort, and right now it is a sentence in sow-platform#94 rather than a working binary. Nothing can be proven correct until it exists.

Done means:

  • nwn_nwsync_write runs locally. It is Nim, from niv/neverwinter.nim. This machine is NixOS — prefer nixpkgs or a transient nix shell; if it is not packaged, record what it actually took.
  • One real .hak from the corpus is on disk, pulled from the depot.
  • Upstream has been run over that single hak, and its output — the blob tree and the NSYM manifest plus sidecar — is stored somewhere durable enough for a byte comparison later.
  • The manifest is inspected against the format written down in #94, and any discrepancy between that spec and what the binary actually produces is reported. The spec was read from source, not from output; this is where it gets checked.

Record in the answer: how the binary was obtained, where the reference output lives, which hak was used and its digest.

## Question Get upstream `nwn_nwsync_write` running on this machine and capture reference output for one real hak. Part of #54. This is the conformance oracle for the whole effort, and right now it is a sentence in [sow-platform#94](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/94) rather than a working binary. Nothing can be proven correct until it exists. Done means: - `nwn_nwsync_write` runs locally. It is Nim, from `niv/neverwinter.nim`. This machine is NixOS — prefer nixpkgs or a transient `nix shell`; if it is not packaged, record what it actually took. - One real `.hak` from the corpus is on disk, pulled from the depot. - Upstream has been run over that single hak, and its output — the blob tree and the NSYM manifest plus sidecar — is stored somewhere durable enough for a byte comparison later. - The manifest is inspected against the format written down in #94, and any discrepancy between that spec and what the binary actually produces is reported. The spec was read from source, not from output; this is where it gets checked. Record in the answer: how the binary was obtained, where the reference output lives, which hak was used and its digest.
archvillainette added the wayfinder:task label 2026-07-27 18:51:20 +00:00
archvillainette commented 2026-07-30 17:31:25 +00:00
Author
Owner
Copy Link
Copy Source

Verification sweep — still valid, and now the highest-value open ticket on this map

Nothing here is stale. Confirmed against the repo: no upstream binary, no reference output, no conformance harness. grep -r nwn_nwsync_write finds one source comment and nothing executable.

PR #71 raises the stakes rather than lowering them. The format shipped in internal/nwsync was read from upstream source (nwsync.nim, compressedbuf.nim, libupdate.nim), and the 12 unit tests check the code against that same reading — as the PR body says, a shared misreading passes them. #59 is the only check that can catch one.

Two additions to "done means", both cheap now that emit exists:

  • Run crucible nwsync emit over the same hak and diff its NSYM against upstream's byte for byte, and its blobs after decompression. That is the whole point of the single-hak split and it is now runnable.
  • Feed specs to upstream reversed when comparing — #62 established module_haks is earliest-wins for us against upstream's last-added-wins.
## Verification sweep — still valid, and now the highest-value open ticket on this map Nothing here is stale. Confirmed against the repo: no upstream binary, no reference output, no conformance harness. `grep -r nwn_nwsync_write` finds one source comment and nothing executable. PR #71 raises the stakes rather than lowering them. The format shipped in `internal/nwsync` was read from upstream **source** (`nwsync.nim`, `compressedbuf.nim`, `libupdate.nim`), and the 12 unit tests check the code against that same reading — as the PR body says, a shared misreading passes them. #59 is the only check that can catch one. Two additions to "done means", both cheap now that emit exists: - Run `crucible nwsync emit` over the same hak and diff its NSYM against upstream's **byte for byte**, and its blobs after decompression. That is the whole point of the single-hak split and it is now runnable. - Feed specs to upstream **reversed** when comparing — #62 established `module_haks` is earliest-wins for us against upstream's last-added-wins.
archvillainette self-assigned this 2026-07-30 17:33:46 +00:00
archvillainette commented 2026-07-30 17:38:01 +00:00
Author
Owner
Copy Link
Copy Source

Resolved — upstream runs, reference output captured, and Crucible matches it byte for byte

How the binary was obtained

Not packaged in nixpkgs (nix search nixpkgs neverwinter finds nothing). Upstream ships prebuilt Linux binaries, so no Nim toolchain was needed:

curl -L https://github.com/niv/neverwinter.nim/releases/download/2.1.2/neverwinter-x86_64-linux-gnu.zip

It runs unpatched on NixOS — the interpreter resolves — but dlopens sqlite at startup and dies with could not load: libsqlite3.so(|.0). Fix is one env var, no patchelf, no FHS wrapper:

LD_LIBRARY_PATH=$(nix build --no-link --print-out-paths nixpkgs#sqlite.out)/lib ./nwn_nwsync_write ...

Version: neverwinter 2.1.2 (/07a475, nim 2.2.4) — the same tree the format was read from.

Where the reference output lives

/home/vicky/nwsync-ref/ on the dev machine (not in any repo — it is 300 MB of bulk):

  • bin/ — the unpacked upstream 2.1.2 toolset plus the crucible-nwsync built from a131b25
  • hak/sow_vfxs_01.hak — the input
  • out-upstream/ — upstream's repository: data/, manifests/cc577365edc6bda2ce7b0f7f1896482fe3478572 + .json, and latest
  • out-crucible/ — crucible nwsync emit --out over the same hak

Which hak

sow_vfxs_01.hak from the current haks release v0.1.9, sha256 521ab5de8e4a8f5569ff189fbf3a04c4f693e239afbf709c932ec2fbfa9f7831, 249,842,292 bytes — the smallest of the 11 published haks, so the loop is cheap to repeat. 2,474 resources, 2,296 distinct blobs.

The comparison — it passes

  • Manifest byte-identical. cmp clean against upstream's manifest, and Crucible's own name for it is the same SHA-1, cc577365edc6bda2ce7b0f7f1896482fe3478572. The entry sort, the mapping demotion of repeated SHA-1s, and the header all reproduce exactly.
  • 2,296 blobs, same names, same paths, data/sha1/<h0h1>/<h2h3>/<sha1>; diff of the two file lists is empty.
  • Blob headers byte-identical — all 20 bytes of NWCompressedBuffer framing before the zstd frame match on every blob, so magic, version, algorithm, uncompressed size, zstd header version and dictionary are all right.
  • Payloads identical after decompression on all 2,296, and sha1(uncompressed) == blob name on all 2,296.
  • Compressed bytes differ as predicted: 58,377,749 on disk upstream vs 59,315,391 for Crucible, +1.6% from Go's zstd against Nim's libzstd. The blob names are unaffected, which is the whole point.
  • Crucible writes no latest and no .origin; upstream writes latest. Correct — we serve an explicit hash.

Reproduce with crucible-nwsync emit --out out-crucible hak/sow_vfxs_01.hak and nwn_nwsync_write out-upstream hak/sow_vfxs_01.hak, then diff. Emit takes 1.8 s against upstream's ~90 s.

Discrepancies found, none fatal

  1. on_disk_bytes means something different upstream. Upstream's sidecar claims 61,359,113 while its own blob tree is 58,377,749; Crucible's claim equals its real tree exactly. The gap is ~5%, and 178 of the 2,474 entries are duplicate SHA-1s (7.2%) — upstream is almost certainly summing per entry, counting a shared blob once per resref, not per stored file. #53 already calls this a display statistic, so nothing to change, but the merged sidecar will be closer to upstream if it keeps counting per entry.
  2. module_name differs. Upstream leaves it empty for a bare hak spec; Crucible writes the artifact name (sow_vfxs_01). Sidecar-only, outside the hashed manifest — but the merged manifest's module_name is the module's, so the per-artifact one is ours to define. Fine as is; noting it so the next byte-diff does not read it as drift.
  3. created_with / emitter_version differ by design (PR #71).
  4. Flag order is positional-hostile. emit hak/x.hak --out DIR fails with "exactly one artifact is required" — Go's flag stops at the first non-flag argument. Cost me a run. Worth fixing in #60's CLI reshape, where the positionals change anyway.

So the format spec in sow-platform#94 is confirmed correct as read, on one real hak, at the byte level. What is not yet proven: multi-artifact assemble against upstream's multi-spec run (upstream's shadowing is last-added-wins, ours is earliest-wins, so specs feed reversed — per #62), and anything about the upload path (#60).

## Resolved — upstream runs, reference output captured, and Crucible matches it byte for byte ### How the binary was obtained Not packaged in nixpkgs (`nix search nixpkgs neverwinter` finds nothing). Upstream ships prebuilt Linux binaries, so no Nim toolchain was needed: ``` curl -L https://github.com/niv/neverwinter.nim/releases/download/2.1.2/neverwinter-x86_64-linux-gnu.zip ``` It runs unpatched on NixOS — the interpreter resolves — but `dlopen`s sqlite at startup and dies with `could not load: libsqlite3.so(|.0)`. Fix is one env var, no patchelf, no FHS wrapper: ``` LD_LIBRARY_PATH=$(nix build --no-link --print-out-paths nixpkgs#sqlite.out)/lib ./nwn_nwsync_write ... ``` Version: `neverwinter 2.1.2 (/07a475, nim 2.2.4)` — the same tree the format was read from. ### Where the reference output lives `/home/vicky/nwsync-ref/` on the dev machine (not in any repo — it is 300 MB of bulk): - `bin/` — the unpacked upstream 2.1.2 toolset plus the `crucible-nwsync` built from `a131b25` - `hak/sow_vfxs_01.hak` — the input - `out-upstream/` — upstream's repository: `data/`, `manifests/cc577365edc6bda2ce7b0f7f1896482fe3478572` + `.json`, and `latest` - `out-crucible/` — `crucible nwsync emit --out` over the same hak ### Which hak `sow_vfxs_01.hak` from the `current` haks release `v0.1.9`, sha256 `521ab5de8e4a8f5569ff189fbf3a04c4f693e239afbf709c932ec2fbfa9f7831`, 249,842,292 bytes — the smallest of the 11 published haks, so the loop is cheap to repeat. 2,474 resources, 2,296 distinct blobs. ### The comparison — it passes - **Manifest byte-identical.** `cmp` clean against upstream's manifest, and Crucible's own name for it is the same SHA-1, `cc577365edc6bda2ce7b0f7f1896482fe3478572`. The entry sort, the mapping demotion of repeated SHA-1s, and the header all reproduce exactly. - **2,296 blobs, same names, same paths**, `data/sha1/<h0h1>/<h2h3>/<sha1>`; `diff` of the two file lists is empty. - **Blob headers byte-identical** — all 20 bytes of NWCompressedBuffer framing before the zstd frame match on every blob, so magic, version, algorithm, uncompressed size, zstd header version and dictionary are all right. - **Payloads identical after decompression** on all 2,296, and `sha1(uncompressed) == blob name` on all 2,296. - Compressed bytes differ as predicted: 58,377,749 on disk upstream vs 59,315,391 for Crucible, +1.6% from Go's zstd against Nim's libzstd. The blob names are unaffected, which is the whole point. - Crucible writes no `latest` and no `.origin`; upstream writes `latest`. Correct — we serve an explicit hash. Reproduce with `crucible-nwsync emit --out out-crucible hak/sow_vfxs_01.hak` and `nwn_nwsync_write out-upstream hak/sow_vfxs_01.hak`, then diff. Emit takes 1.8 s against upstream's ~90 s. ### Discrepancies found, none fatal 1. **`on_disk_bytes` means something different upstream.** Upstream's sidecar claims 61,359,113 while its own blob tree is 58,377,749; Crucible's claim equals its real tree exactly. The gap is ~5%, and 178 of the 2,474 entries are duplicate SHA-1s (7.2%) — upstream is almost certainly summing per **entry**, counting a shared blob once per resref, not per stored file. #53 already calls this a display statistic, so nothing to change, but the merged sidecar will be closer to upstream if it keeps counting per entry. 2. **`module_name` differs.** Upstream leaves it empty for a bare hak spec; Crucible writes the artifact name (`sow_vfxs_01`). Sidecar-only, outside the hashed manifest — but the merged manifest's `module_name` is the module's, so the per-artifact one is ours to define. Fine as is; noting it so the next byte-diff does not read it as drift. 3. **`created_with` / `emitter_version`** differ by design (PR #71). 4. **Flag order is positional-hostile.** `emit hak/x.hak --out DIR` fails with "exactly one artifact is required" — Go's `flag` stops at the first non-flag argument. Cost me a run. Worth fixing in #60's CLI reshape, where the positionals change anyway. So the format spec in sow-platform#94 is confirmed correct as read, on one real hak, at the byte level. What is **not** yet proven: multi-artifact `assemble` against upstream's multi-spec run (upstream's shadowing is last-added-wins, ours is earliest-wins, so specs feed reversed — per #62), and anything about the upload path (#60).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No labels wayfinder:task
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Chass FallenDabus WinterZen XeroTwo Zariu archvillainette gitea-bot xtul
No Assignees archvillainette
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ShadowsOverWestgate/sow-tools#59