ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

crucible nwsync: emit blobs and per-artifact NSYM, assemble the merged manifest #53

New Issue
Closed
opened 2026-07-27 16:27:18 +00:00 by archvillainette · 3 comments
archvillainette commented 2026-07-27 16:27:18 +00:00
Owner
Copy Link
Copy Source

Blocks the NWN go-live pipeline designed in sow-platform#94. Read that ticket's resolution comment first — it holds the byte-level format spec and the reasoning; this issue is the build.

Upstream nwn_nwsync_write cannot be used: it wants every hak, the TLK and the module present in one run, on one disk, and writes to a local directory it also walks to deduplicate. ovh-main has ~13 GB free against a 15.27 GB input set, and the Bunny zone cannot be mounted. So the work splits — blobs are emitted where each artifact is born, and the manifest is assembled later from small per-artifact index files. Crucible owns builder logic (root AGENTS.md), crucible-hak already reads ERFs, so this is two new verbs, not a new package and not a flake input.

crucible nwsync emit <artifact> --out <dir>

Input is one .hak/.erf or one loose file (the TLK). Output:

  • one NWSync blob per resource — path data/sha1/<h0h1>/<h2h3>/<sha1>, body in NWCompressedBuffer framing with magic NSYC, zstd;
  • one NSYM v3 manifest describing only that artifact, plus its .json sidecar carrying total_files, total_bytes, on_disk_bytes and the emitter version.

Rules taken from upstream and not to be re-invented:

  • skip restypes nss, ndb, gic always;
  • an unresolvable restype is a hard error, not a skip;
  • fail closed if any resource exceeds 15 MiB (upstream's --limit-file-size default; our largest today is 13.66 MiB);
  • the sha1 that names a blob is over the uncompressed bytes;
  • no latest file, no .origin file, ever.

crucible nwsync assemble --order <names> --entries <dir> --out <dir>

Reads the per-artifact NSYM files, merges them into one manifest, writes it plus its .json sidecar. It reads no bulk data at all.

Merge rule — resref shadowing, not concatenation. 13 resrefs already exist in two haks (bloodstain1.tga and friends, in sow_core_01 and sow_over_01), and sow_top's 2DAs shadow assets' 2DAs. The winner is the earliest hak in module_haks order, which is how the game resolves it and what upstream's resman reproduces by adding haks in reverse and letting the last one win. Get this backwards and the wrong texture ships with no error anywhere.

The sidecar takes group_id from the caller: per channel, current=1, testing=2. Upstream omits an integer meta field whose value is 0, so 0 means absent.

Conformance

Upstream is the oracle even though it never runs in production. On a developer machine with ~30 GB spare, run the real nwn_nwsync_write over the same inputs and compare:

  • manifests byte for byte — the format is fully deterministic (entries sorted by sha1 hex then resref, repeated sha1s demoted to mappings);
  • blobs after decompression, not before — zstd library and level differ between Nim's libzstd binding and Go's, which changes the compressed bytes but never the blob name.

A single-hak emit is directly comparable against upstream run on that one hak, which is what makes the split testable at all.

Acceptance

A real NWN client syncs a character in from a manifest assemble wrote, served from the Bunny zone. This is the gate #94 could not close by argument, and it is the reason this issue exists.

Notes

  • Manifest scale: 71,436 distinct resrefs, ~42 bytes an entry, so ~3.0 MB fetched once per client. Per-artifact NSYM files sum to the same, roughly 1 MB zstd resident on the zone.
  • on_disk_bytes in the merged sidecar is summed from the per-artifact sidecars, so it overcounts by the handful of cross-hak duplicates. It is a display statistic; do not build a dedupe pass for it.
  • Record the emitter version in every sidecar. assemble refuses to merge across mismatched versions — two producers of blobs means a skewed emitter can otherwise write blobs the manifest quietly disagrees with.
Blocks the NWN go-live pipeline designed in [sow-platform#94](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/94). Read that ticket's resolution comment first — it holds the byte-level format spec and the reasoning; this issue is the build. Upstream `nwn_nwsync_write` cannot be used: it wants every hak, the TLK and the module present in one run, on one disk, and writes to a local directory it also walks to deduplicate. `ovh-main` has ~13 GB free against a 15.27 GB input set, and the Bunny zone cannot be mounted. So the work splits — blobs are emitted where each artifact is born, and the manifest is assembled later from small per-artifact index files. Crucible owns builder logic (root `AGENTS.md`), `crucible-hak` already reads ERFs, so this is two new verbs, not a new package and not a flake input. ## `crucible nwsync emit <artifact> --out <dir>` Input is one `.hak`/`.erf` or one loose file (the TLK). Output: - one NWSync blob per resource — path `data/sha1/<h0h1>/<h2h3>/<sha1>`, body in NWCompressedBuffer framing with magic `NSYC`, zstd; - one NSYM v3 manifest describing **only that artifact**, plus its `.json` sidecar carrying `total_files`, `total_bytes`, `on_disk_bytes` and the emitter version. Rules taken from upstream and not to be re-invented: - skip restypes `nss`, `ndb`, `gic` always; - an unresolvable restype is a hard error, not a skip; - fail closed if any resource exceeds 15 MiB (upstream's `--limit-file-size` default; our largest today is 13.66 MiB); - the sha1 that names a blob is over the **uncompressed** bytes; - no `latest` file, no `.origin` file, ever. ## `crucible nwsync assemble --order <names> --entries <dir> --out <dir>` Reads the per-artifact NSYM files, merges them into one manifest, writes it plus its `.json` sidecar. It reads no bulk data at all. Merge rule — **resref shadowing, not concatenation**. 13 resrefs already exist in two haks (`bloodstain1.tga` and friends, in `sow_core_01` and `sow_over_01`), and `sow_top`'s 2DAs shadow assets' 2DAs. The winner is the earliest hak in `module_haks` order, which is how the game resolves it and what upstream's resman reproduces by adding haks in reverse and letting the last one win. Get this backwards and the wrong texture ships with no error anywhere. The sidecar takes `group_id` from the caller: **per channel**, `current`=1, `testing`=2. Upstream omits an integer meta field whose value is 0, so 0 means absent. ## Conformance Upstream is the oracle even though it never runs in production. On a developer machine with ~30 GB spare, run the real `nwn_nwsync_write` over the same inputs and compare: - manifests **byte for byte** — the format is fully deterministic (entries sorted by sha1 hex then resref, repeated sha1s demoted to mappings); - blobs **after decompression**, not before — zstd library and level differ between Nim's libzstd binding and Go's, which changes the compressed bytes but never the blob name. A single-hak `emit` is directly comparable against upstream run on that one hak, which is what makes the split testable at all. ## Acceptance A real NWN client syncs a character in from a manifest `assemble` wrote, served from the Bunny zone. This is the gate #94 could not close by argument, and it is the reason this issue exists. ## Notes - Manifest scale: 71,436 distinct resrefs, ~42 bytes an entry, so ~3.0 MB fetched once per client. Per-artifact NSYM files sum to the same, roughly 1 MB zstd resident on the zone. - `on_disk_bytes` in the merged sidecar is summed from the per-artifact sidecars, so it overcounts by the handful of cross-hak duplicates. It is a display statistic; do not build a dedupe pass for it. - Record the emitter version in every sidecar. `assemble` refuses to merge across mismatched versions — two producers of blobs means a skewed emitter can otherwise write blobs the manifest quietly disagrees with.
archvillainette commented 2026-07-29 10:18:01 +00:00
Author
Owner
Copy Link
Copy Source

PR up: #71 — crucible nwsync emit and crucible nwsync assemble, format read from niv/neverwinter.nim at HEAD rather than inferred.

Two departures from the letter of the brief, both explained in the PR:

  • Emitter version is its own sidecar field, not the build revision. The brief says "assemble refuses to merge across mismatched versions" and the only version upstream records is created_with. Keyed on that, every unrelated crucible commit would invalidate every published index and force a re-emit of the whole corpus — the opposite of "nothing downstream ever needs the hak again". emitter_version is a constant bumped only when emitted bytes change.
  • SOURCE_DATE_EPOCH pins the sidecar timestamp, so a build reproduces byte for byte. The manifest was already deterministic.

Still open on this issue, deliberately not in the PR:

  • The conformance run. The format was read from upstream source, not from its output; the tests check the layout against the spec, so a shared misreading would pass them. #59 is what closes that gap, and until it does nothing here is proven.
  • The direct upload sink and the mid-hak failure question (#60); only the local --out tree exists, which is the conformance path.
  • The acceptance gate — a real client syncing from an assembled manifest — is untouched.
PR up: https://git.westgate.pw/ShadowsOverWestgate/sow-tools/pulls/71 — `crucible nwsync emit` and `crucible nwsync assemble`, format read from niv/neverwinter.nim at HEAD rather than inferred. Two departures from the letter of the brief, both explained in the PR: - **Emitter version is its own sidecar field**, not the build revision. The brief says "assemble refuses to merge across mismatched versions" and the only version upstream records is `created_with`. Keyed on that, every unrelated crucible commit would invalidate every published index and force a re-emit of the whole corpus — the opposite of "nothing downstream ever needs the hak again". `emitter_version` is a constant bumped only when emitted bytes change. - **`SOURCE_DATE_EPOCH` pins the sidecar timestamp**, so a build reproduces byte for byte. The manifest was already deterministic. Still open on this issue, deliberately not in the PR: - The conformance run. The format was read from upstream *source*, not from its *output*; the tests check the layout against the spec, so a shared misreading would pass them. #59 is what closes that gap, and until it does nothing here is proven. - The direct upload sink and the mid-hak failure question (#60); only the local `--out` tree exists, which is the conformance path. - The acceptance gate — a real client syncing from an assembled manifest — is untouched.
archvillainette commented 2026-07-30 17:30:52 +00:00
Author
Owner
Copy Link
Copy Source

Verification sweep (map #54, under sow-platform#80)

The build half of this brief landed in PR #71 (a131b25): internal/nwsync/{emit,assemble,compressedbuf,manifest,run}.go, cmd/crucible-nwsync, 12 unit tests. Blob framing, blob path, NSYM v3 + sidecar, resref shadowing by order, nss/ndb/gic skip, hard error on unresolvable restype, 15 MB fail-closed, no latest/.origin, .mod refused — all present. zstd is github.com/klauspost/compress, now a third module dependency.

This body's CLI surface is superseded and should not be built from. It was written before #56, #62 and #65 settled the interface, and PR #71 implemented this body rather than those. Current drift, all in internal/nwsync/run.go:46-104:

Shipped Settled
emit <artifact> --out DIR (--out required) emit <artifact-key> <file>, upload is the default sink, --out DIR is the conformance path only (#56, #65)
assemble --order NAMES --entries DIR --out DIR assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>... — keys positional, order is Mod_HakList order, per-artifact NSYM read from its sibling key on the zone (#56, #62)
emit writes <out>/<name>.nsym NSYM key derived by Crucible from the artifact key, fail-closed digest check (#62)

Closing that gap is the remaining build work and now sits inside #60, together with the upload sink itself. Conformance against upstream is still #59 — the format was read from upstream source, and the tests check the code against the same reading, so a shared misreading passes today.

Also: this body opens "blocks the pipeline designed in sow-platform#94". #94's assemble-runs-in-the-resolver claim is wrong; assemble runs in sow-module release CI (#72 corrects the record).

## Verification sweep (map #54, under sow-platform#80) The build half of this brief **landed** in PR #71 (`a131b25`): `internal/nwsync/{emit,assemble,compressedbuf,manifest,run}.go`, `cmd/crucible-nwsync`, 12 unit tests. Blob framing, blob path, NSYM v3 + sidecar, resref shadowing by order, `nss`/`ndb`/`gic` skip, hard error on unresolvable restype, 15 MB fail-closed, no `latest`/`.origin`, `.mod` refused — all present. zstd is `github.com/klauspost/compress`, now a third module dependency. **This body's CLI surface is superseded and should not be built from.** It was written before #56, #62 and #65 settled the interface, and PR #71 implemented *this* body rather than those. Current drift, all in `internal/nwsync/run.go:46-104`: | Shipped | Settled | |---|---| | `emit <artifact> --out DIR` (`--out` required) | `emit <artifact-key> <file>`, upload is the default sink, `--out DIR` is the conformance path only (#56, #65) | | `assemble --order NAMES --entries DIR --out DIR` | `assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...` — keys positional, order **is** `Mod_HakList` order, per-artifact NSYM read from its sibling key on the zone (#56, #62) | | emit writes `<out>/<name>.nsym` | NSYM key derived by Crucible from the artifact key, fail-closed digest check (#62) | Closing that gap is the remaining build work and now sits inside #60, together with the upload sink itself. Conformance against upstream is still #59 — the format was read from upstream *source*, and the tests check the code against the same reading, so a shared misreading passes today. Also: this body opens "blocks the pipeline designed in sow-platform#94". #94's assemble-runs-in-the-resolver claim is wrong; assemble runs in sow-module release CI (#72 corrects the record).
archvillainette commented 2026-07-30 22:04:19 +00:00
Author
Owner
Copy Link
Copy Source

Resolved — the build is done and proven; the client gate moves to #75

Re-verified against main at f9051a2, clean tree, no open PRs.

Build. PR #71 (a131b25) landed internal/nwsync/{emit,assemble,compressedbuf,manifest,run,sink}.go plus cmd/crucible-nwsync. Every rule in this body is present: NSYC + zstd blob framing, data/sha1/<h0h1>/<h2h3>/<sha1> paths, sha1 over uncompressed bytes, NSYM v3 with its .json sidecar, resref shadowing by module_haks order, nss/ndb/gic skipped, unresolvable restype a hard error, 15 MiB fail-closed, no latest and no .origin.

CLI. The command surface written in this body is superseded, as the 2026-07-30 sweep noted. PR #73 (f9051a2) closed that drift against what #56, #62 and #65 settled:

nwsync emit     [--as NAME] [--out DIR] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...

Keys are positional, positional order is Mod_HakList order, upload is the default sink, and --out DIR is the conformance path only.

Conformance. #59 closed it. Upstream nwn_nwsync_write 2.1.2 ran over sow_vfxs_01.hak: manifest byte-identical (cc577365edc6bda2ce7b0f7f1896482fe3478572), 2,296 blobs with identical names and paths, blob headers identical, payloads identical after decompression, and sha1(uncompressed) == blob name on all of them. Compressed bytes differ by +1.6% between Go's zstd and Nim's libzstd, which does not touch a blob name. The "shared misreading" risk this body raised is retired.

go test ./internal/nwsync/... is green.

What is not done. The acceptance gate — a real NWN client syncing a character from an assembled manifest served off the Bunny zone. It needs a published zone and a game client, so it cannot close from this repo's test suite, and it does not belong to the build work this issue tracks. Split to #75 with the duplicate-resref spot check spelled out.

Closing.

## Resolved — the build is done and proven; the client gate moves to #75 Re-verified against `main` at `f9051a2`, clean tree, no open PRs. **Build.** PR #71 (`a131b25`) landed `internal/nwsync/{emit,assemble,compressedbuf,manifest,run,sink}.go` plus `cmd/crucible-nwsync`. Every rule in this body is present: NSYC + zstd blob framing, `data/sha1/<h0h1>/<h2h3>/<sha1>` paths, sha1 over uncompressed bytes, NSYM v3 with its `.json` sidecar, resref shadowing by `module_haks` order, `nss`/`ndb`/`gic` skipped, unresolvable restype a hard error, 15 MiB fail-closed, no `latest` and no `.origin`. **CLI.** The command surface written in this body is superseded, as the 2026-07-30 sweep noted. PR #73 (`f9051a2`) closed that drift against what #56, #62 and #65 settled: nwsync emit [--as NAME] [--out DIR] <artifact-key> <file> nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>... Keys are positional, positional order **is** `Mod_HakList` order, upload is the default sink, and `--out DIR` is the conformance path only. **Conformance.** #59 closed it. Upstream `nwn_nwsync_write` 2.1.2 ran over `sow_vfxs_01.hak`: manifest byte-identical (`cc577365edc6bda2ce7b0f7f1896482fe3478572`), 2,296 blobs with identical names and paths, blob headers identical, payloads identical after decompression, and `sha1(uncompressed) == blob name` on all of them. Compressed bytes differ by +1.6% between Go's zstd and Nim's libzstd, which does not touch a blob name. The "shared misreading" risk this body raised is retired. `go test ./internal/nwsync/...` is green. **What is not done.** The acceptance gate — a real NWN client syncing a character from an assembled manifest served off the Bunny zone. It needs a published zone and a game client, so it cannot close from this repo's test suite, and it does not belong to the build work this issue tracks. Split to #75 with the duplicate-resref spot check spelled out. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Chass FallenDabus WinterZen XeroTwo Zariu archvillainette gitea-bot xtul
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ShadowsOverWestgate/sow-tools#53