Builds sow-tools#53. Format spec followed is the resolution comment of sow-platform#94, checked line by line against niv/neverwinter.nim at HEAD (nwsync.nim, compressedbuf.nim, nwsync/private/libupdate.nim).
What lands
crucible nwsync emit <artifact> --out DIR — explodes one .hak/.erf, or one loose file such as the TLK, into NWSync blobs plus a NSYM v3 manifest covering only that artifact, with the same .json sidecar upstream writes. Blob path data/sha1/<h0h1>/<h2h3>/<sha1>, body in NWCompressedBuffer framing (magic NSYC, version 3, algorithm 2, uncompressed size, zstd header version 1, dictionary 0, raw zstd frame). The sha1 that names a blob is over the uncompressed bytes.
crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N] — merges the per-artifact manifests into one, reading no bulk data at all. Merge rule is resref shadowing, not concatenation: a resref in more than one artifact resolves to the earliest artifact in --order, which is how the game resolves it. --group-id stays caller-supplied (1 current, 2 testing; 0 is absent, matching upstream omitting a zero integer meta field).
Rules taken from upstream and not re-invented: nss/ndb/gic always skipped; an unresolvable restype is a hard error, not a skip; a resource over 15 MB fails closed; no latest file and no .origin file, ever. A .mod is refused outright — a persistent world publishes no module contents, so the module contributes no bytes.
Two deliberate departures
Emitter version is its own field, not the build revision.emitter_version is a constant bumped only when emitted bytes change. Keying the refuse-to-merge check on created_with would invalidate every published index on every unrelated crucible commit and force a re-emit of the whole 15 GB corpus — the opposite of "nothing downstream ever needs the hak again".
SOURCE_DATE_EPOCH pins the sidecar timestamp. The manifest itself was already deterministic; the sidecar's created was not, against the determinism rule in docs/consumer-contract.md.
Not in this PR, and why
Direct upload. Only the local --out sink exists, which is the conformance path. The upload sink and the mid-hak-failure question are sow-tools#60, and the consumer wiring is #65.
The conformance run against upstream. sow-tools#59 owns getting nwn_nwsync_write running and capturing reference output. The format here was read from upstream source rather than from its output, so the byte-for-byte manifest comparison and the after-decompression blob comparison still have to happen — that is what #59 is for. The tests in this PR check the layout against the spec, so a shared misreading would pass them.
The artifacts/haks/sha256/<a>/<b>/<sha256>.nsym location. emit writes <out>/<name>.nsym; where a publisher puts it is the publisher's business (#65).
The acceptance gate — a real client syncing from an assembled manifest — is unchanged and still open.
Checks
make check green (vet, unit tests, shellcheck, yamllint, workflow contract), make smoke green with the new builder, nix build .#crucible produces crucible-nwsync.
Builds sow-tools#53. Format spec followed is the resolution comment of sow-platform#94, checked line by line against niv/neverwinter.nim at HEAD (`nwsync.nim`, `compressedbuf.nim`, `nwsync/private/libupdate.nim`).
## What lands
`crucible nwsync emit <artifact> --out DIR` — explodes one `.hak`/`.erf`, or one loose file such as the TLK, into NWSync blobs plus a NSYM v3 manifest covering only that artifact, with the same `.json` sidecar upstream writes. Blob path `data/sha1/<h0h1>/<h2h3>/<sha1>`, body in NWCompressedBuffer framing (magic `NSYC`, version 3, algorithm 2, uncompressed size, zstd header version 1, dictionary 0, raw zstd frame). The sha1 that names a blob is over the uncompressed bytes.
`crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]` — merges the per-artifact manifests into one, reading no bulk data at all. Merge rule is resref shadowing, not concatenation: a resref in more than one artifact resolves to the earliest artifact in `--order`, which is how the game resolves it. `--group-id` stays caller-supplied (1 current, 2 testing; 0 is absent, matching upstream omitting a zero integer meta field).
Rules taken from upstream and not re-invented: `nss`/`ndb`/`gic` always skipped; an unresolvable restype is a hard error, not a skip; a resource over 15 MB fails closed; no `latest` file and no `.origin` file, ever. A `.mod` is refused outright — a persistent world publishes no module contents, so the module contributes no bytes.
## Two deliberate departures
- **Emitter version is its own field, not the build revision.** `emitter_version` is a constant bumped only when emitted bytes change. Keying the refuse-to-merge check on `created_with` would invalidate every published index on every unrelated crucible commit and force a re-emit of the whole 15 GB corpus — the opposite of "nothing downstream ever needs the hak again".
- **`SOURCE_DATE_EPOCH` pins the sidecar timestamp.** The manifest itself was already deterministic; the sidecar's `created` was not, against the determinism rule in `docs/consumer-contract.md`.
## Not in this PR, and why
- **Direct upload.** Only the local `--out` sink exists, which is the conformance path. The upload sink and the mid-hak-failure question are sow-tools#60, and the consumer wiring is #65.
- **The conformance run against upstream.** sow-tools#59 owns getting `nwn_nwsync_write` running and capturing reference output. The format here was read from upstream source rather than from its output, so the byte-for-byte manifest comparison and the after-decompression blob comparison still have to happen — that is what #59 is for. The tests in this PR check the layout against the spec, so a shared misreading would pass them.
- **The `artifacts/haks/sha256/<a>/<b>/<sha256>.nsym` location.** emit writes `<out>/<name>.nsym`; where a publisher puts it is the publisher's business (#65).
- **The acceptance gate** — a real client syncing from an assembled manifest — is unchanged and still open.
## Checks
`make check` green (vet, unit tests, shellcheck, yamllint, workflow contract), `make smoke` green with the new builder, `nix build .#crucible` produces `crucible-nwsync`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Adds `crucible nwsync emit` and `crucible nwsync assemble`, the split
replacement for upstream nwn_nwsync_write, which cannot be used because it
wants every hak, the TLK and the module present in one run on one disk.
emit explodes one artifact — a .hak/.erf or a loose file such as the TLK —
into NWSync blobs (NWCompressedBuffer framing, magic NSYC, zstd) plus a NSYM
v3 manifest describing only that artifact, with the same .json sidecar
upstream writes. Blob names are the sha1 of the uncompressed bytes, restypes
nss/ndb/gic are always skipped, an unresolvable restype is a hard error, a
resource over 15 MB fails closed, and no latest or .origin file is ever
written.
assemble merges the per-artifact manifests into one, reading no bulk data.
The merge rule is resref shadowing, not concatenation: a resref present in
more than one artifact resolves to the earliest artifact in --order, the way
the game resolves it. It refuses to merge across mismatched emitter versions,
and takes --group-id from the caller (1 current, 2 testing; 0 is absent).
Refs #53.
Review follow-ups on the same work:
- emitter_version is its own sidecar field, not the build revision. Keying
the merge check on created_with invalidated every published index on every
unrelated crucible commit, forcing a re-emit of the whole corpus.
- flake.nix builds cmd/crucible-nwsync, so Nix consumers get the binary the
docs promise.
- readManifest uses io.ReadFull: bytes.Reader.Read can return a short count
with no error, so a truncated manifest parsed into zero-padded garbage.
- emit refuses a .mod outright and names why, rather than failing on an
unknown extension.
- SOURCE_DATE_EPOCH pins the sidecar timestamp; the manifest was already
deterministic.
- Entry.identity replaces the resref/restype key struct duplicated in emit
and assemble.
Refs #53.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
xtul
Builds sow-tools#53. Format spec followed is the resolution comment of sow-platform#94, checked line by line against niv/neverwinter.nim at HEAD (
nwsync.nim,compressedbuf.nim,nwsync/private/libupdate.nim).What lands
crucible nwsync emit <artifact> --out DIR— explodes one.hak/.erf, or one loose file such as the TLK, into NWSync blobs plus a NSYM v3 manifest covering only that artifact, with the same.jsonsidecar upstream writes. Blob pathdata/sha1/<h0h1>/<h2h3>/<sha1>, body in NWCompressedBuffer framing (magicNSYC, version 3, algorithm 2, uncompressed size, zstd header version 1, dictionary 0, raw zstd frame). The sha1 that names a blob is over the uncompressed bytes.crucible nwsync assemble --order NAMES --entries DIR --out DIR [--group-id N]— merges the per-artifact manifests into one, reading no bulk data at all. Merge rule is resref shadowing, not concatenation: a resref in more than one artifact resolves to the earliest artifact in--order, which is how the game resolves it.--group-idstays caller-supplied (1 current, 2 testing; 0 is absent, matching upstream omitting a zero integer meta field).Rules taken from upstream and not re-invented:
nss/ndb/gicalways skipped; an unresolvable restype is a hard error, not a skip; a resource over 15 MB fails closed; nolatestfile and no.originfile, ever. A.modis refused outright — a persistent world publishes no module contents, so the module contributes no bytes.Two deliberate departures
emitter_versionis a constant bumped only when emitted bytes change. Keying the refuse-to-merge check oncreated_withwould invalidate every published index on every unrelated crucible commit and force a re-emit of the whole 15 GB corpus — the opposite of "nothing downstream ever needs the hak again".SOURCE_DATE_EPOCHpins the sidecar timestamp. The manifest itself was already deterministic; the sidecar'screatedwas not, against the determinism rule indocs/consumer-contract.md.Not in this PR, and why
--outsink exists, which is the conformance path. The upload sink and the mid-hak-failure question are sow-tools#60, and the consumer wiring is #65.nwn_nwsync_writerunning and capturing reference output. The format here was read from upstream source rather than from its output, so the byte-for-byte manifest comparison and the after-decompression blob comparison still have to happen — that is what #59 is for. The tests in this PR check the layout against the spec, so a shared misreading would pass them.artifacts/haks/sha256/<a>/<b>/<sha256>.nsymlocation. emit writes<out>/<name>.nsym; where a publisher puts it is the publisher's business (#65).Checks
make checkgreen (vet, unit tests, shellcheck, yamllint, workflow contract),make smokegreen with the new builder,nix build .#crucibleproducescrucible-nwsync.🤖 Generated with Claude Code