Where the per-artifact NSYM lives, now that topdata is not a hak #62

New Issue

2026-07-27T19:00:30Z

archvillainette commented

2026-07-27 19:00:30 +00:00

Question

Where does a per-artifact NSYM live, now that not every artifact is a hak?

Part of #54. sow-tools#53 and sow-platform#94 both put it at artifacts/haks/sha256/<a>//<sha256>.nsym — beside the artifact, derived from the artifact's own digest, so it is immutable, shared by every channel naming that artifact, needs no new index, and its presence is the publication marker.

#58 found that assumption is too narrow: topdata artifacts live under artifacts/topdata/sha256/..., not artifacts/haks/. sow_top.hak and sow_tlk.tlk are published by sow-topdata/scripts/publish-topdata-release.sh:58-66 into the topdata prefix, and both are manifest entries in their own right.

Settle:

Whether the NSYM path is derived from the artifact's own prefix — <whatever>/sha256/<a>//<sha>.nsym, per stream — or whether all NSYMs collect under one shared prefix keyed only by digest.
Whether emit computes the path at all, or whether it writes to a caller-supplied key and each publishing repo decides where.
The consequence for the assembler: how it derives the NSYM key for each artifact it must merge, given that the hak catalog and the topdata catalog use different prefixes.
Whether the TLK — not an ERF at all, a single loose file — changes anything about the path or the emit path itself.

## Question Where does a per-artifact NSYM live, now that not every artifact is a hak? Part of #54. [sow-tools#53](https://git.westgate.pw/ShadowsOverWestgate/sow-tools/issues/53) and [sow-platform#94](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/94) both put it at `artifacts/haks/sha256/<a>//<sha256>.nsym` — beside the artifact, derived from the artifact's own digest, so it is immutable, shared by every channel naming that artifact, needs no new index, and its presence *is* the publication marker. #58 found that assumption is too narrow: topdata artifacts live under `artifacts/topdata/sha256/...`, not `artifacts/haks/`. `sow_top.hak` and `sow_tlk.tlk` are published by `sow-topdata/scripts/publish-topdata-release.sh:58-66` into the topdata prefix, and both are manifest entries in their own right. Settle: - Whether the NSYM path is derived from the artifact's own prefix — `<whatever>/sha256/<a>//<sha>.nsym`, per stream — or whether all NSYMs collect under one shared prefix keyed only by digest. - Whether `emit` computes the path at all, or whether it writes to a caller-supplied key and each publishing repo decides where. - The consequence for the assembler: how it derives the NSYM key for each artifact it must merge, given that the hak catalog and the topdata catalog use different prefixes. - Whether the TLK — not an ERF at all, a single loose file — changes anything about the path or the emit path itself.

archvillainette added the wayfinder:grilling label 2026-07-27 19:00:30 +00:00

archvillainette referenced this issue

2026-07-27 19:01:14 +00:00

Map: NWSync publishing — emit at artifact birth, assemble at module release #54

archvillainette self-assigned this 2026-07-28 07:14:01 +00:00

archvillainette commented

2026-07-28 07:22:08 +00:00

Resolved: the NSYM is a sibling of its artifact, and `emit` derives its key

Settled with the four sub-questions taken in order. The short version: artifacts/<stream>/sha256/<a>//<sha256>.nsym, derived by Crucible from an artifact key the caller already holds.

1. Path rule: sibling, per stream

The NSYM sits beside its artifact, same prefix, extension replaced:

artifacts/haks/sha256/<a>//<sha256>.nsym — the 12 asset haks
artifacts/topdata/sha256/<a>//<sha256>.nsym — sow_top.hak and sow_tlk.tlk

Rejected: one shared artifacts/nsym/ prefix keyed only by digest. It was meant to spare the assembler from knowing which stream an entry came from, but the assembler never needed to know: both catalogs already publish a full url per entry (pack-haks.sh:145-147, publish-topdata-release.sh:60-66), so the key is derived by string-editing that url. A shared prefix buys nothing and splits ownership onto a prefix no repo owns.

Everything sow-platform#94 claimed for the sibling layout survives unchanged: immutable, shared by every channel naming the artifact, no new index, and its presence is the publication marker. Only the assumption that the prefix is always artifacts/haks/ was wrong.

2. Key derivation: replace the final extension

<prefix>/sha256/<a>//<sha256>.nsym, whatever the artifact's own suffix was. The digest is the whole identity, so <sha>.hak and <sha>.tlk can only collide on <sha>.nsym if they are byte-identical — in which case they are the same artifact with the same index.

Rejected: appending (<sha>.hak.nsym). It encodes the artifact type twice, for no reader that wants it, and would force the assembler to know an entry's type before it can build a key — reintroducing exactly the per-stream coupling item 1 removed.

The derivation, in full: take the catalog entry's url, cut at the 64-hex digest, append .nsym. Nothing else is parsed.

3. `emit` derives the key; it is not caller-supplied

emit --artifact-key artifacts/topdata/sha256/aa/bb/<sha>.hak and Crucible applies the item 2 rule itself.

The rule has two users that must agree — emit writing and assemble reading — so it lives in one place, and Crucible is the only place both see. A caller-supplied --nsym-key puts it in neither: two repos each hand-rolling a key while the assembler independently derives its own, with drift surfacing as a missing index at module release, far from the repo that caused it. Also rejected: a --prefix flag with Crucible hashing the file and building the shard path. That makes Crucible own the sha256/<a>// depot convention, which belongs to the publishing repos, and re-derives a digest the caller already computed.

Guard: emit verifies the 64-hex digest in --artifact-key equals the SHA-256 of the file it was handed, and fails closed on mismatch. One hash of bytes already being read; it is the only thing between a copy-paste slip and an index published under another artifact's name.

4. The TLK: one flag, not a second verb

The TLK is not an ERF — one loose file, so its NSYM is a single entry (resref sow_tlk, restype 0x07E2, SHA-1 and size of the whole file) and it emits exactly one blob. 0x07E2 is already in internal/erf/erf.go:89 and matches upstream, so no restype work.

emit --as sow_tlk.tlk <file> treats the input as one loose resource under that name; without --as, the input is parsed as an ERF and names come from inside it.

Rejected: sniffing the header and taking the resref from the local filename. The resref must be the published name, not whatever the runner called the temp file it downloaded — that gets the one thing that matters wrong, and ships a manifest entry named after a scratch path with no error anywhere. Also rejected: a separate emit-file verb, which duplicates the flag surface and the whole upload path for one extra input mode.

The path is unaffected (item 2 already made the extension irrelevant), and so is the emit path itself: blob naming, compression and the NSYM writer are identical either way. Only entry-gathering branches.

Verified against upstream while resolving

Two things that were assumptions in #94 are now read out of nwn_nwsync_write.nim and neverwinter/nwsync/private/libshared.nim:

A hak is a first-class <spec>; the .mod is only convenience. libshared.nim:38-54 sniffs the first three bytes of each spec: "ERF"/"HAK" → resman.add readErf(...); "MOD" → read module.ifo, resolve each hak by name off the search path, and add it with the same readErf call, for hak in reversed(haklist) (:74). The help text says the same: <spec> may be "any valid other erf container (HAK, ERF)" or "single files, including a TLK file". So running the writer on one hak is supported, and the per-hak conformance oracle #59 depends on is real, not hoped for — no 15 GB run needed to prove emit.

Order is inverted between the two worlds. module_haks is earliest-wins; upstream's resman is last-added-wins, which is why the .mod branch reverses. assemble merges per-hak NSYMs and so applies module_haks order directly — sow_top first and therefore winning. Anything comparing a multi-hak result against upstream must feed the specs reversed.

Known, accepted consequence

Per-artifact emission uploads blobs for resources that later get shadowed out of the merged manifest — the 13 duplicate resrefs #94 measured, plus the 2DA overlap between sow_top and the asset haks. Those blobs sit unreferenced in the zone. Small and harmless; the alternative is knowing the shadowing before the hak is born, which is impossible by construction.

Part of #54.

## Resolved: the NSYM is a sibling of its artifact, and `emit` derives its key Settled with the four sub-questions taken in order. The short version: `artifacts/<stream>/sha256/<a>//<sha256>.nsym`, derived by Crucible from an artifact key the caller already holds. ### 1. Path rule: sibling, per stream The NSYM sits beside its artifact, same prefix, extension replaced: - `artifacts/haks/sha256/<a>//<sha256>.nsym` — the 12 asset haks - `artifacts/topdata/sha256/<a>//<sha256>.nsym` — `sow_top.hak` and `sow_tlk.tlk` Rejected: one shared `artifacts/nsym/` prefix keyed only by digest. It was meant to spare the assembler from knowing which stream an entry came from, but the assembler never needed to know: both catalogs already publish a full `url` per entry (`pack-haks.sh:145-147`, `publish-topdata-release.sh:60-66`), so the key is derived by string-editing that url. A shared prefix buys nothing and splits ownership onto a prefix no repo owns. Everything [sow-platform#94](https://git.westgate.pw/ShadowsOverWestgate/sow-platform/issues/94) claimed for the sibling layout survives unchanged: immutable, shared by every channel naming the artifact, no new index, and its presence *is* the publication marker. Only the assumption that the prefix is always `artifacts/haks/` was wrong. ### 2. Key derivation: replace the final extension `<prefix>/sha256/<a>//<sha256>.nsym`, whatever the artifact's own suffix was. The digest is the whole identity, so `<sha>.hak` and `<sha>.tlk` can only collide on `<sha>.nsym` if they are byte-identical — in which case they are the same artifact with the same index. Rejected: appending (`<sha>.hak.nsym`). It encodes the artifact type twice, for no reader that wants it, and would force the assembler to know an entry's type before it can build a key — reintroducing exactly the per-stream coupling item 1 removed. The derivation, in full: take the catalog entry's `url`, cut at the 64-hex digest, append `.nsym`. Nothing else is parsed. ### 3. `emit` derives the key; it is not caller-supplied `emit --artifact-key artifacts/topdata/sha256/aa/bb/<sha>.hak` and Crucible applies the item 2 rule itself. The rule has two users that must agree — `emit` writing and `assemble` reading — so it lives in one place, and Crucible is the only place both see. A caller-supplied `--nsym-key` puts it in neither: two repos each hand-rolling a key while the assembler independently derives its own, with drift surfacing as a missing index at module release, far from the repo that caused it. Also rejected: a `--prefix` flag with Crucible hashing the file and building the shard path. That makes Crucible own the `sha256/<a>//` depot convention, which belongs to the publishing repos, and re-derives a digest the caller already computed. **Guard:** `emit` verifies the 64-hex digest in `--artifact-key` equals the SHA-256 of the file it was handed, and fails closed on mismatch. One hash of bytes already being read; it is the only thing between a copy-paste slip and an index published under another artifact's name. ### 4. The TLK: one flag, not a second verb The TLK is not an ERF — one loose file, so its NSYM is a single entry (resref `sow_tlk`, restype `0x07E2`, SHA-1 and size of the whole file) and it emits exactly one blob. `0x07E2` is already in `internal/erf/erf.go:89` and matches upstream, so no restype work. `emit --as sow_tlk.tlk <file>` treats the input as one loose resource under that name; without `--as`, the input is parsed as an ERF and names come from inside it. Rejected: sniffing the header and taking the resref from the local filename. The resref must be the *published* name, not whatever the runner called the temp file it downloaded — that gets the one thing that matters wrong, and ships a manifest entry named after a scratch path with no error anywhere. Also rejected: a separate `emit-file` verb, which duplicates the flag surface and the whole upload path for one extra input mode. The path is unaffected (item 2 already made the extension irrelevant), and so is the emit path itself: blob naming, compression and the NSYM writer are identical either way. Only entry-gathering branches. ### Verified against upstream while resolving Two things that were assumptions in #94 are now read out of `nwn_nwsync_write.nim` and `neverwinter/nwsync/private/libshared.nim`: **A hak is a first-class `<spec>`; the `.mod` is only convenience.** `libshared.nim:38-54` sniffs the first three bytes of each spec: `"ERF"`/`"HAK"` → `resman.add readErf(...)`; `"MOD"` → read `module.ifo`, resolve each hak by name off the search path, and add it with the same `readErf` call, `for hak in reversed(haklist)` (`:74`). The help text says the same: `<spec>` may be "any valid other erf container (HAK, ERF)" or "single files, including a TLK file". So running the writer on one hak is supported, and the per-hak conformance oracle #59 depends on is real, not hoped for — no 15 GB run needed to prove `emit`. **Order is inverted between the two worlds.** `module_haks` is earliest-wins; upstream's resman is last-added-wins, which is why the `.mod` branch reverses. `assemble` merges per-hak NSYMs and so applies `module_haks` order directly — `sow_top` first and therefore winning. Anything comparing a multi-hak result against upstream must feed the specs reversed. ### Known, accepted consequence Per-artifact emission uploads blobs for resources that later get shadowed out of the merged manifest — the 13 duplicate resrefs #94 measured, plus the 2DA overlap between `sow_top` and the asset haks. Those blobs sit unreferenced in the zone. Small and harmless; the alternative is knowing the shadowing before the hak is born, which is impossible by construction. Part of #54.

archvillainette closed this issue

2026-07-28 07:22:12 +00:00

archvillainette referenced this issue

2026-07-28 07:34:50 +00:00

Where the nwsync verbs live in the command surface #56

archvillainette referenced this issue

2026-07-28 07:35:26 +00:00

Wire the consumer repos to the settled crucible nwsync interface #65

archvillainette referenced this issue

2026-07-30 11:25:21 +00:00

Wire the consumer repos to the settled crucible nwsync interface #65

archvillainette referenced this issue