Restype resolution for emit: what Crucible has, what upstream's table says #57

New Issue

2026-07-27T18:51:20Z

archvillainette commented

2026-07-27 18:51:20 +00:00

Question

How does emit resolve a resource's restype, and what already exists in Crucible for it?

Part of #54. Upstream treats an unresolvable restype as a hard error, not a skip, and always excludes nss, ndb, gic. The 17 restypes in use are dds mdl tga mtr txi wav ssf pwk dwk wok lod shd bmu set 2da itp plt, all in NWN's builtin table, so no registerResType is needed.

Find out:

Whether internal/erf already carries a restype↔extension table, or stores the raw uint16 and maps it somewhere else. Grep the whole module, not just erf.
The authoritative builtin table from neverwinter.nim's restype.nim: which numbers map to which extensions, at least for the 17 in use plus the 3 excluded.
Whether the ERF on-disk format gives emit the restype directly, so nothing needs to be inferred from a filename at all.
The manifest entry writes char[16] lowercase NUL-padded resref plus uint16 restype. Confirm nothing in our corpus needs case folding or truncation beyond what was already measured (no stem exceeds 16 characters).

## Question How does `emit` resolve a resource's restype, and what already exists in Crucible for it? Part of #54. Upstream treats an unresolvable restype as a hard error, not a skip, and always excludes `nss`, `ndb`, `gic`. The 17 restypes in use are `dds mdl tga mtr txi wav ssf pwk dwk wok lod shd bmu set 2da itp plt`, all in NWN's builtin table, so no `registerResType` is needed. Find out: - Whether `internal/erf` already carries a restype↔extension table, or stores the raw `uint16` and maps it somewhere else. Grep the whole module, not just `erf`. - The authoritative builtin table from `neverwinter.nim`'s `restype.nim`: which numbers map to which extensions, at least for the 17 in use plus the 3 excluded. - Whether the ERF on-disk format gives `emit` the restype directly, so nothing needs to be inferred from a filename at all. - The manifest entry writes `char[16]` lowercase NUL-padded resref plus `uint16` restype. Confirm nothing in our corpus needs case folding or truncation beyond what was already measured (no stem exceeds 16 characters).

archvillainette added the wayfinder:research label 2026-07-27 18:51:20 +00:00

archvillainette self-assigned this 2026-07-27 18:53:41 +00:00

archvillainette commented

2026-07-27 18:59:39 +00:00

Resolved: no new table needed, but lowercase the resref and mirror upstream's table

emit needs no new restype table. Crucible already has exactly one, and erf.Read already keeps the raw uint16 straight off the ERF key list — so manifest entries can be built from Resource.Type directly, with zero lookups and nothing inferred from a filename.

Two things do need doing, plus one bug to file separately.

1. The table exists, in one place

internal/erf/erf.go:70-134 — extensionTypes map[string]uint16, 63 entries.
internal/erf/erf.go:136-200 — typeExtensions map[uint16]string, the hand-maintained mirror.
Accessors at erf.go:365,370,374.

Grepped the whole module: no second numeric table. Every other package routes through those accessors (pipeline/chunk_validation.go:22, pipeline/compare.go:255, pipeline/extract.go:118, pipeline/build.go:870,1044,1085,1104, topdata/top_package.go:277,435,441,458, validator/validator.go:515-517). The restype: strings in topdata/autogen.go are a YAML filter compared against the literal "mdl", never a number.

The init() at erf.go:202-212 that should keep the two maps honest is effectively a no-op — it panics only on a number missing from the reverse map, and its if canonicalExt == ext { continue } branch does nothing when they disagree. That is why the wrong numbers below survived.

2. ERF carries the restype per resource, so nothing is inferred

type keyEntry struct {
	ResRef       [16]byte
	ResourceID   uint32
	ResourceType uint16
	Unused       uint16
}

(erf.go:58-63, 24-byte stride read at erf.go:318-326.) The read path assigns key.ResourceType to Resource.Type verbatim (erf.go:339-356). The resource list (erf.go:328-336) carries only offset and size, so the key list is the sole and authoritative source. ERF stores the resref without an extension, so a filename is not even available to infer from.

3. All 20 restypes we care about already match upstream

Checked against restype.nim's makeResTypeTable() (lines 33-121). The 17 in the corpus — tga 3, wav 4, plt 6, bmu 8, mdl 2002, set 2013, wok 2016, 2da 2017, txi 2022, itp 2030, dds 2033, dwk 2052, pwk 2053, ssf 2060, shd 2069, mtr 2072, lod 2078 — and the three excluded, nss 2009, ndb 2064, gic 2046, all agree with internal/erf. No change needed for the corpus.

Upstream's exclusion and hard-error rule, confirmed in nwsync/private/libupdate.nim: the skip list is nss/ndb/gic (:25-29), applied as a filter after a resolvability check that quit(1)s on an unknown restype (:112-115). So an nss still has to resolve; it just is not emitted. Do not mistake restype.nim:167-170 — which prints the number when unresolvable — for graceful degradation; that is a display helper, not the emit policy.

4. Resref case is a real conflict

Upstream lowercases on both sides (nwsync.nim:93-99): read strips trailing NULs and lowercases; write emits resRef.toLowerAscii right-padded with NUL to 16, then the uint16. Max length is 16 (resref.nim:7) and a longer one would make the padding count negative, so length must be validated before writing.

Crucible's reader preserves original case — erf.go:347 is a bare bytes.TrimRight, and there is no ToLower anywhere on the internal/erf read path. Crucible's write paths already lowercase (build.go:874,1077,1089,1108,1114,1130, source_manifest.go:181, top_package.go:279), so HAKs Crucible builds are safe — but foreign HAKs are not, and real HAKs built by other tools do contain mixed-case resrefs.

So emit must lowercase Resource.Name itself rather than trusting the reader, and should reject len(Name) > 16. erf.Write checks that at erf.go:248; Read does not.

5. One decision this leaves open

Crucible registers six extensions upstream does not have at all — lyt 3000, vis 3001, mdx 3008, wlk 3020, xml 3021, gr2 4003 (erf.go:128-133). restype.nim jumps from jui 2083 to ids 9996. If any hak ever contains a .lyt or .vis, upstream would quit(1) while a permissive emit would happily write it.

Mirror upstream and error identically. Matching the reference implementation is the entire point of the conformance exercise, and the corpus contains none of these six. Folded into #60 rather than given its own ticket.

Filed separately

Five numbers in internal/erf/erf.go disagree with upstream: gff 2039 should be 2037, ltr 2067 should be 2036, jpg 2076 should be 2081, mdb 2070 is not registered upstream at all, and dfa 2045 is upstream's dft. A .jpg packed by Crucible today gets 2076, which the game reads as tml. None are in the corpus so none block this map — filed as its own bug.

## Resolved: no new table needed, but lowercase the resref and mirror upstream's table `emit` needs **no new restype table**. Crucible already has exactly one, and `erf.Read` already keeps the raw `uint16` straight off the ERF key list — so manifest entries can be built from `Resource.Type` directly, with zero lookups and nothing inferred from a filename. Two things do need doing, plus one bug to file separately. ### 1. The table exists, in one place - `internal/erf/erf.go:70-134` — `extensionTypes map[string]uint16`, 63 entries. - `internal/erf/erf.go:136-200` — `typeExtensions map[uint16]string`, the hand-maintained mirror. - Accessors at `erf.go:365,370,374`. Grepped the whole module: no second numeric table. Every other package routes through those accessors (`pipeline/chunk_validation.go:22`, `pipeline/compare.go:255`, `pipeline/extract.go:118`, `pipeline/build.go:870,1044,1085,1104`, `topdata/top_package.go:277,435,441,458`, `validator/validator.go:515-517`). The `restype:` strings in `topdata/autogen.go` are a YAML filter compared against the literal `"mdl"`, never a number. The `init()` at `erf.go:202-212` that should keep the two maps honest is effectively a no-op — it panics only on a number missing from the reverse map, and its `if canonicalExt == ext { continue }` branch does nothing when they disagree. That is why the wrong numbers below survived. ### 2. ERF carries the restype per resource, so nothing is inferred ```go type keyEntry struct { ResRef [16]byte ResourceID uint32 ResourceType uint16 Unused uint16 } ``` (`erf.go:58-63`, 24-byte stride read at `erf.go:318-326`.) The read path assigns `key.ResourceType` to `Resource.Type` verbatim (`erf.go:339-356`). The resource list (`erf.go:328-336`) carries only offset and size, so the key list is the sole and authoritative source. ERF stores the resref without an extension, so a filename is not even available to infer from. ### 3. All 20 restypes we care about already match upstream Checked against `restype.nim`'s `makeResTypeTable()` (lines 33-121). The 17 in the corpus — `tga` 3, `wav` 4, `plt` 6, `bmu` 8, `mdl` 2002, `set` 2013, `wok` 2016, `2da` 2017, `txi` 2022, `itp` 2030, `dds` 2033, `dwk` 2052, `pwk` 2053, `ssf` 2060, `shd` 2069, `mtr` 2072, `lod` 2078 — and the three excluded, `nss` 2009, `ndb` 2064, `gic` 2046, all agree with `internal/erf`. No change needed for the corpus. Upstream's exclusion and hard-error rule, confirmed in `nwsync/private/libupdate.nim`: the skip list is `nss`/`ndb`/`gic` (`:25-29`), applied as a filter *after* a resolvability check that `quit(1)`s on an unknown restype (`:112-115`). So an `nss` still has to resolve; it just is not emitted. Do not mistake `restype.nim:167-170` — which prints the number when unresolvable — for graceful degradation; that is a display helper, not the emit policy. ### 4. Resref case is a real conflict Upstream lowercases on both sides (`nwsync.nim:93-99`): read strips trailing NULs and lowercases; write emits `resRef.toLowerAscii` right-padded with NUL to 16, then the `uint16`. Max length is 16 (`resref.nim:7`) and a longer one would make the padding count negative, so length must be validated before writing. Crucible's reader preserves original case — `erf.go:347` is a bare `bytes.TrimRight`, and there is no `ToLower` anywhere on the `internal/erf` read path. Crucible's *write* paths already lowercase (`build.go:874,1077,1089,1108,1114,1130`, `source_manifest.go:181`, `top_package.go:279`), so HAKs Crucible builds are safe — but foreign HAKs are not, and real HAKs built by other tools do contain mixed-case resrefs. **So `emit` must lowercase `Resource.Name` itself rather than trusting the reader**, and should reject `len(Name) > 16`. `erf.Write` checks that at `erf.go:248`; `Read` does not. ### 5. One decision this leaves open Crucible registers six extensions upstream does not have at all — `lyt` 3000, `vis` 3001, `mdx` 3008, `wlk` 3020, `xml` 3021, `gr2` 4003 (`erf.go:128-133`). `restype.nim` jumps from `jui` 2083 to `ids` 9996. If any hak ever contains a `.lyt` or `.vis`, upstream would `quit(1)` while a permissive `emit` would happily write it. **Mirror upstream and error identically.** Matching the reference implementation is the entire point of the conformance exercise, and the corpus contains none of these six. Folded into #60 rather than given its own ticket. ### Filed separately Five numbers in `internal/erf/erf.go` disagree with upstream: `gff` 2039 should be 2037, `ltr` 2067 should be 2036, `jpg` 2076 should be 2081, `mdb` 2070 is not registered upstream at all, and `dfa` 2045 is upstream's `dft`. A `.jpg` packed by Crucible today gets 2076, which the game reads as `tml`. None are in the corpus so none block this map — filed as its own bug.

archvillainette closed this issue

2026-07-27 18:59:40 +00:00

archvillainette referenced this issue

2026-07-27 19:00:31 +00:00

Five restype numbers in internal/erf disagree with upstream #64

archvillainette referenced this issue

2026-07-27 19:00:31 +00:00

The emit upload path and NWCompressedBuffer round-trip #60

archvillainette referenced this issue

2026-07-27 19:01:14 +00:00

Map: NWSync publishing — emit at artifact birth, assemble at module release #54

archvillainette referenced this issue