Emit read the whole artifact into memory and erf.Read then allocated a
second full copy of every payload, so a hak cost about 5x its size in
RAM. The 7 GB runner was OOM-killed on any hak over ~1.4 GB, which
blocks the NWSync backfill and every release that rebuilds a large hak.
Emit now opens the artifact, hashes it by streaming for the key check,
parses only the header and resource table via erf.ReadIndex, and reads,
hashes, compresses and stores one payload at a time. erf.Read keeps its
old shape but returns payloads as subslices instead of fresh copies,
which removes the second copy for the other callers too.
The zstd encoder pool also held one window-sized history per CPU — about
200 MB of live heap on a 24-core runner. EncodeAll is single-threaded
per call, so concurrency 1 gives byte-identical blobs for far less
memory.
Peak heap is now flat at ~22 MB for both an 8 MB and a 64 MB hak, and a
regression test asserts it does not scale with artifact size.
Closes#76
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes#64.
Five entries in `internal/erf/erf.go`'s restype table disagreed with upstream neverwinter.nim (`neverwinter/restype.nim`). `HAKResourceTypeForExtension` packs arbitrary source files, so a `.jpg` packed into a HAK today is written as restype 2076, which the game reads as `tml`.
| ext | was | now |
|-----|-----|-----|
| gff | 2039 (upstream `bte`) | 2037 |
| ltr | 2067 (upstream `bak`) | 2036 |
| jpg | 2076 (upstream `tml`) | 2081 |
| dfa | 2045, name typo | `dft`, 2045 (number was already right) |
| mdb | 2070 (upstream `xbc`) | removed |
Upstream registers no restype for `mdb`, so there is no correct number to give it, and leaving it squatting on `xbc` silently mislabels the resource. It is dropped from `AssetExtensions` too, which turns a `.mdb` source into a loud "unsupported extension" build error instead of a corrupt HAK entry. **This is the one judgment call the issue left open** — if `mdb` should instead stay as a deliberate local addition like `lyt`/`vis`/`gr2`, say so and I will move it to a free number in the 0x0BB8+ local range.
No asset in the current corpus uses any of these paths, so nothing in shipped content changes.
The six local additions upstream does not have (`lyt` 3000, `vis` 3001, `mdx` 3008, `wlk` 3020, `xml` 3021, `gr2` 4003) are left alone and now carry a comment saying they are deliberate.
## Root cause
The `init()` consistency guard only panicked when a number was missing from the reverse map. Its `if canonicalExt == ext { continue }` branch did nothing when the two maps disagreed, so a mismatch was never caught. The guard now panics on disagreement and also checks the reverse direction.
## Tests
- `TestRestypeTableMatchesUpstream` pins every number shared with upstream, and asserts the four numbers upstream reserves for other extensions (2039 `bte`, 2067 `bak`, 2070 `xbc`, 2076 `tml`) stay unclaimed.
- `TestRestypeTablesAreMutualInverses` is the check `init()` is meant to enforce.
`make check` passes.Reviewed-on: #69
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
## What
Two correctness fixes surfaced in review of the direct-depot HAK artifact work.
### Reject resref+type collisions in `chunksFromManifest`
A manifest can name two distinct source paths that collapse to the same
`resref+type` (e.g. `creature/foo.tga` and `placeable/foo.tga` — resref is the
lowercase basename minus extension). The ERF writer keys resources on
`Name:Type`, so the second silently shadowed the first: an asset would vanish
from the packed HAK with no error.
`chunksFromManifest` now runs `ensureUniqueChunkResources` per chunk and fails
the build on a duplicate. This guards **both** build paths — the legacy
`--source-manifest` flow and the direct content-addressed flow
(`chunksFromSourceManifest` → `chunksFromManifest`).
### Document erf post-write hash coupling
`writeResourceData` streams the source into the output while hashing, so
size/SHA mismatches are only detected *after* the bytes are written. A non-nil
return therefore means the writer holds partial, unverified output and the
caller must discard it. Added a comment making that contract explicit;
`writeHAKArchive` already honours it (writes to a temp file, removes on any
Write error, never renames a bad archive into place).
## Tests
- `TestChunksFromManifestRejectsResrefCollision`: collision → error, distinct
resrefs → clean. Asserts only error presence/absence — silent asset loss is
the contract, not any specific wording.
- `go vet ./internal/erf/ ./internal/pipeline/` clean.
- `go test ./internal/erf/ ./internal/pipeline/` green.
## Follow-up
A new crucible release must be cut after this merges so the guard ships in the
binary `sow-assets-manifest` pins.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewed-on: #8
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
archvillainetteandClaude Opus 5
xtulandarchvillainette