Closes#76. Part of #54.
`nwsync emit` held roughly 5× the artifact size in RAM, so ovh-main (7 GB, no swap) OOM-killed it on any hak over ~1.4 GB. That blocked the backfill in sow-assets-manifest and would have killed the next release rebuilding a large hak.
## What it does
- `erf.ReadIndex` / `erf.ReadPayload` — parse the header and resource table only, read one payload on demand. `erf.Read` keeps its shape but returns payloads as subslices of the buffer instead of fresh copies, which removes one full copy for the `pipeline` callers too. Callers must not mutate `Resource.Data`; the doc comment says so and no caller does.
- `nwsync.Emit` opens the artifact, hashes it by streaming for the key check (through a section reader, so the file offset stays put), then hashes, compresses and stores one resource at a time. The archive is never resident. Shadowed duplicate resrefs are now never read at all.
- Bounds checks in `ReadIndex` moved to `int64`, so key/resource-list offsets can no longer overflow.
## Not in the issue, but memory-motivated
The zstd blob encoder ran at the default concurrency, which is one encoder per CPU, each holding a window-sized history — about 200 MB of live heap doing nothing on a 24-core runner. `EncodeAll` is single-threaded per call, so concurrency 1 costs nothing. `TestSingleThreadedEncoderMatchesDefault` pins the claim that blobs come out byte-identical.
## Measured
Peak heap during emit, sampled 1 ms:
| hak | before | after |
|-----|--------|-------|
| 8 MB | 155 MB | 21 MB |
| 64 MB | 289 MB | 22 MB |
Flat, as the acceptance asks. `TestEmitPeakMemoryDoesNotScaleWithArtifactSize` fails if the 64 MB fixture costs more than the 8 MB one plus 24 MB of slack.
## Gaps
- The regression check measures Go heap, not RSS, and its largest fixture is 64 MB — a multi-GB run was not done here. A 2.15 GB hak now needs about the same ~22 MB the 64 MB one does, so the 5 GB budget is not close, but that is inference from the flat curve, not a measurement.
- mmap was suggested in the issue and skipped. Payload buffers are still anonymous, but they are one resource each (≤15 MiB), so making them file-backed buys nothing now.
🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #78
Reviewed-by: xtul <mpiasecki720@protonmail.com>
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Closes#64.
Five entries in `internal/erf/erf.go`'s restype table disagreed with upstream neverwinter.nim (`neverwinter/restype.nim`). `HAKResourceTypeForExtension` packs arbitrary source files, so a `.jpg` packed into a HAK today is written as restype 2076, which the game reads as `tml`.
| ext | was | now |
|-----|-----|-----|
| gff | 2039 (upstream `bte`) | 2037 |
| ltr | 2067 (upstream `bak`) | 2036 |
| jpg | 2076 (upstream `tml`) | 2081 |
| dfa | 2045, name typo | `dft`, 2045 (number was already right) |
| mdb | 2070 (upstream `xbc`) | removed |
Upstream registers no restype for `mdb`, so there is no correct number to give it, and leaving it squatting on `xbc` silently mislabels the resource. It is dropped from `AssetExtensions` too, which turns a `.mdb` source into a loud "unsupported extension" build error instead of a corrupt HAK entry. **This is the one judgment call the issue left open** — if `mdb` should instead stay as a deliberate local addition like `lyt`/`vis`/`gr2`, say so and I will move it to a free number in the 0x0BB8+ local range.
No asset in the current corpus uses any of these paths, so nothing in shipped content changes.
The six local additions upstream does not have (`lyt` 3000, `vis` 3001, `mdx` 3008, `wlk` 3020, `xml` 3021, `gr2` 4003) are left alone and now carry a comment saying they are deliberate.
## Root cause
The `init()` consistency guard only panicked when a number was missing from the reverse map. Its `if canonicalExt == ext { continue }` branch did nothing when the two maps disagreed, so a mismatch was never caught. The guard now panics on disagreement and also checks the reverse direction.
## Tests
- `TestRestypeTableMatchesUpstream` pins every number shared with upstream, and asserts the four numbers upstream reserves for other extensions (2039 `bte`, 2067 `bak`, 2070 `xbc`, 2076 `tml`) stay unclaimed.
- `TestRestypeTablesAreMutualInverses` is the check `init()` is meant to enforce.
`make check` passes.Reviewed-on: #69
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
## What
Two correctness fixes surfaced in review of the direct-depot HAK artifact work.
### Reject resref+type collisions in `chunksFromManifest`
A manifest can name two distinct source paths that collapse to the same
`resref+type` (e.g. `creature/foo.tga` and `placeable/foo.tga` — resref is the
lowercase basename minus extension). The ERF writer keys resources on
`Name:Type`, so the second silently shadowed the first: an asset would vanish
from the packed HAK with no error.
`chunksFromManifest` now runs `ensureUniqueChunkResources` per chunk and fails
the build on a duplicate. This guards **both** build paths — the legacy
`--source-manifest` flow and the direct content-addressed flow
(`chunksFromSourceManifest` → `chunksFromManifest`).
### Document erf post-write hash coupling
`writeResourceData` streams the source into the output while hashing, so
size/SHA mismatches are only detected *after* the bytes are written. A non-nil
return therefore means the writer holds partial, unverified output and the
caller must discard it. Added a comment making that contract explicit;
`writeHAKArchive` already honours it (writes to a temp file, removes on any
Write error, never renames a bad archive into place).
## Tests
- `TestChunksFromManifestRejectsResrefCollision`: collision → error, distinct
resrefs → clean. Asserts only error presence/absence — silent asset loss is
the contract, not any specific wording.
- `go vet ./internal/erf/ ./internal/pipeline/` clean.
- `go test ./internal/erf/ ./internal/pipeline/` green.
## Follow-up
A new crucible release must be cut after this merges so the guard ships in the
binary `sow-assets-manifest` pins.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewed-on: #8
Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
Co-committed-by: vickydotbat <vickydotbat@tutamail.com>
archvillainette
xtulandarchvillainette