ShadowsOverWestgate/sow-tools
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 38 Activity

nwsync emit holds ~5x the artifact size in RAM; OOM-killed on haks over ~1.4 GB #76

New Issue
Closed
opened 2026-07-31 10:44:28 +00:00 by archvillainette · 3 comments
archvillainette commented 2026-07-31 10:44:28 +00:00
Owner
Copy Link
Copy Source

crucible nwsync emit holds roughly 5× the artifact size in RAM, so it is OOM-killed on any hak over about 1.4 GB. This currently blocks the NWSync backfill in sow-assets-manifest, and — since that repo now emits at pack time — it will also kill the next release that rebuilds a large hak.

Evidence

Two kills on ovh-main today, both crucible, both under the runner slice:

Jul 31 10:00:29 ovh-main kernel: oom-kill:constraint=CONSTRAINT_NONE,...,
  task_memcg=/system.slice/gitea-runner-ovh\x2dmain.service,task=crucible,pid=356742
Jul 31 10:00:29 ovh-main kernel: Out of memory: Killed process 356742 (crucible)
  total-vm:7634408kB, anon-rss:5617024kB, file-rss:52kB

Jul 31 10:38:48 ovh-main kernel: Out of memory: Killed process 390899 (crucible)
  total-vm:7634152kB, anon-rss:5773184kB, file-rss:92kB

The job dies about 82 s in, right after [depot] backfilling v0.1.7, with no message from the calling script — the shell's die never runs because the process took a signal.

Measured peak RSS locally, nwsync emit -out against real haks:

artifact size peak RSS ratio
sow_vfxs_01.hak 250 MB 1.11 GB 4.7×
sow_appr_01.hak 485 MB 2.38 GB 4.9×

Linear. Extrapolating to the v0.1.7 catalog, whose first entry is the one that died:

artifact size projected peak
sow_over_01.hak 1.87 GB ~9 GB
sow_core_01.hak 2.15 GB ~10.5 GB

ovh-main has 7 GB total and no swap, so about 5 GB is available to a job. Four of the ten haks in that release are over the limit. No timeout increase helps.

Cause

Emit reads the whole artifact into one buffer, then erf.Read allocates a second full copy of the payload, resource by resource:

internal/nwsync/emit.go:82

artifact, err := os.ReadFile(options.ArtifactPath)

internal/erf/erf.go:358

payload := make([]byte, entry.Size)
copy(payload, data[start:end])

So the artifact is live twice before a single blob is written — the []erf.Resource slice pins every payload for the whole run, because emitResources iterates a fully materialised slice. Compression buffers and GC latency account for the rest of the 5×.

Suggested fix

Stream it. erf.Read already parses the header and the resource table separately from the payloads, so the emit path can seek to one resource at a time, compress it, put the blob, and drop it. Peak memory then tracks the largest single resource — a few MB — instead of the archive.

Two smaller changes would help even without a full rewrite:

  • Have erf.Read return payloads as subslices of the mapped file rather than fresh copies, or offer a ReadLazy that yields offsets. That removes one full copy on its own.
  • mmap the artifact instead of os.ReadFile, so the file-backed pages are evictable under pressure rather than counting as anonymous memory. The OOM lines show anon-rss:5.6GB against file-rss:52kB — all of it is anonymous today.

The index write at the end needs the complete entry list, but entries are small (identity, hash, size), so that part can stay in memory.

Acceptance

nwsync emit completes against a 2.15 GB hak inside a 5 GB budget, with a regression check that asserts peak RSS does not scale with artifact size.

Context

`crucible nwsync emit` holds roughly 5× the artifact size in RAM, so it is OOM-killed on any hak over about 1.4 GB. This currently blocks the NWSync backfill in `sow-assets-manifest`, and — since that repo now emits at pack time — it will also kill the next release that rebuilds a large hak. ## Evidence Two kills on `ovh-main` today, both `crucible`, both under the runner slice: ``` Jul 31 10:00:29 ovh-main kernel: oom-kill:constraint=CONSTRAINT_NONE,..., task_memcg=/system.slice/gitea-runner-ovh\x2dmain.service,task=crucible,pid=356742 Jul 31 10:00:29 ovh-main kernel: Out of memory: Killed process 356742 (crucible) total-vm:7634408kB, anon-rss:5617024kB, file-rss:52kB Jul 31 10:38:48 ovh-main kernel: Out of memory: Killed process 390899 (crucible) total-vm:7634152kB, anon-rss:5773184kB, file-rss:92kB ``` The job dies about 82 s in, right after `[depot] backfilling v0.1.7`, with no message from the calling script — the shell's `die` never runs because the process took a signal. Measured peak RSS locally, `nwsync emit -out` against real haks: | artifact | size | peak RSS | ratio | |---|---|---|---| | `sow_vfxs_01.hak` | 250 MB | 1.11 GB | 4.7× | | `sow_appr_01.hak` | 485 MB | 2.38 GB | 4.9× | Linear. Extrapolating to the v0.1.7 catalog, whose first entry is the one that died: | artifact | size | projected peak | |---|---|---| | `sow_over_01.hak` | 1.87 GB | ~9 GB | | `sow_core_01.hak` | 2.15 GB | ~10.5 GB | `ovh-main` has 7 GB total and no swap, so about 5 GB is available to a job. Four of the ten haks in that release are over the limit. No timeout increase helps. ## Cause `Emit` reads the whole artifact into one buffer, then `erf.Read` allocates a second full copy of the payload, resource by resource: `internal/nwsync/emit.go:82` ```go artifact, err := os.ReadFile(options.ArtifactPath) ``` `internal/erf/erf.go:358` ```go payload := make([]byte, entry.Size) copy(payload, data[start:end]) ``` So the artifact is live twice before a single blob is written — the `[]erf.Resource` slice pins every payload for the whole run, because `emitResources` iterates a fully materialised slice. Compression buffers and GC latency account for the rest of the 5×. ## Suggested fix Stream it. `erf.Read` already parses the header and the resource table separately from the payloads, so the emit path can seek to one resource at a time, compress it, put the blob, and drop it. Peak memory then tracks the largest single resource — a few MB — instead of the archive. Two smaller changes would help even without a full rewrite: - Have `erf.Read` return payloads as subslices of the mapped file rather than fresh copies, or offer a `ReadLazy` that yields offsets. That removes one full copy on its own. - `mmap` the artifact instead of `os.ReadFile`, so the file-backed pages are evictable under pressure rather than counting as anonymous memory. The OOM lines show `anon-rss:5.6GB` against `file-rss:52kB` — all of it is anonymous today. The index write at the end needs the complete entry list, but entries are small (identity, hash, size), so that part can stay in memory. ## Acceptance `nwsync emit` completes against a 2.15 GB hak inside a 5 GB budget, with a regression check that asserts peak RSS does not scale with artifact size. ## Context - Related map: #54 - Blocks: the `backfill-nwsync` workflow in `sow-assets-manifest`, and `pack-haks.sh`, which calls `nwsync_emit` per built archive since ShadowsOverWestgate/sow-assets-manifest#56.
archvillainette commented 2026-07-31 10:47:29 +00:00
Author
Owner
Copy Link
Copy Source

Implementation plan

The peak is bounded by a constant that already exists. internal/nwsync/emit.go:24:

// fileSizeLimit matches upstream's --limit-file-size default of 15 MB.
const fileSizeLimit = 15 * 1024 * 1024

NWSync already refuses any single resource over 15 MB, so a streaming emit peaks at one resource — 15 MB — not one archive. Plus the entry list, which for a 2 GB hak is on the order of 10k resources × ~40 bytes ≈ 400 KB. Peak lands under 100 MB against today's ~10.5 GB, and stops scaling with artifact size, so a 4 GB hak later costs the same.

What makes this cheap: every test in the emitResources pre-pass reads metadata, never payload.

if _, ok := erf.ExtensionForResourceType(resource.Type); !ok   // Type
if slices.Contains(skippedTypes, resource.Type)                // Type
if len(resource.Data) > fileSizeLimit                          // == entry.Size
identity := Identity{ResRef: ..., ResType: resource.Type}      // Name, Type

The size check reads len(resource.Data) today but wants entry.Size, which is already in the resource list. So the whole dedup pass — including last-one-wins ordering — can run on the key list and resource list alone. Those are EntryCount × 32 bytes, and the ERF header points directly at both.

Steps

  1. New erf API alongside Read. An index type over an io.ReaderAt exposing Name / Type / Offset / Size per entry, plus ReadResource(i int, buf []byte). Strictly additive: Read stays as-is for internal/pipeline/compare.go:245 and internal/pipeline/extract.go:55, which pass whole erf.Archive values around and are not worth disturbing for this.

  2. emit.go streams. Open the artifact with os.Open instead of os.ReadFile. Run the dedup pre-pass on the index. Then, for each surviving identity in order, ReadAt its payload into one reused 15 MB buffer, sha1, compressBlob, putBlob, reuse the buffer. Entries accumulate as now — they are small and the index write at the end needs the complete list.

  3. Regression check. Assert peak RSS is flat across a small and a large fixture, so this cannot silently regress into a full read again.

checkArtifactKey currently takes the whole artifact buffer; it only needs the digest, so it either hashes the file streaming or moves behind the same reader.

Roughly 150 lines, two files.

Coordination

sow-assets-manifest pins a crucible build, so it needs a re-pin after this lands before either the backfill-nwsync workflow or a release sees the benefit. Same effort, second repo.

Why not the alternatives

  • One hak at a time — already the case. nwsync_emit_catalog downloads a hak, emits, rm -f, next; peak disk is one hak. The 5× is entirely inside a single Emit call, so sequencing changes nothing.
  • Bigger or different host — ovh-main is where CI builds run, and the game host's CPU is spoken for (ADR-0026, ADR-0023). Fixing the multiplier is the only option that leaves the topology alone.
  • mmap instead of streaming — would make the pages evictable rather than anonymous, which helps, but still leaves the second full copy from erf.Read and leaves peak scaling with archive size. Worth folding in only if streaming turns out awkward.
## Implementation plan The peak is bounded by a constant that already exists. `internal/nwsync/emit.go:24`: ```go // fileSizeLimit matches upstream's --limit-file-size default of 15 MB. const fileSizeLimit = 15 * 1024 * 1024 ``` NWSync already refuses any single resource over 15 MB, so a streaming emit peaks at one resource — 15 MB — not one archive. Plus the entry list, which for a 2 GB hak is on the order of 10k resources × ~40 bytes ≈ 400 KB. Peak lands under 100 MB against today's ~10.5 GB, and stops scaling with artifact size, so a 4 GB hak later costs the same. What makes this cheap: every test in the `emitResources` pre-pass reads metadata, never payload. ```go if _, ok := erf.ExtensionForResourceType(resource.Type); !ok // Type if slices.Contains(skippedTypes, resource.Type) // Type if len(resource.Data) > fileSizeLimit // == entry.Size identity := Identity{ResRef: ..., ResType: resource.Type} // Name, Type ``` The size check reads `len(resource.Data)` today but wants `entry.Size`, which is already in the resource list. So the whole dedup pass — including last-one-wins ordering — can run on the key list and resource list alone. Those are `EntryCount × 32` bytes, and the ERF header points directly at both. ### Steps 1. **New `erf` API alongside `Read`.** An index type over an `io.ReaderAt` exposing `Name` / `Type` / `Offset` / `Size` per entry, plus `ReadResource(i int, buf []byte)`. Strictly additive: `Read` stays as-is for `internal/pipeline/compare.go:245` and `internal/pipeline/extract.go:55`, which pass whole `erf.Archive` values around and are not worth disturbing for this. 2. **`emit.go` streams.** Open the artifact with `os.Open` instead of `os.ReadFile`. Run the dedup pre-pass on the index. Then, for each surviving identity in order, `ReadAt` its payload into one reused 15 MB buffer, sha1, `compressBlob`, `putBlob`, reuse the buffer. Entries accumulate as now — they are small and the index write at the end needs the complete list. 3. **Regression check.** Assert peak RSS is flat across a small and a large fixture, so this cannot silently regress into a full read again. `checkArtifactKey` currently takes the whole `artifact` buffer; it only needs the digest, so it either hashes the file streaming or moves behind the same reader. Roughly 150 lines, two files. ### Coordination `sow-assets-manifest` pins a crucible build, so it needs a re-pin after this lands before either the `backfill-nwsync` workflow or a release sees the benefit. Same effort, second repo. ### Why not the alternatives - **One hak at a time** — already the case. `nwsync_emit_catalog` downloads a hak, emits, `rm -f`, next; peak disk is one hak. The 5× is entirely inside a single `Emit` call, so sequencing changes nothing. - **Bigger or different host** — ovh-main is where CI builds run, and the game host's CPU is spoken for (ADR-0026, ADR-0023). Fixing the multiplier is the only option that leaves the topology alone. - **mmap instead of streaming** — would make the pages evictable rather than anonymous, which helps, but still leaves the second full copy from `erf.Read` and leaves peak scaling with archive size. Worth folding in only if streaming turns out awkward.
archvillainette commented 2026-07-31 10:50:12 +00:00
Author
Owner
Copy Link
Copy Source

Correction to the threshold in the title: at the measured 4.9x, a 5 GB budget is exhausted by a hak of about 1 GB, not 1.4 GB. Six of the ten haks in sow-assets-manifest v0.1.7 are over that line and a seventh is marginal — full table in ShadowsOverWestgate/sow-assets-manifest#57. The evidence in this issue is unchanged; only the cutoff moves, and it moves in the unhelpful direction.

Correction to the threshold in the title: at the measured 4.9x, a 5 GB budget is exhausted by a hak of about **1 GB**, not 1.4 GB. Six of the ten haks in `sow-assets-manifest` v0.1.7 are over that line and a seventh is marginal — full table in ShadowsOverWestgate/sow-assets-manifest#57. The evidence in this issue is unchanged; only the cutoff moves, and it moves in the unhelpful direction.
archvillainette commented 2026-07-31 11:10:27 +00:00
Author
Owner
Copy Link
Copy Source

Fix in #78 (branch fix/nwsync-emit-streaming), not merged yet.

Emit now opens the artifact, hashes it by streaming for the key check, parses only the header and resource table (erf.ReadIndex), and reads/compresses/stores one payload at a time — the archive is never resident. erf.Read also stopped copying payloads; it hands back subslices, which removes the second copy the ticket named.

One thing outside the ticket: the zstd encoder ran at one encoder per CPU, each holding a window-sized history — about 200 MB of live heap idle on a 24-core runner. Concurrency 1 is byte-identical output, so it is set to 1 and a test pins that.

Peak heap during emit, sampled every 1 ms: 8 MB hak went 155 MB -> 21 MB, 64 MB hak 289 MB -> 22 MB. Flat, which is what the acceptance asks for. TestEmitPeakMemoryDoesNotScaleWithArtifactSize fails if that curve tilts again.

Two gaps to be honest about: the check measures Go heap, not RSS, and the biggest fixture is 64 MB — no multi-GB run was done here, so the 2.15 GB / 5 GB acceptance is inference from a flat curve. And mmap was skipped: payload buffers are still anonymous, but each is one resource (<=15 MiB), so file-backing them buys nothing now.

Fix in #78 (branch `fix/nwsync-emit-streaming`), not merged yet. Emit now opens the artifact, hashes it by streaming for the key check, parses only the header and resource table (`erf.ReadIndex`), and reads/compresses/stores one payload at a time — the archive is never resident. `erf.Read` also stopped copying payloads; it hands back subslices, which removes the second copy the ticket named. One thing outside the ticket: the zstd encoder ran at one encoder per CPU, each holding a window-sized history — about 200 MB of live heap idle on a 24-core runner. Concurrency 1 is byte-identical output, so it is set to 1 and a test pins that. Peak heap during emit, sampled every 1 ms: 8 MB hak went 155 MB -> 21 MB, 64 MB hak 289 MB -> 22 MB. Flat, which is what the acceptance asks for. `TestEmitPeakMemoryDoesNotScaleWithArtifactSize` fails if that curve tilts again. Two gaps to be honest about: the check measures Go heap, not RSS, and the biggest fixture is 64 MB — no multi-GB run was done here, so the 2.15 GB / 5 GB acceptance is inference from a flat curve. And mmap was skipped: payload buffers are still anonymous, but each is one resource (<=15 MiB), so file-backing them buys nothing now.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Chass FallenDabus WinterZen XeroTwo Zariu archvillainette gitea-bot xtul
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: ShadowsOverWestgate/sow-tools#76