fix(nwsync): stream emit so peak memory tracks the largest resource (#76)
Emit read the whole artifact into memory and erf.Read then allocated a second full copy of every payload, so a hak cost about 5x its size in RAM. The 7 GB runner was OOM-killed on any hak over ~1.4 GB, which blocks the NWSync backfill and every release that rebuilds a large hak. Emit now opens the artifact, hashes it by streaming for the key check, parses only the header and resource table via erf.ReadIndex, and reads, hashes, compresses and stores one payload at a time. erf.Read keeps its old shape but returns payloads as subslices instead of fresh copies, which removes the second copy for the other callers too. The zstd encoder pool also held one window-sized history per CPU — about 200 MB of live heap on a 24-core runner. EncodeAll is single-threaded per call, so concurrency 1 gives byte-identical blobs for far less memory. Peak heap is now flat at ~22 MB for both an 8 MB and a 64 MB hak, and a regression test asserts it does not scale with artifact size. Closes #76 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -22,9 +22,13 @@ const (
|
||||
blobHeaderBytes = 24
|
||||
)
|
||||
|
||||
// EncodeAll/DecodeAll are single-threaded per call, so the default pool of one
|
||||
// encoder per CPU only buys idle memory: each holds a window-sized history, so
|
||||
// on a 24-core runner that is ~200 MB of live heap doing nothing. Concurrency 1
|
||||
// produces byte-identical output.
|
||||
var (
|
||||
blobEncoder, _ = zstd.NewWriter(nil)
|
||||
blobDecoder, _ = zstd.NewReader(nil)
|
||||
blobEncoder, _ = zstd.NewWriter(nil, zstd.WithEncoderConcurrency(1))
|
||||
blobDecoder, _ = zstd.NewReader(nil, zstd.WithDecoderConcurrency(1))
|
||||
)
|
||||
|
||||
// compressBlob wraps data in NWCompressedBuffer framing.
|
||||
|
||||
Reference in New Issue
Block a user