fix(nwsync): stream emit so peak memory tracks the largest resource (#76)

Conversation 1 Commits 2 Files Changed 5

+313 -93

2 Commits

Author	SHA1	Message	Date
archvillainetteandClaude Opus 5	ba671a76ed	fix(nwsync): hash through a section reader, pin the encoder claim (#76 ) ci / ci (pull_request) Successful in 3m30s Details Review follow-ups. checkArtifactKey drained the *os.File to EOF, so the file offset was left at the end; correct only because everything after it uses ReadAt. Hash a section reader instead, so no later sequential read can silently see nothing. The zstd concurrency comment claimed byte-identical output but nothing asserted it, so add the test. erf.Read now says outright that Data must not be mutated, since payloads alias one buffer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 13:08:35 +02:00
archvillainetteandClaude Opus 5	59b233e3db	fix(nwsync): stream emit so peak memory tracks the largest resource (#76 ) Emit read the whole artifact into memory and erf.Read then allocated a second full copy of every payload, so a hak cost about 5x its size in RAM. The 7 GB runner was OOM-killed on any hak over ~1.4 GB, which blocks the NWSync backfill and every release that rebuilds a large hak. Emit now opens the artifact, hashes it by streaming for the key check, parses only the header and resource table via erf.ReadIndex, and reads, hashes, compresses and stores one payload at a time. erf.Read keeps its old shape but returns payloads as subslices instead of fresh copies, which removes the second copy for the other callers too. The zstd encoder pool also held one window-sized history per CPU — about 200 MB of live heap on a 24-core runner. EncodeAll is single-threaded per call, so concurrency 1 gives byte-identical blobs for far less memory. Peak heap is now flat at ~22 MB for both an 8 MB and a 64 MB hak, and a regression test asserts it does not scale with artifact size. Closes #76 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 13:06:29 +02:00

Author

SHA1

Message

Date

archvillainetteandClaude Opus 5

ba671a76ed

fix(nwsync): hash through a section reader, pin the encoder claim (#76 )

ci / ci (pull_request) Successful in 3m30s

Details

Review follow-ups. checkArtifactKey drained the *os.File to EOF, so the
file offset was left at the end; correct only because everything after it
uses ReadAt. Hash a section reader instead, so no later sequential read
can silently see nothing.

The zstd concurrency comment claimed byte-identical output but nothing
asserted it, so add the test. erf.Read now says outright that Data must
not be mutated, since payloads alias one buffer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-07-31 13:08:35 +02:00

archvillainetteandClaude Opus 5

59b233e3db

fix(nwsync): stream emit so peak memory tracks the largest resource (#76 )

Emit read the whole artifact into memory and erf.Read then allocated a
second full copy of every payload, so a hak cost about 5x its size in
RAM. The 7 GB runner was OOM-killed on any hak over ~1.4 GB, which
blocks the NWSync backfill and every release that rebuilds a large hak.

Emit now opens the artifact, hashes it by streaming for the key check,
parses only the header and resource table via erf.ReadIndex, and reads,
hashes, compresses and stores one payload at a time. erf.Read keeps its
old shape but returns payloads as subslices instead of fresh copies,
which removes the second copy for the other callers too.

The zstd encoder pool also held one window-sized history per CPU — about
200 MB of live heap on a 24-core runner. EncodeAll is single-threaded
per call, so concurrency 1 gives byte-identical blobs for far less
memory.

Peak heap is now flat at ~22 MB for both an 8 MB and a 64 MB hak, and a
regression test asserts it does not scale with artifact size.

Closes #76

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-07-31 13:06:29 +02:00

fix(nwsync): stream emit so peak memory tracks the largest resource (#76) #78