nwsync emit held roughly 5× the artifact size in RAM, so ovh-main (7 GB, no swap) OOM-killed it on any hak over ~1.4 GB. That blocked the backfill in sow-assets-manifest and would have killed the next release rebuilding a large hak.
What it does
erf.ReadIndex / erf.ReadPayload — parse the header and resource table only, read one payload on demand. erf.Read keeps its shape but returns payloads as subslices of the buffer instead of fresh copies, which removes one full copy for the pipeline callers too. Callers must not mutate Resource.Data; the doc comment says so and no caller does.
nwsync.Emit opens the artifact, hashes it by streaming for the key check (through a section reader, so the file offset stays put), then hashes, compresses and stores one resource at a time. The archive is never resident. Shadowed duplicate resrefs are now never read at all.
Bounds checks in ReadIndex moved to int64, so key/resource-list offsets can no longer overflow.
Not in the issue, but memory-motivated
The zstd blob encoder ran at the default concurrency, which is one encoder per CPU, each holding a window-sized history — about 200 MB of live heap doing nothing on a 24-core runner. EncodeAll is single-threaded per call, so concurrency 1 costs nothing. TestSingleThreadedEncoderMatchesDefault pins the claim that blobs come out byte-identical.
Measured
Peak heap during emit, sampled 1 ms:
hak
before
after
8 MB
155 MB
21 MB
64 MB
289 MB
22 MB
Flat, as the acceptance asks. TestEmitPeakMemoryDoesNotScaleWithArtifactSize fails if the 64 MB fixture costs more than the 8 MB one plus 24 MB of slack.
Gaps
The regression check measures Go heap, not RSS, and its largest fixture is 64 MB — a multi-GB run was not done here. A 2.15 GB hak now needs about the same ~22 MB the 64 MB one does, so the 5 GB budget is not close, but that is inference from the flat curve, not a measurement.
mmap was suggested in the issue and skipped. Payload buffers are still anonymous, but they are one resource each (≤15 MiB), so making them file-backed buys nothing now.
Closes #76. Part of #54.
`nwsync emit` held roughly 5× the artifact size in RAM, so ovh-main (7 GB, no swap) OOM-killed it on any hak over ~1.4 GB. That blocked the backfill in sow-assets-manifest and would have killed the next release rebuilding a large hak.
## What it does
- `erf.ReadIndex` / `erf.ReadPayload` — parse the header and resource table only, read one payload on demand. `erf.Read` keeps its shape but returns payloads as subslices of the buffer instead of fresh copies, which removes one full copy for the `pipeline` callers too. Callers must not mutate `Resource.Data`; the doc comment says so and no caller does.
- `nwsync.Emit` opens the artifact, hashes it by streaming for the key check (through a section reader, so the file offset stays put), then hashes, compresses and stores one resource at a time. The archive is never resident. Shadowed duplicate resrefs are now never read at all.
- Bounds checks in `ReadIndex` moved to `int64`, so key/resource-list offsets can no longer overflow.
## Not in the issue, but memory-motivated
The zstd blob encoder ran at the default concurrency, which is one encoder per CPU, each holding a window-sized history — about 200 MB of live heap doing nothing on a 24-core runner. `EncodeAll` is single-threaded per call, so concurrency 1 costs nothing. `TestSingleThreadedEncoderMatchesDefault` pins the claim that blobs come out byte-identical.
## Measured
Peak heap during emit, sampled 1 ms:
| hak | before | after |
|-----|--------|-------|
| 8 MB | 155 MB | 21 MB |
| 64 MB | 289 MB | 22 MB |
Flat, as the acceptance asks. `TestEmitPeakMemoryDoesNotScaleWithArtifactSize` fails if the 64 MB fixture costs more than the 8 MB one plus 24 MB of slack.
## Gaps
- The regression check measures Go heap, not RSS, and its largest fixture is 64 MB — a multi-GB run was not done here. A 2.15 GB hak now needs about the same ~22 MB the 64 MB one does, so the 5 GB budget is not close, but that is inference from the flat curve, not a measurement.
- mmap was suggested in the issue and skipped. Payload buffers are still anonymous, but they are one resource each (≤15 MiB), so making them file-backed buys nothing now.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Emit read the whole artifact into memory and erf.Read then allocated a
second full copy of every payload, so a hak cost about 5x its size in
RAM. The 7 GB runner was OOM-killed on any hak over ~1.4 GB, which
blocks the NWSync backfill and every release that rebuilds a large hak.
Emit now opens the artifact, hashes it by streaming for the key check,
parses only the header and resource table via erf.ReadIndex, and reads,
hashes, compresses and stores one payload at a time. erf.Read keeps its
old shape but returns payloads as subslices instead of fresh copies,
which removes the second copy for the other callers too.
The zstd encoder pool also held one window-sized history per CPU — about
200 MB of live heap on a 24-core runner. EncodeAll is single-threaded
per call, so concurrency 1 gives byte-identical blobs for far less
memory.
Peak heap is now flat at ~22 MB for both an 8 MB and a 64 MB hak, and a
regression test asserts it does not scale with artifact size.
Closes#76
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-ups. checkArtifactKey drained the *os.File to EOF, so the
file offset was left at the end; correct only because everything after it
uses ReadAt. Hash a section reader instead, so no later sequential read
can silently see nothing.
The zstd concurrency comment claimed byte-identical output but nothing
asserted it, so add the test. erf.Read now says outright that Data must
not be mutated, since payloads alias one buffer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
xtul
Closes #76. Part of #54.
nwsync emitheld roughly 5× the artifact size in RAM, so ovh-main (7 GB, no swap) OOM-killed it on any hak over ~1.4 GB. That blocked the backfill in sow-assets-manifest and would have killed the next release rebuilding a large hak.What it does
erf.ReadIndex/erf.ReadPayload— parse the header and resource table only, read one payload on demand.erf.Readkeeps its shape but returns payloads as subslices of the buffer instead of fresh copies, which removes one full copy for thepipelinecallers too. Callers must not mutateResource.Data; the doc comment says so and no caller does.nwsync.Emitopens the artifact, hashes it by streaming for the key check (through a section reader, so the file offset stays put), then hashes, compresses and stores one resource at a time. The archive is never resident. Shadowed duplicate resrefs are now never read at all.ReadIndexmoved toint64, so key/resource-list offsets can no longer overflow.Not in the issue, but memory-motivated
The zstd blob encoder ran at the default concurrency, which is one encoder per CPU, each holding a window-sized history — about 200 MB of live heap doing nothing on a 24-core runner.
EncodeAllis single-threaded per call, so concurrency 1 costs nothing.TestSingleThreadedEncoderMatchesDefaultpins the claim that blobs come out byte-identical.Measured
Peak heap during emit, sampled 1 ms:
Flat, as the acceptance asks.
TestEmitPeakMemoryDoesNotScaleWithArtifactSizefails if the 64 MB fixture costs more than the 8 MB one plus 24 MB of slack.Gaps
🤖 Generated with Claude Code