archvillainette
0 Followers · 0 Following
  • Joined on 2026-03-17
Repositories Projects Packages Public Activity Starred Repositories
archvillainette opened issue ShadowsOverWestgate/sow-tools#79 2026-07-31 13:35:07 +00:00
nwsync emit: upload blobs in parallel with a bounded worker count
archvillainette released v0.3.31 at ShadowsOverWestgate/sow-tools 2026-07-31 11:49:19 +00:00
archvillainette pushed tag v0.3.31 to ShadowsOverWestgate/sow-tools 2026-07-31 11:39:13 +00:00
archvillainette pushed to main at ShadowsOverWestgate/sow-tools 2026-07-31 11:16:25 +00:00
fa32dd411f fix(nwsync): stream emit so peak memory tracks the largest resource (#76) (#78)
archvillainette deleted branch fix/nwsync-emit-streaming from ShadowsOverWestgate/sow-tools 2026-07-31 11:16:25 +00:00
archvillainette closed issue ShadowsOverWestgate/sow-tools#76 2026-07-31 11:16:24 +00:00
nwsync emit holds ~5x the artifact size in RAM; OOM-killed on haks over ~1.4 GB
archvillainette merged pull request ShadowsOverWestgate/sow-tools#78 2026-07-31 11:16:23 +00:00
fix(nwsync): stream emit so peak memory tracks the largest resource (#76)
archvillainette commented on issue ShadowsOverWestgate/sow-tools#54 2026-07-31 11:10:41 +00:00
Map: NWSync publishing — emit at artifact birth, assemble at module release

sow-tools#76 has a fix up in #78: emit streams now, so peak memory tracks the largest single resource instead of the archive.

This is the standing "no bulk data on ovh-main" rule reaching the…

archvillainette commented on issue ShadowsOverWestgate/sow-tools#76 2026-07-31 11:10:27 +00:00
nwsync emit holds ~5x the artifact size in RAM; OOM-killed on haks over ~1.4 GB

Fix in #78 (branch fix/nwsync-emit-streaming), not merged yet.

Emit now opens the artifact, hashes it by streaming for the key check, parses only the header and resource table (erf.ReadIndex),…

archvillainette created pull request ShadowsOverWestgate/sow-tools#78 2026-07-31 11:09:57 +00:00
fix(nwsync): stream emit so peak memory tracks the largest resource (#76)
ba671a76ed fix(nwsync): hash through a section reader, pin the encoder claim (#76)
59b233e3db fix(nwsync): stream emit so peak memory tracks the largest resource (#76)
Compare 2 commits »
archvillainette created branch fix/nwsync-emit-streaming in ShadowsOverWestgate/sow-tools 2026-07-31 11:09:35 +00:00
archvillainette deleted branch docs-nwsync-blob-format from ShadowsOverWestgate/sow-tools 2026-07-31 11:00:51 +00:00
archvillainette pushed to main at ShadowsOverWestgate/sow-tools 2026-07-31 11:00:50 +00:00
7437653f14 docs: describe the on-disk shape of an emitted NWSync tree (#77)
archvillainette merged pull request ShadowsOverWestgate/sow-tools#77 2026-07-31 11:00:49 +00:00
docs: describe the on-disk shape of an emitted NWSync tree
archvillainette created pull request ShadowsOverWestgate/sow-tools#77 2026-07-31 10:51:26 +00:00
docs: describe the on-disk shape of an emitted NWSync tree
archvillainette created branch docs-nwsync-blob-format in ShadowsOverWestgate/sow-tools 2026-07-31 10:51:17 +00:00
a33e39d422 docs: describe the on-disk shape of an emitted NWSync tree
archvillainette commented on issue ShadowsOverWestgate/sow-tools#76 2026-07-31 10:50:12 +00:00
nwsync emit holds ~5x the artifact size in RAM; OOM-killed on haks over ~1.4 GB

Correction to the threshold in the title: at the measured 4.9x, a 5 GB budget is exhausted by a hak of about 1 GB, not 1.4 GB. Six of the ten haks in sow-assets-manifest v0.1.7 are over that…

archvillainette commented on issue ShadowsOverWestgate/sow-tools#76 2026-07-31 10:47:29 +00:00
nwsync emit holds ~5x the artifact size in RAM; OOM-killed on haks over ~1.4 GB

Implementation plan

The peak is bounded by a constant that already exists. internal/nwsync/emit.go:24:

// fileSizeLimit matches upstream's --limit-file-size default of 15 MB.
const…