nwsync emit: upload blobs in parallel with a bounded worker count #79

New Issue

2026-07-31T13:35:07Z

archvillainette commented

2026-07-31 13:35:07 +00:00

Problem

nwsync emit uploads one blob at a time. Every resource costs two serial HTTP round-trips to the Bunny zone — a ProbeKey HEAD, then a PutReader PUT (internal/nwsync/sink.go:94). A hak with a few thousand resources therefore pays a few thousand serialised latencies, and a full backfill across ten haks takes hours.

Evidence

From the live sow-assets-manifest backfill (run 4996, nix-docker on ovh-main, 4 cores):

8.5 min wall for the first hak against 23 s of CPU — 95 %+ of the time is spent waiting on the network, not compressing.
Peak RSS across the whole run: 85 MB (85004 KB), against a 2.15 GB hak. Streaming emit (#78) did its job.
Over 1 h 15 m elapsed and not yet through the ten haks.

The host is neither CPU-bound nor memory-bound. It is latency-bound on serial PUTs.

Proposal

Add a bounded worker pool to the blob loop in emitResources (internal/nwsync/emit.go:214), with a -jobs N flag on nwsync emit (internal/nwsync/run.go:79). Default something conservative like 8; -jobs 1 restores today's behaviour exactly.

Shape:

Keep the order slice as the single source of ordering. Preallocate entries to len(order) and have each worker write its own index, so the manifest stays byte-identical to the serial version regardless of completion order. This is not optional — emitterVersion promises deterministic bytes.
blobs and onDiskBytes become atomics, or each worker returns its own tally and they are summed after the pool drains.
Use errgroup with SetLimit(N) and a context, so the first failure cancels the rest. golang.org/x/sync is not in go.mod yet; a plain semaphore channel plus a sync.Once error is fine too and adds no dependency.
erf.ReadPayload takes an io.ReaderAt, so concurrent reads of the artifact are already safe — no shared file offset. Worth a comment saying so, since it is the reason this is cheap.

Why this does not blow up compute

The memory ceiling is already bounded by fileSizeLimit (15 MB, internal/nwsync/emit.go:24). Each in-flight worker holds at most one payload plus its compressed copy, so worst case is roughly N × 30 MB — about 240 MB at -jobs 8, on top of the ~85 MB baseline. Real resources are far smaller than the limit, so the true figure will be a fraction of that. That fits the runner with room to spare, and it is the reason this is worth doing now rather than before #78: pre-streaming, N workers would have meant N whole archives.

CPU is not a concern either — compression is the only real work and it currently leaves three of four cores idle.

Please confirm the ceiling with a test rather than by argument: the existing internal/nwsync/memory_test.go is the right place to assert peak allocation stays bounded at -jobs N.

Also worth measuring, secondary

Bunny PUT is idempotent, so the ProbeKey HEAD before each upload could be skipped on a first-time emit — halving round-trips. But it doubles uploaded bytes on a re-run, which is exactly what a backfill is. Parallelism is the better lever; mentioning this only so it is not rediscovered later. Do not do both blindly.

Constraints that must survive

Index lands last. putManifestPair must not run if any blob failed — the presence of a .nsym is the publication marker (internal/nwsync/emit.go:76).
Manifest bytes unchanged for a given artifact at any -jobs value. A round-trip test at -jobs 1 vs -jobs 8 producing identical .nsym output covers this.
ProbeKey's fail-closed semantics stay: only a confirmed Present skips an upload; a throttled probe still uploads.

Acceptance

Backfilling the ten sow-assets-manifest haks finishes in materially less wall time on the same runner, with peak RSS still comfortably inside the box, and identical .nsym bytes to a serial run.

Context: sow-assets-manifest#57 / #58, which added an interim NWSYNC_MAX_HAK_BYTES size guard, now obsolete after #78. Related: #54.

### Problem `nwsync emit` uploads one blob at a time. Every resource costs two serial HTTP round-trips to the Bunny zone — a `ProbeKey` HEAD, then a `PutReader` PUT (`internal/nwsync/sink.go:94`). A hak with a few thousand resources therefore pays a few thousand serialised latencies, and a full backfill across ten haks takes hours. ### Evidence From the live sow-assets-manifest backfill (run 4996, `nix-docker` on ovh-main, 4 cores): - 8.5 min wall for the first hak against **23 s of CPU** — 95 %+ of the time is spent waiting on the network, not compressing. - Peak RSS across the whole run: **85 MB** (85004 KB), against a 2.15 GB hak. Streaming emit (#78) did its job. - Over 1 h 15 m elapsed and not yet through the ten haks. The host is neither CPU-bound nor memory-bound. It is latency-bound on serial PUTs. ### Proposal Add a bounded worker pool to the blob loop in `emitResources` (`internal/nwsync/emit.go:214`), with a `-jobs N` flag on `nwsync emit` (`internal/nwsync/run.go:79`). Default something conservative like 8; `-jobs 1` restores today's behaviour exactly. Shape: - Keep the `order` slice as the single source of ordering. Preallocate `entries` to `len(order)` and have each worker write its own index, so the manifest stays byte-identical to the serial version regardless of completion order. This is not optional — `emitterVersion` promises deterministic bytes. - `blobs` and `onDiskBytes` become atomics, or each worker returns its own tally and they are summed after the pool drains. - Use `errgroup` with `SetLimit(N)` and a context, so the first failure cancels the rest. `golang.org/x/sync` is not in `go.mod` yet; a plain semaphore channel plus a `sync.Once` error is fine too and adds no dependency. - `erf.ReadPayload` takes an `io.ReaderAt`, so concurrent reads of the artifact are already safe — no shared file offset. Worth a comment saying so, since it is the reason this is cheap. ### Why this does not blow up compute The memory ceiling is already bounded by `fileSizeLimit` (15 MB, `internal/nwsync/emit.go:24`). Each in-flight worker holds at most one payload plus its compressed copy, so worst case is roughly `N × 30 MB` — about 240 MB at `-jobs 8`, on top of the ~85 MB baseline. Real resources are far smaller than the limit, so the true figure will be a fraction of that. That fits the runner with room to spare, and it is the reason this is worth doing *now* rather than before #78: pre-streaming, N workers would have meant N whole archives. CPU is not a concern either — compression is the only real work and it currently leaves three of four cores idle. Please confirm the ceiling with a test rather than by argument: the existing `internal/nwsync/memory_test.go` is the right place to assert peak allocation stays bounded at `-jobs N`. ### Also worth measuring, secondary Bunny PUT is idempotent, so the `ProbeKey` HEAD before each upload could be skipped on a first-time emit — halving round-trips. But it doubles uploaded bytes on a re-run, which is exactly what a backfill is. Parallelism is the better lever; mentioning this only so it is not rediscovered later. Do not do both blindly. ### Constraints that must survive - Index lands last. `putManifestPair` must not run if any blob failed — the presence of a `.nsym` is the publication marker (`internal/nwsync/emit.go:76`). - Manifest bytes unchanged for a given artifact at any `-jobs` value. A round-trip test at `-jobs 1` vs `-jobs 8` producing identical `.nsym` output covers this. - `ProbeKey`'s fail-closed semantics stay: only a confirmed `Present` skips an upload; a throttled probe still uploads. ### Acceptance Backfilling the ten sow-assets-manifest haks finishes in materially less wall time on the same runner, with peak RSS still comfortably inside the box, and identical `.nsym` bytes to a serial run. Context: sow-assets-manifest#57 / #58, which added an interim `NWSYNC_MAX_HAK_BYTES` size guard, now obsolete after #78. Related: #54.

archvillainette referenced this issue from a commit

2026-07-31 14:28:53 +00:00

feat(nwsync): emit blobs in parallel with a bounded worker pool (#79)

archvillainette referenced a pull request that will close this issue

2026-07-31 14:29:10 +00:00

feat(nwsync): emit blobs in parallel with a bounded worker pool #80

archvillainette commented

2026-07-31 15:12:39 +00:00

Measured numbers from the completed backfill (sow-assets-manifest run 4996, nix-docker on ovh-main, 4 cores, 7.7 GB RAM). Sampler was a read-only ps poll every 5 s, so the figures below are wall time per crucible nwsync emit process and its peak RSS.

The sampler was started ~10 minutes into the run, so sow_top, sow_over_01 and sow_core_01 are not in the table. sow_over_01 was separately measured earlier at 85 MB peak against a 2.15 GB hak.

pass	hak	emit wall	peak RSS
1	`sow_part_01`	21m 0s	46 MB
1	`sow_appr_01`	3m 23s	86 MB
1	`sow_item_01`	19m 0s	55 MB
1	`sow_envi_01`	24m 33s	108 MB
1	`sow_vfxs_01`	4m 2s	53 MB
2	`sow_appr_01`	0m 45s	68 MB
2	`sow_item_01`	11m 17s	74 MB
2	`sow_item_02`	2m 58s	67 MB
2	`sow_envi_01`	5m 57s	74 MB

Sampled emit time: 92 min. Whole job: ~2h 20m.

The latency claim holds across every hak, not just the one

Spot-checked CPU against wall: sow_envi_01 burned 26 s of CPU across 9.5 minutes of wall clock. Downloads are not the cost either — the gaps between emits (depot fetch plus sha256 verify of a multi-gigabyte hak) totalled 258 s across all nine, against 5571 s spent inside emit. So roughly 95 % of the job is emit, and ~95 % of emit is waiting on Bunny round-trips.

Pass 2 is the strongest argument for this issue

Pass 2 is a second tag. The hak digests differ, so NWSYNC_EMITTED does not skip them — but their blobs are almost all already in the zone. Same haks, second time:

sow_appr_01: 3m 23s → 0m 45s
sow_envi_01: 24m 33s → 5m 57s
sow_item_01: 19m 0s → 11m 17s

Those runs upload almost nothing. They are nearly pure ProbeKey latency — one HEAD per resource, answered "already here", serially. Six minutes of a 4-core box doing nothing but waiting for HEADs is exactly the workload a worker pool erases, and it is the common case: every future release re-emits mostly-unchanged haks under new digests.

Memory

Peak across the whole backfill: 108 MB, against haks up to 2.15 GB. #78 holds in production.

That also settles the sizing question for -jobs N. The per-worker ceiling is fileSizeLimit (15 MB) plus its compressed copy, and the observed baseline is ~50-110 MB, so -jobs 16 fits in a few hundred MB on a box with 5 GB free.

Implemented in #80.

Measured numbers from the completed backfill (sow-assets-manifest run 4996, `nix-docker` on ovh-main, 4 cores, 7.7 GB RAM). Sampler was a read-only `ps` poll every 5 s, so the figures below are wall time per `crucible nwsync emit` process and its peak RSS. The sampler was started ~10 minutes into the run, so `sow_top`, `sow_over_01` and `sow_core_01` are not in the table. `sow_over_01` was separately measured earlier at 85 MB peak against a 2.15 GB hak. | pass | hak | emit wall | peak RSS | |---|---|---:|---:| | 1 | `sow_part_01` | 21m 0s | 46 MB | | 1 | `sow_appr_01` | 3m 23s | 86 MB | | 1 | `sow_item_01` | 19m 0s | 55 MB | | 1 | `sow_envi_01` | 24m 33s | 108 MB | | 1 | `sow_vfxs_01` | 4m 2s | 53 MB | | 2 | `sow_appr_01` | 0m 45s | 68 MB | | 2 | `sow_item_01` | 11m 17s | 74 MB | | 2 | `sow_item_02` | 2m 58s | 67 MB | | 2 | `sow_envi_01` | 5m 57s | 74 MB | Sampled emit time: **92 min**. Whole job: ~2h 20m. ### The latency claim holds across every hak, not just the one Spot-checked CPU against wall: `sow_envi_01` burned 26 s of CPU across 9.5 minutes of wall clock. Downloads are not the cost either — the gaps between emits (depot fetch plus sha256 verify of a multi-gigabyte hak) totalled **258 s across all nine**, against 5571 s spent inside emit. So roughly 95 % of the job is emit, and ~95 % of emit is waiting on Bunny round-trips. ### Pass 2 is the strongest argument for this issue Pass 2 is a second tag. The hak digests differ, so `NWSYNC_EMITTED` does not skip them — but their *blobs* are almost all already in the zone. Same haks, second time: - `sow_appr_01`: 3m 23s → **0m 45s** - `sow_envi_01`: 24m 33s → **5m 57s** - `sow_item_01`: 19m 0s → **11m 17s** Those runs upload almost nothing. They are nearly pure `ProbeKey` latency — one HEAD per resource, answered "already here", serially. Six minutes of a 4-core box doing nothing but waiting for HEADs is exactly the workload a worker pool erases, and it is the *common* case: every future release re-emits mostly-unchanged haks under new digests. ### Memory Peak across the whole backfill: **108 MB**, against haks up to 2.15 GB. #78 holds in production. That also settles the sizing question for `-jobs N`. The per-worker ceiling is `fileSizeLimit` (15 MB) plus its compressed copy, and the observed baseline is ~50-110 MB, so `-jobs 16` fits in a few hundred MB on a box with 5 GB free. Implemented in #80.

archvillainette closed this issue

2026-07-31 15:25:34 +00:00

archvillainette referenced this issue from a commit

2026-07-31 15:25:35 +00:00

feat(nwsync): emit blobs in parallel with a bounded worker pool (#80)

archvillainette referenced this issue

2026-07-31 15:36:19 +00:00

Map: NWSync publishing — emit at artifact birth, assemble at module release #54

archvillainette referenced this issue

2026-07-31 16:12:23 +00:00

Map: NWSync publishing — emit at artifact birth, assemble at module release #54

archvillainette referenced this issue