feat(nwsync): emit blobs in parallel with a bounded worker pool (#79)
ci / ci (pull_request) Successful in 3m32s
ci / ci (pull_request) Successful in 3m32s
Emit is latency-bound, not CPU-bound. Every blob costs two serial HTTP round-trips to the zone — an existence probe, then an upload — so a hak with a few thousand resources pays a few thousand serialised latencies. A measured backfill spent 26 seconds of CPU across 9.5 minutes of wall clock, on a host with three of four cores idle and 5 GB free. Emit now hashes, compresses and stores `--jobs N` resources at once (default 16, matching DEPOT_JOBS and the transport's idle connections per host). Three properties had to survive, and each has a test: - The manifest's bytes are promised deterministic by emitterVersion, so entries is index-addressed rather than appended to: a worker owns entries[i] alone and the slice comes back in artifact order whatever order the uploads finish in. - The index is still the publication marker, so any worker's failure aborts the run before a manifest is written. Workers drain the rest of the channel instead of returning, which keeps the feeder from blocking on workers that have gone away. - Two resrefs holding identical bytes still share one blob. Serially the sink's existence check absorbed that; in parallel both workers would probe, both miss, and both upload. Claiming the sha1 in-process restores the dedupe and skips a probe round-trip as well. Peak memory is now the resources in flight rather than one resource, so the ceiling is N times the 15 MB per-resource limit and its compressed copy — bounded by a constant this package enforces itself, and still nowhere near tracking the archive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -81,6 +81,7 @@ func runEmit(args []string, stdout, stderr io.Writer) int {
|
||||
fs.SetOutput(stderr)
|
||||
as := fs.String("as", "", "published name of the artifact, when it differs from the key")
|
||||
out := fs.String("out", "", "write to a local repository tree instead of uploading")
|
||||
jobs := fs.Int("jobs", defaultEmitJobs, "resources to hash, compress and store at once")
|
||||
positional, err := parseArgs(fs, args)
|
||||
if err != nil {
|
||||
return exitUsage
|
||||
@@ -89,12 +90,17 @@ func runEmit(args []string, stdout, stderr io.Writer) int {
|
||||
fmt.Fprintf(stderr, "nwsync emit: <artifact-key> and <file> are both required\n")
|
||||
return exitUsage
|
||||
}
|
||||
if *jobs < 1 {
|
||||
fmt.Fprintf(stderr, "nwsync emit: -jobs must be at least 1, got %d\n", *jobs)
|
||||
return exitUsage
|
||||
}
|
||||
|
||||
result, err := Emit(EmitOptions{
|
||||
ArtifactKey: positional[0],
|
||||
ArtifactPath: positional[1],
|
||||
As: *as,
|
||||
OutDir: *out,
|
||||
Jobs: *jobs,
|
||||
})
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "nwsync emit: %v\n", err)
|
||||
|
||||
Reference in New Issue
Block a user