These land together on purpose. Fixing the encoder alone changes nothing for the blobs already in the zone, because emit skips whatever is already present.
klauspost/compress omits the zstd Frame_Content_Size field for inputs under 256 bytes, which the format permits. Reference libzstd never does, so the NWN client — which sizes its output buffer from ZSTD_getFrameContentSize and has therefore never met a frame without one — rejected roughly 6% of our blobs outright. Any single one stops a sync dead, so no client could complete a sync of the live manifest.
No encoder option changes this, so compressBlob re-headers the affected frames into the shape libzstd itself emits: Single_Segment_flag set, Window_Descriptor dropped, and the freed byte spent on a one-byte Frame_Content_Size. Same length in, same length out, and the same descriptor byte (0x24) the issue recorded from libzstd.
compressBlob then asserts its own output. An encoder upgrade that finds another way to omit the field would otherwise reproduce #86 in silence, and a blob is skipped by every later emit once written.
emitter_version goes to 2, so assemble refuses to merge an index written by the encoder that omitted the field.
Proved against the reference decoder, not just a round trip. A real emitted 175-byte blob:
Frames Skips Compressed Uncompressed Ratio Check Filename
1 0 48 B 175 B 3.646 XXH64 frame.zst
c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 - <- zstd -dc | sha1sum
c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 <- the blob's own name
Before the fix that Uncompressed column was blank.
crucible nwsync verify <manifest-sha1> reads a manifest and its blobs back through the public pull zone, with no credential, because what matters is the bytes a client is served, edge behaviour included. Every distinct blob is decompressed and hashed; failures are reported per blob as missing / malformed framing / size mismatch / hash mismatch, and the exit code is 1.
--sample N makes a routine check cheap against a manifest that is ~69,000 blobs and 15 GB; the default is a full sweep.
--base URL / NWSYNC_PULL_BASE overrides the public host.
The manifest is checked against its own sha1 before a single blob is fetched.
emit --verify applies the same check where emit would otherwise trust presence, and replaces a stored blob that is not what its name claims. This is what makes the #86 blobs repairable.
Why the existing checks missed this
Both new checks assert the frame property, not just a round trip. The conformance suite (#59) compares decompressed bytes, so a frame that decodes correctly passes regardless of its header; and the earlier zone audit decompressed 68 blobs with the zstd CLI, a more capable decoder than the client's, which certified exactly the blobs the client rejects.
Checks
make check and make smoke green. Second commit is the fixes from a two-axis review of the first.
Not in this PR
Three follow-ups, filed separately: the backfill has not been run, replacing a blob does not purge the pull-zone edge cache, and #85's runbook line belongs to sow-platform.
Closes #86. Closes #85.
These land together on purpose. Fixing the encoder alone changes nothing for the blobs already in the zone, because `emit` skips whatever is already present.
## #86 — the framing fix
`klauspost/compress` omits the zstd `Frame_Content_Size` field for inputs under 256 bytes, which the format permits. Reference libzstd never does, so the NWN client — which sizes its output buffer from `ZSTD_getFrameContentSize` and has therefore never met a frame without one — rejected roughly 6% of our blobs outright. Any single one stops a sync dead, so no client could complete a sync of the live manifest.
No encoder option changes this, so `compressBlob` re-headers the affected frames into the shape libzstd itself emits: `Single_Segment_flag` set, `Window_Descriptor` dropped, and the freed byte spent on a one-byte `Frame_Content_Size`. Same length in, same length out, and the same descriptor byte (`0x24`) the issue recorded from libzstd.
`compressBlob` then asserts its own output. An encoder upgrade that finds another way to omit the field would otherwise reproduce #86 in silence, and a blob is skipped by every later emit once written.
`emitter_version` goes to `2`, so `assemble` refuses to merge an index written by the encoder that omitted the field.
**Proved against the reference decoder, not just a round trip.** A real emitted 175-byte blob:
```
Frames Skips Compressed Uncompressed Ratio Check Filename
1 0 48 B 175 B 3.646 XXH64 frame.zst
c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 - <- zstd -dc | sha1sum
c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 <- the blob's own name
```
Before the fix that `Uncompressed` column was blank.
## #85 — `nwsync verify`
`crucible nwsync verify <manifest-sha1>` reads a manifest and its blobs back through the **public pull zone**, with no credential, because what matters is the bytes a client is served, edge behaviour included. Every distinct blob is decompressed and hashed; failures are reported per blob as missing / malformed framing / size mismatch / hash mismatch, and the exit code is 1.
- `--sample N` makes a routine check cheap against a manifest that is ~69,000 blobs and 15 GB; the default is a full sweep.
- `--base URL` / `NWSYNC_PULL_BASE` overrides the public host.
- The manifest is checked against its own sha1 before a single blob is fetched.
- `emit --verify` applies the same check where `emit` would otherwise trust presence, and replaces a stored blob that is not what its name claims. This is what makes the #86 blobs repairable.
## Why the existing checks missed this
Both new checks assert the **frame property**, not just a round trip. The conformance suite (#59) compares decompressed bytes, so a frame that decodes correctly passes regardless of its header; and the earlier zone audit decompressed 68 blobs with the `zstd` CLI, a *more* capable decoder than the client's, which certified exactly the blobs the client rejects.
## Checks
`make check` and `make smoke` green. Second commit is the fixes from a two-axis review of the first.
## Not in this PR
Three follow-ups, filed separately: the backfill has not been run, replacing a blob does not purge the pull-zone edge cache, and #85's runbook line belongs to `sow-platform`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
These two land together on purpose. Fixing the encoder alone changes nothing
for the blobs already in the zone, because emit skips whatever is already
present.
#86 — klauspost/compress omits the zstd Frame_Content_Size field for inputs
under 256 bytes, which the format permits. Reference libzstd never does, so the
NWN client — which sizes its output buffer from ZSTD_getFrameContentSize and has
therefore never met a frame without one — rejected roughly 6% of our blobs
outright. Any single one stops a sync dead, so no client could complete a sync
of the live manifest.
No encoder option changes this, so emit re-headers the affected frames into the
shape libzstd itself emits: Single_Segment_flag set, Window_Descriptor dropped,
and the freed byte spent on a one-byte Frame_Content_Size. Same length in, same
length out. Verified against the zstd CLI end to end: a real emitted 230-byte
blob now reports its decompressed size where it previously reported none.
emitter_version goes to 2, so assemble refuses to merge an index written by the
encoder that omitted the field.
#85 — a published blob was verified by exactly one thing, a player's client, at
download time. `nwsync verify <manifest-sha1>` now reads a manifest and its
blobs back through the public pull zone with no credential, decompresses and
hashes every one, and reports missing / malformed framing / size mismatch / hash
mismatch per blob. `--sample N` makes a routine check cheap against a manifest
that is ~69,000 blobs and 15 GB. `emit --verify` applies the same check where
emit would otherwise trust presence, and replaces a stored blob that is not what
its name claims — which is what makes the #86 blobs repairable.
Both new checks assert the frame property, not just a round trip. Round-tripping
cannot see this class of fault: both the zstd CLI and Go's decoder stream such a
frame happily, being more capable decoders than the client's, which is how the
defect reached production and survived an audit.
Refs #54, #55, #59, #75, sow-platform#79.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- compressBlob now asserts its own output declares a content size, instead of
trusting that the only way to lose the field is the one #86 found. An encoder
upgrade that finds another way would otherwise republish the same undecodable
blob in silence, and a blob is skipped by every later emit once written.
- inspectBlob asserts the frame whenever one is present, rather than only when
the payload is non-empty.
- dirSink stats instead of reading when not verifying. Reading every existing
blob back on the default path was a plain regression, and it contradicted the
documented promise that verifying is a repair pass, not the default.
- The manifest sha1 is now checked as hex before it is interpolated into a URL
path, rather than only measured.
- One blobKey rule for where a blob lives, replacing three copies of the
fanout-path expression.
- Renamed frameSizeThreshold to oneByteContentSizeCeiling, which says whose
threshold it is.
- The read-back in zoneSink reads the storage API on purpose; said so, and
corrected a comment that claimed a failed read-back re-uploads when it
aborts.
- Tests derive their expected blob counts from the run instead of hard-coding
fixture arithmetic, and the size-mismatch category has a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
xtul
Closes #86. Closes #85.
These land together on purpose. Fixing the encoder alone changes nothing for the blobs already in the zone, because
emitskips whatever is already present.#86 — the framing fix
klauspost/compressomits the zstdFrame_Content_Sizefield for inputs under 256 bytes, which the format permits. Reference libzstd never does, so the NWN client — which sizes its output buffer fromZSTD_getFrameContentSizeand has therefore never met a frame without one — rejected roughly 6% of our blobs outright. Any single one stops a sync dead, so no client could complete a sync of the live manifest.No encoder option changes this, so
compressBlobre-headers the affected frames into the shape libzstd itself emits:Single_Segment_flagset,Window_Descriptordropped, and the freed byte spent on a one-byteFrame_Content_Size. Same length in, same length out, and the same descriptor byte (0x24) the issue recorded from libzstd.compressBlobthen asserts its own output. An encoder upgrade that finds another way to omit the field would otherwise reproduce #86 in silence, and a blob is skipped by every later emit once written.emitter_versiongoes to2, soassemblerefuses to merge an index written by the encoder that omitted the field.Proved against the reference decoder, not just a round trip. A real emitted 175-byte blob:
Before the fix that
Uncompressedcolumn was blank.#85 —
nwsync verifycrucible nwsync verify <manifest-sha1>reads a manifest and its blobs back through the public pull zone, with no credential, because what matters is the bytes a client is served, edge behaviour included. Every distinct blob is decompressed and hashed; failures are reported per blob as missing / malformed framing / size mismatch / hash mismatch, and the exit code is 1.--sample Nmakes a routine check cheap against a manifest that is ~69,000 blobs and 15 GB; the default is a full sweep.--base URL/NWSYNC_PULL_BASEoverrides the public host.emit --verifyapplies the same check whereemitwould otherwise trust presence, and replaces a stored blob that is not what its name claims. This is what makes the #86 blobs repairable.Why the existing checks missed this
Both new checks assert the frame property, not just a round trip. The conformance suite (#59) compares decompressed bytes, so a frame that decodes correctly passes regardless of its header; and the earlier zone audit decompressed 68 blobs with the
zstdCLI, a more capable decoder than the client's, which certified exactly the blobs the client rejects.Checks
make checkandmake smokegreen. Second commit is the fixes from a two-axis review of the first.Not in this PR
Three follow-ups, filed separately: the backfill has not been run, replacing a blob does not purge the pull-zone edge cache, and #85's runbook line belongs to
sow-platform.🤖 Generated with Claude Code