Blobs under 256 bytes omit the zstd frame content size, and no client can sync past one #86

New Issue

2026-07-31T22:04:31Z

archvillainette commented

2026-07-31 22:04:31 +00:00

Every client sync fails, and it is our zstd framing

A real NWN client cannot complete a sync against the live zone. It stops with:

Download corrupted; checksum mismatch on file 017f680287c9f1617dd92d7426f51d29577ec38f
(potential compression error: ). This is likely a server-side repository issue.

Deterministic: two runs, same file, from 8593 files remaining to 8590. Not transient, not the client's cache — that blob is absent from the client store (nwsyncdata_0.sqlite3, 64,715 rows), so it is being fetched fresh and rejected each time.

The blob is byte-correct at rest. It decompresses to exactly the 230 bytes its NSYC header declares, and sha1(uncompressed) equals its own filename and the manifest entry (blood_fore, restype 2072, size 230).

What is wrong is the zstd frame header.

accepted by client:  28 b5 2f fd 60 ...   FHD 0x60 -> Frame_Content_Size present
rejected by client:  28 b5 2f fd 04 ...   FHD 0x04 -> Frame_Content_Size ABSENT

zstd -l on the rejected frame reports no "Decompressed Size" at all. A decoder that sizes its output buffer from the frame — ZSTD_getFrameContentSize, which returns ZSTD_CONTENTSIZE_UNKNOWN here — has nothing to allocate against and fails. That is the client's empty (potential compression error: ).

Reproduced against our own encoder

Probing compressBlob directly with the exact failing content produced the byte-identical 203-byte blob, and the threshold is sharp:

input=230 blob=203 FHD=0x4 fcs=false     <- the live failure, reproduced
current encoder declares FCS from 256 bytes up
  size=100/175/186/230 -> FHD 0x04, no FCS
  size=402/1000        -> FHD 0x44, FCS present

klauspost/compress omits Frame_Content_Size for inputs under 256 bytes. No encoder option changes it — tried WithZeroFrames, WithWindowSize, SpeedBestCompression, and streaming; all still 0x04.

Reference libzstd on the identical input does declare it:

libzstd:    FHD 0x24, Decompressed Size: 230 B, 178 bytes
ours (Go):  FHD 0x04, no declared size,        179 bytes

Upstream nwn_nwsync_write compresses through libzstd, so every blob it has ever written declares a content size. The client has never had to cope without one. Our Go emitter is the odd one out.

Scale

Sampling 50 random manifest entries under 300 KB: 3 lacked a content size (6%), all small — 172, 175, 186 bytes. Every affected blob is an unconditional stop for any client that reaches it, so in practice no client can complete a sync of this manifest. This blocks the pre-alpha (sow-platform#79).

Fix

Emit frames that always declare a content size. Since no encoder option does it, the options are to re-frame small outputs by hand (rewrite the FHD and insert the field — the uncompressed size is already known and already in the NSYC header) or to compress through libzstd.
Re-emit the affected blobs. This is the part that will bite: emit skips objects that already exist via a 1-byte range GET (#55), so a corrected encoder alone changes nothing for the ~thousands of small blobs already published. It needs a force or verify-and-replace path — see #85, which this makes urgent rather than merely prudent.

Why nothing caught it

Worth recording, because both gaps are the same shape:

The conformance test (#59) compares decompressed bytes, so a frame that decodes to the right data passes regardless of its header. It cannot see this class of fault.
My own zone audit missed it too. I verified 68 blobs end to end — decompress, hash, compare — and called the zone healthy. The zstd CLI streams happily without a declared size, so a more capable decoder than the client's hid the exact defect the client trips on. A verifier has to assert frame properties, not just round-trip success.

Refs #54, #55, #59, #75, #85, sow-platform#79.

## Every client sync fails, and it is our zstd framing A real NWN client cannot complete a sync against the live zone. It stops with: ``` Download corrupted; checksum mismatch on file 017f680287c9f1617dd92d7426f51d29577ec38f (potential compression error: ). This is likely a server-side repository issue. ``` Deterministic: two runs, same file, from 8593 files remaining to 8590. Not transient, not the client's cache — that blob is absent from the client store (`nwsyncdata_0.sqlite3`, 64,715 rows), so it is being fetched fresh and rejected each time. **The blob is byte-correct at rest.** It decompresses to exactly the 230 bytes its NSYC header declares, and `sha1(uncompressed)` equals its own filename and the manifest entry (`blood_fore`, restype 2072, size 230). **What is wrong is the zstd frame header.** ``` accepted by client: 28 b5 2f fd 60 ... FHD 0x60 -> Frame_Content_Size present rejected by client: 28 b5 2f fd 04 ... FHD 0x04 -> Frame_Content_Size ABSENT ``` `zstd -l` on the rejected frame reports no "Decompressed Size" at all. A decoder that sizes its output buffer from the frame — `ZSTD_getFrameContentSize`, which returns `ZSTD_CONTENTSIZE_UNKNOWN` here — has nothing to allocate against and fails. That is the client's empty `(potential compression error: )`. ## Reproduced against our own encoder Probing `compressBlob` directly with the exact failing content produced the byte-identical 203-byte blob, and the threshold is sharp: ``` input=230 blob=203 FHD=0x4 fcs=false <- the live failure, reproduced current encoder declares FCS from 256 bytes up size=100/175/186/230 -> FHD 0x04, no FCS size=402/1000 -> FHD 0x44, FCS present ``` `klauspost/compress` omits Frame_Content_Size for inputs **under 256 bytes**. No encoder option changes it — tried `WithZeroFrames`, `WithWindowSize`, `SpeedBestCompression`, and streaming; all still 0x04. **Reference libzstd on the identical input does declare it:** ``` libzstd: FHD 0x24, Decompressed Size: 230 B, 178 bytes ours (Go): FHD 0x04, no declared size, 179 bytes ``` Upstream `nwn_nwsync_write` compresses through libzstd, so every blob it has ever written declares a content size. The client has never had to cope without one. Our Go emitter is the odd one out. ## Scale Sampling 50 random manifest entries under 300 KB: **3 lacked a content size (6%)**, all small — 172, 175, 186 bytes. Every affected blob is an unconditional stop for any client that reaches it, so in practice **no client can complete a sync of this manifest**. This blocks the pre-alpha (sow-platform#79). ## Fix 1. Emit frames that always declare a content size. Since no encoder option does it, the options are to re-frame small outputs by hand (rewrite the FHD and insert the field — the uncompressed size is already known and already in the NSYC header) or to compress through libzstd. 2. **Re-emit the affected blobs.** This is the part that will bite: `emit` skips objects that already exist via a 1-byte range GET (#55), so a corrected encoder alone changes nothing for the ~thousands of small blobs already published. It needs a force or verify-and-replace path — see #85, which this makes urgent rather than merely prudent. ## Why nothing caught it Worth recording, because both gaps are the same shape: - **The conformance test (#59) compares decompressed bytes**, so a frame that decodes to the right data passes regardless of its header. It cannot see this class of fault. - **My own zone audit missed it too.** I verified 68 blobs end to end — decompress, hash, compare — and called the zone healthy. The `zstd` CLI streams happily without a declared size, so a *more* capable decoder than the client's hid the exact defect the client trips on. A verifier has to assert frame properties, not just round-trip success. Refs #54, #55, #59, #75, #85, sow-platform#79.

archvillainette referenced this issue

2026-07-31 22:04:45 +00:00

Acceptance gate: a real NWN client syncs from an assembled manifest on the Bunny zone #75

archvillainette referenced this issue from a commit

2026-07-31 22:27:39 +00:00

fix(nwsync): declare Frame_Content_Size, and verify what is published (#86, #85)

archvillainette referenced this issue from a commit

2026-07-31 22:27:39 +00:00

fix(nwsync): close the review findings on the framing fix and verify

archvillainette referenced a pull request that will close this issue

2026-07-31 22:28:00 +00:00

fix(nwsync): declare Frame_Content_Size on every blob, and verify what is published #87