fix(nwsync): declare Frame_Content_Size, and verify what is published (#86, #85)

These two land together on purpose. Fixing the encoder alone changes nothing
for the blobs already in the zone, because emit skips whatever is already
present.

#86 — klauspost/compress omits the zstd Frame_Content_Size field for inputs
under 256 bytes, which the format permits. Reference libzstd never does, so the
NWN client — which sizes its output buffer from ZSTD_getFrameContentSize and has
therefore never met a frame without one — rejected roughly 6% of our blobs
outright. Any single one stops a sync dead, so no client could complete a sync
of the live manifest.

No encoder option changes this, so emit re-headers the affected frames into the
shape libzstd itself emits: Single_Segment_flag set, Window_Descriptor dropped,
and the freed byte spent on a one-byte Frame_Content_Size. Same length in, same
length out. Verified against the zstd CLI end to end: a real emitted 230-byte
blob now reports its decompressed size where it previously reported none.
emitter_version goes to 2, so assemble refuses to merge an index written by the
encoder that omitted the field.

#85 — a published blob was verified by exactly one thing, a player's client, at
download time. `nwsync verify <manifest-sha1>` now reads a manifest and its
blobs back through the public pull zone with no credential, decompresses and
hashes every one, and reports missing / malformed framing / size mismatch / hash
mismatch per blob. `--sample N` makes a routine check cheap against a manifest
that is ~69,000 blobs and 15 GB. `emit --verify` applies the same check where
emit would otherwise trust presence, and replaces a stored blob that is not what
its name claims — which is what makes the #86 blobs repairable.

Both new checks assert the frame property, not just a round trip. Round-tripping
cannot see this class of fault: both the zstd CLI and Go's decoder stream such a
frame happily, being more capable decoders than the client's, which is how the
defect reached production and survived an audit.

Refs #54, #55, #59, #75, sow-platform#79.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-01 00:20:15 +02:00
co-authored by Claude Opus 5
parent 682f920114
commit 56d4054118
11 changed files with 752 additions and 19 deletions
+36 -1
View File
@@ -36,6 +36,7 @@ aliases.
| `depot` | `pull` | Incremental verified pull of every referenced blob. |
| `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
| `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
| `nwsync` | `verify` | Decompress and hash a published manifest's blobs through the pull zone. |
`depot status` and `depot get` pick their backend either with `--out DIR`, a
depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
@@ -48,8 +49,9 @@ beside the artifact itself with the extension replaced, so `emit` and
`assemble` agree on where it is without being told.
```
nwsync emit [--as NAME] [--out DIR] [--jobs N] <artifact-key> <file>
nwsync emit [--as NAME] [--out DIR] [--jobs N] [--verify] <artifact-key> <file>
nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
nwsync verify [--sample N] [--base URL] [--jobs N] <manifest-sha1>
```
`emit` is latency-bound, not CPU-bound: every blob costs an existence probe
@@ -73,6 +75,28 @@ resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
`--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
out of the sidecar.
`nwsync verify` is the only check on a published blob upstream of a player's
client. It reads the **pull zone**, not the storage API, and needs no
credential: what matters is the bytes a client is served, edge behaviour
included. Every blob is decompressed and hashed, and the zstd frame is asserted
to declare its content size. Neither half is optional — a `Content-Length` check
passes a byte-correct-looking object whose contents are short, and a round-trip
check alone passes a frame the game client cannot decode but Go's decoder can.
Failures are reported per blob as missing, malformed framing, size mismatch or
hash mismatch, and the exit code is 1.
A full sweep of the live manifest is roughly 69,000 blobs and 15 GB, so
`--sample N` exists to make verifying routine; the default is a full sweep.
`--base URL` (or `NWSYNC_PULL_BASE`) overrides the public host.
`emit --verify` applies the same check where `emit` would otherwise skip. `emit`
normally reads a blob's presence as proof of its contents, decided by a 1-byte
range GET, so an object written truncated — or written by an emitter since found
broken — is skipped by every later run forever and no backfill repairs it. With
`--verify` the stored copy is read back, unwrapped, hashed against its own name,
and replaced when it does not match. It costs a full GET per existing blob, so
it is a repair pass, not the default.
`emit` uploads blobs first and the index last, so the presence of an index is
the publication marker: an artifact whose emit died halfway leaves real blobs in
the zone and no index. Blob names are content hashes, so re-running skips
@@ -108,6 +132,17 @@ saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
`total_bytes`.
The zstd frame always declares its `Frame_Content_Size`. The game client sizes
its output buffer from that field and cannot decode a frame without one, but the
Go encoder omits it below 256 bytes, so `emit` re-headers those frames into the
shape reference libzstd emits: `Single_Segment_flag` set, `Window_Descriptor`
dropped, and a one-byte content size in its place. `zstd -l <frame>` must print a
decompressed size; a blank column there is the fault, and it is invisible to any
check that only decompresses, because both `zstd -dc` and Go's decoder stream
such a frame happily. This is what the sidecar's `emitter_version` counts:
version 1 omitted the field and no client could sync past such a blob, version 2
declares it. `assemble` refuses to merge indexes that disagree.
## Hidden compatibility aliases
Existing scripts may continue using these names indefinitely. They are accepted