fix(nwsync): declare Frame_Content_Size on every blob, and verify what is published (#87)

Closes #86. Closes #85. These land together on purpose. Fixing the encoder alone changes nothing for the blobs already in the zone, because `emit` skips whatever is already present. ## #86 — the framing fix `klauspost/compress` omits the zstd `Frame_Content_Size` field for inputs under 256 bytes, which the format permits. Reference libzstd never does, so the NWN client — which sizes its output buffer from `ZSTD_getFrameContentSize` and has therefore never met a frame without one — rejected roughly 6% of our blobs outright. Any single one stops a sync dead, so no client could complete a sync of the live manifest. No encoder option changes this, so `compressBlob` re-headers the affected frames into the shape libzstd itself emits: `Single_Segment_flag` set, `Window_Descriptor` dropped, and the freed byte spent on a one-byte `Frame_Content_Size`. Same length in, same length out, and the same descriptor byte (`0x24`) the issue recorded from libzstd. `compressBlob` then asserts its own output. An encoder upgrade that finds another way to omit the field would otherwise reproduce #86 in silence, and a blob is skipped by every later emit once written. `emitter_version` goes to `2`, so `assemble` refuses to merge an index written by the encoder that omitted the field. **Proved against the reference decoder, not just a round trip.** A real emitted 175-byte blob: ``` Frames Skips Compressed Uncompressed Ratio Check Filename 1 0 48 B 175 B 3.646 XXH64 frame.zst c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 - <- zstd -dc | sha1sum c59d6620d4ffd4bf3fe73df43b19b7afcfe8fea4 <- the blob's own name ``` Before the fix that `Uncompressed` column was blank. ## #85 — `nwsync verify` `crucible nwsync verify <manifest-sha1>` reads a manifest and its blobs back through the **public pull zone**, with no credential, because what matters is the bytes a client is served, edge behaviour included. Every distinct blob is decompressed and hashed; failures are reported per blob as missing / malformed framing / size mismatch / hash mismatch, and the exit code is 1. - `--sample N` makes a routine check cheap against a manifest that is ~69,000 blobs and 15 GB; the default is a full sweep. - `--base URL` / `NWSYNC_PULL_BASE` overrides the public host. - The manifest is checked against its own sha1 before a single blob is fetched. - `emit --verify` applies the same check where `emit` would otherwise trust presence, and replaces a stored blob that is not what its name claims. This is what makes the #86 blobs repairable. ## Why the existing checks missed this Both new checks assert the **frame property**, not just a round trip. The conformance suite (#59) compares decompressed bytes, so a frame that decodes correctly passes regardless of its header; and the earlier zone audit decompressed 68 blobs with the `zstd` CLI, a *more* capable decoder than the client's, which certified exactly the blobs the client rejects. ## Checks `make check` and `make smoke` green. Second commit is the fixes from a two-axis review of the first. ## Not in this PR Three follow-ups, filed separately: the backfill has not been run, replacing a blob does not purge the pull-zone edge cache, and #85's runbook line belongs to `sow-platform`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)Reviewed-on: #87 Co-authored-by: vickydotbat <vickydotbat@tutamail.com>
2026-07-31 22:33:12 +00:00
parent 682f920114
commit 7cc53aeb68
13 changed files with 804 additions and 22 deletions
@@ -36,6 +36,7 @@ aliases.
 | `depot` | `pull` | Incremental verified pull of every referenced blob. |
 | `nwsync` | `emit` | Explode one artifact into NWSync blobs plus its own NSYM manifest. |
 | `nwsync` | `assemble` | Merge per-artifact NSYM manifests into one merged manifest. |
+| `nwsync` | `verify` | Decompress and hash a published manifest's blobs through the pull zone. |

 `depot status` and `depot get` pick their backend either with `--out DIR`, a
 depot tree on disk, or with `--target bunny|cdn`, a remote backend. The two
@@ -48,8 +49,9 @@ beside the artifact itself with the extension replaced, so `emit` and
 `assemble` agree on where it is without being told.

 ```
-nwsync emit     [--as NAME] [--out DIR] [--jobs N] <artifact-key> <file>
+nwsync emit     [--as NAME] [--out DIR] [--jobs N] [--verify] <artifact-key> <file>
 nwsync assemble --group-id N [--tlk-key KEY] [--out DIR] <artifact-key>...
+nwsync verify   [--sample N] [--base URL] [--jobs N] <manifest-sha1>
 ```

 `emit` is latency-bound, not CPU-bound: every blob costs an existence probe
@@ -73,6 +75,28 @@ resolves it. `--tlk-key` has its own slot because the TLK shadows nothing.
 `--group-id` is per channel — 1 is current, 2 is testing, and 0 leaves the field
 out of the sidecar.

+`nwsync verify` is the only check on a published blob upstream of a player's
+client. It reads the **pull zone**, not the storage API, and needs no
+credential: what matters is the bytes a client is served, edge behaviour
+included. Every blob is decompressed and hashed, and the zstd frame is asserted
+to declare its content size. Neither half is optional — a `Content-Length` check
+passes a byte-correct-looking object whose contents are short, and a round-trip
+check alone passes a frame the game client cannot decode but Go's decoder can.
+Failures are reported per blob as missing, malformed framing, size mismatch or
+hash mismatch, and the exit code is 1.
+
+A full sweep of the live manifest is roughly 69,000 blobs and 15 GB, so
+`--sample N` exists to make verifying routine; the default is a full sweep.
+`--base URL` (or `NWSYNC_PULL_BASE`) overrides the public host.
+
+`emit --verify` applies the same check where `emit` would otherwise skip. `emit`
+normally reads a blob's presence as proof of its contents, decided by a 1-byte
+range GET, so an object written truncated — or written by an emitter since found
+broken — is skipped by every later run forever and no backfill repairs it. With
+`--verify` the stored copy is read back, unwrapped, hashed against its own name,
+and replaced when it does not match. It costs a full GET per existing blob, so
+it is a repair pass, not the default.
+
 `emit` uploads blobs first and the index last, so the presence of an index is
 the publication marker: an artifact whose emit died halfway leaves real blobs in
 the zone and no index. Blob names are content hashes, so re-running skips
@@ -108,6 +132,17 @@ saving on hak content: a 250 MB hak emitted 2296 blobs totalling 59 MB on disk
 against 249 MB of resources, as recorded in the sidecar's `on_disk_bytes` and
 `total_bytes`.

+The zstd frame always declares its `Frame_Content_Size`. The game client sizes
+its output buffer from that field and cannot decode a frame without one, but the
+Go encoder omits it below 256 bytes, so `emit` re-headers those frames into the
+shape reference libzstd emits: `Single_Segment_flag` set, `Window_Descriptor`
+dropped, and a one-byte content size in its place. `zstd -l <frame>` must print a
+decompressed size; a blank column there is the fault, and it is invisible to any
+check that only decompresses, because both `zstd -dc` and Go's decoder stream
+such a frame happily. This is what the sidecar's `emitter_version` counts:
+version 1 omitted the field and no client could sync past such a blob, version 2
+declares it. `assemble` refuses to merge indexes that disagree.
+
 ## Hidden compatibility aliases

 Existing scripts may continue using these names indefinitely. They are accepted